YL

Y. Li

info

Please Note

4 records found

Binary Weights and Semantic Binary Data Compression

Doctoral thesis (2024) - Y. Li, M.J.T. Reinders, J.C. van Gemert
Improving the efficiency in deploying deep neural networks (DNNs) and processing complex high-dimensional data has drawn increasing attention in recent years. Yet, the deployment of large DNN models is challenged by the high computational complexity and energy consumption, making it difficult to run on resource-constrained devices such as mobile phones. Moreover, the exploding amount of high-dimensional data requires large storage and transmission capacities which is infeasible to be processed on mobile devices.
To alleviate these limitations, this dissertation focuses on binarization techniques, including model binarization and data binarization, to improve the efficiency in terms of storage, computation and energy.
In model binarization, we binarize both the weight and activation of DNN models which can reach up to 32× memory saving and a speed up of 58×. We also develop pruning algorithms to further compress the binarized network while maintaining accuracy. To efficiently train the binarized networks, we discover new optimization methods that has less hyper-parameters and can improve the accuracy.
In data binarization, we propose deep hashing algorithms that learn smaller binary data representation. Deep hashing methods have become an effective technique for fast and efficient similarity search and retrieval of high-dimensional data items in large databases. ...
Conference paper (2023) - Joris Quist, Yunqiang Li, Jan van Gemert
Binary Neural Networks (BNNs) are compact and efficient by using binary weights instead of real-valued weights. Current BNNs use latent real-valued weights during training, where hyper-parameters are inherited from real-valued networks. The interpretation of several of these hyperparameters is based on the magnitude of the real-valued weights. For BNNs, however, the magnitude of binary weights is not meaningful, and thus it is unclear what these hyperparameters actually do. One example is weight-decay, which aims to keep the magnitude of real-valued weights small. Other examples are latent weight initialization, the learning rate, and learning rate decay, which influence the magnitude of the real-valued weights. The magnitude is interpretable for real-valued weights, but loses its meaning for binary weights. In this paper we offer a new interpretation of these magnitude-based hyperparameters based on higher-order gradient filtering during network optimization. Our analysis makes it possible to understand how magnitude-based hyperparameters influence the training of binary networks which allows for new optimization filters specifically designed for binary neural networks that are independent of their real-valued interpretation. Moreover, our improved understanding reduces the number of hyperparameters, which in turn eases the hyperparameter tuning effort which may lead to better hyperparameter values for improved accuracy. Code is available at https: //github.com/jorisquist/Understanding-WM-HP-in-BNNs ...

Deep fisher hashing

Conference paper (2020) - Yunqiang Li, Wenjie Pei, Yufei Zha, Jan Van Gemert
Current massive datasets demand light-weight access for analysis. Discrete hashing methods are thus beneficial because they map high-dimensional data to compact binary codes that are efficient to store and process, while preserving semantic similarity. To optimize powerful deep learning methods for image hashing, gradient-based methods are required. Binary codes, however, are discrete and thus have no continuous derivatives. Relaxing the problem by solving it in a continuous space and then quantizing the solution is not guaranteed to yield separable binary codes. The quantization needs to be included in the optimization. In this paper we push for quantization: We optimize maximum class separability in the binary space. We introduce a margin on distances between dissimilar image pairs as measured in the binary space. In addition to pair-wise distances, we draw inspiration from Fisher's Linear Discriminant Analysis (Fisher LDA) to maximize the binary distances between classes and at the same time minimize the binary distance of images within the same class. Experiments on CIFAR-10, NUS-WIDE and ImageNet100 demonstrate compact codes comparing favorably to the current state of the art. ...
Journal article (2020) - Yufei Zha, Tao Ku, Yunqiang Li, Peng Zhang
Classification-based tracking strategies often face more challenges from intra-class discrimination than from inter-class separability. Even for deep convolutional neural networks that have been widely proven to be effective in various vision tasks, their intra-class discriminative capability is still limited by the weakness of softmax loss, especially for targets not seen in the training dataset. By taking intrinsic attributes of training samples into account, in this paper, we propose a position-sensitive loss coupled with softmax loss to achieve intra-class compactness and inter-class explicitness. Particularly, two additive margins are introduced to encode the position attribute for decision boundary maximization, which is also utilized with the proposed loss to supervise the fine-tuned features on the pre-trained model. With the nearest neighbor ranking measurement in the feature embedding domain, the whole scheme is able to reach an optimized balance between the feature-level inter-class semantic separability and instance-level intra-class relative distance ranking. We evaluate the proposed work on different popular benchmarks, and experimental results demonstrate that our tracking strategy performs favorably against most of the state-of-the-art trackers in the comparison of accuracy and robustness. ...