Deterministic and Stochastic Binary Weight Quantization

Abstract

This project reproduces the BinaryConnect training procedure and evaluates deterministic and stochastic weight binarization on CIFAR-10 and SVHN. In addition to the conventional +1/-1 representation, two 1/0 variants are examined to characterize the relationship between quantization, optimization, and sparsity. In the recorded runs, deterministic +1/-1 binarization produced the highest observed validation accuracy, whereas stochastic sampling exhibited slower convergence without an observed accuracy benefit. Deterministic 1/0 binarization reached approximately 76% validation accuracy while mapping most weights to zero, indicating an accuracy–sparsity trade-off. Because repeated-seed statistics were not reported, these comparisons are specific to the recorded runs.

Problem Formulation

BinaryConnect substitutes binary weights for full-precision weights during forward and backward propagation while retaining full-precision parameters for optimization (Courbariaux et al., 2015). Multiplication by a binary weight can consequently be reduced to operations such as sign changes and additions. The study addresses two questions: whether stochastic binarization provides an empirical advantage over deterministic binarization, and how replacing the conventional values +1 and -1 with 1 and 0 affects predictive performance and sparsity.

Methodology

Binarization Rules

Four binarization rules are compared under a common network and training configuration:

Deterministic +1/-1 binarization applies the sign function, assigning +1 to nonnegative weights and -1 to negative weights.
Stochastic +1/-1 binarization uses a clipped hard-sigmoid of each full-precision weight as the probability of selecting +1; the alternative outcome is -1.
Deterministic 1/0 binarization uses the mean weight of each layer as a layer-dependent threshold.
Stochastic 1/0 binarization replaces the -1 outcome of the stochastic BinaryConnect rule with 0.

The layer-dependent threshold in the deterministic 1/0 method accounts for the possibility that positive and negative weights are not balanced within a layer.

Optimization Procedure

An initial implementation attempted to update binarized weights directly. This procedure produced poor training behavior because optimizer steps that did not cross a quantization threshold were removed by the subsequent binarization operation. The final implementation instead maintains a full-precision copy of every parameter. A temporary binary copy is generated before each convolutional or linear operation; propagation uses this binary path, whereas the resulting gradients update the full-precision parameters. The full-precision values are subsequently clipped to a bounded range. This separation preserves small optimizer updates across iterations.

The convolutional architecture and training parameters follow the original BinaryConnect experiment. All four binarizers are evaluated on CIFAR-10, and the reproduced network is additionally applied to SVHN.

Evaluation and Results

The experiments do not reproduce the reported advantage of stochastic +1/-1 binarization. Within the recorded runs, stochastic sampling exhibited slower convergence and did not improve validation accuracy, while deterministic +1/-1 binarization produced the highest observed validation accuracy. The available artifacts do not contain wall-clock measurements or repeated-seed statistics, so the comparison is descriptive rather than an estimate of computational cost or expected performance.

The stochastic 1/0 method yields substantially lower performance. A plausible explanation concerns batch normalization: running statistics are accumulated during training under sampled binary weights, whereas evaluation uses full-precision weights. The resulting difference between training and evaluation distributions may invalidate the stored statistics. This explanation remains a hypothesis; recomputing batch-normalization statistics before evaluation would be required to test it.

Deterministic 1/0 binarization does not match the accuracy of the +1/-1 models, but it achieves approximately 76% validation accuracy and maps most parameters to zero. Weight histograms show a comparatively balanced +1/-1 distribution and a strongly zero-dominant 1/0 distribution. The latter therefore introduces sparsity during training and has a qualitative relationship to pruning.

Binary representations also provide a potential computational advantage. For example, 32 signs can be packed into a 32-bit word and processed in groups with bitwise XNOR and population-count operations. No hardware speedup was measured in this project; published acceleration results are therefore not reported as experimental outcomes of this study.

Limitations and Discussion

The comparison demonstrates that quantization behavior depends on more than the two representable values. Latent full-precision parameters, thresholds, normalization statistics, and sampling procedures interact with optimization. The available experiments support the deterministic +1/-1 method within the evaluated setting, but they do not establish its superiority across architectures or datasets. Similarly, the cause proposed for the weak stochastic 1/0 result was not isolated experimentally. Repeated trials and an evaluation procedure with recalibrated batch-normalization statistics would provide stronger evidence.

Conclusion

In the recorded runs, deterministic +1/-1 binarization produced the highest observed validation accuracy among the four evaluated methods. Deterministic 1/0 binarization produced lower accuracy but substantial parameter sparsity, motivating further investigation of binary training as a joint quantization and sparsification procedure. The implementation also showed that its optimizer needed persistent full-precision parameters even when propagation used binary weights.

References

M. Courbariaux, Y. Bengio, and J.-P. David. “BinaryConnect: Training Deep Neural Networks with Binary Weights during Propagations.” Advances in Neural Information Processing Systems, 2015.