An Evaluation of Positional-Embedding Integration in Transformers

Abstract

This study examines how positional information is combined with content embeddings in Transformer models. The first experiment compares addition with three concatenation-based alternatives for English-to-German translation on Multi30k. Algebraic analysis shows that addition is a constrained instance of concatenation with shared projection parameters. The variants produce similar loss, sampled BLEU, and attention behavior, whereas large concatenation incurs substantially greater computation. A second experiment tests whether the frequency range of sinusoidal positional encodings should be matched to periodic input signals; no systematic relationship is observed.

Research Questions

The Transformer architecture uses self-attention without encoding token order through sequential computation and therefore introduces an explicit positional representation (Vaswani et al., 2017). This project addresses two questions:

Does concatenating content and positional embeddings provide an empirical advantage over addition?
Does matching positional-encoding frequency to input-signal frequency improve reconstruction of periodic signals?

Formulation

Let Xc denote a content embedding and Xp a positional encoding. Under the standard formulation, the attention input is Xc + Xp, and common query, key, and value projections act on both components. Expansion of the attention score yields four classes of interaction: content–content, content–position, position–content, and position–position.

For a concatenated representation [Xc; Xp], the projection matrices can be partitioned into independent blocks governing these interactions. Equal corresponding blocks recover the additive model. Addition can therefore be interpreted as structured parameter sharing, while a fully connected layer after concatenation permits a learned mixture of the two inputs.

Methodology

Machine Translation

Four positional-integration configurations are evaluated on English-to-German translation using the Multi30K dataset (Elliott et al., 2016):

a 512-dimensional word embedding added to a positional encoding;
256-dimensional word and positional embeddings concatenated to preserve an overall width of 512;
512-dimensional word and positional embeddings concatenated into a larger representation; and
concatenation followed by a fully connected mixing layer.

The base Transformer contains four encoder layers, four decoder layers, and eight attention heads. Training uses Adam with a learning rate of 0.0001, a batch size of 128, and 20 epochs. Evaluation includes training loss, validation loss, sampled BLEU scores, attention maps, and average epoch time. For computational efficiency, BLEU is calculated only for the first sentence of each validation batch; it is consequently treated as an approximate comparative measure.

Periodic-Signal Reconstruction

The second experiment uses 20-second signals sampled at 198 points per second. Controlled sinusoids are combined, transformed into spectrograms with the short-time Fourier transform, and provided to a Transformer trained to reconstruct the waveform. The model contains six encoder layers, two decoder layers, and eight attention heads, and it is optimized using symmetric mean absolute percentage error (SMAPE). A coefficient controls the frequency range of the sinusoidal positional encoding.

Evaluation and Results

Translation Results

Training loss, validation loss, sampled BLEU, and attention patterns are broadly similar across the four configurations. The available measurements do not establish a decisive quality improvement for the large concatenated model, which also uses approximately twice the embedding width. The measured average epoch times are:

Integration method	Average epoch time
Addition	55.10 s
Fixed-width concatenation	54.67 s
Concatenation with fully connected mixing	58.21 s
Large concatenation	138.53 s

Attention maps from corresponding layers focus on similar regions of a test sentence. The fully connected variant exhibits somewhat more diffuse attention, although this observation is not causal evidence. Addition retains a 512-dimensional content representation while providing comparable empirical behavior and substantially lower cost than large concatenation.

Periodic-Signal Results

Validation loss does not exhibit a consistent trend as the positional-frequency coefficient changes. The experiment therefore provides no evidence that matching the selected positional-encoding frequency range to the frequency of the generated signals improves reconstruction.

Limitations and Discussion

The translation comparison indicates that the additional flexibility of concatenation is not sufficient, in this setting, to justify its dimensional or computational cost. However, small differences among methods cannot be interpreted as decisive because BLEU is estimated from only the first sentence of each validation batch. The larger concatenated model also changes representation width and is therefore not a capacity-matched comparison.

The periodic-signal experiment has two further limitations. First, a spectrogram already represents signal frequency explicitly, while positional encoding describes order along the time axis; the generated dataset does not isolate temporal modulation sufficiently to align these concepts. Second, uniformly scaling the positional frequencies creates strong periodic patterns in early encoding dimensions and may alter representation quality independently of the proposed frequency-matching effect. A more conclusive study would require a precise definition of frequency across the input, temporal axis, and positional representation.

Conclusion

The theoretical and empirical results characterize positional addition as an efficient form of structured parameter sharing. Concatenation provides a more general parameterization but does not yield a clear quality advantage in the evaluated translation task, and unconstrained dimensional expansion substantially increases training time. The signal-reconstruction experiment yielded no consistent relationship under the tested representation. Because the design did not isolate temporal-frequency matching, no causal interpretation is warranted.

References

A. Vaswani et al. “Attention Is All You Need.” Advances in Neural Information Processing Systems, 2017.
D. Elliott, S. Frank, K. Sima’an, and L. Specia. “Multi30K: Multilingual English-German Image Descriptions.” Workshop on Vision and Language, 2016.