Efficient and stable training of large language models (LLMs) remains a core challenge in modern machine learning systems. To address this challenge, Reparameterized Orthogonal Equivalence Training (POET), a spectrum-preserving framework that optimizes each weight matrix through orthogonal equivalence transformation, has been proposed. Although POET provides strong training stability, its original implementation incurs high memory consumption and computational overhead due to intensive matrix multiplications. To overcome these limitations, we introduce POET-X, a scalable and memory-efficient variant that performs orthogonal equivalence transformations with significantly reduced computational cost. POET-X maintains the generalization and stability benefits of POET while achieving substantial improvements in throughput and memory efficiency. In our experiments, POET-X enables the pretraining of billion-parameter LLMs on a single Nvidia H100 GPU, and in contrast, standard optimizers such as AdamW run out of memory under the same settings.

高效稳定的大型语言模型（LLM）训练仍然是现代机器学习系统的核心挑战。为了应对这一挑战，人们提出了重新参数化正交等价训练（POET），这是一种通过正交等价变换优化每个权重矩阵的频谱保留框架。尽管 POET 提供了强大的训练稳定性，但其原始实现由于密集的矩阵乘法而导致较高的内存消耗和计算开销。为了克服这些限制，我们引入了 POET-X，这是一种可扩展且内存高效的变体，可以执行正交等价变换，同时显着降低计算成本。 POET-X 保持了 POET 的通用性和稳定性优势，同时在吞吐量和内存效率方面实现了大幅改进。在我们的实验中，POET-X 可以在单个 Nvidia H100 GPU 上预训练十亿参数的 LLM，相比之下，AdamW 等标准优化器在相同设置下会耗尽内存。

We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple.

我们研究 Transformer 语言模型中的两种反复出现的现象：大规模激活（其中少量标记在少数通道中表现出极端异常值）和注意力沉降（其中某些标记无论语义相关性如何都会吸引不成比例的注意力）。先前的研究发现，这些现象经常同时发生，并且经常涉及相同的标记，但它们的功能作用和因果关系仍不清楚。通过系统的实验，我们表明，共现很大程度上是现代 Transformer 设计的建筑产物，并且这两种现象具有相关但不同的功能。大规模激活在全局范围内运行：它们引起跨层持续存在的近乎恒定的隐藏表示，有效地充当模型的隐式参数。注意力接收器在本地运行：它们调节各个头部的注意力输出，并使各个头部偏向于短程依赖性。我们将预规范配置确定为实现共现的关键选择，并表明消除它会导致两种现象解耦。