Modern sequence models are dominated by Transformers, where self-attention mixes information from the visible context in an input-dependent way. However, when retrieval is not sharp and attention remains diffuse over an effective support $S_{\mathrm{eff}}(t)$, the influence of any individual token is diluted, typically scaling as $O(1/S_{\mathrm{eff}}(t))$ and reaching $O(1/\ell)$ for old tokens in full-prefix settings. Structured state-space models process sequences recurrently through an explicit feedback path; selective variants such as Mamba make this feedback input-dependent, yet when freeze time cannot be sustained over long intervals, their long-range sensitivity decays exponentially with lag. Existing architectures therefore either retrieve from the past in a single read or propagate information through a single feedback chain. We introduce Sessa, a decoder that places attention inside a feedback path, enabling recurrent many-path aggregation within a layer. Under stated assumptions, Sessa admits regimes with a power-law memory tail in lag $\ell$ of order $O(\ell^{-β})$ for $0<β<1$, which is asymptotically slower than $1/\ell$; moreover, this rate is tight in an explicit diffuse uniform-routing setting where the influence is $Θ(\ell^{-β})$. Under the same conditions, only Sessa among the compared model classes realizes flexible selective retrieval, including non-decaying profiles. Empirically, under matched architectures and training budgets, Sessa achieves the strongest performance on our long-context benchmarks while remaining competitive with Transformer and Mamba style baselines on short-context language modeling.

现代序列模型以 Transformer 为主，其中自注意力以依赖于输入的方式混合来自可见上下文的信息。然而，当检索不敏锐并且注意力仍然分散在有效支持 $S_{\mathrm{eff}}(t)$ 上时，任何单个令牌的影响都会被稀释，通常缩放为 $O(1/S_{\mathrm{eff}}(t))$ 并在完整前缀设置中达到 $O(1/\ell)$ 旧令牌。结构化状态空间模型通过显式反馈路径循环处理序列；像 Mamba 这样的选择性变体使得这种反馈依赖于输入，但是当冻结时间不能长时间持续时，它们的远程灵敏度会随着滞后呈指数衰减。因此，现有架构要么在单次读取中检索过去，要么通过单个反馈链传播信息。我们引入了 Sessa，一种将注意力放在反馈路径内的解码器，从而实现层内的循环多路径聚合。在规定的假设下，Sessa 承认存在幂律记忆尾部，其滞后 $\ell$ 的阶数为 $O(\ell^{-β})$，$0<β<1$，渐近地慢于 $1/\ell$；此外，在影响为 $θ(\ell^{-β})$ 的显式扩散均匀路由设置中，该速率很紧。在相同条件下，比较模型类中只有Sessa实现了灵活的选择性检索，包括非衰减剖面。根据经验，在匹配的架构和训练预算下，Sessa 在我们的长上下文基准上实现了最强的性能，同时在短上下文语言建模上与 Transformer 和 Mamba 风格基准保持竞争力。

Proximal Policy Optimization (PPO) has become the predominant algorithm for on-policy reinforcement learning due to its scalability and empirical robustness across domains. However, there is a significant disconnect between the underlying foundations of trust region methods and the heuristic clipped objective used in PPO. In this paper, we bridge this gap by introducing the Bounded Ratio Reinforcement Learning (BRRL) framework. We formulate a novel regularized and constrained policy optimization problem and derive its analytical optimal solution. We prove that this solution ensures monotonic performance improvement. To handle parameterized policy classes, we develop a policy optimization algorithm called Bounded Policy Optimization (BPO) that minimizes an advantage-weighted divergence between the policy and the analytic optimal solution from BRRL. We further establish a lower bound on the expected performance of the resulting policy in terms of the BPO loss function. Notably, our framework also provides a new theoretical lens to interpret the success of the PPO loss, and connects trust region policy optimization and the Cross-Entropy Method (CEM). We additionally extend BPO to Group-relative BPO (GBPO) for LLM fine-tuning. Empirical evaluations of BPO across MuJoCo, Atari, and complex IsaacLab environments (e.g., Humanoid locomotion), and of GBPO for LLM fine-tuning tasks, demonstrate that BPO and GBPO generally match or outperform PPO and GRPO in stability and final performance.

近端策略优化（PPO）由于其跨领域的可扩展性和经验鲁棒性，已成为策略强化学习的主要算法。然而，信任域方法的底层基础与 PPO 中使用的启发式裁剪目标之间存在显着脱节。在本文中，我们通过引入有界比率强化学习（BRRL）框架来弥补这一差距。我们提出了一个新颖的正则化和约束策略优化问题，并推导出其解析最优解。我们证明该解决方案可确保单调性能提升。为了处理参数化策略类，我们开发了一种称为有界策略优化 (BPO) 的策略优化算法，该算法可以最大限度地减少策略与 BRRL 的分析最优解决方案之间的优势加权差异。我们进一步根据 BPO 损失函数确定最终政策的预期绩效下限。值得注意的是，我们的框架还提供了一个新的理论视角来解释 PPO 损失的成功，并将信任域策略优化和交叉熵方法（CEM）联系起来。我们还将 BPO 扩展到与集团相关的 BPO (GBPO)，以进行 LLM 微调。对 MuJoCo、Atari 和复杂 IsaacLab 环境（例如人形运动）的 BPO 以及 LLM 微调任务的 GBPO 进行的实证评估表明，BPO 和 GBPO 在稳定性和最终性能方面通常与 PPO 和 GRPO 匹配或优于 PPO 和 GRPO。