Whether language models can systematically generalize remains actively debated. Yet empirical performance is jointly shaped by multiple factors such as training data, training paradigms, and inference-time strategies, making failures difficult to interpret. We introduce a controlled synthetic environment based on shortest-path planning, a canonical composable sequential optimization problem. The setup enables clean separation of these factors and supports two orthogonal axes of generalization: spatial transfer to unseen maps and length scaling to longer-horizon problems. We find that models exhibit strong spatial transfer but consistently fail under length scaling due to recursive instability. We further analyze how distinct stages of the learning pipeline influence systematic problem-solving: for example, data coverage sets capability limits; reinforcement learning improves training stability but does not expand those limits; and inference-time scaling enhances performance but cannot rescue length-scaling failures.

语言模型是否可以系统地泛化仍然存在激烈争论。然而，经验表现是由训练数据、训练范式和推理时间策略等多种因素共同决定的，这使得失败难以解释。我们引入了基于最短路径规划的受控合成环境，这是一个典型的可组合顺序优化问题。该设置可以清楚地分离这些因素，并支持两个正交轴的概括：空间转移到看不见的地图和长度缩放到更长的视野问题。我们发现模型表现出很强的空间转移，但由于递归不稳定而在长度缩放下始终失败。我们进一步分析学习管道的不同阶段如何影响系统问题的解决：例如，数据覆盖范围设置了能力限制；强化学习提高了训练的稳定性，但并没有扩大这些限制；推理时间缩放可以提高性能，但无法挽救长度缩放失败。

LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval: $\textbf{(1)}$ a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates ($\barρ = 0.8$-$4.1\%$), with $33$-$67\%$ of documents exhibiting at least one directed 3-cycle; and $\textbf{(2)}$ split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed $\geq(1{-}α)$ coverage, with set width serving as a per-instance reliability indicator ($r_s = {+}0.576$, $N{=}1{,}918$, $p < 10^{-100}$, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement ($\bar{r} = 0.32$-$0.38$), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size $\approx 3.0$) and coherence moderately so (avg. set size $\approx 3.9$), while fluency and consistency remain unreliable (avg. set size $\approx 4.9$). We release all code, prompts, and cached results.

LLM 作为法官的框架越来越多地用于自动 NLG 评估，但它们的每个实例的可靠性仍然知之甚少。我们提出了一个应用于 SummEval 的双管齐下的诊断工具包：$\textbf{(1)}$ 一种传递性分析，揭示了被低总违规率掩盖的广泛的每个输入不一致性（$\barρ = 0.8$-$4.1\%$），其中 $33$-$67\%$ 的文档至少显示一个定向 3 周期； $\textbf{(2)}$ 将共形预测集分割为 1-5 个 Likert 分数，提供理论上保证的 $\geq(1{-}α)$ 覆盖范围，并将集合宽度用作每个实例的可靠性指标（$r_s = {+}0.576$、$N{=}1{,}918$、$p < 10^{-100}$，汇集所有法官）。至关重要的是，预测集宽度显示出一致的跨评判一致性（$\bar{r} = 0.32$-$0.38$），表明它捕获了文档级别的难度，而不是评判特定的噪音。在四位法官和四个标准中，两种诊断都趋于一致：标准比判断更重要，相关性判断最可靠（平均集合大小$\约3.0$），连贯性中等（平均集合大小$\约3.9$），而流畅性和一致性仍然不可靠（平均集合大小$\约4.9$）。我们发布所有代码、提示和缓存结果。