Predicting and Repairing Merge Collapse in Large Language Models

Let \(\theta_{\text{base}}\) denote the parameters of a pretrained base model and \(\theta_k\) those of specialist \(k\in\{1,\dots,K\}\), fine-tuned from it. The task vector \(\tau_k = \theta_k - \theta_{\text{base}}\) has value \(\tau_k(i)\) at coordinate \(i\). A merge rescales each task vector by a weight \(s_k>0\), set by norm scaling or to one in a raw merge, and averages them into \(\bar{\delta} = \frac{1}{K}\sum_k s_k \tau_k\). At coordinate \(i\) the total power \(P(i)\) splits into signal power \(S(i) = \bar{\delta}(i)^2\) and interference \(I(i) = P(i) - S(i)\), the reduction in mean-squared power due to averaging.

\[I(i) = \underbrace{\tfrac{1}{K}\sum_{k=1}^{K}\big(s_k \tau_k(i)\big)^2}_{P(i)} \;-\; \underbrace{\bar{\delta}(i)^2}_{S(i)} \;=\; \mathrm{Var}_k\big(s_k \tau_k(i)\big) \geq 0\]

This is the ambiguity decomposition of ensembles applied to the scaled task vectors, with the base model as the target. The variance measures task-vector disagreement, which the paper uses as a proxy for task interference, and is computed alongside the mean. Interpreting it as noise requires a model. Drift means the displacement \(s_k\tau_k\) of a specialist from the base. Identical scaled task vectors give zero interference regardless of the size or effect of their shared drift. Aligned vectors with unequal magnitudes can still have positive variance, as can one-sided drift, present in one specialist and absent in the others.

For a layer \(l\) with \(n_l\) merged coordinates, the means \(\overline{P}_l\), \(\overline{S}_l\) and \(\overline{I}_l\) give the coherence ratio \(\overline{\mathrm{CR}}_l = \overline{S}_l / (\overline{P}_l + \epsilon)\) and an interference scale \(\sigma_l^2 = \overline{I}_l = \overline{P}_l - \overline{S}_l\). The standard deviation \(\sigma_l=\sqrt{\overline{I}_l}\) has weight units and can set a threshold on task-vector coordinates. Computing these statistics takes about two CPU-minutes on a 7B model.

The same \(\overline{I}\) that sets the threshold defines the pre-merge score \(c\sqrt{\overline{I}/K}\), computed with \(\overline{I}\) the parameter-count-weighted MLP mean interference. The paper tests whether this score separates destructive and harmless merges on twenty-two configurations across four families and \(K\in[2,8]\). Seven are destructive (perplexity \(>100\) or mean \(>10\)pp below base). Fifteen are harmless (within 1pp or better), with no intermediate cases. The score exceeds the threshold of \(1.9\times10^{-3}\) for five of the seven destructive merges and for none of the harmless ones.

Interference, total power and signal power reach AUC 0.98 or higher, and interference also sets the repair's variance scale. The two direction statistics, mean incoherence \(1-\overline{\mathrm{CR}}\) and \(\rho\), are anti-predictive, placing destructive merges below many harmless ones.

With the statistic, convention and threshold fixed, twelve of the fourteen predictions made before evaluation were correct across three model families, including the harmless and destructive endpoints of the constructed pair. On the other two merges predicted to be destructive, R1-Distill + Coder and Math-PRM + Coder, naive averaging falls 18.4 and 14.4pp below the base, whereas PRISM keeps both within 0.5pp of the base.

A controlled titration continued pretraining two Qwen2.5-1.5B specialists on disjoint Japanese web text and biomedical abstracts. At learning rate \(10^{-4}\), eight measured checkpoints follow an approximate square-root law through a harmless endpoint at one billion tokens for each specialist, and extrapolated, the fitted law crosses the threshold near 7.5 billion tokens. Continuing at 5× the rate reaches the destructive cluster within a further billion tokens and improves own-domain perplexity (Japanese 16.7 → 12.2, PubMed 7.2 → 7.0). As predicted before evaluation, the merge at this endpoint is destructive. Naive averaging loses 20.3pp against the base, while PRISM stays within 0.3pp of it.

The recipe uses naive averaging below the fixed score threshold and the gated shrinkage above it. All fifteen harmless configurations scored below the threshold and therefore keep the plain average, which scores 62.6 against the base's 59.3 on Llama-3.1-8B R1-Distill with Tulu-3. The twenty-eight public pairs without a code specialist that were screened across three families all scored below half the threshold.

Under the working model, injected interference power grows as \(c^2\), and PRISM shrinks the update before applying the coefficient. Across the sweep, PRISM holds a 64.8–68.0% mean. At \(c=0.7\), the four evaluated baselines sit 19 to 33 points below the base at perplexities from 100 to 961. In the generation evaluation at \(c=1.0\), Task Arithmetic, TIES and DELLA score zero on GSM8K. The best baseline in the plotted sweep, LEWIS at 67.6 (\(c=0.3\)), is tied within evaluation uncertainty with the 67.8 that untuned PRISM posts at the shared default.

On the same merge, soft thresholding beats hard truncation at the same threshold by 4.4pp on the six-task mean. Truncation's cost falls almost entirely on tasks that produce text, at 19.2pp on GSM8K and up to 28.6pp on HumanEval against 1.4pp averaged over the five multiple-choice benchmarks. Across threshold scales from \(\kappa=0.25\) to 3, the six-task mean varies by at most 0.5pp, and the default scores highest.