约束导致损失 18 个点，编译 Schema 挽回 14 个点

处理约束导致损失 18 个点，编译 Schema 挽回 14 个点这类问题时，先确认目标场景，再按步骤核对配置或玩法细节。

My previous experiment ended with an uncomfortable result.

On the same 49 audited GSM8K questions, Qwen2.5-7B-Instruct answered 39 correctly

when prompted to produce JSON. When I enforced the declared schema with Outlines or

XGrammar, each backend answered only 30 correctly.

The constraints fixed compliance and cost 18.4 percentage points of recoverable

mathematical accuracy.

That result raised a more useful question than "are constraints bad?"

Was the model failing because it was constrained, or because the external contract

forced it through a poorly aligned lexical representation?

I have now completed the follow-up: 72 targeted and 150 full-confirmation generations

on Qwen2.5-7B, plus local integration probes, boundary traces, paired statistics, and

independent artifact validation.

The short result is this:

Replacing a model-facing signed numeric string with a native JSON integer, then

deterministically restoring the original string contract, improved both constrained

backends from 30/49 to 37/49 contract-valid correct.

That is a 14.3 percentage-point recovery while retaining 100% final external-schema

validity.

This article explains what changed, what stayed frozen, what the trace showed, and

why this is a Green light for a small contract-alignment compiler rather than proof of

a universal solution.

The first study is here:

Structured Output Fixed My JSON and Cut Math Accuracy by 18 Points.

All new code, raw rows, manifests, hashes, traces, and reports are in the repository.

Vaibhav701161/constrained-decoding-lab

A controlled, artifact-validated study of how JSON prompting, grammar-constrained decoding, and output-field order affect mathematical accuracy and schema compliance.

Results | Alignment result | Paired evidence | Study design | Reproduction | Evidence | Public Kaggle artifacts | Technical article | Limitations

Central result

Constrained decoding solved the formatting problem, but it did not preserve all of the model's recoverable mathematical accuracy. On Qwen2.5-7B, prompt-only JSON achieved 79.6% recoverable accuracy and 0% schema compliance. Outlines and XGrammar each achieved 61.2% recoverable accuracy and 100% schema compliance. The paired semantic effect was -18.4 percentage points for both backends (exact McNemar p = 0.003906).

This is not a claim that constrained decoding is universally harmful. It is a controlled reproduction showing that contract compliance and semantic correctness are separate outcomes, and that a decoder can improve the first while reducing the second under a specific, matched…

The failure was more specific than "constraints hurt reasoning"

The external schema required this:

{"reasoning": "...","answer": "18000"}

The model naturally preferred this:

{"reasoning": "...","answer": 18000}

The prompt-only output was valid JSON and often mathematically correct, but it did

not satisfy the contract because answer was a JSON number rather than a string.

Hard constraints solved that type mismatch. They also changed the generated answer.

In the frozen reasoning-first baseline:

Both constrained backends had the same paired transition against prompting:

correct in both:30correct only with prompting: 9correct only with constraints: 0wrong in both:10exact paired p: 0.003906

Eight of the nine losses were shared across Outlines and XGrammar. In seven shared

cases, the reasoning contained the correct positive magnitude and the final answer

field emitted its negative.

For example, the reasoning concluded 18000, then the constrained answer became

-18000.

That pattern suggested a representation problem at the answer boundary, not a reason

to build another grammar engine.

The hypothesis: compile the contract before generation

The caller's external contract remains authoritative. But the model does not

necessarily need to generate that exact wire representation directly.

The intervention was:

External contract{"reasoning": "...", "answer": "18000"}compile ↓Internal model-facing contract{"reasoning": "...", "answer": 18000}constrained generation ↓Deterministic transducerinteger 18000 -> canonical string "18000"external validation ↓Returned contract{"reasoning": "...", "answer": "18000"}

This design has four important properties:

1. It uses one model call.
2. It does not relax the external schema.
3. It does not use an LLM to repair another LLM's output.
4. It fails closed on ambiguous or unsupported values.

The core transducer is deliberately boring:

def canonical_integer_string(value: object) -> str:if isinstance(value, bool) or not isinstance(value, int):raise TypeError("internal answer must be a JSON integer and not a boolean")return str(value)def transduce_integer_object(internal_value):if set(internal_value) != {"reasoning", "answer"}:raise ValueError("internal object must contain exactly reasoning and answer")if not isinstance(internal_value["reasoning"], str):raise TypeError("internal reasoning must be a string")return {"reasoning": internal_value["reasoning"],"answer": canonical_integer_string(internal_value["answer"]),}

After conversion, the rebuilt object is validated against the original signed

numeric-string schema. If parsing, typing, transformation, or external validation

fails, no external object is returned.

This is not post-hoc answer correction. The sign and magnitude generated by the

model are preserved exactly.

I wrote the decision rule before running the new matrix

The gate question was frozen before launching the 7B intervention:

Does a native JSON integer as the hard-constrained, model-facing answer

representation recover the semantic losses observed with a signed numeric string,

while deterministic stringification restores validity under the original external

contract?

The Green criteria were:

1. At least five percentage points of recovery over the matching constrained signed-string baseline.
2. A majority of the shared sign-loss cases repaired.
3. 100% final external validity after transduction.
4. No new systematic semantic failure.

The model, dataset, item order, chat template, greedy decoding, seed, FP32 precision,

256-token cap, and backend versions remained frozen.

Relative to the previous prompt, the integer prompt changed only the symbolic answer

representation:

signed-string template: "answer": ""integer template: "answer":

A regression test compares the new prompt to the first accepted baseline row and

asserts that this replacement is the only change.

Layered scoring prevented a misleading result

The experiment reports four separate outcomes:

1. Semantic correctness: does the extracted numeric value match the gold answer?
2. Internal validity: did the model satisfy the model-facing integer schema?
3. External validity: did deterministic transduction produce an object satisfying the unchanged caller schema?
4. Contract-valid correctness: is the answer both semantically correct and externally valid?

This separation matters. A system that emits perfect JSON with the wrong number is

not successful. A mathematically correct response with the wrong API type is not

immediately usable either.

The primary product metric was contract-valid correctness after transduction.

First gate: an 18-item targeted suite

I mechanically derived the targeted suite from the frozen baseline artifacts. It

contained:

1. The union of all constrained semantic losses.
2. Matched controls where prompting, Outlines, and XGrammar were all correct.
3. Shared difficult cases where all conditions were wrong.
4. A recorded inclusion reason for every item.

The manifest recorded 9 Outlines losses, 9 XGrammar losses, 8 shared losses, and 10

unique losses across the two backends. The resulting suite had 18 items.

I ran four conditions:

1. Prompt-only integer.
2. Outlines integer.
3. XGrammar integer.
4. XGrammar unsigned numeric string, explicitly labeled diagnostic.

The unsigned-string condition was useful because every gold answer in this subset

was positive. But it was not contract-equivalent: the original external schema permits

negative integers, so an unsigned internal language cannot be the general solution.

Targeted result

The aggregate tie hid useful paired information:

1. Outlines integer repaired 7/8 shared signed-string losses.
2. XGrammar integer repaired 8/8.
3. Prompted integer, XGrammar integer, and XGrammar unsigned string had the same item-level correctness set.

The result cleared the preregistered threshold to advance. It did not count as final

confirmation because the suite was deliberately enriched for known failures.

What the answer-boundary trace showed

I wrapped the XGrammar Hugging Face logits processor and captured compact diagnostics

near the answer boundary. The trace records top pre-mask and post-mask candidates,

selected tokens, known sign and digit scores, and the number of masked vocabulary

entries. It does not store a full-vocabulary tensor.

Three items were traced:

1. gsm8k_test_173, a shared sign flip.
2. gsm8k_test_1216, a backend-difference case.
3. gsm8k_test_12, a matched control.

For gsm8k_test_173, the generated reasoning ended with the correct value 18000.

At the integer answer boundary after whitespace:

token "1" pre-mask score:39.63token "-" pre-mask score:-1.33selected token: "1"

The integer grammar still permits legitimate negative values. The intervention did

not simply ban the minus sign. Instead, the changed representation exposed a token

path on which the digit was overwhelmingly preferred.

That is consistent with the hypothesis, but three traces are diagnostic evidence,

not a universal causal proof.

Full confirmation: 150 new 7B generations

After the targeted gate passed, I ran the three integer conditions on the complete

frozen 50-item subset:

1. 50 prompt-only integer generations.
2. 50 Outlines integer generations.
3. 50 XGrammar integer generations.

One contradictory GSM8K reference remained in every raw artifact and was excluded

only from the predeclared cleaned analysis. No other row was removed.

Independent validation accepted all 150 rows:

1. 50 unique item IDs in every condition.
2. Expected dataset, runner, schema, and source hashes.
3. No generation errors.
4. No token-cap hits.
5. 100% final external validity.
6. Complete trace coverage.

The complete result

The architecture figure is generated from the implemented path. The recovery figure

is generated directly from the accepted paired-summary.json; its counts and rates

are not manually typed into the image.

Both constrained backends gained 7 net correct answers, or 14.3 percentage points,

while preserving 100% external validity.

The original constrained gap was 18.4 points. The intervention recovered about 78%

of that gap.

The negative-answer cluster also disappeared in this matrix. Both constrained

backends moved from 12/49 negative answers to 0/49.

Paired effects, including the new failures

Aggregate improvement is not enough. A treatment can repair some items and silently

break others.

Outlines

signed-string correct: 30/49integer + transducer correct:37/49difference:+14.3 pointspaired 95% interval:[+4.1, +26.5]newly correct / newly wrong:8 / 1two-sided exact paired p:0.0391

XGrammar

signed-string correct: 30/49integer + transducer correct:37/49difference:+14.3 pointspaired 95% interval: [0.0, +28.6]newly correct / newly wrong: 10 / 3two-sided exact paired p:0.0923

Outlines clears the conventional 0.05 threshold. XGrammar has the same point

estimate, but more discordant items, a wider interval touching zero, and a p-value

above 0.05.

The project threshold was not defined as "obtain p < 0.05 on every backend." It was

defined as meaningful recovery, majority repair, perfect final validity, and no

systematic replacement failure. Both backends passed that rule, but the uncertainty

around XGrammar belongs in the conclusion.

The incremental constraint cost disappeared under the integer representation

The cleanest comparison is not integer-constrained versus signed-string prompted.

Those conditions use different representations.

The matched comparison is integer-constrained versus integer-prompted.

All three integer conditions scored 37/49.

At the item level:

1. Outlines versus prompted integer had 3 treatment-only and 3 control-only correct items, for zero net difference.
2. XGrammar and prompted integer had the same correctness vector: 0 treatment-only and 0 control-only items.

So, in this experiment, I no longer observed an aggregate semantic tax from applying

the grammar after the representation was aligned.

That is the most important engineering result of the follow-up.

What did not work perfectly

The intervention did not restore every answer.

Prompted signed-string generation had 39/49 recoverable semantic correctness, while

all integer conditions had 37/49. The prompted representation difference was -4.1

points with a paired interval from -12.2 to +4.1 and exact p = 0.625.

The aligned systems also retained ordinary reasoning errors:

1. 12/49 items were still wrong.
2. Outlines repaired 8 baseline errors and newly missed 1.
3. XGrammar repaired 10 and newly missed 3.

Representation alignment fixes a representation-associated failure. It does not

turn a 7B model into a perfect arithmetic solver.

An integration bug the local smoke test caught

The local Qwen2.5-0.5B smoke test exposed a separate XGrammar issue in my runner.

I initially reused one stateful Hugging Face logits processor across multiple

generations. The first item passed and later items failed with assertions. The fix was

to build a fresh processor for every generation.

The failed attempt is preserved as a diagnostic. The corrected five-item XGrammar

smoke passed 5/5 external validity, and the accepted cloud artifacts contain no such

errors.

This was a useful reminder: a structured-generation benchmark can be invalidated by

state management even when its schema and scoring logic are correct.

What this result proves

Within the declared setup, the evidence supports these statements:

1. The model-facing answer representation was materially associated with the constrained semantic loss.
2. A native integer plus deterministic external transduction recovered 14.3 points for both tested backends.
3. The original external signed-string contract remained 100% valid.
4. The observed negative-answer cluster disappeared.
5. Under the integer representation, neither backend had an aggregate accuracy loss relative to matched integer prompting.

What it does not prove

The evidence does not establish that:

1. Native integers improve every model or schema.
2. Every constrained-decoding failure is a representation problem.
3. The mechanism generalizes beyond Qwen2.5-7B, GSM8K, greedy FP32 decoding, and the tested library versions.
4. The result holds for legitimate negative gold answers. The transducer is tested on negatives, but this GSM8K subset contains positive targets.
5. The 49-item confirmation is an independent replication. The intervention was frozen before the new runs, but the underlying evaluation set also produced the original failure observation.
6. The complete contract compiler already exists.

The XGrammar confidence interval touching zero is another reason to keep the claim

narrow even though its point recovery matches Outlines.

The project decision

The gate is Green.

The result justifies building a bounded compiler layer with a small, auditable set of

safe transformations:

1. Canonical integer-string representation.
2. Field-order restoration.
3. Key aliases with exact inverse mapping.
4. Canonical whitespace policies.
5. Final validation against the unchanged external contract.
6. Explicit refusal when a transformation is ambiguous or unsupported.

The compiler should sit above existing engines such as Outlines and XGrammar. The

goal is not to compete with their grammar execution. It is to choose a safer internal

language for the model, prove that the transformation back is sound, and preserve the

caller's contract.

Before making a general product claim, the next evidence gates are:

1. Replication on an independent model family.
2. An executable tool-call task.
3. A fresh evaluation split.
4. Property tests over the supported schema subset.
5. Integration through a production inference path.

Inspect or reproduce the evidence

The repository contains the protocol, failure catalogue, schema variants,

transducer, tests, raw JSONL rows, manifests, compact traces, validation reports, and

paired summary:

1. Complete repository
2. Representation-alignment results
3. Frozen gate protocol
4. Full artifact validation
5. Machine-readable paired summary
6. Compact XGrammar trace

The practical lesson is simple:

Do not assume the caller's wire format is the best language for the model.

Sometimes the safest way to preserve the external contract is to compile it into a

different internal representation, generate there, and transform back with code that

is deterministic enough to audit.

In this experiment, that small change recovered most of the lost accuracy without

weakening the guarantee.