Compound Prompt Constraints: LLM Code Generation Degrades 3–12% Under Combined Format, Persona, Urgency

Production prompts for LLM code generation are never one thing. They specify an output format (JSON, XML, markdown), assign a persona ("expert Python developer"), and set the tone ("this is critical — ship it fast"). Each constraint looks reasonable in isolation. But a new full-factorial study reveals something unsettling: combining individually neutral or beneficial constraints can degrade code generation reliability by 3–12 percentage points beyond what additive predictions would suggest — and the effect depends on which model architecture you use, not how large it is.

The Problem: Compound Constraints Are Untested

Prompt engineering research typically varies one factor at a time: does a persona help? Does JSON output hurt? But real prompts compound them. A CI pipeline asks for "JSON output, you are a senior engineer, this deployment is urgent." The authors of this paper argue that the interaction between constraints — not their individual effects — is what matters, and that the community has been testing single factors while deploying compounds.

They designed a controlled 3×3×3 factorial experiment across three dimensions: output format (free-form, JSON, XML), persona (none, novice, expert), and urgency framing (none, moderate, high). That's 27 distinct prompt conditions, tested on all 164 HumanEval+ problems across five OpenAI models spanning three families (GPT-4o, GPT-4.1, o3-mini), for a total of 22,140 greedy-decoding evaluations. Each output was run through a format-aware extraction pipeline to separate formatting failures from reasoning failures, with significance assessed via McNemar's test and 95% confidence intervals.

Method: Full-Factorial Design With Interaction Decomposition

The key analytical innovation is decomposing each compound condition's pass rate into an additive prediction (sum of individual constraint effects) plus a residual interaction term. A negative residual means the combination degrades performance more than the sum of its parts — a super-additive failure mode that single-factor experiments cannot detect.

The authors controlled for prompt length effects, randomized condition ordering, and verified that constraint semantics were preserved across all 27 templates. The format-aware extraction pipeline was critical: it allowed them to distinguish between "the model couldn't produce valid JSON" and "the model produced valid JSON with wrong logic," isolating where each constraint actually bites.

mindmap
  root((Prompt Constraints
3×3×3 Design))
    Format
      Free-form
      JSON
      XML
    Persona
      None
      Novice
      Expert
    Urgency
      None
      Moderate
      High
    Interaction
      Super-additive
      Architecture-dependent
      Size-independent

Results: Architecture-Dependent Degradation Up to 12.2 Percentage Points

The results are stark and architecture-specific:

• GPT-4o family: Consistent super-additive degradation. The largest interaction was -12.2 percentage points on GPT-4o-mini for the combination of JSON format + expert persona + moderate urgency. JSON combinations consistently produced larger negative interactions than XML.
• GPT-4.1 family: Largely resistant to compound constraint interactions. The more recent architecture handled the compound conditions without notable super-additive degradation.
• o3-mini: A qualitatively different pattern — structured output constraints (JSON, XML) improved performance relative to free-form, even when combined with persona and urgency framing. The reasoning model appears to benefit from the scaffolding.

These results establish a critical finding: vulnerability to compound constraints is architecture-dependent, not size-dependent. GPT-4o and GPT-4.1 are different architectures from the same lab, trained on similar data, yet respond entirely differently to the same prompt combinations. The GPT-4o family's pass@1 dropped 3–12 pp beyond additive predictions; GPT-4.1 was flat.

Limitations

The study is limited to OpenAI models and a single benchmark (HumanEval+). All models are from one provider, and HumanEval+ is a relatively simple coding benchmark — the interaction effects may differ on more complex, multi-file coding tasks. The experiment uses greedy decoding (temperature 0), so the interaction effects under stochastic sampling remain unknown. Additionally, the persona and urgency conditions use fixed phrasings; different wordings could shift results. The paper also cannot determine why the interaction occurs mechanistically — whether from attention dilution, token distribution shifts, or something else in the forward pass.

Why This Matters for Anyone Building With LLMs

This paper has immediate practical implications. If you're building a coding agent or CI pipeline that layers constraints onto prompts, you cannot assume the effects are additive. A persona that helps in isolation might hurt when combined with JSON output and urgency framing. The safe path: test your specific compound prompt on your specific model. Don't extrapolate from single-factor benchmarks.

The results also suggest a hierarchy of fragility: GPT-4o → GPT-4.1 → o3-mini, from most to least vulnerable to compound interactions. If reliability matters, prefer architectures that show resistance to interaction effects, or test compounds exhaustively before deployment. The paper's recommendation to make "compound-prompt testing standard in reliability assessment" is not academic hand-wringing — it's a concrete engineering takeaway backed by 22,140 data points.