{"path":"listening/first-time-right-2026-11-05/report-llm-constraint-methods-2026-11-05.md","content":"# First-pass compliance for readability-gated scripts: the literature as a helper read it (2026-11-05)\n\nThis is the report of a read-only helper (a language model, Sonnet) made on 2026-11-05 for\n[first-time-right-2026-11-05.md](../first-time-right-2026-11-05.md), kept verbatim as it came\nback. Every number carries its source and a read-status tag the helper assigned; its interpretation is marked as its own\nat the end. The helper was handed the gates' numbers and today's failure (median 10 against 9, ninetieth percentile 22\nagainst 16, an 84-word stretch, pace 119) and asked for the measured literature on length and readability control,\nverifier loops, constrained decoding, slotted writing, chunking, and the linter-against-formatter frame.\n\n---\n\nRESEARCH REPORT: first-pass compliance for readability-gated spoken scripts\nTags: [full text] read in full, [part] read excerpts or sections, [abstract only], [secondary] reported by someone else. I paraphrase except where marked. Most studies use GPT-3.5/GPT-4-era models. I found no 2026 measurement of sentence-length-cap compliance for Claude.\n\nBOTTOM LINE\n- Prompts alone move averages, not tails or narrow bands.\n- Gains come from externalizing the counter, shrinking the unit, and giving a program (not the model) the job of judging.\n- Hosted APIs enforce structure, not sentence length or readability.\n- Verifier loops converge in 1-2 rounds, then flatten.\n- Product tools fix only the cases with one right answer.\n\n1. LENGTH AND READABILITY FROM INSTRUCTIONS\n- LIFEBench (26 models, 10,800 items, targets 16-8,192 words): models follow short targets, then \"deteriorate sharply\" past a threshold. Under \"equal to\", 23 of 26 models score below 60; best is o3-mini at 75.4. Authors' diagnosis: models cannot tell how many words they have written. [part] https://proceedings.neurips.cc/paper_files/paper/2025/file/c84f00ccee3d35cee1901acb0e258dc7-Paper-Datasets_and_Benchmarks_Track.pdf\n- Max-word limits: GPT-4 Turbo broke them 49.3% (AlpacaEval-LI) and 44.2% (MT-Bench-LI). Training on length instructions (LIFT-DPO) cut one model's violations from 65.8% to 7.1%. [part] https://arxiv.org/html/2406.17744\n- Per-sentence length (\"the 7th sentence has 29 words\"): best model GPT-4 got 12% exact, 68% within plus or minus 3 words. Errors skew short. [part] https://ar5iv.labs.arxiv.org/html/2310.16343\n- Examples can hurt numeric targets. Models copy example length and miss when the requested number is not in the examples. [part] https://arxiv.org/pdf/2310.14542v1.pdf. ChatGPT word-count success was 0.41. [secondary] https://liner.com/review/evaluating-large-language-models-on-controlled-generation-tasks\n- Readability by instruction (Flesch or Fumble): ChatGPT mean FKGL was 8.83 unprompted, 5.16 once a grade was named, and 4.57 with metric plus description. No model or prompt hit the Grade 2 band (FKGL 1-3). ChatGPT's CEFR accuracy was 0-13%, usually one level too high. [full text] https://aclanthology.org/2023.gem-1.18.pdf\n- Putting the standard's content in the prompt (Standardize): GPT-4 exact CEFR accuracy went 0.227 to 0.540, adjacent accuracy 0.630 to 0.906. [abstract only] https://doi.org/10.48550/arxiv.2402.12593\n- Few-shot beat zero-shot for Lexile targeting, but I saw no delta. Manual review found edits unevenly spread, so readability varied across an article. [part] https://arxiv.org/html/2406.12787v1\n- TSAR 2025: one-shot demos helped Claude and o1, hurt Gemini at B1, and helped easy-to-spot levels (short sentences) more than diffuse ones. [full text] https://aclanthology.org/2025.tsar-1.20.pdf\n- Visible counters (CAPEL): a countdown-marker suffix lifted gpt-4.1 exact-length match from 9.7% to 96.3%. On MT-Bench-LI, strict compliance rose from under 30% to over 95%, beating draft-then-revise. [part] https://arxiv.org/html/2508.13805\n- Anthropic's docs say sentence or paragraph limits work better than word limits. [part] https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-latency\n- Decode-time readability control exists in research: lookahead decoding hit requested FRE 90/70/30 as 87.1/70.3/30.9 in one illustrated example. It needs logit access. [part] https://aclanthology.org/2023.emnlp-main.714.pdf\n\n2. GENERATE-THEN-VERIFY\n- Self-checking: without external feedback, accuracy fell after self-correction in every model and benchmark tested. Earlier gains came from oracle labels. GPT-3.5 kept its GSM8K answer 74.7% of the time. [part] https://arxiv.org/abs/2310.01798\n- Kamoi et al. survey: no prior work shows prompted-LLM feedback working in general tasks. Self-correction works where reliable external feedback exists, with code execution as the example. [abstract + part] https://arxiv.org/abs/2406.01297\n- Multi-constraint instructions (DeCRIM): Self-Refine gave marginal gain on one benchmark and a drop on IFEval. A self-critic gave minimal gain; an oracle critic gave +16.9 and +20.3 for Mistral. [part] https://arxiv.org/html/2410.06458v2\n- Self-Refine's own gain (~20 points absolute average, up to 4 iterations) came with weak initial prompts and two calls per round (feedback, refine). [abstract + part] https://proceedings.neurips.cc/paper_files/paper/2023/file/91edff07232fb1b55a505a9e9f6c0ff3-Paper-Conference.pdf. Kamoi flags the weak-baseline issue. For the total-cost part, see section 5.\n- External classifier loop, readability (HIT-YOU, GPT-5, max 5 rounds, 200 items): 129 (64.5%) passed untouched. Rounds needed: 1 for 31, 2 for 14, 3 for 7, 4 for 2; 17 (8.5%) never converged. My cumulative pass rates: 80%, 87%, 90.5%, 91.5%. Meaning similarity fell from 0.85 to 0.79 on refined items. [full text] https://aclanthology.org/2025.tsar-1.20.pdf\n- Constraint-violation feedback on code-instruction tasks: Claude 3.7 Sonnet went 63.0, 71.2, 76.5, 82.3, 83.4% over rounds 0-4; GPT-4o 62.1, 70.6, 74.2, 77.6, 78.6%. Most gain came in rounds 1-2. [part] https://arxiv.org/html/2507.00699\n- Flesch-score loop with GPT-4o over ~2,000 articles: about 650 passed first try. Round 2 sometimes made sentences longer before round 3 fixed it. Authors call two rounds the practical optimum. Preprint, unreviewed. [part] https://www.arxiv.org/pdf/2602.22524\n- Best-of-n versus pass rate for style constraints: I found no rigorous study. Theory says best-of-n matches pass@n only with a perfect verifier. [part] https://arxiv.org/html/2507.12399. A deterministic checker is perfect for what it measures.\n- Weak evidence: an LLM-generated repo with 30 samples reports Claude Sonnet 4 length compliance of 80% single-shot, 98% with 3-sample filtering, 99% with up to 2 revisions. [secondary, low reliability] https://github.com/Hypogenic-AI/llm-output-length-claude\n- Gates stack: GPT-4 scored 76.89% prompt-level versus 83.57% instruction-level on IFEval. [part] https://doi.org/10.48550/arxiv.2311.07911. On nested constraints the best models stayed just above 50% prompt-level. [part] https://arxiv.org/html/2607.27912v1\n\n3. CONSTRAINED DECODING VIA HOSTED APIs\n- Anthropic: JSON schema via output_config.format and strict tools, enforced by constrained decoding. minLength and maxLength, numeric bounds, and minItems above 1 are not enforced. SDKs strip them and validate afterward. Regex pattern is supported in a subset: no lookaround, and complex {n,m} with large ranges is rejected. [part] https://platform.claude.com/docs/en/build-with-claude/structured-outputs\n- Opus 5.5 rejects temperature, top_p, top_k and prefill. [snippet] https://platform.claude.com/docs/en/models/opus-5-5/migration-guide\n- OpenAI: strict JSON schema (100% on its schema evals versus under 40% for gpt-4-0613) [part] https://openai.com/index/introducing-structured-outputs-in-the-api/. GPT-5 and later offer custom tools whose output is held to a Lark or regex grammar (llguidance). The docs advise bounded terminals using {M,N}. [part] https://developers.openai.com/cookbook/examples/gpt-5/gpt-5_new_params_and_tools. A word-capped sentence is expressible there; I found no test of it.\n- Gemini: minItems and maxItems are enforced; minLength and maxLength are silently dropped. [part] https://ai.google.dev/gemini-api/docs/structured-output\n- Research decoders (NeuroLogic, lookahead, llguidance/XGrammar masks) need token-level access to open weights. [part] https://aclanthology.org/2021.naacl-main.339.pdf, https://github.com/guidance-ai/llguidance/blob/92604a74/README.md\n- Net: I found no hosted parameter that enforces sentence length or readability. Slot counts and (OpenAI only) regex terminals are the nearest.\n\n4. PLAN-THEN-WRITE AND SLOTS\n- AgentWrite: plan with per-paragraph word counts, then write serially with earlier text in context. GPT-4o length score on 4k-20k-word targets went 5.6 to 86.8; overall 65.3 to 86.6; quality 91.8 to 91.6 (LLM judge). Writing in parallel cost 6% coherence. [part] https://arxiv.org/abs/2408.07055. LongEval also found plan-based beats direct. [part] https://arxiv.org/html/2502.19103v1\n- Both measure total length, not sentence density.\n- Format cost: one study found stricter JSON schemas lower reasoning accuracy [abstract only] https://arxiv.org/abs/2408.02442. A rebuttal says prompts were unequal and structured did as well or better (GSM8K 0.77 vs 0.78). [part] https://blog.dottxt.ai/say-what-you-mean.html. Both concern reasoning, not prose.\n- Not found: any paper comparing one-sentence-per-slot writing with free prose plus edit.\n\n5. CHUNKING AND WHERE ERRORS CLUSTER\n- Adherence falls as output grows: marked near 4,000 tokens, substantial failure by 16,000, and not explained by \"lost in the middle\" since instructions sat in the first 500 tokens. [part] https://arxiv.org/html/2409.02076v2. Your ~3,000 words is near that edge.\n- DocFollow: constraints hold better early and decay later; topic coherence ~0.9 early to 0.65-0.7 at the end. Sentence-level constraint scores ranged 33.8-62.7%. [part] https://aclanthology.org/2026.emnlp-main.261.pdf\n- Not found: any measurement of sentence-length violations by position.\n- Engineering blogs, unreviewed: per-section briefs with section-only regeneration cost ~50% more tokens than one prompt, no pass-rate data [part] https://dev.to/lukasz_blania_4b7d226fa2a/per-section-briefs-how-to-stop-ai-agents-losing-the-plot-at-2000-words-431o. Block-addressed edits cut five edits from 8,128 to 775 output tokens. [part] https://featuringcode.com/block-addressed-editing-a-10x-output-token-cost-reduction-for-document-llm-agents\n\n6. LINTER VERSUS FORMATTER\n- Prettier: formatting rules become unnecessary because it reprints from scratch; quality rules stay with linters. It declines cases with no good rule (blank lines, multi-line objects, decorators) and keeps the author's choice. [full text] https://prettier.io/docs/comparison, https://prettier.io/docs/rationale\n- Vale: rules can carry fixes (replace, remove, edit, convert, suggest), meant for cases with one right answer. Readability and metric checks only flag. [part] https://docs.vale.sh/topics/actions, https://docs.vale.sh/checks/readability. A third-party guide says Vale is \"not a formatter like Prettier\" because prose fixes need judgment. [secondary] https://docsio.co/blog/vale-linter\n- Hemingway flags hard sentences and offers an AI rewrite. It does not charge a credit if the rewrite fails to lower the grade, a built-in verify step. It warns scores are only a guide and over-shortening reads choppy. [part] https://hemingwayapp.com/help/docs/highlighted-issues, https://hemingwayapp.com/blog/posts/20240201-feb-2024-update\n- Yoast: green when no more than 25% of sentences exceed 20 words; red above 30%; flags only. [part] https://github.com/Yoast/wordpress-seo/blob/trunk/packages/yoastseo/src/scoring/assessments/SCORING%20READABILITY.md. Over 40% of articles initially scored red; they relaxed the transition-word demand until about 35% were red and 35% green. [part] https://yoast.com/content-analysis-methodological-choices-explained/\n- Rule-based splitting: a splitter (semantic roles, splitting on \"and\", clause extraction) was judged correct on 64% of 100 samples, the same as a neural model. The benchmark has exploitable cues. [part] https://aclanthology.org/2020.emnlp-main.91.pdf\n\n7. PRACTITIONER REPORTS (all engineering blogs, no first-pass rates)\n- Podcast pipeline whitepaper, median 4,436-word scripts: under 90% of minimum words gets one expansion pass; over 105% of maximum gets one trim; rejected if it overshoots 1.8x or 0.6x. A deterministic polish pass removed 124 of 138 echoed hand-offs that had survived the older pass. An LLM review pass found switched off in production led to the rule: whatever must hold goes in the prompt or in code. [full text] https://myweirdprompts.com/docs/mwp-pipeline-whitepaper.pdf\n- Video-script pipeline: retries naming the specific failed field or budget were \"the piece that mattered most\"; pacing handled by a deterministic trim to word budget. No numbers. [full text] https://dharmeshjoshi.hashnode.dev/turning-prompt-engineering-into-an-actual-discipline\n- LLM critic plus a cheap deterministic check beside it, because they fail differently. [part] https://brokeit.dev/posts/five-daily-podcasts-no-humans-the-pipeline/\n- Not found: published first-pass pass rates from TTS, audiobook, newsroom or documentation teams.\n\nNOT FOUND\n- Best-of-n curves for stylistic constraints.\n- Slot-writing versus free-then-edit for sentence length.\n- Violation-by-position data for sentence density.\n- Token-cost multipliers for deterministic-verifier loops (Self-Refine's two calls per round are known; the rest is arithmetic).\n\nMY INTERPRETATION (not from the sources)\n1. A model with no running counter writing a whole-text distribution (median, P90, longest stretch, pace) will often miss. If four gates passed independently at 65% each, the joint first-pass rate would be about 18%. That is an illustration, not a measurement, and your gates are correlated.\n2. Split the gates. Pause insertion and pace arithmetic have one right answer, so make them formatter-style: insert a pause line at the sentence boundary nearest the 50-word mark, then recompute pace. Keep sentence length, nested relatives, glossary order and voice as checks.\n3. Check and patch per section (~300-500 words), rewriting only offending sentences. Cap at two rounds, then hand the rest to a splitter or a person.\n4. At the prompt, give budgets in sentences and paragraphs, not words. Test one scratch-pad pass with visible per-sentence word counts (CAPEL logic), stripped before the final. Test numbered slots too. Both are unproven for your exact gates.\n5. Use exemplar passages cautiously: they steer cadence but can pin lengths to what they show.\n6. Cheap probe worth one API call: an Anthropic schema string with a bounded-repeat pattern. A 400 means no.\n7. Keeping the gates is right: external verifiers are the part with solid evidence. The open design question is placement.\n"}