One of our generators had two ingredients mixed together: a rule that links the end of one word to the start of the next ("coupling"), and a rule that invents brand-new words to keep the vocabulary growing ("novelty"). Each ingredient worked fine on its own. Combined, something went wrong — a test that measures whether whole words follow each other in suspiciously predictable order kept failing, every single time.
The first guess was wrong — and we said so
Our first idea was simple: maybe the problem shows up whenever the next word happens to start with one of a small handful of letters the coupling rule favors. We tested it directly. It wasn't true — those particular pairs were actually less predictable than average, not more. Rather than quietly dropping the idea, we published that it failed and moved on to a better question.
Finding the real cause
We went back and read the actual code line by line instead of guessing from the outside. The real cause turned out to be narrower and more mechanical than our first guess: whenever the coupling rule fires, it doesn't pick a word's first letter from the whole alphabet — it forces it into one of just 4 possible values. That's an artificial narrowing, and it's exactly the kind of thing that would make word order look more predictable than it should. We measured this directly (not the letter it produced, but whether the narrowing rule had actually fired) and found a clear, consistent jump in predictability exactly where we expected it.
Then we proved it — the hard way
Finding a plausible cause isn't the same as proving it. So we built a small, targeted fix aimed at only that one narrowing effect, and — before running it even once — we wrote down exactly what we expected to happen: the fix should help a lot when the coupling rule is switched off, and barely help at all when it's switched on, because the coupling rule's own narrowing would still be doing its damage regardless of the fix. We ran it. That is exactly what happened.
That's the part worth bragging about. It's easy to notice a pattern after the fact and tell a convincing story about it. It's much harder to write down a specific, falsifiable prediction in advance and then watch it come true. That's the difference between a hunch and evidence.