We were checking whether two of our generators matched a specific pattern in the real manuscript — how predictable text gets as you group letters into larger and larger chunks. A paper published by outside researchers had already measured this same pattern for the real Voynich text, so instead of remeasuring it ourselves, we planned to compare our generators directly against the paper's published numbers.
Before trusting an outside number, we checked it ourselves
Before running that comparison, we did something we've made a habit of on this project: we ran the exact same measurement on the real manuscript text ourselves, through our own pipeline, just to make sure our numbers and the paper's numbers actually meant the same thing. They didn't. Our own measurement of the real manuscript came out meaningfully different from the paper's published values — not a rounding error, but a real difference, at every single comparison point.
Digging in, the cause was clear: the outside paper had measured this pattern using a different method than our own pipeline uses (holding out different sections of the manuscript for testing). Comparing our generators to the paper's numbers directly would have been comparing two different kinds of measurement and calling it a fair test — a mistake that could have produced a misleading conclusion in either direction.
What we did about it
We didn't quietly swap in whichever number made our result look better. We wrote up the mismatch plainly, in public, as its own finding — and then used only our own consistently-measured numbers as the fair comparison basis for testing our generators, treating the outside paper's numbers as background context only.
This wasn't a one-time save. It's a pattern we've built into how this project works: earlier, we also caught a case where a word-length statistic we'd cited from an earlier pass turned out to come from a different way of splitting up the text than our newer scripts use, and — separately — a formatting quirk in the raw manuscript transcription file that was silently shifting where six word boundaries fell on one page. In every case: found, disclosed, fixed, and rechecked — before it could distort a conclusion.