Early on, we found a real academic paper by outside researchers that had already tested some of the same questions we were asking about the manuscript. Rather than start over, we wanted to build on their work — reuse their code, check their numbers, and add our own tests on top. Before we could trust any of it, though, we had to make sure we were actually looking at the same thing they were.
The paper pointed to the wrong version of its own code
The paper named a specific snapshot of its own code and data — like a paper citing "page 42" of a book — so anyone could go find exactly the version they used. We went and got it. It was empty. That exact snapshot didn't contain any of the scripts the paper said it used to produce its results.
That's a real problem: if you can't find what a paper says it used, you can't actually check its work, and neither can anyone else.
We didn't guess — we checked
Instead of assuming the paper was simply wrong, or quietly picking whatever version looked close enough, we did the boring, careful thing: we looked at the project's full history of saved versions, one at a time, to see exactly when the missing scripts actually appeared. We found them — in the very next saved version after the one the paper named, saved on the same day the paper itself was submitted. That's about as clear as evidence gets: the authors almost certainly meant to point to that version and cited the wrong one by a single step.
We wrote this up plainly as a real defect in the paper's own citation — not a mistake in their science, not a reason to distrust their results, just an easy, fixable error in the paperwork pointing to those results. Then we used the correct version going forward, and said so, so anyone checking our work later would know exactly which version we mean.
Why this matters more than it might sound like
This project's whole approach depends on results being checkable, not just believable. If we had shrugged and used the "close enough" version, or hadn't checked at all, every later comparison we made against that paper's numbers could have been quietly comparing against the wrong thing — and neither we nor anyone reading our work would have known. Catching it early meant every later use of that paper's results (and we used them a lot — see our "tried to fake it a dozen different ways" highlight) is standing on solid ground.