Here’s a habit that sounds dull and has turned out to be one of the most useful things we do.
Before we run an experiment, we write a short note: what we’re testing, what result would count as a win, and what we’ll do if it doesn’t win. We commit that note to version control. Only then do we run anything and look at the numbers.
Clinical trials have worked this way for years; it’s called pre-registration. In machine learning it’s still rare, and it’s easy to see why. It feels like paperwork standing between you and the fun part.
Why bother
Anomaly detection makes it very easy to fool yourself. Anomalies are rare, so a test set might hold only a handful of real events, and a handful of events can swing a score a long way. Try enough settings and something will look great by luck. Decide what “great” means after seeing the results, and you’ll find it every time.
Writing the test down first has already caught us out more than once:
- An idea that looked like a clear win only won in one corner of the parameter grid.
- A baseline’s score moved a long way with a default setting nobody had chosen.
- A result that looked solid didn’t repeat on a second dataset.
None of these were fun to find. All of them were much better found by us than by someone relying on the result.
Losing counts too
The other half of the habit: we report every run, including the ones that lose. A negative result tells you where the real problem is, and it’s often not where you expected.
Try it
Our note fits on half a page:
- The question, in one sentence.
- The data, and how it’s split into training and test.
- The metric, and the number that counts as a win.
- What we’ll do if it wins, and what we’ll do if it doesn’t.
- The date and the commit, before any results exist.
That’s all there is to it, and it has saved us from a few very convincing mistakes.
TupleTrace builds anomaly detection for spacecraft telemetry that explains every alert. See how it works, or download the brochure.