I should start with a confession: almost every interesting result in this project showed up first as a bug.
The value function looked wrong before it looked right, the defensive zones came out more valuable than the penalty box, which is the kind of thing that should make you stop and distrust everything downstream of it. The headline policy comparison flipped sign between two runs of the same code with a different random seed, which meant the number I nearly wrote down as “the effect” was, for a while, just noise wearing a confidence interval as a costume. None of this is a special property of football, or of me being particularly careless. It’s what happens whenever you try to build something quantitative that continuously simulates itself: every layer you add is a new place for a plausible-looking number to be wrong in a way that doesn’t announce itself.
I find this to be an important insight about statistical models -- they do not, and possibly cannot, account for the fact that there is another team on the field who will notice and react to changes you make