What changed, and do we care?

Keeping up with Minecraft means a lot of reading. Since Mojang started churning snapshots like there’s no tomorrow, we have to check each snapshot for anything that might affect our projects. Some changes are obvious. Others turn out to matter more than the changelogs suggest.

We’ve started using AI to automate parts of this.

Reading between the lines

Reading a changelog is easy. Figuring out what it means for Oraxen or an anticheat is harder.

For Oraxen, even small changes can matter.

On Paper 1.21.2+, breaking or placing a block above or below a custom note block could briefly make it look like a vanilla note block. The client predicts the block change and recalculates the neighboring note block’s instrument, which also changes the model our pack shows.

Oraxen had to adjust how it handles those block changes and send the correct states back to the client. Nothing about that sounds like a resource pack change, but players still end up seeing the wrong block.

So we use AI to flag possible breaking changes. We also use it to look for new features or detection ideas worth exploring. If block interactions change, for example, we want to know whether our existing checks still work, whether there’s an edge case worth testing, or whether the new behavior could be useful for detecting modified clients.

Somewhere to start

A useful finding gives us a reason to open the source code or explore. A rewording of the release notes doesn’t help much.

Instead of going through the entire snapshot ourselves, we can start with a few suspected changes and why they might matter. The findings aren’t always correct, but checking a specific claim is usually easier than finding it ourselves.

It’s also useful for coming up with ideas. A new item component might let Oraxen support something that previously needed a workaround mechanic.

Not every idea leads anywhere, and that’s fine.

Quite impressive

Snapshots can break things in unexpected ways, including stuff that has nothing to do with the changes listed in the changelog. For Oraxen, assuming a snapshot is safe just because it doesn’t mention the shaders we use is a good way to end up with broken packs.

So we’ve started automating some of the testing too.

Snapshots got us started, but we now use agents for regular testing too. Reproducing bugs, checking fixes, or testing existing features before releasing a new version.

Using Codex’s Computer Use feature and a few scripts for managing Minecraft servers, we can let AI join a server and try to reproduce issues on its own.

Some of the things it does are pretty impressive. One agent enabled subtitles to check whether a sound feature was working. Another downloaded and set up WorldEdit without us ever mentioning it, and ended up finding a bug while testing a feature.

But it also misses things.

Computer Use doesn’t see every frame of the game. It works with screenshots, which makes it easy to miss brief visual glitches or performance issues.

We’ve had cases where an agent reproduced and patched an issue but completely missed the lag that came with it. It checked the relevant behavior, saw nothing wrong in the screenshots, and concluded that everything worked.

And that’s really the problem with relying on it too much.

I still think it’s worth using. It can do a surprising amount on its own, and having it handle the initial research and testing saves us from repeating the same work.

But I wouldn’t trust it to tell us that everything works just yet.