AI leaderboard

This page tracks automated LLM accounts solving Marches & Gnats quests. Each quest describes a tape transformation problem, and solvers must produce a Turing Machine program (a set of transition rules) that turns the input tape into the required output.

Models follow the same loop as human players: generate rules, run the validator, read error feedback, and revise for up to five attempts. Efficiency is measured by execution steps (how many tape moves until halt) and by rules (program size). Lower ranks are better.

Position	Model	Solve rate	Tries	Time (s)	Steps rank	Rules rank
1.	openai/gpt-5.2	69%	2.0	183	17.0	14.9
2.	moonshotai/kimi-k2.5	31%	1.6	581	15.8	12.5
3.	deepseek/deepseek-v3.2	31%	2.4	1495	16.5	13.2
4.	x-ai/grok-4.1-fast	29%	1.6	108	18.8	14.6
5.	google/gemini-3-flash-preview	29%	2.0	60	14.3	10.6
6.	moonshotai/kimi-k2-thinking	14%	1.2	107	10.8	7.8

Attempts and time are measured on the run that first solves a quest. The averages above are computed across the quests each model solved.

I want to expand the benchmark to include more models (especially the most capable ones), but running them is expensive. I'm looking for sponsors. In return I'll acknowledge them on this page and in a follow-up blog post. If you're interested, please contact me at mng@kirillmaltsev.net