Designing AI resistant technical evaluations

The core challenge is creating technical evaluations that remain meaningful as AI capabilities improve. A test that distinguishes human skill levels today may be "trivially solved by models tomorrow," forcing continuous redesign to maintain signal about candidate abilities.

Effective technical evaluations should be representative of real work, high-signal with wide scoring distribution, free of narrow domain knowledge requirements, and genuinely engaging. The original test used a simulated accelerator with features like manually managed scratchpad memory, VLIW, SIMD, and multicore parallelism.

Claude 3.7 Sonnet reached 50% of human performance, Claude Opus 4 outperformed most human applicants, and Claude Opus 4.5 matched even the strongest human candidates within the 2-hour limit. Each advancement forced test redesign.

The simulated machine included accelerator-typical features—manually managed memory, VLIW instruction packing, SIMD vector operations—testing skills needed for real TPU/GPU optimization work. The parallel tree traversal problem was deliberately non-deep-learning flavored to avoid requiring specific domain knowledge.