Scaling RSI on Inference Beyond the Scaling Law with Ave and Data-Flow Invariants
Scaling Self-Improving Inference Kernels Beyond the Scaling Law
Modern large language models (LLMs) increasingly exhibit broad, human-level capabilities and may be approaching a tipping point for recursive self-improvement (RSI). GPU kernel optimization is one of the first places where that improvement can happen: a faster kernel makes the model that wrote it cheaper to serve. It is a small step, but a critical one toward a faster flywheel for artificial intelligence. Frontier models such as GPT 6 Astra and Claude Fable can already write functionally correct GPU kernels. The essential question is whether RSI can beat the scaling law on efficiency, so that each round of self-improvement is economically sustainable.
Today's kernel optimization agents scale beyond human capacity through massive trial and error. They burn tokens in parallel for days or weeks, prototyping ideas, compiling and debugging, reading profiles, and trying again, and they reach good performance only after a long and uncertain search. The search is expensive because the feedback is sparse: unit tests and profiles report that a kernel is wrong or slow, but not which transformation broke it, and reasoning about aliasing and data races at the assembly level fills the context until the model loses track of the kernel it is editing. Guided by taste and calculation rather than random trials, an experienced GPU engineer understands the performance impact of an optimization before writing it, makes sure it is implemented faithfully, and evaluates the result across multiple layers with rigorous profiling. This approach is principled and resource-efficient, but its velocity is bounded by human speed.
We developed Ave to combine the best of both worlds: the faithful implementation of deep optimizations that an expert engineer provides, at a velocity that exceeds human capacity. Ave is built on data-flow invariants, properties of the data that every transformation must preserve. A matrix multiplication, for example, may be tiled, vectorized, and shuffled into whatever layout the matrix cores want, but each output element must still be the dot product of matching elements from the two inputs. Ave makes specifying and validating such invariants a first-class part of kernel development: a tile-based domain-specific language (DSL) states them, the compiler validates them at compile time, and a violation returns concrete values, such as the thread and the mismatched elements, so the agent pinpoints the bug instead of searching for it.
To our knowledge, Ave is the first kernel optimization agent that reliably delivers real-world production kernels, including flash attention, general matrix multiplication (GEMM), and fused Mixture-of-Experts (MoE), at the level of human experts on AMD platforms. These three kernels account for up to 90% of GPU time in real-world inference workloads. On AMD Instinct MI300X, Ave reaches performance comparable to AMD's vendor libraries, AITER and hipBLASLt, both of which represent tens of engineer-years of hand optimization, and 1.62–1176x the throughput of existing agentic approaches. It is also extremely time- and token-efficient: with GPT-5.6 Sol, it takes a naive GEMM kernel to state-of-the-art performance in well under an hour using roughly $6 of tokens, about 100x more efficient in tokens per unit of performance than other approaches. Ave opens a path toward a realizable RSI in which the infrastructure improves faster than the scaling law of the model itself. It is open source at github.com/causalflow-ai/avelang under the Apache license.
Ave is an optimization agent rather than an exploration agent. It implements complex, multi-level transformations faithfully when their correctness depends on cross-layer data flow; discovering new algorithms or selecting tile sizes remains the job of its knowledge base or of a separate exploration front end. The front end chooses what to try, and Ave implements it correctly at low token cost.
This is the first of two blog posts on Ave. It focuses on Ave's capabilities and on how Ave optimizes production kernels such as attention and GEMM; the full design and evaluation are in our SOSP 2026 paper. The second post describes how we integrate Ave with Hyperloom to push the frontier of kernel optimization, targeting novel kernels such as MegaMoE on AMD platforms.
A Motivating Example: Optimizing an Attention Kernel on MI300X
Attention is the central mechanism of modern LLMs. Conceptually, it computes scores S = QK^T, a row-wise P = softmax(S), and an output O = PV. Context lengths now reach one million tokens, which makes materializing the full N×N score matrix prohibitively expensive. Flash attention uses online softmax to avoid it: it keeps a block of Q in registers, streams K and V through shared memory in tiles, and accumulates O as it goes, so the working set always fits in the memory hierarchy.
A naive implementation of this algorithm is still roughly 10x slower than a vendor-optimized kernel, and the gap is closed entirely by hardware-specific engineering. On the AMD Instinct MI300X, the optimized kernel reshuffles data to feed the matrix cores, driven by MFMA (matrix fused multiply-add) instructions, that accelerate both matrix multiplications. It tunes the layouts and tile sizes of Q, K, and V to fit shared memory and L2, and transposes V on the fly with cooperative, vectorized loads so that O = PV can also run on the matrix cores. It uses an XOR-swizzled shared-memory layout to eliminate bank conflicts. One level deeper, it pipelines and specializes wavefronts (AMD's 64-thread warps) to overlap computation with data movement. Finally, experts hand-tune the assembly for instruction scheduling and register allocation to minimize pipeline stalls. AITER's hand-written attention kernel uses 11 pipeline stages across two wavefront groups to get all of this right.
Trial and error is ineffective here for two reasons. First, each optimization reshuffles data layouts, and the kernel is correct only if every reshuffle agrees with every other: the shuffles introduced by a transpose, a pipeline stage, and a register assignment must all line up, or the result is a silent bug that is hard to pinpoint. A unit test reports only that the output is wrong, which leaves humans and LLMs alike no choice but to debug by trial and error. Second, the order of optimizations matters. Tuning instruction scheduling or register allocation is meaningless until the kernel uses the matrix cores and is pipelined well enough to nearly saturate the hardware. A random walk through this space spends most of its tokens on steps that cannot pay off yet, or on repairs it cannot localize.
Data-Flow Invariants in Ave
Recall the matrix-multiplication invariant from the introduction, applied to S = QK^T: however many transformations the kernel has applied, every multiply must pair an element of Q with an element of K from the same attention head and the same position along the reduction dimension d. Ave expresses such invariants with two constructs. Ave is a tile-based DSL in which all memory is accessed through tiles, and the data in a tile can carry tags, user-defined tuples of coordinates and other variables of the programmer's choosing. A tag function labels the elements of a tile with their tags, and a tag assertion is a compile-time check that two tags are equal, or that they are not.
Figure 1 shows the two constructs enforcing the QK^T invariant. The tag function labels every element of Q and K with its reduction index d, shown as a color. Q is loaded once per wavefront into registers. K is transposed while it is staged in shared memory and then read back with 16-byte vector loads into a swizzled register layout, so its elements reach the registers by a very different route. At the matrix-core instruction that multiplies them, a single assertion requires the paired elements of rQ and rK to carry the same tag.
The assertion survives every transformation because the tags ride with the data rather than with the code. Every load, store, view, transpose, and swizzle carries each element's tag along; the compiler re-derives tags when the access width changes, as when V is read as 4-byte words, stored as 8-byte halves, and read back as 16-byte vectors; and when a pipeline stage reuses a shared-memory buffer, the tags in it are reset so the next stage starts clean. The assertion therefore holds or fails on the data that actually reaches the multiply, and it constrains only the pairing: where rows of Q or columns of K end up is left to the kernel. An inequality assertion covers a different class of bugs, such as one wavefront overwriting a shared-memory region that another is still reading.
All of this tracking is program analysis inside the compiler, so the tags never exist in the generated kernel and there is no runtime cost. The next section describes what happens when a check fails and how that feedback drives the optimization loop.
The Ave Optimization Loop
Figure 2 shows how Ave uses these invariants in an agentic loop. Each round proposes a few optimizations, implements them with their invariants attached, and learns from what the compiler and the profiler report.
The planner retrieves candidate optimizations from a persistent, expert-curated knowledge base of hardware facts, DSL references, and optimization recipes. Each recipe pairs a transformation with an invariant template, so proposing "use MFMA for QK^T" also proposes "assert that the Q and K operands carry matching head and reduction tags." A selector orders compatible proposals into a plan, and a lowering agent turns the plan into DSL code, instantiating the templates as concrete tag functions and assertions. The compiler checks every assertion. Only a kernel that passes them all proceeds to unit tests and profiling, and a candidate replaces the current best only on a measured speedup.
When an assertion fails, the compiler reports more than the failure. It encodes the propagated tags and the assertion as constraints for an SMT solver, which returns a concrete witness: the thread ID, loop index, and tensor coordinates at which the tags disagree. For the agent, the difference is between "the output is wrong" and "thread 0 pairs K element (0, 1) with Q element (0, 0) at the second MFMA." The witness tells the agent where to look instead of leaving it to search.
Both the witness and the measured latency feed the planner's next round through in-context reinforcement learning (ICRL) with text gradients. Each round is scored on whether the candidate compiled, how many of its assertions held, whether it passed its tests, and how much faster it ran. The model weights never change; what changes is the planner prompt. The LLM writes a textual critique of the round and applies it as a constrained edit to the prompt that drives the next one, so the planner learns the order and the pitfalls of a particular kernel as it goes.
A concrete example from the paper illustrates the process. The planner proposed tiling the scalar QK^T dot products onto 32×32×16 MFMA accumulations. The lowering agent staged K in shared memory and materialized the tags, but read K back through a flattened (512, 2) layout. The kernel compiled, but the compiler reported tag(tK[0]) ≠ tag(tQ[0]) with a witness at thread 0 and loop index 1, so tests and profiling did not run. The critique traced the mismatch to the flattened access, and in the next round the planner specified the (2, 32, 8, 2) view of shared memory and the exact pairing of tK with rQ. That candidate passed every check and was profiled. Without the witness, the agent would have seen only a wrong output tensor, with no indication of which staged load was responsible.
Evaluation
We evaluated Ave on a server with eight AMD Instinct MI300X GPUs over three production kernel families: BF16 GEMM, causal grouped-query flash attention matching LLaMA-3 70B prefill under eight-way tensor parallelism, and fused MoE using DeepSeek-V3.2's block-quantized FP8 configuration. Each workload started from a naive kernel and ran for up to ten optimization rounds with GPT-5.6 Sol. The expert baselines are hipBLASLt for GEMM and AITER for attention and MoE; the agentic baselines are KernelFalcon, KSearch, CUDAForge, and Atrex.
Figure 3 compares Ave, KernelFalcon, and Atrex using the performance and token costs reported in Table 1 of the paper. Token costs include failed attempts.
Figure 3 shows that Ave reaches 0.89x of hipBLASLt on GEMM, 0.99x of AITER on attention, and 0.94x of AITER on fused MoE, and improves geometric-mean throughput by 1.62–1176x over the uncontaminated agentic baselines. The gap comes from global, intrusive optimizations that Ave applies and the baselines do not, such as transposing V on the fly in attention and quantizing intermediate values to FP8 in swizzled order in fused MoE. KernelFalcon stops at 0.12–0.55x because its Triton-level stack cannot express the low-level optimizations that deliver the last 2x, and agents that generate CUDA or HIP directly, including KSearch and CUDAForge, fall furthest behind, largely because they never reach the matrix cores for attention.
Ave also reaches this performance at a small fraction of the cost. It finishes each operator in 33–44 minutes of model time for $4.67–7.01 and under one million tokens. Atrex, the only other agent that approaches library performance, spends 50–200 million tokens per operator, 86x to 263x more than Ave, and reaches 0.91x on attention only by copying the AITER reference kernel from its knowledge base. KernelFalcon spends fewer tokens than Ave, but on kernels that stay at a fraction of library throughput.
To see what each part of the harness contributes, we ran it with the knowledge base but without ICRL to order the recipes. The planner attempted instruction scheduling before pipelining; scheduling had no effect, and performance stalled at the bank-conflict stage. With ICRL but without invariants, the planner found the right order, but the agent could not diagnose and repair the V^T layout in the first step, so performance stayed near the naive kernel at 194 TFLOPS, against 588 TFLOPS for the full system. Only the full system completes the sequence. ICRL supplies the order, and invariants make each step checkable.
Ave's reach extends beyond these three kernel families and one model. On KernelBench, it produces valid kernels within three attempts for 100 of 100 Level 1 tasks and 88 of 100 Level 2 tasks, and we observe similar results with DeepSeek V4 Pro. Supporting a new architecture is also inexpensive: enabling NVIDIA H200 took one person-week, and the same harness reaches parity with cuBLASLt, FlashAttention-3, and DeepGEMM there.
Lessons Learned
Our work on Ave demonstrates the power of symbolic feedback, of which invariants are one concrete example. With LLMs, the bottleneck of development is no longer writing code but identifying bugs effectively. Program analysis detects errors far more efficiently and precisely than tests do, and invariants with witnesses give a succinct way to specify correctness globally. A witness turns a multi-round repair into a one-round repair and keeps the context small. Stronger models do not substitute for this structure. More tokens alone did not close the gap in our experiments; lowering agents without invariants still exhausted their contexts on trial-and-error repairs, and weaker models benefited from invariants the most.
The construct also opens up a new set of optimization opportunities. Optimizing at the lowest level of the hardware is expensive without a map, and invariants provide exactly the guardrails that make it tractable. With them, an LLM can work on low-level optimizations directly instead of relying on the fixed heuristics of a compiler.
Conclusion
Ave demonstrates that data-flow invariants, checked at compile time, let a frontier model apply assembly-level optimizations and reach the performance of hand-optimized libraries for about $6 and well under one million tokens per operator, on AMD MI300X and NVIDIA H200 alike.
The broader lesson is that recursive self-improvement on inference does not have to scale with tokens. Wherever correctness depends on cross-layer data flow, including collective communication, storage engines, and compiler pipelines, pairing a domain abstraction with actionable compile-time feedback offers a path to agentic optimization that gets cheaper as it goes.
Ave is open source at github.com/causalflow-ai/avelang, and the paper is available on arXiv. In the next post in this series, we describe how we combine Ave with Hyperloom to extend this approach to MegaMoE kernels on AMD GPUs.