Running AlphaFold3 (single seq mode) 3 times faster on TPUs
Background
Recently I’ve been training some protein design flow matching models on TPUs. To evaluate them protein folding models play an important role. So to keep things simpler (GCP buckets etc), I wanted to use AF3 on TPUs as well. However, the current AF3 codebase didn’t seem to support inference on TPUs.
Correct, we don’t support TPU at the moment.
— GitHub issue #550 [1], October 2025
Running AF3 with XLA
Unsupported doesn’t mean we cannot run it on TPUs.
AF3 is written in JAX and it compiles through XLA, which runs natively on TPUs. So what’s the issue? For TPUs it does not have an optimized kernel for pair-biased attention.44 AF3 actually has four pair-biased attentions: triangle attn., the single attn in the pairformer, the diffusion transformer and the atom transformers. Only triangle attention goes through tokamax (since its the most heaviest). Because of this, when you run AF3 on a v6e chip it runs slow even compared to a 5090 55 Actually on both GCP and Modal, the CPU perf was slower compared to my 5090.
To dig a little deeper: AF3 calls tokamax, an OpenXLA library of fused JAX kernels, and --flash_attention_implementation picks which tokamax backend runs.
However, on a TPU we hit two dead ends:
- tokamax’s fused GPU kernels (
triton, the default, andcudnn) don’t run on TPUs. - tokamax’s TPU kernel (
mosaic) doesn’t accept an attention bias at all 66 The check is inpallas_mosaic_tpu_common.py[3] (as of tokamax 0.0.17), and AF3′s triangle attention needs one: the pair bias.
If we don’t care about performance, we can just run the standard AF3 inference code with this flag 77 Defined in run_alphafold.py [4].
--flash_attention_implementation=xla
With it, AF3 computes attention with ordinary JAX operations instead of a hand-written GPU kernel. XLA compiles those for the TPU like the rest of AF3′s JAX code. But since this automatic compilation doesn’t result in a fused kernel, it writes the full attention-logit tensor to memory and reads it back 88 Flash attention (Dao et al., 2022 [5]) computes softmax(QKᵀ + bias)·V in tiles held in fast on-chip memory, keeping a running maximum and sum for the softmax, so the N × N matrix of logits is never written to the slower main memory (HBM). That removes the largest memory traffic in attention. The TPU kernel later in this post uses the same idea in a simpler form: at these sizes a whole row of logits fits on-chip, so it never has to tile across keys.. How much does this hurt us?
Optimizations in a nutshell
If we look at the timing for a full end to end run on a v6e chip, it runs in about 5.8 seconds. For context I measured this on a 5090 as well.
How much do precision errors propagate?
Here are all the optimizations in one place, with what each one does to AF3′s predictions. Two of them just reorganize when and where the work happens, so the outputs stay the same as the original AF3 inference code’s.
The other four change the floating-point arithmetic. The kernels round the same way as AF3′s own code on more than 99% of values, but through 200 denoising steps even that can change the final structure slightly.
| Change | What it does | Time per fold | Outputs vs stock AF3 |
|---|---|---|---|
| Pipelined driver | featurizes the next inputs on the CPU while the TPU runs the current one | 5.81 → 3.80 s | identical, bit for bit |
| Hoisting | computes the diffusion head’s position-independent work once per fold, not at each of the 200 steps | → 3.56 s | identical, bit for bit |
| Pallas triangle attention | fused attention that keeps the logits on-chip | → 3.21 s | rounding: 99.996% of values identical |
| Pallas transition | LayerNorm, SwiGLU and output projection in one kernel | → 3.08 s | rounding: 99.97% identical |
| Pallas triangle multiplication | fused triangle multiplication | → 2.77 s | rounding: 99.2–99.6% identical |
| Tighter padding | pads to the next multiple of 16 tokens (192 here) instead of 256 | → 1.81 s | rounding, plus new starting noise for 4 of the 5 diffusion samples |
Details about the kinds of optimizations I did
Pipelining: keeping the TPU busy
Standard AF3 inference runs a fold one step at a time. First the host CPU featurizes the input. In summary, it
- parses it,
- builds the token and atom features and
- generates reference conformers for the ligands.
On the TPU VM that takes 1.85 s, on a single core 99 on the modal instance (h100) this cpu time was 1.69 s. Interestingly my 5090 CPU was way faster (0.77 s). Then the TPU runs the model, and then the CPU unpacks the results. While one side works, the other waits.
The pipelined driver overlaps them. A pool of CPU worker processes featurizes the next inputs while the TPU is busy, so the features are ready when it finishes. The model call is sent without waiting for the previous fold, and the main process unpacks and writes that fold’s results while the TPU runs the next one. The weights also stay on the TPU, instead of being handed over again with every call.While this seems like a text book simple optimization, I was honestly suprised how helpful this becomes for heavy workloads 1010 say you fold 1000s of structures.
None of this changes the numerical results. The same functions run on the same inputs in the same order, so the outputs are bit for bit identical to the standard runner’s: all 10 folds on the v6e, and on an H100 every output array of the fold I compared. A fold drops from 5.81 to 3.80 s on the v6e 1111 How much it helps depends on the host CPU, because featurization runs on one core. The TPU VM’s AMD EPYC 9B14 is about 2.4× slower per core than the RTX 5090 workstation’s Core Ultra 9 285K, so hiding featurization pays off most on the TPU: 1.53× on the v6e and 1.61× on the H100 (5.57 → 3.45 s), against 1.24× on the 5090..
Hoisting: what AF3 recomputes 1,000 times
AF3 predicts a structure in two stages. The trunk (48 Pairformer blocks, run 11 times with recycling) builds a single representation for each token and a pair representation for each pair of tokens, once per fold. The diffusion module then turns noise into atom coordinates over 200 denoising steps, for each of 5 samples: 1,000 calls per fold.
Each call does two kinds of work. Anything that touches the noisy coordinates or the noise level has to run again:
- the atom encoder,
- the transformer and
- the decoder, and
- the single conditioning that carries the noise level into every layer.
However, we are lucky that the pair conditioning is different. It is built only from the trunk’s pair representation and where the tokens sit in the sequence, so it never sees the noise level or the coordinates, and the diffusion transformer only reads it, to make the attention bias of each of its 24 layers. Those biases come out the same in all 1,000 calls.
Standard AF3 inference still recomputes them every time. XLA usually moves work like this out of a loop by itself, but here it refuses: one of the steps is a gather whose output is about 12 times larger than its input, and XLA does not hoist operations that grow their data. So I compute them once per fold and reuse them. On the v6e the outputs stay bit for bit identical 1212 On GPUs, XLA compiles the reorganized computation slightly differently, so the outputs agree only up to rounding. and a fold drops from 3.80 to 3.56 s1313 This hoisting comes at a cost: memory. The bias maps (24 layers × 16 heads, one map each) now stay in memory for the whole run instead of one block at a time, about 1.5 KB per token pair. That is under 0.1 GB for the 192-token designs here, but about 1.6 GB at 1,024 tokens.
The same trick does not work in the trunk. There, triangle multiplication and triangle attention rewrite the pair representation in every block, so nothing derived from it stays fixed.
Better kernels in Pallas
I’m hoping to write a separate explainer on this soon. As it needs to go deep in Pallas kernel writing. Take a look at that note if you are interested in learning more!
Do the predictions change?
Pipelining and hoisting just organizes the computations, so that does not change results. However, the kernels can change outputs slightly through rounding 1414 especially when propagating through 48 pairformer blocks, so I checked them in two ways.
First, against real structures. I folded 40 proteins from the PDB from their deposited sequences (with their ligands, no MSA), once with original AF3 code and once with the kernels, with the same padding and seed, and scored each prediction against the deposited structure: lDDT over all the protein’s heavy atoms, where 1 is a perfect match.
If the kernels changed accuracy, the points below would move off the diagonal. For scale, the other two panels make changes that have nothing to do with the kernels: a new random seed 1515 Here the seed means the diffusion model’s seed. Rest of the Pairformer outputs stay the same., and running original AF3 inference code on an RTX 5090 instead of the TPU.
TPU v6e, same seed
typical change 0.027
stock AF3, RTX 5090
typical change 0.015
stock AF3, same seed
typical change 0.025
On average kernels do not shift the predicted structures, but I did see 3 structures with larger lDDT deviations. We also see similar deviations when it comes to hardware changes anyway.
Measured as Cα RMSD to the deposited structure instead of lDDT, the picture is the same:
TPU v6e, same seed
typical change 1.89 Å
stock AF3, RTX 5090
typical change 1.74 Å
stock AF3, same seed
typical change 2.31 Å
Related implementations
After I worked on this with claude + codex on optimizing AF3 for TPUs, I realized that I didn’t do much of a existing literature search. So we revisited Anthropic’s FlashPairformer code [6] that include three close comparisons, all targeting GPUs through Pallas’s Triton backend:
- Triangle attention: tiled flash attention with the pair bias shared across triangle rows.
- Transition: fused LayerNorm, SwiGLU and output projection.
- Triangle multiplication: fused input processing, a contraction, and fused output processing. Its source explicitly notes that fusion changes BF16 rounding and is not bitwise identical to stock execution.
However, Anthropic’s repo mainly targets NVIDIA GPUs through the Pallas kernels. With only their compiler settings switched to TPU ones, the triangle-attention and triangle-multiplication kernels do not compile on a TPU: their memory layouts break the TPU’s block-shape rules.
Apart from Anthropic’s repo, JAX’s own Pallas FlashAttention for TPUs [2] does accept an additive attention bias, but only as a separate copy for every row.1616 Triangle attention runs one attention per row of the pair grid, and every row adds the same bias: one N × N matrix per head, computed from the pair representation. JAX’s kernel takes the bias as an input shaped [batch, heads, queries, keys], so with the rows as the batch, that shared bias has to be repeated once per row. At 256 tokens with AF3′s 4 heads, that is about 67 million bias values instead of 262 thousand, written to memory before the kernel runs and read back inside it. The kernel in this post reads the one shared bias directly. But it was 1.6 – 4.4x slower than the kernel discussed in this post. Most of the gap is its handling of AF3′s 32-wide heads; building the per-row bias costs the rest.
Separately, JAXBench: Benchmarking Autonomous TPU Kernel Optimization [7] includes an AlphaFold2-style triangle-multiplication workload in its TPU v6e benchmark; its reference workload [8] differs from the exact AF3 module.
In a future post, I’d like to dive into the details of the 3 kernels.
References
- AlphaFold 3 contributors (2025). “AlphaFold 3 TPU support discussion: GitHub issue 550.”
- The JAX Authors (n.d.). “JAX Pallas TPU FlashAttention implementation.”
- OpenXLA contributors (n.d.). “Tokamax: Pallas Mosaic TPU attention input-support checks.”
- Google DeepMind (n.d.). “AlphaFold 3 inference runner: flash-attention implementation flag.”
- Dao et al. (2022). “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.”
- Anthropic (n.d.). “FlashPairformer: Pallas kernels for biomolecular modeling on GPUs.”
- Tschand et al. (2026). “JAXBench: Benchmarking Autonomous TPU Kernel Optimization.”
- JAXBench contributors (n.d.). “JAXBench: AlphaFold2-style triangle-multiplication reference workload.”
MoveThanks!

Comments
Join the conversation with your GitHub account.