Running AlphaFold3 (single seq mode) 3 times faster on TPUs

Background

Recently I’ve been training some protein design flow matching models on TPUs. To evaluate them protein folding models play an important role. So to keep things simpler (GCP buckets etc), I wanted to use AF3 on TPUs as well. However, the current AF3 codebase didn’t seem to support inference on TPUs.

Correct, we don’t support TPU at the moment.

— GitHub issue #550 [1], October 2025

Running AF3 with XLA

Unsupported doesn’t mean we cannot run it on TPUs.

AF3 is written in JAX and it compiles through XLA, which runs natively on TPUs. So what’s the issue? For TPUs it does not have an optimized kernel for pair-biased attention.44 AF3 actually has four pair-biased attentions: triangle attn., the single attn in the pairformer, the diffusion transformer and the atom transformers. Only triangle attention goes through tokamax (since its the most heaviest). Because of this, when you run AF3 on a v6e chip it runs slow even compared to a 5090 55 Actually on both GCP and Modal, the CPU perf was slower compared to my 5090.

To dig a little deeper: AF3 calls tokamax, an OpenXLA library of fused JAX kernels, and --flash_attention_implementation picks which tokamax backend runs.

However, on a TPU we hit two dead ends:

  1. tokamax’s fused GPU kernels (triton, the default, and cudnn) don’t run on TPUs.
  2. tokamax’s TPU kernel (mosaic) doesn’t accept an attention bias at all 66 The check is in pallas_mosaic_tpu_common.py [3] (as of tokamax 0.0.17), and AF3′s triangle attention needs one: the pair bias.
Figure 1: Why AF3 has no fast attention path on a TPU. AF3 hands its pair-biased triangle attention to tokamax. tokamax’s fused GPU kernels don’t run on TPUs, and its TPU kernel doesn’t accept a bias, which leaves the plain-XLA path: it runs anywhere, but unfused.

If we don’t care about performance, we can just run the standard AF3 inference code with this flag 77 Defined in run_alphafold.py [4].

--flash_attention_implementation=xla

With it, AF3 computes attention with ordinary JAX operations instead of a hand-written GPU kernel. XLA compiles those for the TPU like the rest of AF3′s JAX code. But since this automatic compilation doesn’t result in a fused kernel, it writes the full attention-logit tensor to memory and reads it back 88 Flash attention (Dao et al., 2022 [5]) computes softmax(QKᵀ + bias)·V in tiles held in fast on-chip memory, keeping a running maximum and sum for the softmax, so the N × N matrix of logits is never written to the slower main memory (HBM). That removes the largest memory traffic in attention. The TPU kernel later in this post uses the same idea in a simpler form: at these sizes a whole row of logits fits on-chip, so it never has to tile across keys.. How much does this hurt us?

Optimizations in a nutshell

If we look at the timing for a full end to end run on a v6e chip, it runs in about 5.8 seconds. For context I measured this on a 5090 as well.

featurization (host CPU)stock AF3, XLA attentionstock AF3, fused Triton attentionpipelining + hoisting+ Pallas kernels+ tighter padding
Figure 2: End to end time for one fold of a 180-residue designed protein with two ligands at AF3′s standard settings (5 diffusion samples, 10 recycles, 200 denoising steps, no MSA): the median of 10 folds (2 designs × 5 seeds), compilation excluded. Stock AF3 featurizes on the host CPU (hatched) before each fold; the pipelined driver overlaps that with the accelerator, so it no longer adds to the time. The Pallas kernels are TPU-only; the GPUs get the same driver changes and tighter padding. With everything applied, the v6e (1.81 s) is 1.7× faster than the RTX 5090 (3.11 s) and 1.34× faster than the H100 (2.43 s).

How much do precision errors propagate?

Here are all the optimizations in one place, with what each one does to AF3′s predictions. Two of them just reorganize when and where the work happens, so the outputs stay the same as the original AF3 inference code’s.

The other four change the floating-point arithmetic. The kernels round the same way as AF3′s own code on more than 99% of values, but through 200 denoising steps even that can change the final structure slightly.

ChangeWhat it doesTime per foldOutputs vs stock AF3
Pipelined driverfeaturizes the next inputs on the CPU while the TPU runs the current one5.81 → 3.80 sidentical, bit for bit
Hoistingcomputes the diffusion head’s position-independent work once per fold, not at each of the 200 steps→ 3.56 sidentical, bit for bit
Pallas triangle attentionfused attention that keeps the logits on-chip→ 3.21 srounding: 99.996% of values identical
Pallas transitionLayerNorm, SwiGLU and output projection in one kernel→ 3.08 srounding: 99.97% identical
Pallas triangle multiplicationfused triangle multiplication→ 2.77 srounding: 99.2–99.6% identical
Tighter paddingpads to the next multiple of 16 tokens (192 here) instead of 256→ 1.81 srounding, plus new starting noise for 4 of the 5 diffusion samples
Table 1: Every change in this post, in the order it is applied, with the warm time per fold after it (the bench and settings of the timing figure above). “Identical” means every raw output of the model, all 5 diffusion samples included, matches stock AF3 bit for bit on all 10 bench folds.

Details about the kinds of optimizations I did

Pipelining: keeping the TPU busy

Standard AF3 inference runs a fold one step at a time. First the host CPU featurizes the input. In summary, it

On the TPU VM that takes 1.85 s, on a single core 99 on the modal instance (h100) this cpu time was 1.69 s. Interestingly my 5090 CPU was way faster (0.77 s). Then the TPU runs the model, and then the CPU unpacks the results. While one side works, the other waits.

The pipelined driver overlaps them. A pool of CPU worker processes featurizes the next inputs while the TPU is busy, so the features are ready when it finishes. The model call is sent without waiting for the previous fold, and the main process unpacks and writes that fold’s results while the TPU runs the next one. The weights also stay on the TPU, instead of being handed over again with every call.While this seems like a text book simple optimization, I was honestly suprised how helpful this becomes for heavy workloads 1010 say you fold 1000s of structures.

STOCK AF3: ONE THING AT A TIME5.81 s per foldCPUTPUfeaturizefold 1fold 2fold 33 folds: 17.4 sPIPELINED: CPU WORK HIDES BEHIND THE TPU3.80 s per foldCPU workersTPUCPU mainfeaturizefold 1fold 2fold 33 folds: 13.4 s024681012141618 sfeaturization (host CPU)model, stockmodel, pipelinedunpack and write results
Figure 3: The pipelined driver (bottom) featurizes ahead on CPU worker processes and unpacks each fold while the TPU runs the next, so the TPU runs the folds back to back.

None of this changes the numerical results. The same functions run on the same inputs in the same order, so the outputs are bit for bit identical to the standard runner’s: all 10 folds on the v6e, and on an H100 every output array of the fold I compared. A fold drops from 5.81 to 3.80 s on the v6e 1111 How much it helps depends on the host CPU, because featurization runs on one core. The TPU VM’s AMD EPYC 9B14 is about 2.4× slower per core than the RTX 5090 workstation’s Core Ultra 9 285K, so hiding featurization pays off most on the TPU: 1.53× on the v6e and 1.61× on the H100 (5.57 → 3.45 s), against 1.24× on the 5090..

Hoisting: what AF3 recomputes 1,000 times

AF3 predicts a structure in two stages. The trunk (48 Pairformer blocks, run 11 times with recycling) builds a single representation for each token and a pair representation for each pair of tokens, once per fold. The diffusion module then turns noise into atom coordinates over 200 denoising steps, for each of 5 samples: 1,000 calls per fold.

Each call does two kinds of work. Anything that touches the noisy coordinates or the noise level has to run again:

However, we are lucky that the pair conditioning is different. It is built only from the trunk’s pair representation and where the tokens sit in the sequence, so it never sees the noise level or the coordinates, and the diffusion transformer only reads it, to make the attention bias of each of its 24 layers. Those biases come out the same in all 1,000 calls.

ONCE PER FOLDTrunkPairformer48 blocks× 11 passeszᵢⱼsᵢEVERY DENOISING CALL · 200 steps × 5 samples = 1,000 calls per foldxₜAtom encoderatoms → tokensTransformer24 blocks · 16 headsAtom decodertokens → atomsx̂₀denoisedSingle conditioningsᵢ + noise level σₜscales every layerPair conditioningzᵢⱼ + relative positionAttention biasestransformer + atom windowsNo σₜ, no xₜ:same in all 1,000 calls↑ hoisted: computed once per foldsame in every callchanges every call
Figure 4: One fold of AF3′s diffusion. Green is computed from the trunk’s output alone and is the same in every denoising call; blue depends on the noisy atoms or the noise level and changes every call. Hoisting moves the green row out of the loop.

Standard AF3 inference still recomputes them every time. XLA usually moves work like this out of a loop by itself, but here it refuses: one of the steps is a gather whose output is about 12 times larger than its input, and XLA does not hoist operations that grow their data. So I compute them once per fold and reuse them. On the v6e the outputs stay bit for bit identical 1212 On GPUs, XLA compiles the reorganized computation slightly differently, so the outputs agree only up to rounding. and a fold drops from 3.80 to 3.56 s1313 This hoisting comes at a cost: memory. The bias maps (24 layers × 16 heads, one map each) now stay in memory for the whole run instead of one block at a time, about 1.5 KB per token pair. That is under 0.1 GB for the 192-token designs here, but about 1.6 GB at 1,024 tokens.

The same trick does not work in the trunk. There, triangle multiplication and triangle attention rewrite the pair representation in every block, so nothing derived from it stays fixed.

Better kernels in Pallas

I’m hoping to write a separate explainer on this soon. As it needs to go deep in Pallas kernel writing. Take a look at that note if you are interested in learning more!

Do the predictions change?

Pipelining and hoisting just organizes the computations, so that does not change results. However, the kernels can change outputs slightly through rounding 1414 especially when propagating through 48 pairformer blocks, so I checked them in two ways.

First, against real structures. I folded 40 proteins from the PDB from their deposited sequences (with their ligands, no MSA), once with original AF3 code and once with the kernels, with the same padding and seed, and scored each prediction against the deposited structure: lDDT over all the protein’s heavy atoms, where 1 is a perfect match.

If the kernels changed accuracy, the points below would move off the diagonal. For scale, the other two panels make changes that have nothing to do with the kernels: a new random seed 1515 Here the seed means the diffusion model’s seed. Rest of the Pairformer outputs stay the same., and running original AF3 inference code on an RTX 5090 instead of the TPU.

AF3′s confidence (mean pLDDT), left half before, right half after:very high (≥ 90)confident (70–90)low (50–70)very low (< 50)
Pallas kernels
TPU v6e, same seed
with the kernels
0.20.20.40.40.60.60.80.8mean change −0.002
typical change 0.027
stock AF3
New random seed
stock AF3, RTX 5090
seed 231
0.20.20.40.40.60.60.80.8mean change +0.001
typical change 0.015
seed 230
Other hardware
stock AF3, same seed
TPU v6e
0.20.20.40.40.60.60.80.8mean change +0.008
typical change 0.025
RTX 5090
Figure 5: Each dot is one PDB protein: how close AF3′s top structure is to the deposited one before (x) and after (y) a change, measured by all-atom lDDT (1 = perfect match). Dots on the diagonal did not change. Colours show AF3′s own confidence (mean pLDDT): left half before, right half after. “Mean change” is the average of after − before; “typical change” is the average size of the change. Single sequence (no MSA), AF3-LA weights, AF3′s standard settings.

On average kernels do not shift the predicted structures, but I did see 3 structures with larger lDDT deviations. We also see similar deviations when it comes to hardware changes anyway.

Measured as Cα RMSD to the deposited structure instead of lDDT, the picture is the same:

AF3′s confidence (mean pLDDT), left half before, right half after:very high (≥ 90)confident (70–90)low (50–70)very low (< 50)
Pallas kernels
TPU v6e, same seed
with the kernels (Å)
00101020203030mean change +0.07 Å
typical change 1.89 Å
stock AF3 (Å)
New random seed
stock AF3, RTX 5090
seed 231 (Å)
00101020203030mean change +0.30 Å
typical change 1.74 Å
seed 230 (Å)
Other hardware
stock AF3, same seed
TPU v6e (Å)
00101020203030mean change +0.45 Å
typical change 2.31 Å
RTX 5090 (Å)
Figure 6: The same comparison in Cα RMSD to the deposited structure (Å; lower is better). Most of these predictions are far from the deposited structure (median about 17 Å), so this mostly tracks where the uncertain parts end up. On the residues AF3 is confident about (pLDDT above 70), the before and after structures differ by a median 0.73 Å with the kernels, 0.52 Å with a new seed and 0.70 Å on other hardware.

Related implementations

After I worked on this with claude + codex on optimizing AF3 for TPUs, I realized that I didn’t do much of a existing literature search. So we revisited Anthropic’s FlashPairformer code [6] that include three close comparisons, all targeting GPUs through Pallas’s Triton backend:

However, Anthropic’s repo mainly targets NVIDIA GPUs through the Pallas kernels. With only their compiler settings switched to TPU ones, the triangle-attention and triangle-multiplication kernels do not compile on a TPU: their memory layouts break the TPU’s block-shape rules.

Apart from Anthropic’s repo, JAX’s own Pallas FlashAttention for TPUs [2] does accept an additive attention bias, but only as a separate copy for every row.1616 Triangle attention runs one attention per row of the pair grid, and every row adds the same bias: one N × N matrix per head, computed from the pair representation. JAX’s kernel takes the bias as an input shaped [batch, heads, queries, keys], so with the rows as the batch, that shared bias has to be repeated once per row. At 256 tokens with AF3′s 4 heads, that is about 67 million bias values instead of 262 thousand, written to memory before the kernel runs and read back inside it. The kernel in this post reads the one shared bias directly. But it was 1.6 – 4.4x slower than the kernel discussed in this post. Most of the gap is its handling of AF3′s 32-wide heads; building the per-row bias costs the rest.

Separately, JAXBench: Benchmarking Autonomous TPU Kernel Optimization [7] includes an AlphaFold2-style triangle-multiplication workload in its TPU v6e benchmark; its reference workload [8] differs from the exact AF3 module.

In a future post, I’d like to dive into the details of the 3 kernels.

References

  1. AlphaFold 3 contributors (2025). “AlphaFold 3 TPU support discussion: GitHub issue 550.”
  2. The JAX Authors (n.d.). “JAX Pallas TPU FlashAttention implementation.”
  3. OpenXLA contributors (n.d.). “Tokamax: Pallas Mosaic TPU attention input-support checks.”
  4. Google DeepMind (n.d.). “AlphaFold 3 inference runner: flash-attention implementation flag.”
  5. Dao et al. (2022). “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.”
  6. Anthropic (n.d.). “FlashPairformer: Pallas kernels for biomolecular modeling on GPUs.”
  7. Tschand et al. (2026). “JAXBench: Benchmarking Autonomous TPU Kernel Optimization.”
  8. JAXBench contributors (n.d.). “JAXBench: AlphaFold2-style triangle-multiplication reference workload.”
KUDOSDon’t
Move
Thanks!