Computing & Software
Dense Attention Matrix & QKV Memory Calculator
Estimate explicit attention-score and Q/K/V tensor payloads for a dense multi-head attention layer.
How the calculation works
Explicit score matrix payload = batch*heads*tokens**2*bytes_per_value/1048576 MiB. Three Q/K/V tensor payloads = 3*batch*tokens*heads*head_dim*bytes_per_value/1048576 MiB. Combined named tensor payload = batch*heads*tokens**2*bytes_per_value/1048576+3*batch*tokens*heads*head_dim*bytes_per_value/1048576 MiB.
Worked example
Example inputs — Sequence length: 1024; Batch size: 1; Attention heads: 8; Features per head: 64; Bytes stored per tensor value: 2. Expected results — Explicit score matrix payload: 16 MiB; Three Q/K/V tensor payloads: 3 MiB; Combined named tensor payload: 19 MiB.
Assumptions and limits
Analytical software/hardware scenario, not a benchmark or a live provider quote. Use the actual architecture, workload and billing definitions; overheads not named in the model are excluded. Explicit dense score storage, excluding gradients, outputs, allocator overhead and temporary workspaces. Fused/FlashAttention kernels may avoid materializing the full score matrix, so this is not a GPU-memory requirement guarantee.
Reference and further reading
Read the underlying subject reference. Calculith is an independent implementation; the reference does not endorse or certify this tool.
Related calculators
- Password Entropy & Brute-Force Time Estimate
- CPU Instruction Count & CPI Runtime Calculator
- Amdahl & Gustafson Parallel Speedup Calculator
- Roofline Arithmetic Intensity Performance Calculator
- Memory Bus Theoretical Bandwidth Calculator
Example values are editable. Results are estimates where assumptions apply. Calculations run in your browser. Updated 2026-11-16.