Computing & Software

Dense Attention Matrix & QKV Memory Calculator

Estimate explicit attention-score and Q/K/V tensor payloads for a dense multi-head attention layer.

How the calculation works

Explicit score matrix payload = batch*heads*tokens**2*bytes_per_value/1048576 MiB. Three Q/K/V tensor payloads = 3*batch*tokens*heads*head_dim*bytes_per_value/1048576 MiB. Combined named tensor payload = batch*heads*tokens**2*bytes_per_value/1048576+3*batch*tokens*heads*head_dim*bytes_per_value/1048576 MiB.

Worked example

Example inputs — Sequence length: 1024; Batch size: 1; Attention heads: 8; Features per head: 64; Bytes stored per tensor value: 2. Expected results — Explicit score matrix payload: 16 MiB; Three Q/K/V tensor payloads: 3 MiB; Combined named tensor payload: 19 MiB.

Assumptions and limits

Analytical software/hardware scenario, not a benchmark or a live provider quote. Use the actual architecture, workload and billing definitions; overheads not named in the model are excluded. Explicit dense score storage, excluding gradients, outputs, allocator overhead and temporary workspaces. Fused/FlashAttention kernels may avoid materializing the full score matrix, so this is not a GPU-memory requirement guarantee.

Reference and further reading

Read the underlying subject reference. Calculith is an independent implementation; the reference does not endorse or certify this tool.

Related calculators

Browse all 1000 calculators →

Example values are editable. Results are estimates where assumptions apply. Calculations run in your browser. Updated 2026-11-16.