Throughput and IOPS

A team running Loki on gp3 EBS gets a one-day ingest spike to 250 MB/s; their normal baseline is 50 MB/s. The bucket side scales fine; the disk side does not. The Loki ingester hits the gp3 burst credit balance, the volume drops to its baseline 125 MB/s, the ingester queue fills, and the front-end log shippers start returning 429s. The on-call engineer discovers that gp3 burst credits accrue at 3 K IOPS and deplete at the burst rate; their spike exceeded the credit balance within 20 minutes.

Throughput and IOPS are different budgets. Choosing the wrong one turns a working observability stack into a throttled one.

What throughput and IOPS are

Two storage performance characteristics that are easy to confuse and almost always conflated.

• IOPS (input/output operations per second). The number of discrete read or write operations the disk can complete per second. Each operation is small (4 KB to 16 KB on NVMe). IOPS matters for random-access workloads like the Prometheus TSDB head block.
• Throughput (megabytes per second). The number of bytes per second the disk can transfer. Throughput matters for sequential workloads like Loki chunk uploads to S3 or Tempo block compaction reads.

   IOPS-bound                          Throughput-bound
   +--------------------+              +--------------------+
   | Random 4 KB writes |              |  Sequential 1 MB   |
   | at high rate       |              |  reads/writes      |
   +--------------------+              +--------------------+
            |                                  |
            v                                  v
   Prometheus TSDB head                Loki chunk upload
   WAL fsync                           Tempo block compaction
   Compaction                          Object store GET
   page cache fault                    Backup reads

The two budgets interact. A disk that delivers 1 M IOPS at 4 KB delivers roughly 4 GB/s throughput. A disk that delivers 1 GB/s throughput with 4 KB operations delivers roughly 250 K IOPS. The interaction is the reason the trade-off exists: you cannot have both at the maximum.

Why a sysadmin cares

Storage performance is the budget that constrains how much data the observability stack can ingest before the stack starts dropping samples. Three failure shapes appear when the budget is unmeasured.

1. The Prometheus head block stalls at 50 K samples per second. A gp3 volume without provisioned IOPS delivers roughly 3 K IOPS. The WAL fsync at 50 K samples per second needs roughly 3.3 K IOPS at 15 s scrape interval. Symptom: head block stops draining; WAL segment count climbs past 10; alerts evaluate as stale.
2. The Loki ingester throttles on a sustained burst. A gp3 volume delivers 125 MB/s baseline throughput plus burst credits. A sustained ingest of 250 MB/s exceeds the baseline and burns through the credits within 20 minutes. Symptom: ingesters report back-pressure; front-end shippers return 429.
3. The Tempo compactor falls behind on read. A low-throughput disk delivers 100 MB/s sequential read. The compactor reads 200 MB/s of blocks during a compaction window. Symptom: compaction windows slip; the block count on disk grows; query latency rises because more blocks must be scanned.

How it works

The Prometheus budget

Prometheus writes 4 KB per sample on the WAL fsync and 4 KB per chunk in the head block. The IOPS budget is:

IOPS = samples_per_second * (WAL_writes + head_writes_per_second)
     = samples_per_second * (1 + 1 / scrape_interval_in_seconds)

For 100 K samples per second at 15 s scrape interval:

IOPS = 100_000 * (1 + 1 / 15)
     = 100_000 * 1.067
     = ~107_000 IOPS

The throughput budget is small by comparison:

throughput = samples_per_second * sample_size_bytes
           = 100_000 * 4
           = 400 KB/s

The TSDB is overwhelmingly an IOPS-bound workload. A disk with high throughput but low IOPS will starve the WAL; a disk with high IOPS but low throughput will serve the WAL and the head block fine.

The Loki budget

Loki ingests log streams into compressed chunks. The throughput budget is:

throughput = ingest_bytes_per_second
           * (1 + replication_factor / compression_ratio)

For 50 MB/s ingest at 3x replication and 5x compression:

throughput = 50 * (1 + 3 / 5)
           = 50 * 1.6
           = 80 MB/s

The IOPS budget is small by comparison (chunk upload is sequential):

IOPS = ingest_bytes_per_second / chunk_size_bytes
     = 50_000_000 / 1_000_000
     = 50 IOPS

Loki is overwhelmingly a throughput-bound workload for the chunk upload path. The ingester itself (which holds chunks in memory before flushing) has a memory budget, not an IOPS budget.

The Tempo budget

Tempo ingests spans into blocks. The throughput budget is:

throughput = spans_per_second * span_size_bytes
           / compression_ratio

For 50 K spans per second at 1 KB per span and 2x compression:

throughput = 50_000 * 1024 / 2
           = 25 MB/s

The IOPS budget is larger than Loki’s because spans are written as separate records:

IOPS = spans_per_second * writes_per_span
     = 50_000 * 4
     = 200 K IOPS

Tempo is throughput-bound on the block upload path and IOPS-bound on the WAL. The WAL is local; the blocks are remote. The two budgets are independent.

Burst vs sustained

EBS volumes have a burst credit model. gp3 accrues 3 K IOPS burst credits per second at the baseline rate; spending credits at a higher rate depletes the balance. The credit balance starts at the full bucket (5.4 M credits for a 100 GB volume).

gp3 baseline IOPS = 3 K (at 125 MB/s baseline throughput)
gp3 burst IOPS    = up to 16 K (with credit balance)
gp3 max IOPS      = 16 K (with provisioned IOPS)

A workload that sustains 5 K IOPS will burn through the initial credit balance within 30 minutes and then be limited to the 3 K baseline. A workload that bursts to 5 K IOPS for 5 minutes every hour will never deplete the credit balance.

io2 Block Express has a different model: it delivers the provisioned IOPS sustained, with no burst concept. The cost is roughly 10x gp3 per GB-month but the performance is predictable.

Under the hood

The relevant kernel and device layers:

• Queue depth. The number of in-flight I/O operations the device can hold. NVMe devices support queue depths in the thousands; SATA SSDs support tens. A workload that generates more concurrent I/O than the device can queue will saturate at the queue depth.
• Block size. The size of each I/O operation. A 4 KB random write and a 1 MB sequential write generate different IOPS-to-throughput ratios on the same device.
• fsync behaviour. A synchronous fsync forces the device to flush its volatile cache. NVMe devices have low fsync latency (sub-millisecond); SATA SSDs have higher fsync latency (milliseconds). The WAL fsync budget is bounded by the fsync latency.

How to configure it

The configuration of throughput and IOPS is mostly a hardware and cloud configuration, not a software one. The relevant choices:

# AWS CLI: create a gp3 volume with provisioned IOPS.
aws ec2 create-volume \
  --volume-type gp3 \
  --size 500 \
  --iops 10000 \
  --throughput 500 \
  --availability-zone eu-west-1a

# AWS CLI: create an io2 Block Express volume.
aws ec2 create-volume \
  --volume-type io2 \
  --size 500 \
  --iops 50000 \
  --availability-zone eu-west-1a

The TSDB path and retention are command-line flags, not prometheus.yml keys:

# /etc/default/prometheus
ARGS="--storage.tsdb.path=/var/lib/prometheus \
      --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=200GB"

WAL compression reduces the bytes-per-second write to the WAL but does not change the IOPS count. It is on by default since Prometheus 2.20; the --storage.tsdb.wal-compression flag only exists to turn it off.

The Loki ingester configures the chunk size:

limits_config:
  ingestion_rate_mb: 100
  ingestion_burst_size_mb: 200
  chunk_target_size: 1572864

chunk_target_size is the target uncompressed size of a chunk in bytes. Larger chunks mean fewer chunks and fewer requests but more memory pressure on the ingester.

How to validate it

# READ-ONLY: fio random 4 KB write benchmark (Prometheus-shaped).
fio --name=randwrite \
    --ioengine=libaio --iodepth=64 \
    --rw=randwrite --bs=4k --size=10G \
    --filename=/var/lib/prometheus/test.bin \
    --runtime=60 --time_based
# ...
#   iops        : avg=107,234, min=98,432, max=112,567
#   bw (KiB/s)  : avg=419,234
#   lat (usec)  : avg=596.3

# READ-ONLY: fio sequential 1 MB write benchmark (Loki-shaped).
fio --name=seqwrite \
    --ioengine=libaio --iodepth=8 \
    --rw=write --bs=1M --size=10G \
    --filename=/var/lib/loki/test.bin \
    --runtime=60 --time_based
# ...
#   iops        : avg=512
#   bw (MiB/s)  : avg=512.3

# READ-ONLY: burst credit balance on EBS gp3.
aws cloudwatch get-metric-statistics \
  --namespace AWS/EBS --metric-name BurstBalance \
  --dimensions Name=VolumeId,Value=vol-0123456789abcdef0 \
  --start-time 2026-08-13T00:00:00Z \
  --end-time   2026-08-13T01:00:00Z \
  --period 300 --statistics Average
# {"Datapoints":[{"Timestamp":"...","Average":72.4,"Unit":"Percent"}], ...}

# READ-ONLY: per-disk IOPS in use.
iostat -x 1
# Device   r/s     w/s   rMB/s  wMB/s  aqu-sz  await  svctm  %util
# nvme0n1  12.0  89.4    0.4   91.2    3.2   0.4    0.2   28.4

A clean validation: fio with the workload-shaped pattern delivers the expected IOPS or throughput; the burst credit balance is healthy (above 50%) for gp3; iostat shows the disk is below 70% utilisation under peak load.

How it can fail

The most expensive throughput-and-IOPS failures, in order of how often they appear in incident reviews.

1. Prometheus head block stalls on IOPS exhaustion. A gp3 volume at baseline 3 K IOPS cannot sustain a 50 K samples-per-second WAL. The head block stalls; WAL segments accumulate; samples are dropped. Symptom: prometheus_tsdb_head_series is flat for several scrape intervals.
2. Loki ingester throttles on throughput exhaustion. A gp3 volume at baseline 125 MB/s cannot sustain a 250 MB/s spike. Burst credits deplete within 20 minutes; throughput drops to 125 MB/s; the ingester queue fills. Symptom: ingester logs report back-pressure; shippers return 429.
3. Tempo compactor falls behind on read throughput. A gp3 volume delivers 125 MB/s read. The compactor reads 200 MB/s during a compaction window. The compaction cannot complete in the window; the compactor falls behind. Symptom: tempo_compaction_window_seconds rises past the configured window; block count grows on disk.
4. Read latency spikes under contention. A backup job reads 500 MB/s; the page cache is evicted by the backup reads; the queries hit the disk. Symptom: query latency rises from 200 ms to 5 s; iostat shows high read IOPS.
5. fsync latency on consumer-grade SSD. A team uses a consumer-grade SATA SSD that delivers high IOPS but has high fsync latency under write pressure. The WAL fsync exceeds the scrape interval. Symptom: head block stalls; WAL segments accumulate; samples are dropped.
6. Volume provisioned IOPS less than baseline need. A team provisions a gp3 volume with 5 K IOPS expecting burst behaviour. The workload sustains 5 K IOPS and depletes burst credits within minutes. Symptom: the volume performs at baseline (3 K IOPS) for the rest of the day; the head block stalls during the dip.

How to troubleshoot it

The diagnostic order is “is the disk saturated?”, “is the workload IOPS or throughput bound?”, “is the burst credit balance healthy?”.

1. Start with iostat -x 1. Look at %util and aqu-sz. A util above 70% sustained is a sign the disk is the bottleneck. An aqu-sz above the device’s queue depth is a sign the device is overloaded.
2. Determine the workload shape. Prometheus is IOPS-bound (random 4 KB writes). Loki is throughput-bound (sequential large writes). Tempo is mixed (random small writes to WAL, sequential large writes to blocks).
3. Check the burst credit balance. On AWS, the BurstBalance CloudWatch metric. A balance below 20% is a sign the burst capacity is exhausted.
4. Check the provisioned IOPS. On AWS, the VolumeConsumedReadWriteOps CloudWatch metric. A workload that sustains more than the provisioned IOPS is throttled.
5. Validate with fio. Reproduce the workload shape on the candidate device with fio and confirm the expected IOPS or throughput.

Security implications

• Disk encryption is not affected by IOPS or throughput. LUKS on Linux or EBS encryption at the cloud layer adds CPU cost but no I/O cost. The performance penalty is in the single-digit percent range.
• Volumes are the access boundary. The volume is attached to a single instance by default. Multi-attach volumes exist for shared filesystems (ocfs2, gfs2) but are not used for Prometheus or Loki.
• Snapshots preserve the workload shape. A snapshot of a hot volume preserves the data but not the provisioned IOPS. Restoring a snapshot to a lower-IOPs volume silently throttles the workload.

Performance implications

• NVMe outperforms SATA SSD by roughly 10x on the relevant axes. A 2026 NVMe device delivers 1 M IOPS and 7 GB/s throughput. A SATA SSD delivers 100 K IOPS and 600 MB/s throughput.
• EBS gp3 without provisioned IOPS delivers 3 K IOPS and 125 MB/s. A workload above 30 K samples per second needs provisioned IOPS.
• EBS io2 Block Express delivers 1 K IOPS per GB provisioned, up to 256 K IOPS and 4 GB/s. io2 is the right choice for sustained high-IOPS workloads.
• Instance store is fast and ephemeral. AWS instance store and equivalent offerings on other clouds deliver high IOPS but lose data on instance stop. Useful for short-retention workloads where the long retention lives elsewhere.

Production guidance

• Measure before you deploy. fio on the candidate device with the workload-shaped pattern is the test. Do not assume a volume type delivers the documented performance; measure on your hardware in your region.
• Right-size the IOPS budget to the workload. A Prometheus at 100 K samples per second needs roughly 100 K IOPS for the WAL plus compactor headroom. A gp3 volume with 100 K provisioned IOPS or an io2 Block Express volume with 100 K provisioned IOPS is the right shape.
• Right-size the throughput budget to the workload. A Loki at 50 MB/s sustained needs roughly 80 MB/s throughput for chunk uploads plus ingester headroom. A gp3 volume with 250 MB/s provisioned throughput is the right shape.
• Alert on burst credit balance. On AWS, alert when BurstBalance falls below 20%. The volume is about to drop to baseline; the workload will throttle.
• Co-locate the WAL with the blocks by default. On hosts with multiple NVMe devices, splitting the WAL onto a separate device is an optimisation for high-IOPS hosts.

Verification

You should now be able to answer:

• Is Prometheus IOPS-bound or throughput-bound, and why?
• Is Loki chunk upload IOPS-bound or throughput-bound, and why?
• What is the difference between burst credits and provisioned IOPS on EBS gp3?
• What fio invocation validates the IOPS budget for a Prometheus workload?
• What CloudWatch metric warns that an EBS gp3 volume is about to drop to baseline throughput?

Quiz