Reading a single PG with pg query

Why this matters in production

ceph -s tells you a PG is unhappy. ceph pg query tells you why. It is a large JSON document and the instinct is to skim it, but almost every stuck-PG investigation ends at one of four fields, and knowing where they are turns a twenty-minute read into a thirty-second one.

Running it

ceph pg 7.3d query > /tmp/pg-7.3d.json
jq 'keys' /tmp/pg-7.3d.json

The query goes to the PG’s primary OSD. If the primary is down, the query blocks or fails — which is itself a diagnosis.

The four fields that matter

state — the same string ceph -s shows, at the top level.

jq -r '.state' /tmp/pg-7.3d.json
# active+undersized+degraded

recovery_state — an array of state-machine states, most recent first, each with an enter_time. This is where you learn how long the PG has been where it is.

jq -r '.recovery_state[] | "\(.name)  \(.enter_time)"' /tmp/pg-7.3d.json
# Started/Primary/Active/Degraded  2026-08-18T02:14:07
# Started/Primary/Active           2026-08-18T02:14:07
# Started                          2026-08-18T02:13:55

A PG whose innermost state was entered four hours ago is stuck. One entered forty seconds ago is working.

blocked_by — a list of OSD ids the PG is waiting on. When it is non-empty, your investigation is over; go look at those OSDs.

jq -r '.recovery_state[] | select(.blocked_by) | .blocked_by[]' /tmp/pg-7.3d.json

peer_info — one entry per peer, each with that peer’s view of the PG’s version and log. Divergent last_update values across peers explain why peering has not completed.

A worked read

jq -r '{state, up: .up, acting: .acting,
        newest: .info.stats.last_update,
        inner: .recovery_state[0].name,
        since: .recovery_state[0].enter_time}' /tmp/pg-7.3d.json

{
  "state": "peering",
  "up": [12, 47, 83],
  "acting": [12, 47, 83],
  "newest": "41207'8823914",
  "inner": "Started/Primary/Peering/GetInfo",
  "since": "2026-08-18T01:58:12"
}

GetInfo means the primary is waiting for peers to report their PG info and has been for over an hour. That is not peering; that is a peer that is not answering.

Quiz

Production discipline

Capture pg query output to a file at the moment you find a problem, before you change anything. The state machine moves, and an intervention that fixes the symptom destroys the evidence. A saved JSON document is what lets you explain afterwards what was actually wrong, and it is the single most useful attachment on a Ceph support case.

Cross-course references

• Kubernetes: kubectl describe pod events serve the same role — a timestamped state history rather than a status word
• Linux: reading enter_time deltas is the same technique as reading process state durations from /proc