1. Q1. The p99 of the checkout API has doubled over two hours. Every node CPU, memory, disk and network panel is unchanged, the 5xx rate is flat, and the request rate is flat. Which move produces the evidence that says where the extra time is being spent?

   Add a `customer_id` label to the latency histogram so the slow customers can be isolated Open a trace for one slow checkout request and read the span durations along its critical path Raise the application log level to DEBUG and grep the request lines for their elapsed-time fields Plot `node_load5` beside the latency panel and look for a correlation over the same two hours

2. Q2. A Redis cache is returning errors to about one request in twenty. Every USE panel on its host is comfortable: CPU 45%, memory 50%, disk and network well inside capacity. Which methodology answers the question the on-call actually has?

   USE on the host, because resource saturation is the signal that precedes every user-visible failure USE on the host plus NetUSE on the link, because the resource here is the broker, not the service RED on the host and USE on the service, because the host is what the user contacts over the wire RED on the service, because rate, errors and duration describe the user-visible state directly

3. Q3. A platform instrumented only with USE metrics can show every resource on every host as healthy while users are being served errors; RED on the service is what makes the user-visible state visible.

   True False

4. Q4. `http_requests_total` carries `method` (5 values), `status` (8), `route` (40) and `instance` (12). A developer proposes adding `request_id`, one value per request, so that failed requests can be found from the metric. What happens to the series count?

   It becomes unbounded: the count is the product of the label domains and one domain is now per-request It rises by one series per route, because Prometheus folds repeated label values into an existing series It is unchanged until the next compaction, when the head block writes the new label into the block index It rises by 19 200 series, because the new label multiplies the existing set by its first observed value

5. Q5. Two Prometheus servers each have 32 GiB of RAM. Server A holds 6 million active series with almost no series created or removed per hour. Server B holds 2 million active series but creates and removes about 1.5 million series every hour. Which is the more fragile of the two, and why?

   A, because active-series level is the only input to Prometheus memory and 6 million is three times B A, because a level that high with no churn pins every series in the head block until retention expires B, because churn drives head garbage collection, index rebuilds and WAL growth on top of the level B, because a series removed and recreated is counted twice in the head until the next compaction runs

6. Q6. Prometheus normally exhausts CPU before it exhausts memory, because query evaluation and rule evaluation dominate its resource profile on a busy platform.

   True False

7. Q7. kube-state-metrics is adding `pod_template_generation`, `controller_revision_hash` and `pod_template_hash` to every sample, and the head block has reached 26 million active series. You want those three labels to cost nothing in memory, in the WAL, in the blocks or in the remote-write queue. Where does the rule belong?

   In `relabel_configs` on the scrape job, with `action: labeldrop` and a regex naming all three of those labels In `metric_relabel_configs` on the job, with `action: labeldrop` and a regex naming the three labels In `write_relabel_configs` under `remote_write`, with `action: labeldrop` naming the three labels In a recording rule that re-aggregates the metric without the three labels every thirty seconds

8. Q8. `up{job="api", instance="10.0.4.7:8080"}` has been 0 for twenty minutes. From the Prometheus host, `curl` against that target's metrics path returns a full exposition body in 40 ms. What is the most likely explanation?

   A scrape limit was breached — `sample_limit`, `label_limit` or `body_size_limit` — so the scrape failed A `relabel_configs` `drop` rule removed the target, so Prometheus records that target as `up` 0 Every sample was dropped by `metric_relabel_configs`, so Prometheus wrote `up` as 0 for the scrape The series went stale, so the stale marker is rendered by the panel and by the rule engine as 0

9. Q9. If a configuration reload fails to parse, Prometheus keeps running the previous configuration and sets `prometheus_config_last_reload_successful` to 0, so nothing pages unless somebody alerted on that metric.

   True False

10. Q10. A remote-write receiver is unreachable for six hours. Prometheus keeps scraping normally and its local retention is fifteen days. Head compaction, and therefore WAL truncation, happens roughly every two hours. What arrives at the receiver when it returns?

    All six hours, because the remote-write queue buffers to a dedicated on-disk spool until delivery All six hours, because the queue manager re-reads the local blocks for any window it has not sent Only what is still in the WAL; the truncated segments are a permanent gap in the remote store Nothing at all, because the queue manager discards its checkpoint once `max_backoff` is reached

11. Q11. A blackbox_exporter is deployed with an `http_2xx` module, and a Prometheus job rewrites `__address__` to the exporter while passing the real URL through `__param_target`. What performs each probe, and when?

    The exporter runs its own scheduler and pushes each probe result to Prometheus over remote write The exporter probes continuously on its own interval and Prometheus scrapes the cached last result Prometheus performs the probe itself; the exporter only renders the result as exposition format Prometheus scrapes the exporter probe path, and the act of scraping is what performs the probe

12. Q12. A fleet panel plots `node_memory_MemFree_bytes` over `node_memory_MemTotal_bytes` and reads 2-4% on every healthy host. The alert built on it pages most nights and nothing is ever wrong. What is the correct fix?

    Lower the alert threshold to 1%, so that only genuine memory exhaustion is able to cross it Use `node_memory_MemAvailable_bytes`, the kernel estimate counting reclaimable page cache and slab Subtract `node_memory_Cached_bytes` from used memory, the way the `free -m` buffers/cache line does Use `node_memory_AnonPages_bytes`, since only anonymous pages are able to provoke an OOM kill

13. Q13. A counter pushed to a Pushgateway by a batch job at 02:00 is expired automatically once that job's next scheduled run completes, so a stale value cannot outlive one cycle.

    True False

14. Q14. A panel plots `rate(http_requests_total[1m])` for an application fleet. Between 02:50 and 02:56 the pipeline rolls every replica. The panel climbs to about 2 000 req/s at 02:55 and falls back to 1 000 req/s by 02:57. Load-balancer traffic was flat throughout. What is the panel showing?

    A genuine doubling, because a rolling restart concentrates all traffic onto the remaining replicas A scrape artefact, because the scrape interval halved during the restart and doubled the sample count A rate() artefact: each restart reset the counter and rate() extrapolated across every reset it found A rate() artefact: rate() ignored the resets and divided the raw counter difference by the window

15. Q15. A job is scraped every 30 seconds. An engineer writes `rate(http_requests_total[15s])` in a new panel and gets "No data" on every series, while `rate(http_requests_total[2m])` on the same metric renders normally. Why?

    A 15-second window holds fewer than two samples, and rate() returns nothing without a sample pair A 15-second window holds one sample, so rate() divides by a zero interval and returns NaN as no data The counter did not increment inside the window, so rate() returned zero and Grafana hid the line The series was stale inside the window, so the five-minute lookback delta returned no samples at all

16. Q16. A latency panel reads `histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))`. It shows 80 ms and no alert has fired, but one of the four clusters behind it is serving a p99 of 740 ms against a 200 ms SLO. Which change makes the breach visible?

    Wrap the existing query in `max by (cluster)` so the worst cluster dominates the single global figure Keep the cluster in the inner grouping — `sum by (le, cluster)` — so a quantile is estimated per cluster Replace the histogram with a summary and read its pre-computed `quantile="0.99"` series per cluster Raise the quantile from 0.99 to 0.999 so that the slow cluster stops being averaged away by the fast

17. Q17. A `checkout` deployment is scaled to zero at 03:00. The rule `expr: up{job="checkout"} == 0` never fires and the panel simply goes blank. Which replacement rule pages on the disappearance?

    `up{job="checkout"} != 1`, which matches both a missing series and any value other than one `up{job="checkout"} < 1` with `for: 5m`, so that the pending window outlives the scrape gap `count(up{job="checkout"}) == 0`, which counts the series that are still being reported `absent(up{job="checkout"})`, which returns one synthetic sample when no series matches

18. Q18. Which statements about missing data in PromQL are correct? Select all that apply.

    A comparison between an empty vector and a scalar returns an empty vector, so the alert never fires A Grafana panel no-data setting converts an empty vector into a series carrying the value 0 `absent(metric)` returns a single sample carrying the metric name as a label when nothing matches A series with no sample inside the query lookback delta is returned with its last value carried forward A stale marker ends a series, so graphs go blank rather than flatlining at the last observed value An empty vector and a series whose latest sample is 0 behave identically in every binary operator

19. Q19. A recording rule is named `job:http_requests_total:rate5m` and its expression is `sum by (job, instance) (rate(http_requests_total[5m]))`. A dashboard panel built on the recorded series renders about a thousand lines and takes thirty seconds to load. What is wrong?

    The name advertises a per-job level while the expression keeps `instance`, so consumers get per-instance series The name is right and the panel is at fault: it needs its own `sum by (job)` around the recorded series The rate window in the name disagrees with the window in the expression, so the rule records raw counters Recording rules always keep every label of the source metric, so the rule name is documentation and not a contract

20. Q20. A rule has `for: 5m` and its group has `evaluation_interval: 30s`. The expression returns a result series continuously from 09:00:00 to 09:04:30, returns nothing at exactly 09:05:00, and returns a result series again from 09:05:30 onward. When does the alert start firing?

    At 09:05:00, because the alert had already accumulated 4m30s of dwell before the empty evaluation At 09:05:30, because a single empty evaluation only pauses the `for:` timer rather than clearing it At 09:10:30, because one empty evaluation resets the timer and the dwell restarts from 09:05:30 At 09:11:00, because the evaluation interval is added to the `for:` budget every time it restarts

21. Q21. An `inhibit_rule` is written with `source_matchers: [alertname = ClusterDown]`, `target_matchers: [severity = critical]`, and no `equal:` clause. What is the production consequence?

    None in practice: without `equal:` the rule can only suppress alerts that share the source alert name A ClusterDown anywhere suppresses every critical in the fleet, including unrelated hardware failures The configuration is rejected by `amtool check-config`, because `equal:` is required with a target matcher The suppressed criticals are removed from `nflog`, so they are missing from the post-incident review

22. Q22. An Alertmanager tree has a root, then a child route matching `severity = critical` whose own children match `team = payments`, `team = dba` and `team = infra`, and then — as a later peer of that critical route — a route matching `team = payments`. An alert arrives with `severity=critical, team=payments`. Which receiver is notified?

    The later peer route, because a matcher naming the team is more specific than one naming the severity Both routes, because Alertmanager evaluates every peer at each level and notifies once per match The root receiver, because two peer routes matching the same alert make the selection ambiguous The payments child of the critical route, because the walk is depth-first and the first match wins

23. Q23. A single-window burn-rate alert — a 1-hour window at 14.4x — has paged every Friday at 17:30 for six months as the weekday traffic dip briefly inflates the error ratio. The team has stopped reading it. Which change keeps the fast-burn page and removes the Friday noise?

    Require the 1h/14.4x condition and a 6h/6x condition together, so one sharp hour cannot page alone Raise the threshold from 14.4x to 30x, so that only much sharper incidents are able to cross it Lengthen `for:` on the rule to 30 minutes, so that the Friday dip resolves before the dwell elapses Shorten the SLO window from 30 days to 7 days, so the same burn consumes a smaller share of budget

24. Q24. A sustained burn rate of 14.4 against a 30-day error budget exhausts that budget in roughly two days, which is why 14.4x over a 1-hour window is written as a page rather than a ticket.

    True False

25. Q25. A team asks you to review a proposed rule before it goes on the pager rota. List the properties you would require before approving it, and say what each one prevents.

26. Q26. A Grafana 11 install on an internal subnet ships with `[auth.anonymous] enabled = true` and `org_role = Viewer`. The team describes it as "internal Grafana". What is the accurate description of the boundary?

    Only dashboards explicitly marked as public are reachable; everything else still requires a login Anonymous users may open dashboards but cannot query data sources, which need their own credentials Anyone who can reach the port is a shared Viewer, and every audit entry for them says "Anonymous" Grafana refuses anonymous requests to any data source that has `basicAuth` disabled on the datasource

27. Q27. A team builds a dashboard in the Grafana UI on an instance whose folders are file-provisioned. The dashboard is good and everyone uses it. After the next deploy it is gone. What happened?

    The provider reconciled the folder against its source directory and removed what was not on disk The Grafana database was recreated by the container image, so every UI-built dashboard was lost The dashboard `schemaVersion` was older than the running Grafana, so the loader refused to render it The provider ran with `allowUiUpdates: false`, which reverts UI edits but never deletes a dashboard

28. Q28. A dashboard takes eight seconds to load. Every panel, once it renders, returns in under 200 ms. The five variable dropdowns at the top say "loading" for the first five seconds. Where is the time going?

    In the Grafana query cache, which is cold on the first load and warm on every load after that one In the variable queries, which run before the panels and block them until every one has resolved In the data source proxy, which serialises all requests from one browser session onto one connection In the panels themselves, each of which re-runs once per value of every variable on the dashboard

29. Q29. A dashboard variable's regex is pushed down to the data source, so a `label_values()` variable filtered by a regex only ever fetches the values that the regex keeps.

    True False

30. Q30. Which of the following does the course name as dashboard anti-patterns? Select all that apply.

    A wall of panels nobody reads, kept on a screen because removing panels feels like losing information A panel stating its unit explicitly and carrying a threshold that matches the alert on the same query A row of gauges whose needles have no threshold, no unit and no relationship to any alerting rule A dashboard whose description names the team that owns it and the rota paged for its alerts A panel carrying a link to the runbook for the failure that the panel is designed to make visible A palette assigning a different hue to every series, so the eye cannot pick out the anomalous one

31. Q31. A Loki ingester is at 9.4 GiB resident and climbing. The stream-count panel shows 11.3 million active streams on one tenant, up from 40 000 a week ago. Queries on every other tenant are timing out. What is the cause, and where is the cost paid?

    Retention was lengthened, so the ingester now holds every chunk in the retention window in memory The compactor stopped, so the ingester is buffering all of the chunks it cannot flush to object storage A query is scanning chunks, and the ingester holds each scanned chunk until that query finishes A stream label with unbounded values was added; each distinct label set is a stream held in memory

32. Q32. The same team still needs to locate a specific `request_id` in Loki. Where should that identifier live so that it stays queryable without multiplying the stream count?

    As structured metadata on the entry, indexed per entry rather than as part of the stream label set As a stream label, with a `labeldrop` rule in the compactor so the index shrinks after compaction As a stream label on a separate low-traffic tenant, so that the cardinality is charged to that tenant As a Prometheus label on the log-derived metric, where the cardinality budget is larger than Loki has

33. Q33. Loki 3.x deletes nothing out of the box: the compactor runs but `retention_enabled` defaults to false, so a cluster deployed without an explicit retention decision keeps every log line forever.

    True False

34. Q34. A Grafana panel for `{job="payments"}` is empty. The collector is running; its log shows `server returned HTTP status 429 Too Many Requests` and its remote-write error counter is climbing. What is the correct next move?

    Restart the collector to clear its buffer, then re-push the missing window from its disk queue Increase the collector batch size so that fewer, larger pushes stay under the request-rate limit Read `loki_discarded_samples_total` by `reason` on the distributor to name the limit being breached Widen the query time range, because a 429 delays the batch rather than discarding any of its entries

35. Q35. Every line from a service is JSON and carries `"level":"error"`. The query `{service="checkout", level="error"}` returns nothing at all. Why, and what is the correct query?

    Stream selectors need a regex for quoted values; write `{service="checkout", level=~"error"}` instead The field is in the line, not in the stream labels; write `{service="checkout"} | json | level="error"` The line filter has to come first; write `{service="checkout"} |= "error" | level="error"` in that order JSON fields are indexed lazily, so the field is queryable only after the compactor rewrites the chunk

36. Q36. Running a second Loki compactor against the same object-storage bucket doubles retention throughput, because the two instances divide the tenants between them.

    True False

37. Q37. A failed checkout has a Tempo trace with four spans — gateway, inventory, pricing, payment. The warehouse service's spans for the same request exist, with sensible parent and child links among themselves, but under a different trace id. Where is the defect?

    The warehouse clock is skewed, so Tempo assigned its spans to a different trace by their timestamps The warehouse spans were tail-sampled out of the trace and re-ingested later under a fresh trace id The inventory service emitted `tracestate` but no `traceparent`, so the break is at the previous hop The warehouse service does not extract the incoming `traceparent`, so it starts a new trace at that hop

38. Q38. A service samples traces at 1% at the producer. A post-mortem needs the trace of one customer's failed cart from 14:03 and it is not in Tempo. Which change makes that class of trace available next time, and what does it cost?

    Raise producer sampling to 100% in production, at a storage cost that scales with total traffic Move the decision to a gateway tail sampler that keeps every errored trace, at the cost of buffering Add a `customer_id` span attribute so that the head sampler always selects the traces from those customers Extend Tempo retention, since the trace was written and then aged out before the post-mortem opened

39. Q39. A tail sampler has to buffer every span of a trace until its decision window expires, so the gateway's memory cost scales with total incoming traffic rather than with the fraction of traces it eventually keeps.

    True False

40. Q40. A p99 latency alert fired on a single 1.4 s outlier while the histogram curve sat at 80 ms. What mechanism puts the trace of that one request under the on-call engineer directly from the histogram panel?

    A `trace_id` label on the histogram series, letting the engineer filter the metric to the slow request A Grafana annotation recording the trace id of every request that crosses the alerting threshold An exemplar on the histogram bucket, carrying the `trace_id` and `span_id` of one observation in it A TraceQL search, which is the only route because Prometheus cannot carry trace identity in any form

41. Q41. A trace is split in two by a `traceparent` that one hop failed to propagate. Which capabilities are lost as a result? Select all that apply.

    The individual spans stop being recorded, because a span with no parent is rejected by the ingester End-to-end latency for the request, because the two halves of the journey are in different traces Failure attribution across the boundary, because the two halves cannot be compared as one request Span-level timings inside each service, because durations are computed relative to the trace root The trace-to-logs pivot for the downstream services, which filters Loki by the trace id of the trace The metrics-generator service graph entirely, because it derives every edge from the trace id alone

42. Q42. A gateway OpenTelemetry Collector is OOM-killed twice a week. Its pipeline declares `processors: [batch, memory_limiter, resource, transform]` and the limiter is configured with `limit_percentage: 80`. What is the fix?

    Put `memory_limiter` first in the chain, so it gates the data before `batch` can accumulate any of it Put `batch` first, so that records are coalesced before the limiter has to inspect each one of them Raise `limit_percentage` from 80 to 95, so the limiter admits more data before it starts refusing Drop `memory_limiter` and cap the process with a cgroup memory limit on the container instead

43. Q43. A single OpenTelemetry Collector pipeline can carry more than one signal, so a `logs` pipeline may also accept a trace receiver provided the exporter it feeds understands both.

    True False

44. Q44. A new logs pipeline is added to a gateway collector. Lines arrive in Loki — the ingest counters move — but nobody on the owning team can see them in their Grafana. What is the first thing to check?

    The `resource` processor, which may have stamped a `job` value that no dashboard query selects on The `filelog` receiver path glob, which may be reading a rotated file rather than the live one The distributor stream hash, which assigns a tenant and may have moved the lines to another one The exporter headers, since a missing `X-Scope-OrgID` puts the lines into the wrong Loki tenant

45. Q45. Describe the metric-to-log-to-trace pivot as an on-call workflow, and state what must be true of the telemetry at each hop for the pivot to work.

46. Q46. A CI job runs a container that lives for about twelve seconds, several hundred times a day. Prometheus scrapes cAdvisor every fifteen seconds. The container dashboard shows almost nothing for that job. What is happening, and what is the remedy?

    cAdvisor never polls the cgroup at that lifetime; shorten its housekeeping interval below the container lifetime Prometheus dropped the target by relabeling; add a `keep` rule matching the job container name Prometheus rarely scrapes before the cgroup is removed; have the job push its metrics to a Pushgateway cAdvisor exports the metrics to Loki as log lines; read them back with a LogQL `unwrap` expression

47. Q47. The only certificate alert on the platform is `TLSCertExpiresIn24Hours`, built on `probe_ssl_earliest_cert_expiry`. Reviewing the alert set, what do you change?

    Nothing: 24 hours is enough lead time and anything earlier wakes the on-call for a routine renewal Add earlier tickets at 30, 14 and 7 days; by 24 hours the alert is already announcing an emergency Replace it with an alert on `probe_success`, since a failed handshake is the only visible symptom Move it to 90 days to match the ACME issuance period and drop the shorter-horizon alerts entirely

48. Q48. Applying USE to a network interface on a Linux host, which node_exporter series is the *saturation* signal?

    The receive and transmit drop counters, which count the packets the interface buffers could not hold The receive and transmit error counters, which count the frames the interface itself rejected as malformed The receive bytes counter, whose rate against the link capacity gives the interface utilisation The interface MTU gauge, whose value bounds how much payload each frame can carry across the link

49. Q49. Two Loki single-binary processes each hold their own chunks directory. Half the fleet's agents are configured to send to A and half to B. A dashboard reports "Loki HA, 2/2 healthy". What is the honest description of this deployment?

    It is HA for ingest but not for query, because either process can accept a push from any agent It is HA once a load balancer is placed in front, because the agents then fail over between the two It is HA for query but not for ingest, because a query against either process sees the whole fleet It is not HA at all: neither process holds the other data, so losing one loses half of the logs

50. Q50. A capacity plan for a new Prometheus multiplies samples per second by retention in seconds by 2 bytes per sample, then divides the result by a compression ratio of 10 to reach the disk budget. What is wrong with it?

    The 2-byte figure is the in-memory size, so the plan should multiply by ten rather than divide by it The 2-byte figure is already post-compression, so dividing again understates the disk that is needed Samples per second must be multiplied by the scrape interval, not divided, before the retention term Retention has to be counted in two-hour blocks rather than seconds, because compaction rewrites them

51. Q51. A team's observability bill has doubled over four months. Active series is flat, the service count is flat, query volume is flat, and no new tenant has been onboarded. What is the most likely explanation?

    Retention was never decided, so stored volume — and therefore cost — grows linearly with elapsed time The compactor fell behind, so uncompacted blocks are being stored twice until compaction finally catches up Sample compression degrades as blocks age, so the same data occupies more bytes after some months Object-storage egress on dashboard queries is the dominant line, and new dashboards were added

52. Q52. A Prometheus host sized correctly for the memory its active series require is, by that fact alone, also sized correctly for its ingest rate and for the disk its retention window needs.

    True False

53. Q53. A platform keeps thirteen months of full-resolution metrics on local NVMe so that year-old incidents remain investigable. Before adopting tiered retention instead, what is the honest trade-off the course asks you to state?

    Tiering costs nothing operationally; the only change is which storage class the same bytes land on Tiering costs query latency across the board, because every query is then served from object storage Tiering costs the long-tail investigation: a once-a-year failure needs last year at full resolution Tiering costs compliance, because a retention policy shorter than the maximum breaches every regime

54. Q54. A Prometheus runs with `--web.enable-admin-api` and `--web.enable-lifecycle` so an internal tool can take snapshots and reload configuration. Its Service is ClusterIP, but a NetworkPolicy gap lets the management subnet reach it. What is the worst realistic outcome, and what is the production default?

    An attacker can read metrics only; leave both flags on and rely on the read-only nature of the API An attacker can delete series through the admin API; leave the flags on and snapshot more often An attacker can stop the process through the lifecycle path; leave the flags on and supervise it An attacker can snapshot the TSDB and take the scrape configs and their credentials; disable both

55. Q55. An OTel Collector `transform` processor deletes attributes matching a pattern covering `password`, `token` and `secret`. A developer ships a field called `authToken`; the pattern has no case-insensitive flag and the credential lands in Tempo. What is the right response?

    Fix the pattern and drop the field at the call site too: the agent regex is depth, not the control Fix the pattern with a case-insensitive flag; a correct agent regex is a complete control on its own Leave the pattern and mask the value at query time in Grafana, which is where redaction belongs Leave the pattern and set the affected retention to zero, which removes it from every stored chunk

56. Q56. Once a secret has been written into Loki, Tempo or Prometheus, nothing in that stack masks it at query time; the remaining remedies are access control, a shorter retention period and — in Loki alone — an explicit delete request.

    True False

57. Q57. A reviewer needs a gate that catches an alert rule whose author wrote `for: 5s` where they meant `for: 5m`. Which one catches it?

    `promtool check rules`, which validates the file against the rule schema including duration units `promtool test rules`, which advances a synthetic clock across the dwell and asserts the alert state `promtool check config`, which resolves the rule files and rejects a dwell below the eval interval `yamllint`, which flags a duration written in seconds where every surrounding rule uses minutes

58. Q58. `promtool test rules` loads its synthetic series into in-memory storage and uses the same evaluation engine as the daemon, so it needs no network, no credentials and no running Prometheus.

    True False

59. Q59. Which defect does `promtool check config` NOT catch?

    A rule file named under `rule_files:` whose path does not resolve from the configuration directory A `relabel_configs` entry whose `action` is not one of the actions Prometheus recognises at all An alert expression that parses correctly but names a metric that no exporter emits any more A YAML indentation error in `alerting:` that makes the block parse as some other top-level key

60. Q60. A meta-Prometheus and meta-Alertmanager run on the same Kubernetes cluster as the production observability stack they watch. The cluster API server fails at 02:14; the meta pods stay running but cannot refresh their service-account tokens, their federated scrape fails, and no page is delivered. What root cause should the post-mortem record?

    The federation interval was too long, so the outage fell between two scrapes of the production stack The meta Alertmanager grouped the page with an earlier alert, delaying it past the group interval The meta stack alerted correctly and the receiver was misrouted, so the page went to another rota The meta stack shared the failure domain it watched, so the outage it existed to catch disabled it

61. Q61. Which of the following, when shared between a production observability stack and its meta stack, will silently defeat the meta? Select all that apply.

    The OIDC or LDAP backend that both layers authenticate their scrapes and their web UIs against The Prometheus minor version that both layers happen to be running at the time of the outage The dashboard JSON that both layers load from the same provisioning repository held in Git The notification destination through which the meta page and the production pages both go out The DNS resolver that both layers use to resolve the hostnames of their own scrape targets The metric naming convention that both layers apply to their recording rules and alert names

62. Q62. A backup job tars the Prometheus data directory nightly while the process is running. What should it do instead, and why?

    Call the admin snapshot endpoint first; it returns a consistent, read-only directory to archive Stop Prometheus, tar the data directory and restart it; a cold copy is the only consistent one Tar the WAL directory alone; the head block can always be rebuilt by replaying it on restore Copy the compacted blocks only; the head is transient and is not really part of a backup at all

63. Q63. A team upgrading Loki from schema v11 to v13 edits `schema_config.configs`, removes the v11 entry and leaves only a v13 entry. The cluster restarts, new writes work, and six months of historical queries return nothing. Which rule did they break?

    The `from:` date of a new entry must be in the past, so the new schema covers historical timestamps The list is append-only: the old period entry is what keeps historical data readable at all A schema change needs a migration job to rebuild the v13 index from the existing chunks first A schema change needs the compactor stopped, or it deletes the chunks it cannot index any more

64. Q64. A P0 is open: a user-facing service is returning errors and the error budget is burning at 40x. The primary on-call recognises the symptom and believes a restart will clear it, but the cause is not yet established. What does the operating model say to do?

    Establish the cause first, so that the mitigation addresses the real fault rather than merely masking it Preserve the evidence and change nothing, because a restart destroys the state that proves cause Mitigate now and run the RCA in parallel; on a P0 the SLO burn is the cost driver during the event Run both lanes with the same engineer, because splitting attention doubles the time to mitigate

65. Q65. A card transaction fails for 30% of users for forty minutes and no alert fires at any point. Which investigation order does the alert-failure method prescribe?

    Telemetry exists, then the rule evaluates, then the threshold is right, then routing and receiver Receiver reachable, then routing correct, then the rule evaluates, then whether telemetry existed Dashboards render, then the on-call schedule is right, then the rule, then the telemetry pipeline Rule syntax parses, then the runbook link resolves, then the severity label, then the receiver

66. Q66. `HostClockSkewCritical` fires on `node_timex_offset_seconds` with a value of 1.412. The engineer connects to the host and `chronyc tracking` reports the system time 12 microseconds fast of NTP time. Which reading is right?

    The metric is wrong: the timex offset series reports the frequency correction, not the offset The chrony output is wrong: it reports the offset to the local source, not the upstream stratum The metric is stale: node_exporter caches the timex reading until the collector is restarted Both are right: the metric is a scrape from minutes ago and chronyd has since slewed the clock back

67. Q67. Sketch the reference architecture for a production observability platform. Name the layers, say what runs in each, and state one failure mode that each layer is responsible for surviving.

Passing score: 80%. Answers are checked in this browser.