Lab: Deploy Grafana

Objective

By the end of this lab you will have a Grafana 11.x installed from the signed apt repository, listening only on loopback, reached over TLS through nginx, supervised by a hardened systemd unit, with an admin password that was never admin and never appeared on a command line. You will then validate it on the five layers from the installation-validation lesson, including one layer you make fail on purpose so you can see which layers stay green while it does.

Architecture

One host. Two processes. Nothing on the network can reach the Grafana port.

   client
     |  https://grafana.example.com
     v
  +---------------------------------------------+
  |  host (disposable VM)                        |
  |                                              |
  |   nginx  :443 (TLS)  :80 (301 -> https)      |
  |     |                                        |
  |     |  proxy_pass http://127.0.0.1:3000      |
  |     |  X-Forwarded-Proto / -For / -Host      |
  |     v                                        |
  |   grafana-server  127.0.0.1:3000             |
  |     |                                        |
  |     +-- /etc/grafana/grafana.ini             |
  |     +-- /etc/grafana/grafana.env  (password) |
  |     +-- /var/lib/grafana/grafana.db (sqlite) |
  |     +-- /var/log/grafana/grafana.log         |
  +---------------------------------------------+

The important structural decision is the loopback bind. Grafana 11.x defaults to 0.0.0.0:3000, which means a proxy misconfiguration does not expose the backend — the backend was already exposed. Setting http_addr = 127.0.0.1 makes the proxy the only path in, so a mistake in the proxy fails closed.

Requirements

A disposable Debian 12 or Ubuntu 24.04 host with sudo and outbound HTTPS to apt.grafana.com. The lab installs two packages, writes into /etc, and edits /etc/hosts. Do not run it on a machine you care about.
About 500 MB of free disk and 1 GB of RAM.
curl, jq, openssl and sqlite3. Task 1 records which of these were already installed so Cleanup removes only what this lab added.
Ports 80, 443 and 3000 free. Task 1 checks.
No out-of-band access requirement. This lab does not touch SSH, the firewall, or the primary interface. nginx binds 80 and 443, so if you are reaching this host through a web server on either port, stop reading and use a different host — that is the one way this lab can cut you off.

Scenario

A new region needs a Grafana this week. The last two regions were built by different people: one from a tarball dropped in /opt, one from a container started by hand on a host nobody has rebooted since. Neither has an upgrade procedure, and the second one has been serving cleartext on port 3000 to the office VLAN for eleven months.

You are building the third, and it is going to be the one whose install, validation, and teardown are all written down. The target is: package-managed, version-pinned, loopback-bound, TLS in front, password from a secret file, and a validation that answers “is it useful” rather than “is the process alive”.

Tasks

Task 1: Capture the starting state

LAB="$HOME/grafana-deploy-lab"
mkdir -p "$LAB"
cd "$LAB"

# What is already listening? 80, 443 and 3000 must be free.
ss -ltnp 2>/dev/null | awk 'NR==1 || /:(80|443|3000)\s/' | tee ports.pre-lab

# Which of the lab's tools were already here? Cleanup reads this.
for p in curl jq openssl sqlite3 nginx; do
  printf '%s %s\n' "$p" "$(dpkg-query -W -f='${Status}' "$p" 2>/dev/null || echo notinstalled)"
done | tee packages.pre-lab

# The hosts file, which Task 6 edits.
sudo cp -a /etc/hosts "$LAB/hosts.pre-lab"
ls -l "$LAB"

If ports.pre-lab shows anything bound to 80, 443 or 3000, resolve it before continuing. A port conflict later in this lab surfaces as nginx failing to reload with a message about a socket, three tasks after the cause.

Install the tools the lab needs:

sudo apt-get update
sudo apt-get install -y curl jq openssl sqlite3 ca-certificates

Task 2: Add the signed repository and pin a version

Modern apt refuses an unsigned source, so the key and the signed-by= clause are one step, not two.

sudo install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://apt.grafana.com/gpg.key \
  | gpg --dearmor \
  | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
sudo chmod 0644 /etc/apt/keyrings/grafana.gpg

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt-get update

Now find out what the repository actually offers rather than assuming a version string. This is the step people skip, and it is why apt-get install grafana=11.3.0 fails with a message about a version that has no installation candidate:

apt-cache madison grafana | head -10
apt-cache policy grafana

Pick a version from that list and pin it. The example below uses 11.3.0; substitute whatever madison printed:

Configuration changelab host— Installs grafana, creates the grafana user, and installs the unit. Does not start it.

$ GF_VER=11.3.0; sudo apt-get install -y grafana=$GF_VER

The package does not enable or start the service. That is deliberate on Grafana’s part and it is the window this lab uses: you get to set the admin password before the first boot creates the admin user.

# What did the package put where?
dpkg -L grafana | grep -E '^/(etc|usr/sbin|lib/systemd)' | sort | head -20
systemctl is-enabled grafana-server    # expect: disabled
systemctl is-active grafana-server     # expect: inactive

If either reports otherwise, the service started before you set a password and the admin row already exists with the shipped default. Close that window before going on — on a host that has nothing in it yet, the cheapest fix is to throw the database away and let the next start create it again:

if [ "$(systemctl is-active grafana-server)" = active ]; then
  sudo systemctl stop grafana-server
  sudo rm -f /var/lib/grafana/grafana.db
  echo "database removed; first boot will happen after Task 3"
fi

Task 3: Set the admin password before first boot

The admin user row is created during the first-boot schema migration, using whatever password Grafana can find at that moment. Give it one from a file, and admin / admin never exists on this host at all.

sudo install -o root -g grafana -m 0640 /dev/null /etc/grafana/admin_password
openssl rand -base64 24 | tr -d '=/+' \
  | sudo tee /etc/grafana/admin_password > /dev/null

sudo install -o root -g grafana -m 0640 /dev/null /etc/grafana/grafana.env
printf 'GF_SECURITY_ADMIN_PASSWORD__FILE=/etc/grafana/admin_password\n' \
  | sudo tee /etc/grafana/grafana.env > /dev/null

Grafana reads GF_<SECTION>_<KEY> from the environment, and the __FILE suffix tells it to read the value from a path instead. The systemd drop-in in the next task is what puts that variable into the unit’s environment.

Task 4: Configure Grafana, then harden the unit

grafana.ini is an override file. Grafana reads /usr/share/grafana/conf/defaults.ini for every key you do not set, so the file in /etc only has to carry the differences. The shipped copy is several hundred lines of commented defaults; replacing it with the handful of keys that are actually yours makes the next .dpkg-dist diff readable instead of enormous.

# /etc/grafana/grafana.ini  (the whole file; everything else comes from
# /usr/share/grafana/conf/defaults.ini)
[server]
# Loopback only. The proxy is the public face, and a proxy mistake now
# fails closed instead of exposing 3000.
http_addr = 127.0.0.1
http_port = 3000
domain = grafana.example.com
# Forwarded headers are honoured only from these sources. nginx is on
# this host, so loopback is the whole trusted set. Blank trusts nothing;
# a wildcard lets any client choose the IP that lands in the audit log.
trusted_proxies = 127.0.0.1

[database]
type = sqlite3
path = grafana.db
# The shipped busy_timeout of 1s is too short for an alert-evaluation
# burst; sqlite has one writer and the queue is real.
busy_timeout = 10000
max_open_conn = 1
max_idle_conn = 2

[security]
admin_user = admin
cookie_secure = true
cookie_samesite = lax

[users]
allow_sign_up = false

[auth.anonymous]
enabled = false

Keep the shipped file — you will want to diff against it after an upgrade — then write yours:

sudo cp -a /etc/grafana/grafana.ini /etc/grafana/grafana.ini.packaged

sudo tee /etc/grafana/grafana.ini > /dev/null <<'INI'
[server]
http_addr = 127.0.0.1
http_port = 3000
domain = grafana.example.com
trusted_proxies = 127.0.0.1

[database]
type = sqlite3
path = grafana.db
busy_timeout = 10000
max_open_conn = 1
max_idle_conn = 2

[security]
admin_user = admin
cookie_secure = true
cookie_samesite = lax

[users]
allow_sign_up = false

[auth.anonymous]
enabled = false
INI

sudo chown root:grafana /etc/grafana/grafana.ini
sudo chmod 0640 /etc/grafana/grafana.ini

The hardening drop-in. A drop-in rather than an edit of the shipped unit, so a package upgrade cannot silently drop it:

# /etc/systemd/system/grafana-server.service.d/99-lab.conf
[Service]
EnvironmentFile=/etc/grafana/grafana.env

# Dashboards, alert rules and data source connections all want file
# descriptors. The shipped default is the distribution default.
LimitNOFILE=65536

ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadWritePaths=/var/lib/grafana /var/log/grafana
RestrictSUIDSGID=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictNamespaces=true
LockPersonality=true

Score the unit before and after, so the hardening is a measurement rather than a belief:

systemd-analyze security grafana-server.service | tail -3

sudo mkdir -p /etc/systemd/system/grafana-server.service.d
sudo tee /etc/systemd/system/grafana-server.service.d/99-lab.conf > /dev/null <<'DROPIN'
[Service]
EnvironmentFile=/etc/grafana/grafana.env
LimitNOFILE=65536
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadWritePaths=/var/lib/grafana /var/log/grafana
RestrictSUIDSGID=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictNamespaces=true
LockPersonality=true
DROPIN

sudo systemctl daemon-reload
systemd-analyze security grafana-server.service | tail -3

Task 5: Start it and validate layers 1 and 2

Service impact possiblelab host— First boot: runs the schema migration and creates the admin user

$ sudo systemctl enable --now grafana-server

# Layer 1 - the process.
systemctl is-active grafana-server
systemctl is-enabled grafana-server

# Layer 2 - the local HTTP probe. No credential needed.
for i in $(seq 1 30); do
  curl -fsS http://127.0.0.1:3000/api/health && break
  sleep 2
done

A healthy layer 2 returns database as ok alongside the version. Anything else means the process is up and the store is not, and the log will say which:

sudo journalctl -u grafana-server -n 40 --no-pager | grep -iE 'error|migrat|listen'

Confirm the password you set is the password that took effect, and that admin / admin never worked:

GF_PW=$(sudo cat /etc/grafana/admin_password)
curl -fsS -u "admin:$GF_PW" http://127.0.0.1:3000/api/org | jq -r '.name'

# Expect HTTP 401 for the published default.
curl -s -o /dev/null -w 'default creds: HTTP %{http_code}\n' \
  -u admin:admin http://127.0.0.1:3000/api/org

Task 6: Terminate TLS at nginx

Install nginx and issue a self-signed certificate. Self-signed is right for a lab and wrong for production — the point here is to exercise the proxy path and the certificate validation, not to model your CA:

sudo apt-get install -y nginx

sudo install -d -m 0755 /etc/ssl/lab
sudo openssl req -x509 -newkey rsa:2048 -nodes -days 2 \
  -keyout /etc/ssl/lab/grafana.key -out /etc/ssl/lab/grafana.crt \
  -subj '/CN=grafana.example.com' \
  -addext 'subjectAltName=DNS:grafana.example.com'
sudo chmod 0600 /etc/ssl/lab/grafana.key

# Resolve the name locally. Task 1 backed this file up.
echo '127.0.0.1 grafana.example.com' | sudo tee -a /etc/hosts

The site. Note the two header details that cause most proxy incidents: X-Forwarded-Proto must carry the scheme the client used, and the WebSocket upgrade needs HTTP/1.1 plus the Connection map:

# /etc/nginx/sites-available/grafana.conf
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

upstream grafana_upstream {
  server 127.0.0.1:3000 max_fails=3 fail_timeout=10s;
  keepalive 32;
}

server {
  listen 80;
  server_name grafana.example.com;
  return 301 https://$host$request_uri;
}

server {
  # HTTP/2 is deliberately not enabled here. The standalone `http2 on;`
  # directive arrived in nginx 1.25.1; Debian 12 ships 1.22 and Ubuntu
  # 24.04 ships 1.24, where the spelling is `listen 443 ssl http2;` and
  # the newer form fails `nginx -t`. Check `nginx -v` before copying a
  # TLS block from anywhere.
  listen 443 ssl;
  server_name grafana.example.com;

  ssl_certificate     /etc/ssl/lab/grafana.crt;
  ssl_certificate_key /etc/ssl/lab/grafana.key;
  ssl_protocols       TLSv1.2 TLSv1.3;

  add_header Strict-Transport-Security "max-age=15768000" always;

  # The nginx host is the Grafana host, so the only trusted hop is
  # loopback. This must agree with trusted_proxies in grafana.ini.
  set_real_ip_from 127.0.0.1;
  real_ip_header X-Forwarded-For;

  location / {
    proxy_pass http://grafana_upstream;
    proxy_http_version 1.1;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header Upgrade           $http_upgrade;
    proxy_set_header Connection        $connection_upgrade;
    proxy_read_timeout 300s;
  }
}

Write it, enable it, and let nginx check its own syntax before you reload:

sudo tee /etc/nginx/sites-available/grafana.conf > /dev/null <<'SITE'
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

upstream grafana_upstream {
  server 127.0.0.1:3000 max_fails=3 fail_timeout=10s;
  keepalive 32;
}

server {
  listen 80;
  server_name grafana.example.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;
  server_name grafana.example.com;

  ssl_certificate     /etc/ssl/lab/grafana.crt;
  ssl_certificate_key /etc/ssl/lab/grafana.key;
  ssl_protocols       TLSv1.2 TLSv1.3;

  add_header Strict-Transport-Security "max-age=15768000" always;

  set_real_ip_from 127.0.0.1;
  real_ip_header X-Forwarded-For;

  location / {
    proxy_pass http://grafana_upstream;
    proxy_http_version 1.1;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header Upgrade           $http_upgrade;
    proxy_set_header Connection        $connection_upgrade;
    proxy_read_timeout 300s;
  }
}
SITE

sudo ln -sf /etc/nginx/sites-available/grafana.conf \
            /etc/nginx/sites-enabled/grafana.conf
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

Validate layer 3 — and validate it against the certificate, not around it:

# The redirect from cleartext.
curl -sI http://grafana.example.com/api/health | head -3

# The TLS path, verifying the chain we created.
curl -fsS --cacert /etc/ssl/lab/grafana.crt \
  https://grafana.example.com/api/health

# HSTS is present and the response is not a redirect.
curl -fsSI --cacert /etc/ssl/lab/grafana.crt \
  https://grafana.example.com/api/health | grep -iE '^(HTTP|strict-transport)'

Task 7: Make layer 4 fail on purpose

Layers 1 to 3 are all green. Now create a data source whose URL is wrong in the way real ones are wrong — right host, wrong port — and watch what each layer says:

GF_PW=$(sudo cat /etc/grafana/admin_password)

curl -fsS -u "admin:$GF_PW" -H 'Content-Type: application/json' \
  -X POST https://grafana.example.com/api/datasources \
  --cacert /etc/ssl/lab/grafana.crt \
  -d '{"name":"Prometheus","uid":"prom-lab","type":"prometheus","access":"proxy","url":"http://127.0.0.1:9999"}' \
  | jq -r '.message'

# Layers 1-3: still perfect.
systemctl is-active grafana-server
curl -fsS --cacert /etc/ssl/lab/grafana.crt \
  https://grafana.example.com/api/health | jq -r '.database'

# Layer 4: the only one that notices.
curl -sS -u "admin:$GF_PW" --cacert /etc/ssl/lab/grafana.crt \
  https://grafana.example.com/api/datasources/uid/prom-lab/health \
  | jq '{status, message}'

The data source health endpoint runs a real query against the upstream with the configured credentials. It is the first layer in the ladder that leaves the Grafana host. A monitoring system that probes only /api/health reports this install as healthy for as long as nobody opens a dashboard.

Now correct the URL and watch the same probe change its mind. Leave a real backend out of it — this lab has none, and pointing the data source at the provisioning lab’s Prometheus is that lab’s job:

curl -fsS -u "admin:$GF_PW" --cacert /etc/ssl/lab/grafana.crt \
  -X DELETE https://grafana.example.com/api/datasources/uid/prom-lab \
  | jq -r '.message'

Task 8: Prove the backup unit

For a sqlite install the backup unit is one file, and the restore is a stop, a copy and a start. Prove it by creating something to lose:

GF_PW=$(sudo cat /etc/grafana/admin_password)

curl -fsS -u "admin:$GF_PW" -H 'Content-Type: application/json' \
  --cacert /etc/ssl/lab/grafana.crt \
  -X POST https://grafana.example.com/api/dashboards/db \
  -d '{"dashboard":{"uid":"lab-canary","title":"Lab canary","panels":[],"schemaVersion":39},"overwrite":true}' \
  | jq -r '.status'

Service impact possiblelab host— Stops Grafana so the sqlite file is quiescent before the copy

$ sudo systemctl stop grafana-server

sudo sqlite3 /var/lib/grafana/grafana.db 'PRAGMA integrity_check;'
sudo cp -a /var/lib/grafana/grafana.db "$HOME/grafana-deploy-lab/grafana.db.bak"
sudo systemctl start grafana-server

An integrity_check that returns anything other than ok means the copy you are about to take is a copy of a damaged database. Run it before the backup, not after the restore, when the answer is still useful.

Now delete the dashboard, restore, and confirm it came back:

GF_PW=$(sudo cat /etc/grafana/admin_password)
curl -fsS -u "admin:$GF_PW" --cacert /etc/ssl/lab/grafana.crt \
  -X DELETE https://grafana.example.com/api/dashboards/uid/lab-canary | jq -r '.title'

sudo systemctl stop grafana-server
sudo cp -a "$HOME/grafana-deploy-lab/grafana.db.bak" /var/lib/grafana/grafana.db
sudo chown grafana:grafana /var/lib/grafana/grafana.db
sudo systemctl start grafana-server

sleep 5
curl -fsS -u "admin:$GF_PW" --cacert /etc/ssl/lab/grafana.crt \
  https://grafana.example.com/api/dashboards/uid/lab-canary | jq -r '.dashboard.title'

Validation

The five layers, run in order, as one transcript. Layer 4 has no backend in this lab and says so rather than pretending.

GF_PW=$(sudo cat /etc/grafana/admin_password)
CA=/etc/ssl/lab/grafana.crt
URL=https://grafana.example.com

echo "== layer 1: process"
systemctl is-active grafana-server
systemctl show -p NRestarts --value grafana-server

echo "== layer 2: local probe"
curl -fsS http://127.0.0.1:3000/api/health | jq '{database, version}'

echo "== layer 3: through the proxy, with certificate validation"
curl -fsSI --cacert "$CA" "$URL/api/health" | head -1
curl -sI http://grafana.example.com/api/health | head -1

echo "== layer 4: data sources"
curl -fsS -u "admin:$GF_PW" --cacert "$CA" "$URL/api/datasources" \
  | jq 'if length == 0 then "no data sources configured - layer 4 not exercised" else . end'

echo "== layer 5: measured latency, not assumed"
for i in $(seq 1 20); do
  curl -fsS -u "admin:$GF_PW" --cacert "$CA" -o /dev/null -w '%{time_total}\n' \
    "$URL/api/search?limit=1"
done | sort -n | awk '{a[NR]=$1} END {printf "p50=%.3fs p95=%.3fs n=%d\n", a[int(NR*0.5)], a[int(NR*0.95)], NR}'

Confirm the security posture holds:

# Grafana is NOT reachable on the network, only on loopback.
ss -ltnp 2>/dev/null | grep ':3000'

# Anonymous access is off.
curl -s -o /dev/null -w 'anon: HTTP %{http_code}\n' \
  --cacert /etc/ssl/lab/grafana.crt https://grafana.example.com/api/org

# The unit is running the hardening you wrote.
systemctl show grafana-server -p ProtectSystem -p ReadWritePaths -p LimitNOFILE

Expected Outcome

grafana-server is active and enabled, with NRestarts at 0.
ss shows 3000 bound to 127.0.0.1 only; 80 and 443 answer on all interfaces via nginx.
https://grafana.example.com/api/health returns 200 with certificate validation on, and http:// returns a 301.
admin / admin returns 401; the generated password returns the org.
systemd-analyze security scores lower after the drop-in than before.
The canary dashboard survived a delete followed by a restore from the sqlite backup.

Troubleshooting

apt-get install grafana=11.3.0 reports no installation candidate. That exact version is not in the repository for your architecture or suite. Run apt-cache madison grafana and pin one it lists. Pinning to a version you guessed is how a “reproducible” install becomes an unpinned one when somebody later removes the = to make it work.

/api/health reports the database as something other than ok. Almost always a permissions or a ReadWritePaths problem, not a Grafana bug. Check ls -ld /var/lib/grafana — it must be owned by grafana — and then systemctl show grafana-server -p ReadWritePaths. Under ProtectSystem=strict a path missing from that list is read-only to the unit no matter what the filesystem says.

nginx fails to start with an address-already-in-use message. Something else owns 80 or 443. Compare against ports.pre-lab from Task 1: if it was there before the lab, the lab is on the wrong host.

curl --cacert fails with a name mismatch. The certificate’s SAN is grafana.example.com and you asked for something else, or the /etc/hosts line did not take. getent hosts grafana.example.com should return 127.0.0.1. This is a real production failure in miniature: the name in the certificate, the name in the request, and the name in DNS all have to be the same name.

Grafana logs the user out on every page load behind the proxy. The scheme Grafana thinks the client used does not match reality. Confirm nginx is sending X-Forwarded-Proto and that trusted_proxies in grafana.ini includes the proxy’s source address. Grafana ignores forwarded headers from an untrusted source, which looks identical to the proxy not sending them.

The data source health check returns an error you did not expect. Read the message field, then reproduce it from the Grafana host with curl against the same URL. Layer 4 failing while a direct curl succeeds points at the Grafana process — egress rules, a proxy environment variable in the unit — and not at the upstream.

Cleanup

LAB="$HOME/grafana-deploy-lab"

# 1. Stop and remove the services.
sudo systemctl disable --now grafana-server
sudo rm -f /etc/systemd/system/grafana-server.service.d/99-lab.conf
sudo rmdir --ignore-fail-on-non-empty /etc/systemd/system/grafana-server.service.d
sudo systemctl daemon-reload

sudo rm -f /etc/nginx/sites-enabled/grafana.conf /etc/nginx/sites-available/grafana.conf
sudo ln -sf /etc/nginx/sites-available/default /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

# 2. Remove the package and everything it owns.
sudo apt-get purge -y grafana
sudo rm -rf /var/lib/grafana /var/log/grafana /etc/grafana

# 3. Remove the repository, the keyring and the certificate.
sudo rm -f /etc/apt/sources.list.d/grafana.list /etc/apt/keyrings/grafana.gpg
sudo rm -rf /etc/ssl/lab
sudo apt-get update

# 4. Restore /etc/hosts from the Task 1 capture.
sudo cp -a "$LAB/hosts.pre-lab" /etc/hosts
getent hosts grafana.example.com || echo "hosts entry removed"

# 5. Remove only the packages this lab installed. Read packages.pre-lab
#    first and drop from this list anything that was already present.
cat "$LAB/packages.pre-lab"
# sudo apt-get purge -y nginx sqlite3   # uncomment for what you added
sudo apt-get autoremove -y

# 6. The lab directory, including the database backup.
rm -rf "$LAB"

Confirm the host is as you found it: ss -ltn should match ports.pre-lab, and systemctl status grafana-server should report the unit as not found.

What You Learned

The install method is an upgrade and backup procedure you are choosing. The apt path gave you a versioned package, a systemd unit, a grafana user, and a .dpkg-dist diff on upgrade. Pinning the version made the install reproducible; discovering the version with madison is what makes the pin honest.
The safest moment to set the admin password is before first boot. The admin row is created during the first schema migration. Set the password first and the published default never exists on the host, so there is no window to defend and nothing to rotate afterwards.
Loopback binding makes proxy mistakes fail closed. With Grafana on 0.0.0.0, a mistake in the nginx site leaves the backend reachable anyway. On 127.0.0.1 the same mistake produces a 502 — loud, immediate and safe.
systemctl is-active is layer 1 of five. You saw layers 1 to 3 stay green through a data source that could not answer a single query. A monitoring system built on layer 1 alone reports this install healthy right up to the moment somebody needs it.
A backup you have not restored is a hypothesis. The integrity_check, the copy of a stopped database, and the delete-and-restore of the canary dashboard are three separate claims, and you tested all three.

Taking this to a real change window

This install maps onto a change window in four blocks, each with its own rollback. Repository and package: rollback is apt-get purge, no user impact because nothing has started. Configuration and unit: rollback is removing the drop-in and daemon-reload. First start: the only irreversible step, because it migrates the schema — take the host out of the load balancer first if it is a rebuild rather than a new build. Proxy: rollback is a symlink and a reload, measured in seconds.

Two things this lab simplifies that production does not. The certificate is self-signed here; in production it comes from your CA with an expiry you monitor, because a Grafana that is unreachable during an incident is indistinguishable from a Grafana that is down. And sqlite is the right default for one host and the wrong one for a pair behind a load balancer — that decision belongs before the first boot, not after the database has six months of dashboards in it.

Objectives

Prerequisites