Ransomware: determining scope and the compromise timeline

Separating the identity that writes backups from the identity that can destroy them limits what one stolen credential reaches. It does not tell you when that credential stopped being yours, and that is the question standing between an incident and its first authorised restore. Every recovery point taken after the intruder arrived is suspect, not because its bytes are damaged but because it is a faithful copy of a compromised system. Restoring one reinstates the access that produced the incident, and the encryption runs again from inside the recovery. So the first deliverable of a ransomware response is not a restore plan. It is a timestamp, per system, with evidence behind it.

Two clocks, and the loud one is the wrong one

Every ransomware incident has two times, and they are separated by days or months.

The encryption time is the one everybody knows. Files are rewritten in place, extensions change, a note appears, a database refuses to start, and somewhere a monitoring check finally goes red. It is precise, it is easy to establish, and it is the last act of the operation rather than the first.

The intrusion time is the one that matters and the one nobody has. A valid credential used on a normal port. A session that authenticates successfully, because the password was genuinely correct. A package installed from a repository the host already trusts. A unit file created by root, which is what root does. Nothing in that sequence is anomalous in kind, only in context, so none of it fires anything.

The interval between them is dwell time, and its length is not a number you can look up. It is a property of this incident on this estate, and you learn it by measuring. What you can say in advance is the direction: the intrusion is always earlier, and a recovery point chosen against the encryption time is therefore always at risk of sitting on the wrong side of the boundary.

This produces two questions that incident calls routinely merge into one. Which copies are intact? is a question about bytes, and the earlier parts of this course answer it with verification that reads the data, with versioning, and with retention that a compromised credential cannot shorten. Which copies predate the intrusion? is a question about time, and no property of the storage answers it. Immutability, in the object-lock sense this part teaches, guarantees that a copy still exists for the length of its retention window. It says nothing whatsoever about whether that copy is clean.

The evidence set, and what each source can bound

Timeline reconstruction is interval narrowing. Each source gives an upper bound — the intrusion happened no later than this — and occasionally a lower one, where a source you trust shows nothing anomalous before some point. Confidence comes from independent sources agreeing, not from any single artefact, and the interval is reported honestly as an interval until it stops shrinking.

Authentication and session records are the first place to look and the easiest to over-trust. You want the first session that is anomalous in source rather than in outcome: a successful login for an account that has never authenticated from that network, a service account with an interactive session, a sudo invocation by someone who was on leave. Local records are a starting point, but a root-level intruder can edit the local journal, so the copy that carries weight is the one shipped off-host.

Package and binary changes are recorded by the package manager with its own timestamps, which is a second, independent clock on the same host. Alongside it, binaries whose content timestamp looks old while their inode-change timestamp is recent are the classic tell, for the reason set out below.

File modification times across the estate are the broad sweep: what changed under /etc, /root, /usr/local, service home directories and the web root since a candidate boundary. This is the noisiest source and the one most often misread, so it is best used to generate candidates that another source confirms.

Scheduled tasks and service units are frequently the tightest early bound available on the victim host, because persistence is established early. An operator who intends to spend three weeks in an estate needs to survive a reboot in the first hour, so a timer, a cron entry or a unit file created days before the encryption is exactly the artefact you are looking for.

Outbound connections are recorded where the attacker usually could not reach: the firewall, the forward proxy, the resolver, the flow collector. The first connection from a production host to a destination nobody can account for is often the single best bound in the whole exercise, and it is the one that survives the host being reimaged.

The logs of the backup system itself are the source teams forget, and they are frequently the earliest. Operators enumerate before they destroy: they list repositories, read catalogues, attempt a restore to confirm what is recoverable, and edit retention so that the copies age out quietly. Every one of those is a timestamped operation on a system the victim host does not control. And when the destructive attempt finally lands against a versioned target, it leaves a dated object rather than an absence.

$ mc ls --versions prod/rbdr-immutable/backup-0900.tar

[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar

>>> exit code: 0

Read as forensics rather than as storage behaviour, that listing is a witness statement. The lower row records when the backup was written. The upper row records when a credential holding delete rights was used against it, to the second, from a system the compromised host does not administer. In the capture the two are two seconds apart because it is a capture; in an estate the gap is the interesting quantity, and the DEL row bounds the compromise of that credential. On a target without versioning the same action produces no row at all — just an object that is no longer listed, with nothing to say when it stopped being listed or who asked.

The mechanics of the sweep follow from that. Read everything in a single timezone, keep the two timestamps side by side, and write the output somewhere the host being examined does not control.

# Every timestamp below is read in UTC so hosts can be merged later.
export TZ=UTC
HOST=$(hostname -s)
OUT="/evidence/$HOST"
mkdir -p "$OUT"

# authenticated sessions, in an unambiguous format, oldest first
journalctl --no-pager -o short-iso _COMM=sshd > "$OUT/ssh-sessions.log"

# units and scheduled jobs, with the time each file was last changed on disk
find /etc/systemd/system /etc/cron.d -type f \
  -printf '%TY-%Tm-%TdT%TH:%TM:%TS %p\n' 2>/dev/null \
  | sort > "$OUT/units-and-jobs.txt"

With a candidate boundary in hand, the same tool tests it. The first sweep asks what changed after the candidate; the second asks which files disagree with themselves, which is the question an attacker cannot easily answer for.

# T0 is the earliest evidence of intrusion so far - NOT the encryption time.
T0='2026-08-14 00:00:00'

# what changed on disk after T0, on this filesystem only
find /etc /root /usr/local -xdev -newermt "$T0" \
  -printf '%TY-%Tm-%TdT%TH:%TM:%TS %p\n' 2>/dev/null | sort

# a backdated file gives itself away: content date old, inode date recent
find /usr/bin /usr/sbin -xdev -newerct "$T0" \
  -printf 'mtime %TY-%Tm-%Td ctime %CY-%Cm-%Cd %p\n' 2>/dev/null | sort

One scheduled interval, exit code 0, monitoring green throughout

The reason the timeline outranks the alert is easiest to see on the simplest possible replication arrangement. In a three-night capture on rsync 3.4.1, a source tree of three files is mirrored to another host by rsync -a --delete at 01:00. On the third night the source is encrypted in place, and the schedule does what schedules do.

$ rsync -a --delete src/ mirror/

--- mirror after that run ---
mirror/orders.csv.locked
mirror/reports/q3.txt.locked

The mirror is now a faithful copy of ciphertext, and the capture’s own count of what remains is plaintext copies of the ledger remaining anywhere: 0. Its closing sentence is the one to carry into an incident call: “The mirror did exactly what it was configured to do, on schedule, with exit code 0, and it did it to the only other copy of the data.”

Three consequences bear directly on the timeline exercise. First, the elapsed time between the injury and the loss of the second copy was one scheduled interval — the capture states the general form on the previous night, where an operator deletion propagated and the note reads “Elapsed time between the mistake and the loss of the only other copy: one scheduled interval. Nobody had to make a second mistake.” Night three is the same interval with an attacker in place of an operator. Second, the alert that eventually fires is keyed to the encryption, and by the time it fires the copy nearest the incident may already be a copy of the encrypted data. Third, nothing anywhere went red on the way past: the run exited 0, so a dashboard watching job outcomes showed the same green it had shown for months. The job ran. That is the entire content of the signal, and it is not a statement about which nights are usable.

The estate has no single clean line

The most expensive assumption in a ransomware response is that the estate can be rolled back to one moment. It usually cannot, because encryption is staged.

Operators move laterally over days, and they encrypt when each system is ready rather than when the calendar says so. Backup infrastructure is commonly neutralised first, file servers and databases next, endpoints last. Even where the payload is scripted to fire everywhere at once, it does not land everywhere at once: hosts are powered off, in a maintenance window, behind a segment the operator had not yet crossed, or running an agent that failed. The result is a spread of encryption times across the estate, sometimes across days.

That spread orders nothing. A host encrypted last may have been compromised first, because it was the foothold and the operator had no reason to burn it early. Two hosts encrypted in the same minute may have been entered three weeks apart. Encryption time is a property of the attacker’s scheduling, and inferring intrusion order from it is the error this section exists to prevent.

Two failure modes follow, and they cost in opposite currencies. Set a boundary too late on one system and you reinstate the intruder on that system, at which point the whole recovery is suspect again. Set a single estate-wide boundary at the earliest evidence found anywhere and you are safe, but you discard weeks of legitimate data on systems that were never touched — a real, quantifiable loss that the business will feel and that you will have to justify.

The defensible position is between them, and it has to be written down rather than held in someone’s head: a boundary per system, with the evidence that produced it. Systems sharing an identity share a boundary, because a credential compromised on one is compromised for every system that accepts it, so the directory service, the configuration-management controller and the CI system propagate their boundaries to everything downstream. Systems with independent credentials and no evidence of contact keep their own, later, less costly boundary — provided you can say why.

What to preserve before anything is touched

Recovery destroys evidence. A restore that overwrites the affected volume removes the only copy of the intruder’s traces and the only fallback if the restore itself is wrong, and it is usually done under pressure by people who have been awake for a long time. The ordering principle is standard and worth following exactly: capture in decreasing order of volatility, because the most fragile evidence is the first to disappear and the last to be missed.

In practice, before any host is rebooted, reimaged or restored: take a storage-layer snapshot or a disk image of each affected system; capture the volatile state, accepting that on a compromised kernel it is best-effort; retrieve the off-host copies while they still exist — central logs, firewall and proxy records, identity-provider sign-in logs, and the backup server’s own catalogue and audit trail. Suspend the lifecycle rules that would expire noncurrent versions, and leave delete markers exactly where they are; on a versioned target the marker is a dated record of the destructive attempt, and removing it to tidy the bucket destroys the timestamp shown earlier in this lesson. Freeze log retention explicitly, because retention expires on schedule during an incident and the record you need is often the oldest one you have.

INCIDENT=inc-2026-0828
SAFE="/mnt/evidence/$INCIDENT/$(hostname -s)"
mkdir -p "$SAFE/var-log"

# volatile first: running processes with start times, sockets, sessions
ps -eo pid,ppid,lstart,user,args > "$SAFE/processes.txt"
ss -tunap                        > "$SAFE/sockets.txt"
who -a                           > "$SAFE/sessions.txt"

# then the on-disk logs, copied out rather than read in place
cp -a /var/log/. "$SAFE/var-log/"

# a manifest makes the set defensible weeks later
find "$SAFE" -type f -exec sha256sum {} + > "$SAFE.sha256"

One more preservation rule is procedural rather than technical: record what the responders do, with times. A week later, in a merged timeline, an unrecorded administrative login is indistinguishable from an intrusion, and hours get spent chasing a colleague.

What all of this produces is deliberately narrow. Not a plan, not a root cause, not an attribution — a table with one row per system, an earliest-known-good timestamp, the evidence that supports it, and a note saying which other systems share that boundary because they shared an identity. That table is the input to the next lesson, where recovery points are actually chosen and the restore sequence is built on top of it. Until it exists, every restore date under discussion is a guess wearing a timestamp.

Production discipline

1. Bound the intrusion, not the encryption. The encryption time is precise, easy and late; it tells you when the operation ended. Write the boundary against the earliest evidence of access, and treat every recovery point after it as suspect until something independent says otherwise.
2. Take the evidence from where the attacker could not write. Local journals, local package logs and local file timestamps are leads. The off-host copies — central logs, firewall and proxy records, and the backup server’s own catalogue, which usually records the reconnaissance before the encryption — are what the timeline rests on.
3. Freeze retention before it expires under you. Suspend lifecycle rules that would remove noncurrent versions, leave delete markers in place, and extend log retention on day one. A v2 DEL row timestamped 13:28:22 above a v1 PUT at 13:28:20 is a dated witness; a tidied bucket is an absence.
4. Give every system its own boundary, and say which systems share one. Encryption is staged, so the observed times order nothing. Systems that trust a compromised identity inherit its boundary; systems that do not, keep their own and keep the data that comes with it.
5. Distrust the green signal, and check what it measured. A rsync -a --delete mirror carried an in-place encryption to the only other copy on rsync 3.4.1, on schedule and with exit code 0, ending with plaintext copies of the ledger remaining anywhere: 0. Job success is a statement that a process ran, and the timeline is what tells you which of those runs are usable.

Cross-course references

• Observability for Production Sysadmins — Part CVIII (Clock Skew) is the direct dependency of this lesson: every bound assembled here is a timestamp written in the frame of the host that produced it, and merging hosts whose clocks disagree yields an ordering that is wrong in exactly the direction that makes a recovery point look safer than it is.
• Linux for Production Sysadmins — Part XXXI (Audit and Security Logging) supplies the records this lesson consumes. The first-anomalous-session question is only answerable when authentication and audit events were being generated and shipped off-host before the incident, which is a decision taken months earlier.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVIII (Incidents and Recovery) takes the same boundary and applies it to credentials rather than data: every secret readable from a system inside its compromise window is in scope for rotation, so the timestamp produced here sizes the rotation work as well as the restore.

Quiz