perf - finding the code that is burning the CPU

top tells you which process is burning CPU. That is often as far as an investigation gets, because the answer - “the application” - is not actionable. perf answers the next question: which function, in which library, reached from which call path.

perf is a sampling profiler. It interrupts the CPU at a fixed frequency, records the instruction pointer and optionally the call stack, and aggregates. It does not instrument your code, it does not need a restart, and its overhead at sensible sampling rates is small enough to run on production.

The first attempt usually fails

Run this and read the error, because you will meet it:

perf stat -e task-clock true

Error:
No supported events found.
Access to performance monitoring and observability operations is limited.
Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
access to performance monitoring and observability operations for processes
without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
More information can be found at 'Perf events and tool security' document:
https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
perf_event_paranoid setting is 4:

perf_event_paranoid gates what an unprivileged process may measure:

cat /proc/sys/kernel/perf_event_paranoid

4

Ubuntu and Debian ship 4, which blocks unprivileged profiling entirely. Two ways forward:

# Option 1: run perf under sudo. Simplest, and the usual answer on a server.
sudo perf stat -e task-clock,context-switches,cycles,instructions -- sleep 5

# Option 2: relax the gate for this boot, for a developer host.
sudo sysctl -w kernel.perf_event_paranoid=1

A first profile

perf top is the live view - the profiler equivalent of top:

sudo perf top -F 99

Samples: 12K of event 'cycles:P', 4000 Hz, Event count (approx.): 9873445120
Overhead  Shared Object         Symbol
  21.44%  libcrypto.so.3        [.] sha512_block_data_order_avx2
  11.02%  [kernel]              [k] copy_user_enhanced_fast_string
   7.83%  postgres              [.] heap_page_prune
   5.11%  libc.so.6             [.] __memmove_avx_unaligned_erms
   3.02%  [kernel]              [k] finish_task_switch

For anything you want to keep or share, record to a file instead:

cd /var/tmp
sudo perf record -F 99 -a -g -- sleep 30

[ perf record: Woken up 14 times to write data ]
[ perf record: Captured and wrote 3.412 MB perf.data (14882 samples) ]

Reading the flags, because each one is a decision:

• -F 99 - sample at 99 Hz per CPU. The odd number is deliberate: a round 100 Hz can beat against timers that also run at 100 Hz and systematically over-sample whatever runs at that cadence.
• -a - all CPUs, system-wide. Use -p PID to profile one process instead.
• -g - record call stacks, not just the leaf function.
• -- sleep 30 - profile for 30 seconds. perf profiles for as long as the given command runs; sleep is the idiom for “for this long”.

Then read it:

sudo perf report --stdio --no-children

# Overhead  Command     Shared Object        Symbol
# ........  ..........  ...................  ..............................
#
    18.20%  postgres    libcrypto.so.3       [.] sha512_block_data_order_avx2
            |
            ---sha512_block_data_order_avx2
               SHA512_Update
               scram_verify_plain_password
               CheckSASLAuth
               ClientAuthentication

     9.44%  postgres    [kernel.kallsyms]    [k] copy_user_enhanced_fast_string

That call graph is the actionable form. Not “postgres is using CPU” but “18% of all CPU on this host is SHA-512 inside SCRAM authentication” - which points at connection churn and a missing connection pooler, a fix nobody would have found from top.

When the symbols are missing

A profile full of hexadecimal addresses is the second thing that goes wrong:

    31.77%  myapp    myapp                [.] 0x00000000000d41a0
    12.03%  myapp    [unknown]            [.] 0x00007f2a1c4b8e21

Three distinct causes, with three distinct fixes.

Stripped binaries. The symbol table was removed at packaging time. Install the matching debug symbols package - -dbgsym on Debian and Ubuntu, -debuginfo on RHEL - and re-run perf report. The recording does not need to be repeated; symbols are resolved at report time.

Kernel symbols hidden. [kernel] frames show as addresses when kptr_restrict hides them:

cat /proc/sys/kernel/kptr_restrict

1

sudo sysctl -w kernel.kptr_restrict=0

JIT-compiled code. Java, .NET and Node generate machine code at runtime that exists in no file, so there is nothing for perf to look up. These runtimes emit a /tmp/perf-PID.map file when asked - -XX:+PreserveFramePointer plus an agent for the JVM, --perf-basic-prof for Node. Without it, a JIT workload profiles as one large [unknown] block.

perf stat: is it even compute-bound?

Before optimising a hot function, find out whether the CPU is doing work or waiting. perf stat reads hardware counters:

sudo perf stat -d -- ./bench

 Performance counter stats for './bench':

          4,012.44 msec task-clock                #    0.998 CPUs utilized
             1,204      context-switches          #  300.066 /sec
                12      cpu-migrations            #    2.991 /sec
            88,301      page-faults               #   22.006 K/sec
    15,881,204,331      cycles                    #    3.958 GHz
     6,104,882,190      instructions              #    0.38  insn per cycle
     1,204,338,012      branches                  #  300.150 M/sec
        44,209,881      branch-misses             #    3.67% of all branches
     4,881,203,441      L1-dcache-loads           #    1.217 G/sec
       881,204,993      L1-dcache-load-misses     #   18.05% of all L1-dcache accesses
        92,004,118      LLC-loads                 #   22.930 M/sec
        41,882,004      LLC-load-misses           #   45.52% of all LL-cache accesses

       4.019880443 seconds time elapsed

The number to read first is instructions per cycle (IPC).

• IPC above roughly 2 - the CPU is retiring work efficiently. This is genuinely compute-bound, and the fix is a better algorithm.
• IPC below roughly 1 - the CPU is stalled most of the time, usually waiting on memory. Look at the cache miss rates. Here, IPC of 0.38 with 45% of last-level cache loads missing says the workload is memory-bound: it is not executing too many instructions, it is waiting for data.

A memory-bound workload does not get faster from a faster CPU or from micro-optimising the hot function. It gets faster from changing the data layout, and that is a very different piece of work to schedule.

perf stat also reads a running process, which is how you use it on production:

sudo perf stat -p "$(pgrep -f myapp | head -1)" -- sleep 10

Overhead, and where it is not small

At 99 Hz system-wide, perf record costs a fraction of a percent and is routinely run on production hosts. What changes that:

Two habits keep it safe. Bound the run with -- sleep N so a forgotten profiler cannot fill a filesystem, and write perf.data somewhere with room - a 30-second DWARF profile on a busy 64-core host is easily a gigabyte.

cd /var/tmp
sudo perf record -F 99 -a -g -o /var/tmp/perf.data -- sleep 30
ls -lh /var/tmp/perf.data

Check the sample count in the output. perf record reports lost samples if the ring buffer overflowed, and a profile that dropped a large fraction of its samples is biased toward whatever was cheap to record.

Taking the profile somewhere else

perf report resolves symbols against the local filesystem, so a perf.data copied to your laptop resolves nothing. perf archive bundles the binaries and symbol files it needs:

sudo perf archive /var/tmp/perf.data

Unpack the resulting archive into ~/.debug on the analysis machine, and perf report finds the symbols there. Failing that, run perf report --stdio > profile.txt on the host itself and move the text - less flexible, but it always works.

Knowledge check