Threads and cgroup views · Linux

Many production processes are multithreaded. nginx, mysqld, Java applications, and most modern daemons have many threads sharing one address space. The kernel tracks each thread as a separate schedulable entity with its own TID; the set of threads is the process.

Threads from ps

Read-only / Safeps -L— ps -L shows every thread (LWP = light-weight process; TID = thread ID). All four lines share PID 12345 — they are threads of one nginx process. Each thread is independently scheduled and uses its own CPU time.

$ ps -L -p 1234

  PID   LWP   TID  STAT   TIME COMMAND
12345 12345 12345 Sl   00:00:00 nginx: master process
12345 12346 12346 Sl   00:00:00 nginx: worker process
12345 12347 12347 Sl   00:00:00 nginx: worker process
12345 12348 12348 Sl   00:00:00 nginx: worker process

Illustrative output

Read-only / Safe/proc/<pid>/task— Every thread has a subdirectory under /proc/<pid>/task/. Each contains per-thread state — the same files as /proc/<pid>, but scoped to that thread. /proc/<pid>/task/<tid>/status shows the thread's state.

$ ls /proc/12345/task/ | head

Illustrative output

Threads vs processes

Property	Process	Thread
Address space	Independent	Shared
File descriptors	Independent	Shared (in most threads)
PID	One per process	One per thread (the TID)
Scheduling	Independent	Independent
Signal handlers	Independent	Process-wide
Resource accounting	Per-process	Per-thread (CPU time) but per-cgroup (memory)

How threads are implemented

Linux threads are processes created by clone() with the CLONE_VM | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD flags. The result is a new task that shares the address space, file descriptors, signal handlers, and other state with the parent. The kernel still treats the thread as a separate schedulable task — it has its own kernel stack, its own register set, and its own scheduling state.

The kernel exposes threads as task_structs, the same data structure as a regular process. The difference is the flags set on that struct and the shared resources they point to. There is no separate “thread” data structure in the kernel.

The practical implication: the kernel’s view of “processes” and “threads” is the same set of objects. What userspace calls a “thread” is a kernel task with shared resources.

Resource accounting with cgroups v2

The kernel’s process accounting is split between two layers:

Per-task (per-thread): CPU time, scheduling state, signals. Visible in /proc/<pid>/stat and top -H.
Per-cgroup: memory, I/O, PIDs, network. Visible in /sys/fs/cgroup/<controller>/<path>/.

For a multi-threaded process, the memory and I/O accounting is shared across all threads (one cgroup). CPU time is per-thread.

Read-only / Safecgroup memory— The cgroup for sshd.service shows total memory used by every thread in every sshd process. The number is in bytes; 4194304 = 4 MiB.

$ cat /sys/fs/cgroup/system.slice/sshd.service/memory.current

Illustrative output

Read-only / Safesystemd-cgtop— systemd-cgtop is top for cgroups. Each row is a cgroup (typically one per service). CPU% is the aggregate across all threads of all processes in the cgroup. Memory is the total RSS.

$ systemd-cgtop

Control Group                          Tasks   %CPU   Memory
/                                          234    12.5   1.2G
/system.slice                             168    10.0   900M
/system.slice/sshd.service                 4      0.1   16M
/user.slice/alice.service                  12     1.2   200M
...

Illustrative output

Inspecting thread-level CPU usage

Read-only / Safetop -H— top -H shows per-thread CPU usage. The PID column here is the TID (thread ID), not the process ID. The master thread (12345) is at 95% CPU — typical when nginx is processing a slow request on the main thread.

$ top -H -p 1234

top - 12:00:00 up 30 days,  3:45,  1 user,  load average: 4.20, 3.10, 2.50
Threads:   4 total,   1 running,   3 sleeping
PID   USER   PR  NI   VIRT   RES   SHR  S  %CPU  %MEM     TIME+  COMMAND
12345  root   20   0  1234m  456m  123m  R  95.0   1.1   0:42.15  nginx: master
12346  www-d  20   0  1234m  456m  123m  S   2.0   1.1   0:01.20  nginx: worker
12347  www-d  20   0  1234m  456m  123m  S   1.5   1.1   0:01.05  nginx: worker
12348  www-d  20   0  123m  45m  12m  S   0.5   0.1   0:00.30  nginx: worker

Illustrative output

Setting thread affinity

Configuration changetaskset— taskset -cp PID shows the current CPU affinity. taskset -cp MASK PID sets it. Use taskset to pin a single-threaded process to a specific CPU, or to keep two cache-thrashing threads on the same CPU.

$ taskset -cp 1234; taskset -cp 0-3 1234

pid 1234's current affinity mask: 0-3
pid 1234's current affinity mask: 0-3

Illustrative output

Production use of cgroups

cgroups are how systemd enforces resource limits:

[Service]
CPUQuota=200%
MemoryMax=2G
IOWeight=100

These directives write to the cgroup’s cpu.max, memory.max, and io.weight files. systemd translates the high-level directives to the underlying kernel interface.

Threads from ps

Threads vs processes

Resource accounting with cgroups v2

Inspecting thread-level CPU usage

Setting thread affinity

Production use of cgroups

Knowledge check