Redirection, pipes, and three-channel I/O

Redirection and pipes are what turn a collection of small commands into a production-grade text-processing toolkit. Most senior sysadmins can read a pipeline the way other engineers read code.

File descriptors

Every process starts with three open file descriptors inherited from its parent:

You can redirect any of them with the shell’s redirection operators.

Redirecting to files

$ echo hello > out.txt; cat out.txt

hello

$ echo again >> out.txt; cat out.txt

hello
again

$ ls /nonexistent 2> err.txt; cat err.txt

ls: cannot access '/nonexistent': No such file or directory

$ ls /etc/passwd /nonexistent > out.txt 2> err.txt; cat out.txt; echo ---; cat err.txt

/etc/passwd
---
ls: cannot access /nonexistent: No such file or directory

Pipelines

A pipeline chains commands by feeding each command’s stdout into the next command’s stdin.

$ ps -eo pid,comm | head -10

    PID COMM
1 systemd
124 systemd-journald
135 systemd-udevd
...

Pipelines have one important subtlety for production scripts: the exit status of a pipeline is the exit status of the last command, not the first. If cmd1 | cmd2 succeeds at cmd1 but cmd2 fails, the pipeline as a whole reports cmd2’s failure — which is what you want.

But if any earlier command in the pipeline fails, the failure is silently lost unless you enable pipefail (covered in the scripting lesson):

$ false | true; echo "pipeline exit: $?"; set -o pipefail; false | true; echo "with pipefail: $?"

pipeline exit: 0
with pipefail: 1

/dev/null and the black hole

/dev/null is a device file that accepts any input and produces no output. Two common uses:

$ command > /dev/null 2>&1

$ command < /dev/null

Here-documents and here-strings

A here-document lets you embed a block of text as stdin to a command:

$ cat <<'EOF' > config.txt
setting_one=alpha
setting_two=beta with spaces
EOF
cat config.txt

setting_one=alpha
setting_two=beta with spaces

A here-string is the one-line version:

$ bc <<< "2 + 2"

4

tee — the fork in the pipeline

tee reads stdin and writes it to both stdout and one or more files. This is the canonical way to capture intermediate pipeline state without breaking the chain:

$ journalctl -u ssh --since "1 hour ago" | tee /tmp/ssh.log | grep -i 'failed' | head -20

...

WhyThisMatters

Knowledge check