Engineering Publication · Syed Omar Ibn Feroj← Back to portfolio

Tux

Engineering Notes · Open Knowledge Repository

Linux — Engineering
Notes for Operators

A working reference for Linux as you actually run it — the shell and its expansion rules, permissions, processes and signals, systemd, the grep/sed/awk toolkit, and the troubleshooting habits that resolve incidents fast.

Chapters

POSIX

Baseline

Living

Document

Free

License

Ch. 01

How the Shell Runs

What actually happens between hitting Enter and the program running.

1.1 — The processing order

The shell expands a line in a fixed order: brace → tilde → parameter/variable → command substitution → arithmetic → word splitting → filename (glob). Knowing the order explains almost every "why did it do that" surprise.

Shell

$ echo ~/x{a,b}      # /home/u/xa /home/u/xb
$ echo "$HOME"        # quoted: one word, no splitting
$ echo $(date +%s)

■

Word splitting happens after variable expansion. rm $f where f="a b" deletes two files. Always quote: rm "$f". Unquoted variables are the single biggest source of shell bugs.

Ch. 02

Filesystem Layout

The FHS — where things live and why.

2.1 — The directories that matter

Path	Holds
`/etc`	system config (text, version-controllable)
`/var/log`	logs
`/var/lib`	app state/data
`/usr/bin`, `/usr/local/bin`	binaries (distro vs locally installed)
`/proc`, `/sys`	kernel/process virtual filesystems
`/tmp`	scratch, world-writable, cleared on boot
`~/.config`	per-user app config (XDG)

▹

/proc/<pid>/ is a live window into any process — environ, cmdline, fd/, limits. Indispensable when a process misbehaves and you can't attach a debugger.

Ch. 03

Navigation

Moving around fast, and the path traps.

3.1 — Paths, links, and cd -

Shell

$ cd -            # back to previous dir
$ pwd -P         # resolve symlinks
$ readlink -f x   # canonical absolute path

▲

A relative path is resolved against the current directory at execution time — a script that cds and then uses relative paths is fragile. Resolve to absolute early: cd "$(dirname "$0")" or use $(readlink -f …).

Ch. 04

Files & Directories

cp/mv/rm/ln and the irreversible ones.

4.1 — The destructive trio

Shell

$ cp -a src dst        # archive: recurse + preserve
$ rm -rf ./build       # no undo, no trash
$ ln -s /opt/app cur   # symlink (atomic deploy trick)

■

There is no recycle bin. rm -rf "$DIR/" with an unset $DIR becomes rm -rf /. In scripts: set -u, and guard with [ -n "$DIR" ] || exit 1 before any recursive delete.

Ch. 05

Globs & Expansion

Filename patterns are not regex.

5.1 — Glob vs regex

Shell

$ ls *.log           # glob: * = any chars
$ ls file?.txt       # ? = exactly one char
$ shopt -s globstar; ls **/*.js  # recursive

▲

A glob that matches nothing is passed through literally by default (bash), so a loop can run once with the pattern itself as the value. Set shopt -s nullglob in scripts so an empty match yields zero iterations.

Ch. 06

Permissions

rwx, octal, and the bits people forget.

6.1 — Read the mode, set it precisely

Shell

$ ls -l          # -rwxr-xr--  user/group/other
$ chmod 640 f     # rw- r-- ---
$ chmod u+x,go-w s
$ chmod 600 ~/.ssh/id_ed25519   # SSH refuses looser

▹

On a directory, x means "may enter / traverse", not "execute". A dir with r but no x lets you list names but not stat or cd into it — a common "permission denied despite readable" cause.

Ch. 07

Users & Groups

UID/GID, supplementary groups, and the login-again gotcha.

7.1 — Identity and membership

Shell

$ id                    # uid, gid, groups
$ sudo usermod -aG docker $USER
$ newgrp docker         # or log out/in

▲

Adding yourself to a group does not affect your current shell — group membership is set at login. People burn time wondering why docker still says permission denied; the fix is a new session, not a retry.

Ch. 08

sudo & Privilege

Least privilege, and why 'sudo su -' is a smell.

8.1 — Targeted, audited elevation

Shell

$ sudo systemctl restart nginx    # one command, logged
$ sudo -l                          # what may I run?

■

Each sudo <cmd> is logged to the audit trail; a root shell (sudo -i) runs everything unattributed. Prefer per-command sudo, and scope rights in /etc/sudoers.d/ rather than granting blanket root.

Ch. 09

Pipes & Redirection

stdin/stdout/stderr, and the order that bites everyone.

9.1 — Redirect order matters

Shell

$ cmd > out.log 2>&1     # stdout+stderr → file (correct)
$ cmd 2>&1 > out.log     # stderr → old stdout (terminal!)
$ cmd &> out.log           # both, bash shorthand
$ a | b | c                # exit = c's; see pipefail

■

2>&1 means "make fd 2 point where fd 1 points now". Put it after the file redirect, not before. And a pipeline's exit status is the last command's unless you set pipefail.

Ch. 10

grep

Search, but know which regex flavour you're in.

10.1 — The flags you'll reuse

Shell

$ grep -rn "TODO" src/        # recursive + line numbers
$ grep -i -w error log     # case-insensitive whole word
$ grep -E "warn|fail" log     # ERE alternation
$ grep -v -c "^#" conf       # count non-comment lines

▹

Basic regex (BRE, default) needs \(, \+, \? escaped; -E (ERE) makes them special. When a pattern "doesn't work", you're usually in the wrong flavour — use -E and stop escaping.

Ch. 11

find & xargs

Locate by predicate, act in bulk safely.

11.1 — NUL-delimited is the safe way

Shell

$ find . -name "*.tmp" -mtime +7 -delete
$ find . -type f -print0 | xargs -0 grep PASS
$ find . -name "*.log" -exec gzip {} +

■

Filenames can contain spaces and newlines. find … | xargs breaks on them and can act on the wrong file. Always pair -print0 with xargs -0, or use -exec … +.

Ch. 12

sed

Stream editing — substitution and in-place edits.

12.1 — Substitute, range, in-place

Shell

$ sed 's/old/new/g' f          # to stdout
$ sed -i.bak 's|/a|/b|g' f     # in place + backup
$ sed -n '10,20p' f            # print a line range

▲

sed -i without a backup suffix and a bad expression destroys the file with no undo. Test the expression without -i first; use -i.bak until you trust it. GNU vs BSD sed -i syntax differs — pin your environment.

Ch. 13

awk

Column-oriented processing in one line.

13.1 — Fields, filters, aggregation

Shell

$ awk '{print $1, $NF}' f           # first & last field
$ awk -F: '$3 >= 1000' /etc/passwd  # real users
$ awk '{s+=$2} END{print s}' f       # sum a column

▹

Reach for awk the moment a task is "per-line, by column, with a running total". A 3-line awk beats a 20-line loop with cut + arithmetic, and it's faster.

Ch. 14

Processes

ps, top, and reading the state column.

14.1 — Inspect what's running

Shell

$ ps aux --sort=-%mem | head
$ pgrep -fl nginx
$ top -o %CPU      # or htop

ℹ

A process in state D (uninterruptible sleep) is stuck in a kernel I/O wait — kill -9 won't touch it. That usually means a slow/hung disk or network mount, not a runaway process.

Ch. 15

Signals & Jobs

TERM vs KILL, traps, and backgrounding.

15.1 — Ask nicely before forcing

Shell

$ kill -TERM <pid>   # 15: clean shutdown (default)
$ kill -KILL <pid>   # 9: last resort, no cleanup
$ cmd & ; disown; jobs
$ nohup long.sh &> out &

■

kill -9 gives the process no chance to flush buffers, close files, or release locks — it can leave corrupt state. Always try -TERM first; reserve -KILL for the genuinely wedged.

Ch. 16

systemd

Units, the journal, and enable vs start.

16.1 — Manage a service

Shell

$ systemctl status nginx
$ systemctl enable --now nginx   # start + boot-persist
$ systemctl daemon-reload          # after editing a unit
$ journalctl -u nginx -f

▲

start runs it now; enable makes it survive reboot — they're independent. A service that "works until the box reboots" was started but never enabled.

Ch. 17

Packages

apt/dnf basics and the update vs upgrade distinction.

17.1 — The common operations

Shell

$ sudo apt update          # refresh index ONLY
$ sudo apt upgrade         # actually install updates
$ apt policy <pkg>          # installed vs available
$ dpkg -S /usr/bin/curl     # which package owns a file

ℹ

apt update only refreshes the package list — it installs nothing. apt upgrade applies them. Running only update and assuming you're patched is a common security gap.

Ch. 18

Networking

ip, ss, and curl as the universal probe.

18.1 — Modern tooling (not ifconfig/netstat)

Shell

$ ip a; ip route
$ ss -tlnp             # listening TCP + owning process
$ curl -fsS -w '%{http_code} %{time_total}\n' -o /dev/null URL
$ dig +short example.com

▹

ss -tlnp ("what's listening and who owns it") is the first command in 90% of "connection refused / port already in use" investigations. ifconfig/netstat are deprecated — use ip/ss.

Ch. 19

SSH

Keys, config, and tunnels.

19.1 — Keys + a config that saves typing

Shell

$ ssh-keygen -t ed25519 -C "laptop"
$ ssh-copy-id user@host
# ~/.ssh/config
Host prod
  HostName 10.0.0.4
  User deploy
  IdentityFile ~/.ssh/id_ed25519

■

SSH ignores a private key whose permissions are too open. chmod 600 ~/.ssh/id_* and 700 ~/.ssh — "Permissions 0644 are too open" means exactly this.

Ch. 20

Disks & Filesystems

df vs du, inodes, and the disk-full mystery.

20.1 — Find what's eating the disk

Shell

$ df -h; df -i           # space; inodes
$ du -sh * | sort -rh | head
$ lsof +L1                  # deleted-but-open files

■

"Disk full" but du shows space free? Either you're out of inodes (df -i) or a process holds a deleted-but-open file (lsof +L1) — the space frees only when that process is restarted, not when you rm the file.

Ch. 21

Logs & journald

Where the evidence is, and how to read it fast.

21.1 — Filtered, time-bounded, followed

Shell

$ journalctl -u app --since "1 hour ago" -p err
$ journalctl -k -b            # kernel, this boot
$ tail -F /var/log/nginx/error.log

▹

Use tail -F (capital), not -f — capital re-opens the file after log rotation, so you don't silently stop seeing new lines when logrotate runs.

Ch. 22

Cron & Timers

Scheduling, and why your cron job 'doesn't run'.

22.1 — crontab and the environment trap

Shell

# m h dom mon dow  command
*/5 * * * * /opt/app/sync.sh >> /var/log/sync.log 2>&1

■

cron runs with a minimal PATH and no shell profile. "Works in my terminal, not in cron" is almost always an unset variable or a bare command name. Use absolute paths, set PATH at the top of the script, and redirect output to a log so failures aren't silent.

Ch. 23

Bash Basics

Variables, quoting, command substitution.

23.1 — Quoting is the whole language

Bash

name="Omar"            # no spaces around =
echo "hi $name"       # expands
echo 'hi $name'       # literal
files=$(ls *.txt)       # command substitution
echo "${name:-guest}"   # default if unset

■

Unquoted $var is split on whitespace and glob-expanded. Quote every expansion unless you have a specific reason not to. "$@" (quoted) preserves arguments correctly; $* does not.

Ch. 24

Bash Control Flow

test, [[ ]], loops, case.

24.1 — Prefer [[ ]] over [ ]

Bash

if [[ -f "$f" && "$n" -gt 0 ]]; then ...; fi
for f in *.log; do gzip "$f"; done
while read -r line; do echo "$line"; done < f
case "$1" in start) ...;; stop) ...;; *) ...;; esac

▹

[[ ]] (bash) doesn't word-split or glob its operands and supports &&/||/=~. [ ] (POSIX test) does split — a frequent source of "unary operator expected". Use [[ ]] in bash scripts.

Ch. 25

Robust Scripts

The header every production script needs.

25.1 — Strict mode

Bash

#!/usr/bin/env bash
set -Eeuo pipefail
trap 'echo "failed at line $LINENO" >&2' ERR
# -e exit on error  -u unset = error
# -o pipefail: a failing stage fails the pipe

✓

set -euo pipefail + an ERR trap turns a script that "silently did half the job" into one that stops loudly at the first failure with a line number. This single header prevents a whole class of automation incidents.

Ch. 26

Performance

The four-number triage: CPU, memory, disk, network.

26.1 — Fast situational awareness

Shell

$ uptime                 # load avg vs core count
$ vmstat 1 5             # cpu/mem/io over time
$ iostat -xz 1           # per-disk utilisation
$ free -h                # is it really low on RAM?

ℹ

A load average above core count isn't automatically bad — it includes processes blocked on I/O (state D), not just CPU demand. Correlate with vmstat's r/b columns and iostat before concluding "CPU bound".

Ch. 27

Hardening

The baseline that stops the opportunistic attacker.

27.1 — The non-negotiables

SSH: key-only auth, PermitRootLogin no, non-default port optional
A host firewall (ufw/nftables) default-deny inbound
Unattended security updates enabled
No services bound to 0.0.0.0 that should be localhost-only (ss -tlnp to verify)
Run apps as a non-root service user with least privilege
Audit sudo -l and /etc/sudoers.d/ regularly

Ch. 28

Troubleshooting

A repeatable workflow beats guessing.

28.1 — The order to check

Is it actually down? systemctl status + curl the health endpoint
What changed? recent deploys, config, journalctl --since the incident
Resources? df -h, df -i, free -h, uptime (full disk/inodes is the silent killer)
Connectivity? ss -tlnp (listening?), curl from the box, DNS
The logs. Always the logs — filtered by unit and time, error priority first

▹

Form a hypothesis, change one thing, re-test. Changing several things at once means that even when it works, you don't know why — and the incident recurs.

REF

Linux Cheatsheet

The first commands in an incident.

Triage in 60 seconds

Shell

$ uptime                       # load
$ df -h && df -i             # disk + inodes
$ free -h                      # memory
$ ss -tlnp                     # listeners
$ systemctl --failed           # broken units
$ journalctl -p err -b | tail  # recent errors

Linux — EngineeringNotes for Operators

Linux — Engineering
Notes for Operators