Owl0vv1.io/documentation

A brief Introduction to Linux and BSD based Operating Systems

by Alexander Puls, last updated
license: Creative Commons BY-NC-SA

man *nix(7)

Basics


It all starts with the two least dangerous commands:


note: Both commands take either absolute or relative paths as an argument. An absolute path always starts with /, i. e. the root of the filesystem. A relative path starts inside the (current) working directory.
cmd ls lists directory content

cmd cd changes current (working) directory
So-called shell builtin, i. e. provided by the shell itself.
Other than the obvious parameters:
  • cd all alone and cd ~ switch to the home directory of the invoking user as specified by its ${HOME} variable (set by cmd login according to the respective entry in /etc/passwd)
  • cd - returns to the last working directory (i. e. where one came from)
  • cd .. changes the working directory to the directory 'above', since .. is a link to the parent directory, . a link to the current dir (cd . 'changes' from the current dir to the current one –i. e. you stay where you are)
Online resources:

from here on: every character counts, a shell and its environment is not a just a programming language and the most exhaustive IDE one can imagine, it's a complete interface to the underlying operating system and every reachable remote one
– think before you RETURN!

System Architecture №101

Determine and Configure Hardware Settings №101.1

cmd /bin/cat concatenates files and prints them on stdout (standard output)
Easiest call: cat [FILE[S]] prints the (vertically concatenated) content of given FILE(s) to stdout (i. e. (usually) the terminal).
One may want to redirect the output via > (which overwrites(!) the content of a given TARGET FILE, if it is not empty (e. g. new)) or >> (attaches output to given TARGET FILE).
  • synopsis: cat [FILE[S]] [>|>> [TARGETFILE]
    • notice: one may redirect an empty string with > to a not yet existing file in order to create it with no content
    (0 byte) 
    $ >NEWFILE
$ ls -l NEWFILE
-rw-r----- 1 user group 0 Nov 15 13:39 NEWFILE
    (same goes for command touch: touching a non-existing file leads to creation)
Online resources:

dir /sys/ provides information about kernel objects  Linux specific
Via the sysfs filesystem.
Similiar functionality to the sysctl mechanism in BSD operating systems.
Dynamically generated.
Represents the driver modules of the running kernel.
Online resource: man sysfs(5) at man7.org

dir /proc/ provides information about processes via the procfs filesystem
Subdirectory for every running PID (process ID).
Subsequent dirs, e. g.: cpuinfo/, dma/, interrupts/, ioports/, meminfo/, modules/.
Boot parameter for the kernel in /proc/cmdline.
Online resources:

dir /dev/ is the location for interfaces to the various device drivers
Every instance is represented as a single file, e. g.: /dev/tty[0-9]* being the terminals (teletype), /dev/random being an entropy generator, /dev/[s]?da1 the first partition on the first SCSI disk et c.

daemon dbus used for IPC (inter process communication)
Focus on desktop applications (desktop bus).
Gets its information from the daemon hald, representing HAL (hardware abstraction layer), e. g. when hotplug devices get attached.

cmd lsmod shows the status of modules in the Linux kernel  Linux specific
.. and nothing more (just prints out a formated /proc/modules).
Online resource: man lsmod(8) at man7.org

cmd modstat displays the status of modules in the kernel  NetBSD specific
.. with lots of information (if needed).
Online resources:

cmd kldstat displays the status of the dynamic kernel linker  FreeBSD specific
AKA modules.
Online resource: man kldstat(8) at FreeBSD.org

cmd modprobe adds and removes modules from the Linux kernel  Linux specific
Expects an up-to-date modules.dep.bin (to be generated by depmod).
modprobe MODULE[S] loads the given module(s), no path is needed.
Further options:
  • -r removes the given module
  • --show-depends shows dependencies of given modules (*DUH*)
Online resource: man modprobe(8) at man7.org

cmd /usr/bin/lspci lists all PCI devices  Linux specific
Useful options:
  • -v, -vv, -vvv sets verbosity level
  • -x, -xxx, -xxxx shows hexadecimal dump of the configuration space
  • -k shows which kernel drivers and modules are used for the device
  • -m prints device data in a machine readable format
Online resource: man lspci(8) at man7.org

cmd lsusb lists USB devices  Linux specific
Online resource: man lsusb(8) at man7.org

Boot the System №101.2

cmd dmesg prints and controls the kernel ring buffer

cmd journalctl queries log files provided by systemd

BIOS meaning the basic I/O system (firmware interface)
In Ancient Greek βίος means 'Live'.

UEFI is another interface between the OS and the firmware of a platform
In Ancient Greek υεφι means nothing.
Developed by Intel® and intended to be a successor of the 'old' BIOS.
Online resource: Hundreds of MBs of official documentation at Intel.com. 😂

svc SysVinit originally being the init system of System V
Online resource: wiki article at wiki.gentoo.org

bootloader is a program being started by the firmware to load the kernel into memory

kernel is the core of an OS, resident in memory at runtime

file initramfs is a compressed archive containing a filesystem
Can be mounted to / at boot time.

daemon init is the first process of the system

svc systemd originally was an init daemon Linux specific
Promoted as universal system control service of all Linux based operating systems.
Online resource: A few hundred man pages at freedesktop.org. 😂

Runlevels & Shutdown №101.3

file /etc/inittab is the main configuration file for SysV-style init programs

cmd /sbin/shutdown halts, powers-off or reboots the machine
Option -k generates only a message (like wall) and suppresses any new logins 5 minutes (default) before the given time.
Online resources:

daemon init is the first process of the system (PID 1)
Controls all other daemons.
Online resource: man init(8) at man.NetBSD.org

dir /etc/init.d/ (or /etc/rc.d/) is the directory for storing init scripts of traditional SysV-style init systems
Online resource: man rc(8) at man.NetBSD.org

cmd telinit tells init to change SysV runlevel
Takes the runlevel no. (as configured in /etc/inittab) as an argument.
Online resource: man init(8) at man.NetBSD.org

daemon systemd

cmd systemctl

dir /etc/systemd/

dir /usr/lib/systemd/

cmd wall writes a message to users
Option -g limits message printing to specified group.
Online resources:

Linux Installation and Package Management №102

Design Hard Disk Layout №102.1

FHS (Filesystem Hirarchy Standard)
Online resources:
attention: The systemd project reinterpretes the filesystem hierachy. Kinda up-to-date documentation of 'their' specification can be found on platforms using this service under man file-hierachy(7) (copy at man7.org).

Important directories on Unix-like platforms:

  • dir / (root)
  • dir /boot/ holding static data of the bootloader
  • dir /boot/efi/ being the favorite mountpoint of the EFI System Partition (ESP) on platforms using UEFI and might be shared with other systems, e. g. MSW
  • dir /etc/ holding the main configuration files of the local machine
  • dir /var/ holding variable data (of significance for platforms providing servers)

swap

mountpoint indicates a directory at which a block device is mounted

partition

Install a Boot Manager №102.2

dir /boot/grub/ et c. & associated commands as an example

  • grub-install (re)installs the bootloader to the MBR (master boot record)
  • grub-mkconfig (re)writes the configuration in file /boot/grub/grub.cfg
    • templates from /etc/default/grub and below /etc/grub.d/
  • legacy files: menu.lst and grub.conf
  • grub shell
    • e. g. to boot manually: 
      grub> set root='(hd0,0)' # 1st partition on 1st harddisk
      sets the place to find the kernel (and its initrd if necessary) 
      grub> kernel /boot/vmlinuz-[VERSION]root=/dev/sd[a-z][1-9]*
      specifies two parameters: the kernel file and the partition on which the root filesystem resides 
      grub> initrd /boot/initrd.img-[VERSION]         
grub> boot
  • /boot/grub/device.map holds the assignments to the GRUB devices

Manage Shared Libraries №102.3

cmd ldd prints shared object dependencies (AKA libraries)
Online resources:

cmd ldconfig configures/rebuilds the library cache
Online resources:

dir /etc/ld.so.conf stores the information of the run-time link-editor
Online resources:

var LD_LIBRARY_PATH stores directories in which to search for `ELF` libraries at execution time
Online resources:

Use Debian Package Management №102.4 Debian/GNU/Linux specific

Important files and commands:

  • file /etc/apt/sources.list
  • cmd dpkg
  • cmd dpkg-reconfigure
  • cmd apt-get
  • cmd apt-cache


Use RPM and YUM Package Management №102.5 Red Hat Linux specific

Important commands and files:

  • cmd rpm
  • cmd rpm2cpio
  • file /etc/yum.conf
  • dir /etc/yum.repos.d/
  • cmd yum
    • provides shows which package holds a given program
  • cmd zypper


Linux as a Virtualization Guest №102.6

Buzzwords:

  • Virtual machine
  • Linux container
  • Application container
  • Guest drivers
SSH host keys

DBus machine ID

GNU and Unix Commands №103

Work on the Command Line №103.1

cmd echo writes arguments to stdout
Online resources:

cmd env runs a program in a modified environment
shell builtin.
Online resources:

cmd export exports shell variables to the following processes
shell builtin.
Option -f refers to functions, -n removes attribute
Online resource: man export(1p) at man7.org

cmd pwd prints working directory
shell builtin.
Online resources:

cmd set lists shell variables
shell builtin.

cmd unset deletes shell variables
shell builtin.
Online resource: man unset(1p) at man7.org

cmd type indicates how a given command would be interpreted

cmd which locates a command in the users $PATH

cmd man interacts with the system reference manuals (interface)
Online resources:
Offline resource: man man
  • defaulting to section 1 (more precise would be man 1 man or, if your system has another $LANG then an English one and you want to hit the original (in section 1), type man -L en 1 man)
cmd whatis is closely related:
  • displays one-line manual page descriptions
  • lists all available man pages regarding the given term(s) including its/their man page section(s), which might be several
  • e. g.: 
    $ whatis whatis ed man
whatis (1)           - display one-line manual page descriptions
ed (1)               - line-oriented text editor
man (1)              - an interface to the on-line reference manuals
man (7)              - macros to format man pages
  • online resources:

cmd uname prints a few system informations (unix name)
Appeared first in PWB UNIX® 1.0 (Programmers Workbench) from AT&T Bell Laboratories.
The most prominent option -r gives the kernel release no. as a string.
Option -a gives all information it can gather.
Online resources:

cmd history manages a command history on most (i. e. more feature rich) shells
Mostly delivered as a shell built-in.

Process Text Streams Using Filters №103.2

cmd bzcat decompresses files to stdout

cmd cat concatenates files

cmd cut removes sections from each line of files
Online resources:

cmd head outputs the first part of files
Online resources:

cmd less is the opposite of more (pager)
Basically less is more(1p), but kinda more.
Online resources:

cmd md5sum computes and checks MD5 message digest  Linux specific
Online resource: man md5sum(1) at man7.org

cmd cksum displays file checksums and block counts  BSD specific
To be used as md5, sum -a MD5 or cksum -a MD5 to display MD5 checksums.
Online resources:

cmd nl numbers lines of files
Online resources:

cmd od dumps files in octal and other formats
Online resources:

cmd paste merges lines of files
Online resources:

cmd sed is the stream editor for filtering and transforming text

cmd sha256sum computes and checks SHA256 message digest  Linux specific
Online resource: man sha256sum(1) at man7.org

cmd cksum displays file checksums and block counts  BSD specific
To be used as sha256, sum -a SHA256 or cksum -a SHA256 to display SHA256 checksums.
Online resources:

cmd sha512sum computes and checks SHA512 message digest  Linux specific
Online resource: man sha512sum(1) at man7.org

cmd cksum displays file checksums and block counts  BSD specific
To be used as sha512, sum -a sha512 or cksum -a SHA512 to display SHA512 checksums.
Online resources:

cmd sort sorts lines of text files
Online resources:

cmd split splits a file into pieces
Online resources:

cmd tail outputs the last part of files
Online resources:

cmd tr translates or deletes characters
Online resources:

cmd uniq reports or omits repeated lines
Online resources:

cmd wc prints newline, word and byte counts for each file
Online resources:

cmd xzcat compresses or decompresses .xz and .lzma files

cmd zcat compresses or expands files
Belongs to gzip.

Perform Basic File Management №103.3

cmd cp copies files (and directories)
Online resources:

cmd find searches for files in a directory hirarchy

cmd mkdir makes directories
Online resources:

cmd mv moves files
And thus is used to rename files too.
Works on directories either, since they are treated like files (of type 'directory').
Online resources:

cmd ls lists directories

cmd rm removes directory entries
Works recursively with options -r or -R.
Online resources:

cmd rmdir removes directories
.. if they are empty.
Online resources:

cmd touch changes file access and modification times
Online resources:

cmd tar is the tape archiver
Important options:
  • c compresses,
  • x extracts,
  • z uses gzip algorithm,
  • j uses bzip2 algorithm,
  • J uses xz algorithm,
  • t shows table of content,
  • w brings an interactive mode up,
  • f specifies an archive name
E. g.: tar xzf ARCHIVE extracts a gzip compressed file named ARCHIVE to the current directory (if the output dir is not changed by option -C)
Online resources:

cmd cpio copies files in and out of archives
Ancient (appeared first in 1977 in PWB UNIX®).
Has certain limitations.
Online resources:

cmd dd converts and copies files
Works on block level, thus can dump disks too.
Common syntax: dd if=INPUT-FILE of=OUTPUT-FILE [ .. ]
More options with the same syntax (i. e. bs=BLOCK-SIZE).
Online resources:

cmd file determines file type
Online resources:

cmd gzip compresses files
Uses the Lempel-Ziv coding.
cmd gunzip decompresses files (basically is gzip -d or --decompress).
cmd zcat or gzcat decompresses files to stdout (gzip -c (or --stdout) and -d(--decompress) are used).
Online resource: man gzip(1) at man.NetBSD.org

cmd bzip2 compresses files
Uses the Burrows-Wheeler block sorting text compression algorithm.
cmd bunzip decompresses files (basically is bzip2 -d or --decompress).
cmd bzcat decompresses files to stdout (bzip2 -c or --stdout).
Online resource: man bzip2(1) at man.NetBSD.org

cmd xz is a general-purpose data compression tool
Native file format is .xz.
unxz is equivalent to xz -d.
xzcat is equivalent to xz -c -d.
lzma is equivalent to xz --format=lzma.
unlzma is equivalent to xz --format=lzma -d.
lzcat is equivalent to xz --format=lzma -cd.
Online resource: man xz(1) at man.NetBSD.org

file globbing

Create, Monitor and Kill Processes №103.5

&

cmd bg

cmd fg

cmd jobs

cmd kill terminates or signals processes
Online resources:

cmd nohup invokes a command imune to hangups
Online resources:

cmd ps reports a snapshot of the current processes

cmd top displays processes

cmd free displays amount of free and 'used' memory Linux specific
Online resource: man free(1) at man7.org

cmd vmstat reports virtual memory statistics BSD specific
Online resource: man vmstat(1) at man.NetBSD.org

cmd pstat displays system data structures  BSD specific
Online resource: man pstat(8) at man.NetBSD.org

cmd systat displays system statistics in a full screen view  BSD specific
Online resource: man systat(1) at man.NetBSD.org

cmd uptime shows how long the system has been running since last boot
Option -p pretties output (on Linux).
Online resources:

cmd pgrep finds processes by name
Online resources:

cmd pkill signals processes by name
Online resources:

cmd killall kills processes by name  Linux & FreeBSD specific
Or kills all(!) processes that user is able to kill (System V).
Online resources:

cmd watch executes a program periodically  Linux common
.. and shows the output fullscreen.
Part of the procps-ng project (installable on other Unix-like systems too).
Online resource: man watch(1) at man7.org

cmd screen is a screen manager with terminal emulation  Linux specific
Online resource: man screen(1) at man7.org

cmd tmux is the terminal multiplexer
Online resources:

Modify Process Execution Priorities №103.6

cmd nice runs a program with an altered scheduling priority
Online resources:

cmd ps reports a snapshot of the current processes

cmd renice alters priority of running processes
Online resources:

cmd top displays processes

Search Text Files Using Regular Expressions №103.7

cmd grep prints lines matching a pattern
cmd egrep is grep -E.
cmd fgrep is grep -F.
Online resources:

cmd sed is the stream editor for filtering and transforming text
Online resources:

lang regex i. e. Regular Expressions
Online resource: man regex(7) at man7.org

Basic File Editing №103.8

cmd ed is a line-oriented text editor
Was one of the first parts of the Unix operating system developed by Ken Thompson in August 1969.
Remains part of the POSIX and Open Group standards.
Still useful for automation processes from stdin.
Online resources:

cmd vi is a screen-oriented (visual) display editor
Nowadays often an alias of (or link to) its 'successor' vim on many 'modern' desktop operating systems, but is part of POSIX.1-2008 and can be found on literally every system.
Online resources:

cmd vim is vi improved
VIMawesome, a plugin aggregator
dark corners of vim forum post at stackoverflow.com

cmd emacs is an operating system by GNU Corp.
.. it lacks a good editor, though. (Thomer M. Gil), but there are numerous efforts undertaken to integrate mature and sane solutions like vim.
Repetitve Strain Injury entry in the EmacsWiki, stating that it is not clear what causes it, recommending foot switches for control keys and vi like key bindings.
man emacs(1) - ah, no.. - Emacs has its own manual system, called info(?) and a program called woman within Emacs, short for 'without manual' to not read manuals.
Highlights: It has orgmode, the most brilliant calendaring and GTD tool in this spiral arm of our home galaxy.

var EDITOR to not store emacs in
Environment variable to assign an editor of choice.

The assigned value to EDITOR (i. e. ones choice of an 'editor') should be executable inside a shell and should not have too many dependencies.
Important to some tools (like sudo and its derived commands) as they rely on the default editor. E. g.:

$ EDITOR="vi"
$ echo $EDITOR
vi

Optionally include a qualified path (e. g. EDITOR=${HOME}/.local/bin/vim) if the chosen editor cannot be found inside $PATH (i. e. installed somewhere else).

Optionally use export (e. g. export EDITOR=vi) to let the variable get inherited by every following command (e. g. invoked subshells).
May be used with env (*e. g.* env EDITOR=vim CMD) to let the variable value (possibly among others) be valid for a certain command CMD.

cmd edit invokes the standard editor on most distributions, but the corresponding systemwide $EDITOR can only be set by root

var VISUAL can also be set for 'visual' (i. e. non-line-oriented) editors, since $EDITOR originally was used to hold a line-oriented editor (e. g. ed)

Devices, Filesystems, Filesystem Hierarchy Standard №104

Create Partitions and Filesystems №104.1

Important filesystems on Linux:

  • ext2/ext3/ext4
  • XFS
  • VFAT
  • exFAT
  • BtrFS

cmd fdisk manipulates the disk partition table
  • cmd gdisk is used to handle GPTs (GUID partition tables)
Online resources:

cmd parted manipulates partitions too  Linux common
Part of the GNU project.
Online resource: man parted(8) at man7.org

cmd mkfs builds a Linux file system Linux specific
Is deprecated in favour of:
  • cmd mkfs.exfat, executed as mkexfatfs
  • cmd mkfs.ext2, executed as mke2fs -t ext2
  • cmd mkfs.ext3, executed as mke2fs -t ext3
  • cmd mkfs.ext4, executed as mke2fs -t ext4
  • cmd mkfs.vfat, executed as mkfs.fat
Online resources: man mkfs.fat(8) and man mkfs(8) at man7.org

cmd newfs constructs a new file system  BSD specific
Online resource: man newfs(8) at man.NetBSD.org

cmd mkswap sets up a Linux swap area  Linux specific
Online resource: man mkswap(8) at man7.org

Maintain the Integrity of Filesystems №104.2

cmd du estimates file (disk) space usage
Online resources:

cmd df displays free disk space
Online resources:

cmd fsck checks file system consistancy
.. and optionally repairs interactively.
Online resources:

cmd e2fsck checks an ext2/ext3/ext4 file system Linux (fs) specific
Online resource: man e2fsck(8) at man7.org

cmd mke2fs creates an ext2/ext3/ext4 file system Linux (fs) specific
Online resource: man mke2fs(8) at man7.org

cmd tune2fs adjusts tunable file system parameters Linux (fs) specific
.. on ext2/ext3/ext4 file systems.
Online resource: man tune2fs(8) at man7.org

Important commands regarding the XFS file system:

  • cmd xfs_repair
  • cmd xfs_fsr
  • cmd xfs_db

Control Mounting and Unmounting of Filesystems №104.3 & №712.3

file /etc/fstab/ is the file system table
Holds static information about the file systems and their mount points.
Online resources:

Example for an fstab on a small private system with UEFI boot, with a seperate /opt partition, a small ram drive and a dynamic mount (partitions identified by LABEL): 

# file:  fstab(5)
# brief: static file system information
# -------------------------------------
UUID=C1D7-A93J    /boot/efi vfat  umask=0077                       0 2
LABEL=SYSTEM      /         ext4  errors=remount-ro,relatime,rw    0 1
LABEL=OPT         /opt      ext4  relatime,rw                      0 2
tmpfs             /tmp      tmpfs nodev,nosuid,relatime,rw,size=2g 0 0
LABEL=PERSISTENT  /mnt/pers ext4  relatime,x-systemd.automount     0 2
# EOF /etc/fstab ----------------------

dir /media/ intended to contain subdirectories as mount points for removable media

cmd mount mounts file systems
Online resources:

cmd umount unmounts file systems
Online resources:

cmd blkid locates/prints block device attributes Linux specific
Interface to libblkid(3).
Online resource: man blkid(8) at man7.org

cmd lsblk lists block devices Linux specific
Reads from dir /sys/dev/block/.
Online resource: man lsblk(8) at man7.org

Manage File Permissions and Ownership №104.5 & №712.4

cmd chmod changes file modes
AKA access permissions in octal notation:
Online resources:

Determination of file (or directory) mode

Every file (or directory) has three octets (in raw Unix) defining permissions:

  • the 1st represents the owner of the file
  • the 2nd its group
  • the 3rd all others users in the universe

Every of the three octets can be split into three binary digit positions:

  • every binary position is either 1 or 0 (TRUE or FALSE, YES or NO)
  • a binary 100 translates to a 4 (octal)
  • a binary 010 translates to a 2
  • a binary 001 translates to a 1

In Unix:

  • the 1st position represents the right to read (or list dir content)
  • the 2nd position represents the right to write (or change dir content)
  • the 3rd position represents the right to execute (or enter dir and see file attributes)

Adding them gives the permission set.

E.g. for a file with mode 754 (owner is allowed to do everything, group members are not allowed to write, others cannot even execute):

user group others
1st position (read) 1 0 0 = 4oct 1 0 0 = 4oct 1 0 0 = 4oct
2nd position (write)   1 0 = 2oct   0 0 = 0oct   0 0 = 0oct
3rd position (exec)     1 = 1oct     1 = 1oct     0 = 0oct
sum (binary) 1 1 1 = 7oct 1 0 1 = 5oct 1 0 0 = 4oct
logical (True/False) T T T T F T T F F
unixal r w x r - x r - -

A somewhat reasonable default for self-written scripts.
For unpriviliged users even numbers like 644 (rw-r--r--) or 640 are common, with just very few files having the execution bit set.

cmd umask (gets or) sets the file mode creation mask
Online resources:

cmd chown changes file owner and group
Online resources:

cmd chgrp changes group ownership
Online resources:

Create and Change Hard and Symbolic Links №104.6

cmd ln links files
Online resources:

cmd ls lists directories
With lots of options to format the output.
  • synopsis on BSD: ls [-1AaBbCcdFfghikLlMmnOoPpqRrSsTtuWwXx] [file ...]
On Linux mostly implemented as part of the GNU core utilities.
Online resources:

Find System Files and Place Files in the Correct Location №104.7

cmd find (an euphemism) searches for files in a directory hirarchy
Online resources:

cmd locate lists files in databases that match a pattern
Online resources:

cmd updatedb updates a file name database
On BSDs: cmd /usr/libexec/locate.updatedb
Gets configured in file /etc/updatedb.conf.
Online resources:

cmd whereis locates programs
.., their corresponding sources and man-pages.
Online resources:

cmd which locates a command in the users $PATH
shell builtin.
Option -a continues search after first find.
Online resource: man which(1) at man.NetBSD.org

cmd type indicates how a given command would be interpreted
shell builtin.
Online resource: man type(1p) at man7.org

To be continued..