TIL: Advanced Linux System Administration and Monitoring Tools

Today I explored advanced Linux system administration tools and discovered powerful utilities for monitoring, process management, and productivity enhancement that go far beyond basic system commands.

Advanced Command-Line Utilities

MoreUtils - Extended Linux Commands

MoreUtils provides additional commands that fill gaps in standard Linux utilities:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# Essential moreutils commands for system administration

# sponge - soaks up stdin and writes to a file
# Prevents truncation when reading from and writing to the same file
sort file.txt | uniq | sponge file.txt

# vipe - vi pipe editor - edit the pipeline interactively
ps aux | vipe | head -10

# ts - timestamp lines from stdin
tail -f /var/log/syslog | ts '[%Y-%m-%d %H:%M:%S]'

# pee - tee for pipes - duplicate stdin to multiple processes
echo "data" | pee 'wc -c' 'wc -w' 'wc -l'

# parallel - run multiple jobs in parallel
find . -name "*.jpg" | parallel -j4 convert {} {.}.png

# ifdata - query network interface information
ifdata -pN eth0  # Print network statistics for eth0

# isutf8 - check if file is valid UTF-8
isutf8 suspicious_file.txt && echo "Valid UTF-8" || echo "Invalid UTF-8"

# lckdo - run command with file locking
lckdo /tmp/mylock.lock important_singleton_script.sh

# combine - combine sets of lines from files using boolean operations
combine file1.txt and file2.txt    # Intersection
combine file1.txt not file2.txt    # Difference
combine file1.txt or file2.txt     # Union

📋 Practical MoreUtils Applications

Log Processing:

1
2
3
4
5


# Add timestamps to real-time logs
tail -f /var/log/nginx/access.log | ts '%H:%M:%S'

# Edit log filtering pipeline interactively
journalctl -f | grep ERROR | vipe | mail [email protected]

File Operations:

1
2
3
4
5


# Safe in-place editing (no race conditions)
cat config.yml | yq '.database.host = "new-host"' | sponge config.yml

# Parallel file processing
find /data -name "*.csv" | parallel -j8 'gzip {}'

System Monitoring:

1
2
3
4
5


# Network interface monitoring
while true; do ifdata -pN eth0; sleep 1; done

# Process monitoring with timestamps
ps aux | ts | grep postgres

Process Monitoring with PV and Progress

PV (Pipe Viewer) and Progress provide visual feedback for long-running operations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


# PV - Pipe Viewer for monitoring data flow

# Monitor file copy progress
pv /path/to/large_file > /destination/large_file

# Database backup with progress
pg_dump large_database | pv | gzip > backup.sql.gz

# Monitor network transfer
curl -s http://example.com/large_file.zip | pv > large_file.zip

# Limit transfer rate (bandwidth throttling)
pv -L 1M /dev/zero > /dev/null  # Limit to 1MB/s

# Progress with size estimation
pv -s 1G /dev/zero > /dev/null

# Multiple progress bars
pv input1.txt > output1.txt &
pv input2.txt > output2.txt &
wait

# Progress - monitor existing processes
# Install: apt install progress
progress -m  # Monitor all processes with progress info
progress cp  # Monitor specific command
progress -w  # Wide output format

# Advanced PV usage for system administration
# Disk imaging with progress
sudo dd if=/dev/sda | pv -s 500G | dd of=/dev/sdb

# Monitor log file growth
pv /var/log/syslog > /dev/null

# Network throughput testing
pv -L 10M /dev/zero | nc target_host 9999

# File system monitoring
while true; do
    du -sh /var/log | pv -l -s 1 | tail -1
    sleep 60
done

Cheat - Offline Command Reference

cheat provides a community-driven cheatsheet system:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


# Install and configure cheat
pip install cheat
# or
snap install cheat

# Basic usage
cheat tar        # Show tar command examples
cheat grep       # Show grep examples
cheat ssh        # SSH usage patterns

# List available cheatsheets
cheat -l

# Edit cheatsheets
cheat -e tar     # Edit tar cheatsheet

# Search across all cheatsheets
cheat -s "recursive"

# Create custom cheatsheets
mkdir -p ~/.config/cheat/cheatsheets/personal
cheat -e mycmd   # Create custom cheatsheet

# Example custom cheatsheet content
# ~/.config/cheat/cheatsheets/personal/docker-admin

# Docker container management
docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Remove stopped containers
docker container prune -f

# View container resource usage
docker stats --no-stream

# Export/import container filesystem
docker export container_name > container_backup.tar
docker import container_backup.tar new_image:tag

# Container networking
docker network ls
docker network inspect bridge

# Volume management
docker volume ls
docker volume inspect volume_name
docker volume prune

System Resource Monitoring

Advanced Process Management

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70


# Comprehensive process analysis and management

# Process tree with resource usage
ps aux --forest | head -50

# Find processes by resource usage
ps aux --sort=-%mem | head -10  # Top memory users
ps aux --sort=-%cpu | head -10  # Top CPU users

# Process monitoring with specific intervals
watch -n 2 'ps aux --sort=-%mem | head -10'

# Find processes using specific files or directories
lsof /var/log/nginx/
lsof +D /tmp/  # Recursively find processes using /tmp

# Network connections by process
lsof -i :80    # Processes using port 80
lsof -i TCP:LISTEN  # All listening TCP ports
netstat -tulpn | grep :22  # SSH connections

# Process signals and management
# List all available signals
kill -l

# Send specific signals
kill -TERM 1234    # Graceful termination
kill -KILL 1234    # Force kill
kill -STOP 1234    # Suspend process
kill -CONT 1234    # Resume process
kill -USR1 1234    # User-defined signal 1

# Process group management
killall -TERM nginx     # Kill all nginx processes
pkill -f "python script.py"  # Kill by command pattern

# Advanced process monitoring script
#!/bin/bash
monitor_processes() {
    echo "=== System Resource Monitor ==="
    echo "Date: $(date)"
    echo
    
    echo "=== Load Average ==="
    uptime
    echo
    
    echo "=== Memory Usage ==="
    free -h
    echo
    
    echo "=== Top CPU Processes ==="
    ps aux --sort=-%cpu | head -6
    echo
    
    echo "=== Top Memory Processes ==="
    ps aux --sort=-%mem | head -6
    echo
    
    echo "=== Disk Usage ==="
    df -h | grep -v tmpfs | head -10
    echo
    
    echo "=== Network Connections ==="
    ss -tuln | head -10
    echo
}

# Run monitoring
monitor_processes

System Performance Analysis

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91


# Comprehensive system performance monitoring

# I/O statistics
iostat -x 1 5    # Extended I/O stats every second, 5 times
iotop -o        # Show only processes with I/O activity

# Network statistics
iftop           # Real-time network bandwidth by connection
nethogs         # Network usage by process
ss -i           # Socket statistics with details

# Memory analysis
vmstat 1 5      # Virtual memory statistics
slabtop         # Kernel slab allocator information
cat /proc/meminfo | grep -E "(MemTotal|MemFree|Cached|Buffers)"

# CPU analysis
mpstat 1 5      # Multi-processor statistics
pidstat -u 1 5  # Per-process CPU usage
cat /proc/loadavg  # Current load average

# Advanced system monitoring script
#!/bin/bash
comprehensive_monitor() {
    local duration=${1:-300}  # Default 5 minutes
    local interval=${2:-5}    # Default 5 seconds
    local logdir="/var/log/system-monitor"
    
    mkdir -p "$logdir"
    local timestamp=$(date +%Y%m%d_%H%M%S)
    local logfile="$logdir/monitor_$timestamp.log"
    
    echo "Starting comprehensive system monitoring..."
    echo "Duration: ${duration}s, Interval: ${interval}s"
    echo "Log file: $logfile"
    
    # System info header
    {
        echo "=== System Monitor Report ==="
        echo "Start time: $(date)"
        echo "Hostname: $(hostname)"
        echo "Kernel: $(uname -r)"
        echo "Uptime: $(uptime)"
        echo
    } > "$logfile"
    
    # Monitoring loop
    local iterations=$((duration / interval))
    for ((i=1; i<=iterations; i++)); do
        {
            echo "=== Sample $i/$(iterations) at $(date) ==="
            
            # CPU usage
            echo "--- CPU ---"
            mpstat | tail -1
            
            # Memory usage
            echo "--- Memory ---"
            free -m | grep -E "(Mem|Swap)"
            
            # Load average
            echo "--- Load ---"
            cat /proc/loadavg
            
            # Top processes
            echo "--- Top CPU Processes ---"
            ps aux --sort=-%cpu | head -6 | tail -5
            
            echo "--- Top Memory Processes ---"
            ps aux --sort=-%mem | head -6 | tail -5
            
            # Disk I/O
            echo "--- Disk I/O ---"
            iostat -x | grep -E "(Device|[s|h|n]d[a-z])" | tail -5
            
            # Network
            echo "--- Network ---"
            cat /proc/net/dev | grep -E "(eth|ens|enp)" | head -3
            
            echo "----------------------------------------"
            echo
        } >> "$logfile"
        
        sleep "$interval"
    done
    
    echo "Monitoring complete. Report saved to: $logfile"
}

# Usage: comprehensive_monitor [duration_seconds] [interval_seconds]
# comprehensive_monitor 600 10  # Monitor for 10 minutes, sample every 10 seconds

Security and Process Control

Chesterton’s Fence Principle

Chesterton’s Fence - understanding why systems exist before changing them:

⚠️ System Administration Wisdom

Chesterton’s Fence Applied to System Administration:

“Do not remove a fence until you know why it was put up.”

Practical Applications:

Configuration files - Understand why settings exist before changing them
Running processes - Research process purpose before killing it
Firewall rules - Understand rule purpose before removal
Cron jobs - Investigate job history before deletion
User permissions - Understand access patterns before modification

Safe Investigation Process:

Document current state
Research configuration history (git logs, documentation)
Test changes in staging environment
Implement changes gradually with rollback plan

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115


# Safe system administration practices

# Document system state before changes
document_system_state() {
    local backup_dir="/var/backups/system-state/$(date +%Y%m%d_%H%M%S)"
    mkdir -p "$backup_dir"
    
    # Configuration backups
    cp -r /etc "$backup_dir/etc_backup"
    
    # Process list
    ps aux > "$backup_dir/processes.txt"
    
    # Network configuration
    ip addr show > "$backup_dir/network_config.txt"
    ss -tuln > "$backup_dir/listening_ports.txt"
    
    # Installed packages
    dpkg -l > "$backup_dir/installed_packages.txt"
    
    # System services
    systemctl list-units --type=service > "$backup_dir/services.txt"
    
    # Disk usage
    df -h > "$backup_dir/disk_usage.txt"
    
    echo "System state documented in: $backup_dir"
}

# Investigate before modification
investigate_process() {
    local pid=$1
    
    if [[ -z "$pid" ]]; then
        echo "Usage: investigate_process <PID>"
        return 1
    fi
    
    echo "=== Process Investigation: PID $pid ==="
    
    # Basic process info
    ps -p "$pid" -o pid,ppid,user,start,command
    
    # Process tree
    pstree -p "$pid"
    
    # Open files
    echo "--- Open Files ---"
    lsof -p "$pid" | head -10
    
    # Network connections
    echo "--- Network Connections ---"
    lsof -i -p "$pid"
    
    # Memory maps
    echo "--- Memory Usage ---"
    pmap -d "$pid" | tail -1
    
    # System calls (if strace available)
    if command -v strace >/dev/null; then
        echo "--- Recent System Calls (5 second sample) ---"
        timeout 5 strace -p "$pid" -c 2>&1 | tail -10
    fi
}

# Safe configuration editing
safe_config_edit() {
    local config_file=$1
    
    if [[ -z "$config_file" ]]; then
        echo "Usage: safe_config_edit <config_file>"
        return 1
    fi
    
    if [[ ! -f "$config_file" ]]; then
        echo "Error: File does not exist: $config_file"
        return 1
    fi
    
    # Create backup
    local backup_file="${config_file}.backup.$(date +%Y%m%d_%H%M%S)"
    cp "$config_file" "$backup_file"
    echo "Backup created: $backup_file"
    
    # Test configuration if possible
    case "$config_file" in
        */nginx/*)
            if command -v nginx >/dev/null; then
                echo "Testing nginx configuration..."
                nginx -t
            fi
            ;;
        */apache*/*)
            if command -v apache2ctl >/dev/null; then
                echo "Testing apache configuration..."
                apache2ctl configtest
            fi
            ;;
        */ssh/sshd_config)
            echo "Testing SSH configuration..."
            sshd -t
            ;;
    esac
    
    # Edit the file
    ${EDITOR:-nano} "$config_file"
    
    # Offer to restore backup
    echo "Configuration edited. Restore backup? (y/N)"
    read -r response
    if [[ "$response" =~ ^[Yy]$ ]]; then
        cp "$backup_file" "$config_file"
        echo "Configuration restored from backup"
    fi
}

Productivity Enhancement Tools

Mnemonic - CLI Memory Aid

Mnemonic helps remember complex commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# Install mnemonic
cargo install mnemonic

# Basic usage
mnemonic add "complex-docker-cmd" "docker run -it --rm -v \$(pwd):/workspace ubuntu:latest bash"

# Use saved command
mnemonic run complex-docker-cmd

# List all mnemonics
mnemonic list

# Search mnemonics
mnemonic search docker

# Edit existing mnemonic
mnemonic edit complex-docker-cmd

# Advanced usage with variables
mnemonic add "deploy-service" "docker build -t \$1 . && docker push \$1 && kubectl set image deployment/\$2 \$2=\$1"

# Use with parameters
mnemonic run deploy-service myapp:v1.2.3 web-service

System Administration Shortcuts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63


# Create a comprehensive system admin toolkit
# ~/.bashrc or ~/.zshrc additions

# Quick system status
alias systat='echo "=== System Status ===" && uptime && echo "=== Memory ===" && free -h && echo "=== Disk ===" && df -h | head -10'

# Process management shortcuts
alias topcpu='ps aux --sort=-%cpu | head -10'
alias topmem='ps aux --sort=-%mem | head -10'
alias ports='ss -tuln'

# Log monitoring
alias syslog='tail -f /var/log/syslog'
alias authlog='tail -f /var/log/auth.log'
alias nginxlog='tail -f /var/log/nginx/error.log'

# Service management
alias services='systemctl list-units --type=service --state=running'
alias failed='systemctl list-units --type=service --state=failed'

# Network utilities
alias myip='curl -s ipinfo.io/ip'
alias netstat='ss -tuln'
alias listen='ss -tuln | grep LISTEN'

# File operations
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'

# Safety aliases
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Advanced functions
monitor_service() {
    local service=$1
    if [[ -z "$service" ]]; then
        echo "Usage: monitor_service <service_name>"
        return 1
    fi
    
    watch -n 2 "systemctl status $service"
}

find_large_files() {
    local size=${1:-100M}
    local path=${2:-.}
    find "$path" -type f -size +"$size" -exec ls -lh {} + | sort -k5 -hr
}

check_service_logs() {
    local service=$1
    local lines=${2:-50}
    
    if [[ -z "$service" ]]; then
        echo "Usage: check_service_logs <service_name> [lines]"
        return 1
    fi
    
    journalctl -u "$service" -n "$lines" --no-pager
}

Key System Administration Insights

Effective Monitoring Strategies

💡 System Monitoring Best Practices

Proactive Monitoring:

Baseline Establishment - Know normal system behavior
Threshold Setting - Define alert thresholds based on baselines
Log Aggregation - Centralize logs for correlation analysis
Automated Responses - Script common remediation tasks
Documentation - Maintain runbooks for common issues

Essential Metrics:

CPU usage and load averages
Memory utilization and swap usage
Disk I/O and space utilization
Network throughput and connection counts
Process states and resource consumption

Command-Line Productivity Principles

📋 Productivity Multipliers

Tool Selection:

Specialized tools over general-purpose commands (htop vs top)
Interactive tools for exploration (iotop, iftop)
Scriptable tools for automation (jq, awk, sed)
Visual tools for complex data (pv, progress)

Workflow Optimization:

Aliases for frequently used commands
Functions for complex operations
History management and search
Tab completion for efficiency
Screen/tmux for session management

This exploration of advanced Linux system administration tools demonstrates that effective system management requires both deep tool knowledge and systematic approaches to monitoring, troubleshooting, and maintenance.

These Linux system administration insights from my archive showcase the evolution from basic command usage to sophisticated system management practices, emphasizing the importance of understanding system behavior and implementing proactive monitoring strategies.

Advanced Command-Line Utilities#

MoreUtils - Extended Linux Commands#

Process Monitoring with PV and Progress#

Cheat - Offline Command Reference#

System Resource Monitoring#

Advanced Process Management#

System Performance Analysis#

Security and Process Control#

Chesterton’s Fence Principle#

Productivity Enhancement Tools#

Mnemonic - CLI Memory Aid#

System Administration Shortcuts#

Key System Administration Insights#

Effective Monitoring Strategies#

Command-Line Productivity Principles#