Today’s learning focused on container technologies, systems programming concepts, and distributed systems architecture.

Docker Networking and Graph Drivers

Brutally Honest Guide to Docker Graph Drivers and Container Networking provide deep insights into Docker’s internal mechanisms:

Docker Graph Drivers:

Storage Driver Comparison:

1
2
3
4
5
6
7
8
9

# Check current storage driver
docker info | grep "Storage Driver"

# Available drivers and characteristics:
# overlay2 (recommended): Copy-on-write, good performance
# aufs: Legacy, being phased out
# devicemapper: Block-level storage, complex configuration
# btrfs: Advanced filesystem features, snapshot support
# zfs: Enterprise features, high memory usage

Overlay2 Driver Deep Dive:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Inspect layer structure
docker pull ubuntu:20.04
docker history ubuntu:20.04

# Examine overlay2 directory structure
sudo ls -la /var/lib/docker/overlay2/

# Layer composition
# Lower layers: read-only base layers
# Upper layer: read-write container changes
# Merged layer: unified view presented to container

# Example layer structure:
# /var/lib/docker/overlay2/
# ├── abc123...def/
# │   ├── diff/          # Layer changes
# │   ├── link           # Short identifier
# │   ├── lower          # Parent layers
# │   └── work/          # Temporary work directory
# └── l/                 # Symbolic links to layers

Performance Implications:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Inefficient: creates many layers
FROM ubuntu:20.04
RUN apt-get update
RUN apt-get install -y python3
RUN apt-get install -y python3-pip  
RUN apt-get install -y git
RUN apt-get clean

# Efficient: fewer layers, better caching
FROM ubuntu:20.04
RUN apt-get update && \
    apt-get install -y \
        python3 \
        python3-pip \
        git && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Docker Container Networking:

Network Types:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# List networks
docker network ls

# Default networks:
# bridge: Default network for containers
# host: Container uses host networking directly
# none: No networking
# Custom networks: User-defined bridge networks

# Create custom network
docker network create --driver bridge my-network
docker network create --driver bridge --subnet=172.20.0.0/16 my-subnet

# Inspect network configuration
docker network inspect bridge
docker network inspect my-network

Bridge Network Deep Dive:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Default bridge network (docker0)
ip addr show docker0
brctl show docker0

# Container networking internals
docker run -d --name web nginx
docker exec web ip addr show
docker exec web ip route show

# Network namespace inspection
docker inspect web | grep NetworkMode
sudo nsenter -t $(docker inspect -f '{{.State.Pid}}' web) -n ip addr

Custom Networking:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

# docker-compose.yml with custom networks
version: '3.8'
services:
  web:
    image: nginx
    networks:
      - frontend
      - backend
    ports:
      - "80:80"
  
  app:
    image: python:3.9
    networks:
      - backend
      - database
    depends_on:
      - db
  
  db:
    image: postgres:13
    networks:
      - database
    environment:
      POSTGRES_PASSWORD: secret

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
  database:
    driver: bridge
    internal: true  # No external access

Container Communication:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Service discovery within custom networks
docker network create app-network
docker run -d --name db --network app-network postgres:13
docker run -d --name app --network app-network my-app:latest

# Containers can reach each other by name
# app container can connect to: db:5432
# No need for --link (deprecated)

# Port publishing vs exposure
docker run -p 8080:80 nginx    # Publish to host
docker run --expose 80 nginx   # Only expose to other containers

Julia Evans’ Systems Programming Insights

Julia Evans provides exceptional explanations of complex systems concepts:

Networking Fundamentals:

Computer Networking Basics:

  • TCP vs UDP: Reliability vs speed trade-offs
  • DNS resolution: How domain names become IP addresses
  • HTTP request flow: Complete journey from browser to server
  • Load balancing: Distributing traffic across multiple servers

HTTP Request Routing:

Client Request Flow:
1. DNS lookup: domain.com → IP address
2. TCP connection: Three-way handshake
3. TLS handshake: Certificate validation, encryption setup
4. HTTP request: Headers, body, method
5. Server processing: Route matching, business logic
6. HTTP response: Status code, headers, body
7. Connection handling: Keep-alive or close

SNI (Server Name Indication):
- Allows multiple HTTPS sites on same IP
- Client sends domain name during TLS handshake
- Server selects appropriate certificate
- Critical for shared hosting and CDNs

Kubernetes Architecture Understanding:

Kubernetes Learning Journey:

  • Pods: Basic deployment units, shared networking/storage
  • Services: Stable network endpoints for dynamic pods
  • Deployments: Declarative pod management and rolling updates
  • ConfigMaps/Secrets: Configuration and sensitive data management

Kubernetes from the Ground Up

Kamal Marhubi’s series provides deep architectural understanding:

Core Kubernetes Components:

The API Server:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# API Server responsibilities:
# 1. RESTful API for all Kubernetes resources
# 2. Authentication and authorization
# 3. Admission control and validation
# 4. etcd storage interface
# 5. Resource change notifications

# Example API interaction
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  namespace: default
spec:
  containers:
  - name: web
    image: nginx:1.20
    ports:
    - containerPort: 80
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"

The Kubelet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Kubelet responsibilities:
# 1. Watch API server for pod assignments
# 2. Manage container lifecycle (start, stop, restart)
# 3. Health checking and reporting
# 4. Resource monitoring and management
# 5. Volume mounting and networking setup

# Kubelet configuration
sudo systemctl status kubelet
sudo journalctl -u kubelet -f

# Pod lifecycle management
# kubelet receives pod spec from API server
# Downloads container images
# Creates container runtime (containerd/docker)
# Sets up networking (CNI plugins)
# Mounts volumes
# Starts containers
# Monitors health and reports status

The Scheduler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

// Simplified scheduler algorithm
func Schedule(pod *Pod, nodes []Node) *Node {
    // 1. Filter nodes (predicates)
    feasibleNodes := []Node{}
    for _, node := range nodes {
        if canSchedule(pod, node) {
            feasibleNodes = append(feasibleNodes, node)
        }
    }
    
    // 2. Score nodes (priorities)
    scoredNodes := make(map[Node]int)
    for _, node := range feasibleNodes {
        score := calculateScore(pod, node)
        scoredNodes[node] = score
    }
    
    // 3. Select highest scoring node
    bestNode := selectBestNode(scoredNodes)
    return bestNode
}

func canSchedule(pod *Pod, node *Node) bool {
    // Resource constraints
    if node.AvailableCPU < pod.RequestedCPU {
        return false
    }
    if node.AvailableMemory < pod.RequestedMemory {
        return false
    }
    
    // Node selectors
    if !nodeMatchesSelectors(node, pod.NodeSelector) {
        return false
    }
    
    // Taints and tolerations
    if !podToleratesNodeTaints(pod, node.Taints) {
        return false
    }
    
    return true
}

Advanced Kubernetes Concepts:

Networking Model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# CNI (Container Network Interface) plugins
# Provide pod-to-pod networking across nodes

# Example: Calico network policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-netpol
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 80
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432

Storage and Persistence:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Persistent Volume and Claims
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-example
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/pv-example

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-example
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Everything is a File Philosophy:

The Unix principle that everything is a file extends to container technologies:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Container processes visible in host /proc
docker run -d --name test nginx
docker exec test ps aux

# Container networking through Linux primitives  
ip netns list
docker exec test cat /proc/net/tcp

# Container storage as filesystem layers
ls -la /var/lib/docker/overlay2/
mount | grep overlay

# Socket communication
stat /var/run/docker.sock  # Docker daemon socket
file /var/run/docker.sock  # Shows socket type

These concepts form the foundation for understanding modern containerized and orchestrated systems, from basic Docker networking to sophisticated Kubernetes cluster management.