The Internet Post Office
Every time you browse a website, your computer performs an intricate dance involving multiple layers of wrapping, addressing, and routing. Think of it like a multinational postal system:
The Network as a Postal System
Your message
→ The letter content (HTTP request, file data)TCP envelope
→ Tracking number + delivery confirmation (ensures nothing gets lost)
IP envelope
→ Street address (tells routers where to send it)Ethernet envelope
→ Local mailroom routing (MAC addresses for the local network)
Security checkpoint
→ iptables/netfilter (inspects and filters every package)
Let's open each envelope and see how your data actually travels from application to wire.
Packet Encapsulation: The Russian Nesting Doll
When you send data, it gets wrapped in headers at each layer—like putting a letter in progressively larger envelopes. Watch the encapsulation process:
Your application writes "Hello World" to a socket
- Payload
- 12B
- Headers
- 0B
- On wire
- 12B
Layer Responsibilities
Each layer has a specific job:
| Layer | Protocol | What It Adds | Why |
|---|---|---|---|
| Application | HTTP | Request/response | Your actual data |
| Transport | TCP | Ports + sequencing | Which app, reliable delivery |
| Network | IP | IP addresses | Where to route globally |
| Data Link | Ethernet | MAC addresses | Where on local network |
Why So Much Overhead?
For a 12-byte "Hello World", you add 54 bytes of headers—that's 82% overhead! This is why small packets are inefficient, and why protocols like HTTP/2 batch multiple requests together. The overhead is the same whether you send 12 bytes or 1,400 bytes.
The TCP Handshake: Establishing Trust
Before any data flows, TCP performs a "three-way handshake" to establish a reliable connection. This is one of the most asked-about networking concepts:
Why Three Packets?
The handshake proves that both sides can send AND receive:
Client Server
| |
| -------- SYN (seq=1000) ------> | Client proves: "I can send"
| |
| <-- SYN-ACK (seq=2000,ack=1001) | Server proves: "I can receive AND send"
| |
| -------- ACK (ack=2001) ------> | Client proves: "I can receive"
| |
| Connection ESTABLISHED |
The sequence numbers (1000, 2000, etc.) are how TCP tracks every byte—they're essential for detecting lost or reordered packets.
Socket Programming: The Application Interface
Sockets are the API between applications and the network stack. Under the hood, every socket operation is a system call that transitions from user space into the kernel. Every socket is an integer file descriptor indexing a per-process kernel table: fd 0, 1, and 2 are stdin, stdout, and stderr; fd 3 and above can name sockets, files, or pipes. accept() returns a new fd for each client while the listening socket remains open. Follow the system calls that create, configure, and use a socket.
// Server starting up...- Role
- server
- State
- CLOSED
- File descriptor
- not allocated
Socket Types
// TCP Socket (reliable, ordered delivery)
int tcp_sock = socket(AF_INET, SOCK_STREAM, 0);
// UDP Socket (fast, no guarantees)
int udp_sock = socket(AF_INET, SOCK_DGRAM, 0);
// Raw Socket (direct IP access, requires root)
int raw_sock = socket(AF_INET, SOCK_RAW, IPPROTO_ICMP);
// Unix Domain Socket (local IPC, no network overhead)
int unix_sock = socket(AF_UNIX, SOCK_STREAM, 0);
TCP Server Pattern
#include <sys/socket.h>
#include <netinet/in.h>
int create_server(int port) {
int server_fd = socket(AF_INET, SOCK_STREAM, 0);
// Allow port reuse (avoid "Address already in use")
int opt = 1;
setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
struct sockaddr_in addr = {
.sin_family = AF_INET,
.sin_addr.s_addr = INADDR_ANY,
.sin_port = htons(port)
};
bind(server_fd, (struct sockaddr *)&addr, sizeof(addr));
listen(server_fd, 128); // Backlog queue
while (1) {
int client = accept(server_fd, NULL, NULL);
handle_client(client);
close(client);
}
}
High-Performance I/O with epoll
For handling thousands of connections:
int epfd = epoll_create1(0);
struct epoll_event ev = {
.events = EPOLLIN | EPOLLET, // Edge-triggered
.data.fd = socket_fd
};
epoll_ctl(epfd, EPOLL_CTL_ADD, socket_fd, &ev);
// Event loop
struct epoll_event events[MAX_EVENTS];
while (1) {
int n = epoll_wait(epfd, events, MAX_EVENTS, -1);
for (int i = 0; i < n; i++) {
handle_event(events[i].data.fd);
}
}
Routing and Forwarding
The kernel decides where to send each packet based on its routing table. Understanding how routes are selected is crucial for network debugging. The longest matching prefix wins: a /32 host route overrides a /24 network route, which overrides /16 and /8; 0.0.0.0/0 matches everything but is the fallback. Try the destination samples to see which route wins.
Viewing Routes
# Modern way
ip route show
# Output:
# default via 192.168.1.1 dev eth0
# 192.168.1.0/24 dev eth0 scope link
# 172.17.0.0/16 dev docker0 scope link
Adding Routes
# Static route to a network
ip route add 10.0.0.0/8 via 192.168.1.1 dev eth0
# Default gateway
ip route add default via 192.168.1.1
# Source-based routing (advanced)
ip rule add from 192.168.2.0/24 table custom
ip route add default via 10.0.0.1 table custom
Enable IP Forwarding (Router Mode)
# Temporarily
echo 1 > /proc/sys/net/ipv4/ip_forward
# Permanently
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p
Netfilter: The Kernel's Security Checkpoint
Every packet entering or leaving your system passes through netfilter—a core part of the Linux kernel architecture. This is what iptables (and its successor nftables) uses for packet filtering. Try each packet path to see the hooks it traverses; the path determines where a filtering rule belongs.
HTTP request to your web server
- Path
- PREROUTING → INPUT
Understanding Netfilter Hooks
Packets traverse different hooks depending on their destination:
┌─────────────────┐
│ Local Process │
└────────┬────────┘
│
Network ──► PREROUTING ──► INPUT
│ ▲
│ │ (local destination)
│ │
└──────► FORWARD ─────────┴──► POSTROUTING ──► Network
│ ▲
└─────────────────────────┘
(forwarded packets)
OUTPUT ────────────► POSTROUTING ──► Network
▲
│
┌───────┴───────┐
│ Local Process │
└───────────────┘
Common iptables Rules
# Allow established connections (most efficient rule first!)
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# Allow SSH from specific network
iptables -A INPUT -p tcp --dport 22 -s 192.168.1.0/24 -j ACCEPT
# Allow HTTP/HTTPS
iptables -A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT
# Drop everything else (default deny)
iptables -P INPUT DROP
Rule Order Matters!
iptables processes rules top-to-bottom—first match
wins. If you put -j DROP before -j ACCEPT,
packets will be dropped before reaching the accept rule. Always put your
most-matched rules (like ESTABLISHED) first for performance.
Performance Tuning
Network performance depends heavily on memory management—the kernel allocates buffers (sk_buff structures) for every packet in flight. The most critical tuning parameter is the TCP buffer size. TCP can keep only one buffer of data in flight while it waits for acknowledgments. A 64KB buffer with a 600ms round trip limits throughput to 853 Kbps; a BDP-sized buffer can use the full link, while an oversized one wastes memory and may cause bufferbloat.
- BDP
- 100 Mbps × 50 ms = 610.4 KB
- TCP buffer
- 128.0 KB
- Achieved throughput
- 21.0 Mbps
BDP is the data in flight when the link is fully utilized.
# Set maximum buffer sizes
sysctl -w net.core.rmem_max=131072
sysctl -w net.core.wmem_max=131072
# Set TCP buffer auto-tuning range
sysctl -w net.ipv4.tcp_rmem="4096 87380 131072"
sysctl -w net.ipv4.tcp_wmem="4096 65536 131072"TCP Buffer Tuning Commands
# Increase buffer sizes for high-bandwidth links
sysctl -w net.core.rmem_max=134217728
sysctl -w net.core.wmem_max=134217728
sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728"
# Use BBR congestion control (Google's algorithm)
sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl -w net.core.default_qdisc=fq
Network Interface Tuning
# Increase ring buffer
ethtool -G eth0 rx 4096 tx 4096
# Enable offloading
ethtool -K eth0 gso on gro on tso on
# Receive Packet Steering (multi-core)
echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus
Network Namespaces
Linux can create isolated network environments—this is how containers get their own network stack. Network namespaces work alongside other namespace types (PID, mount, user namespaces) to provide full container isolation.
# Create namespace
ip netns add mycontainer
# Create virtual ethernet pair
ip link add veth0 type veth peer name veth1
# Move one end into namespace
ip link set veth1 netns mycontainer
# Configure host side
ip addr add 10.0.0.1/24 dev veth0
ip link set veth0 up
# Configure container side
ip netns exec mycontainer ip addr add 10.0.0.2/24 dev veth1
ip netns exec mycontainer ip link set veth1 up
ip netns exec mycontainer ip link set lo up
# Test connectivity
ping 10.0.0.2
Debugging Tools
Essential Commands
# Active connections
ss -tunap
# Packet capture
tcpdump -i eth0 'tcp port 80'
# Route tracing
traceroute google.com
mtr google.com # Better interactive version
# Performance testing
iperf3 -s # Server
iperf3 -c host # Client
Common Pitfalls
Common Networking Mistakes
- Small buffers on high-latency links. Default 64KB buffers throttle satellite/VPN connections. Calculate BDP and size buffers accordingly.
- iptables rule ordering mistakes. Putting DROP before ACCEPT, or putting expensive rules before ESTABLISHED. First match wins!
- Forgetting to enable ip_forward. Linux won't route packets between interfaces unless
net.ipv4.ip_forward=1. - Blocking I/O for high-concurrency servers. Use epoll/kqueue for thousands of connections. Blocking I/O means one thread per connection.
- Ignoring connection state tracking overhead. conntrack uses memory per connection. High-traffic servers may need to tune
nf_conntrack_max.
Related concepts
Visualize the complete Linux boot sequence from BIOS/UEFI to login. Learn how GRUB, kernel, and systemd work together with interactive visualizations.
Learn how initramfs enables Linux boot by loading essential drivers before the root filesystem mounts. Explore early userspace initialization.
Explore Linux kernel architecture: user and kernel space, cooperating subsystems, cached and uncached file reads, and how containers share one kernel.
Master Linux kernel modules through interactive visualizations. Learn how to load, unload, develop, and debug kernel modules that extend Linux functionality.
Explore Linux memory management through interactive visualizations. Understand virtual memory, page tables, TLB, swapping, and memory allocation.
Learn nvidia-modeset for display configuration on Linux. Understand kernel mode-setting, DRM integration, and GPU drivers.
