Containers Under the Hood: From Primitives to Docker

What is a Container, Really?

Linux containers are just Linux processes with three primitives layered on top — and that framing surprises many developers when they first see it. There's no special "container" system call. No kernel module named "docker". When you run docker run nginx, you're ultimately just running the nginx process with some clever configuration:

• Namespaces make it see an isolated system
• cgroups limit what resources it can use
• OverlayFS gives it its own filesystem
• Security features restrict what it can do

That's it. A container is a regular process, wrapped in isolation and limits.

The Container Stack

Before diving into primitives, let's understand the layers involved when you run docker run:

Building a Container from Scratch

Watch step-by-step how a container runtime creates a container. Each step uses specific Linux syscalls:

The Simplified Flow

# What docker run actually does (simplified)
1. Pull image layers → /var/lib/docker/overlay2/
2. Create container directory
3. fork() → clone(CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS | ...)
4. Set up cgroups → /sys/fs/cgroup/docker/<id>/
5. Mount overlay filesystem → /var/lib/docker/overlay2/<id>/merged
6. pivot_root() → switch root to container filesystem
7. Configure network (veth pair → bridge)
8. Apply security (drop capabilities, seccomp)
9. execve() → run the application

Container Image Layers

Docker images use a layered filesystem. This enables efficient storage and fast distribution. Try modifying files to see Copy-on-Write in action:

How OverlayFS Works

OverlayFS merges multiple directories into a single unified view:

Key concepts:

# Mount an overlay filesystem manually
mount -t overlay overlay \
  -o lowerdir=/layer3:/layer2:/layer1,\
     upperdir=/container/upper,\
     workdir=/container/work \
  /merged

Container Networking

Containers get network isolation via network namespaces, but need connectivity. Docker uses several patterns:

Bridge Networking (Default)

Port Mapping

When you run docker run -p 8080:80 nginx:

1. Docker creates an iptables DNAT rule
2. Traffic to host:8080 redirects to container:80
3. Container responds through same path

# View Docker's iptables rules
iptables -t nat -L DOCKER -n
# DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 to:172.17.0.2:80

Container vs Virtual Machine

Understand when to use each technology:

The Shared Kernel Trade-off

The biggest difference is the kernel:

This is why container security requires defense-in-depth: seccomp, capabilities, AppArmor/SELinux, and careful image curation.

Security Layers

Containers need multiple security mechanisms since they share the kernel:

1. Capabilities

Linux capabilities split root's powers into ~40 individual capabilities:

# Default Docker drops these (among others):
CAP_SYS_ADMIN     # Mount, namespace operations
CAP_NET_ADMIN     # Network configuration
CAP_SYS_PTRACE    # Debug other processes
CAP_SYS_MODULE    # Load kernel modules

# Check container capabilities
docker run --rm alpine cat /proc/self/status | grep Cap

2. seccomp

System call filtering - block dangerous syscalls entirely:

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "syscalls": [
    {
      "names": ["read", "write", "exit", "futex", ...],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

Docker's default profile blocks ~44 syscalls including reboot, kexec_load, mount (in most cases).

3. User Namespace Remapping

With userns-remap enabled on the Docker daemon, container root maps to an unprivileged host UID. The host UID range depends on the configured subordinate IDs:

# On a daemon with userns-remap enabled, inspect the UID mapping.
docker run --rm alpine cat /proc/self/uid_map
# Example: container UID 0 maps to host UID 231072 across 65536 IDs.
#          0     231072      65536

--userns=host disables remapping for that container. Rootless mode also runs the Docker daemon without root privileges. See Docker's user namespace documentation for configuration and limitations.

4. Read-only Filesystems

docker run --read-only nginx

# Or with temporary write areas
docker run --read-only --tmpfs /tmp nginx

The OCI Specification

The Open Container Initiative standardizes container formats:

OCI Image Spec

Defines how images are structured:

• Manifest: List of layers and config
• Config: Runtime settings (env vars, entrypoint)
• Layers: Filesystem tarballs

OCI Runtime Spec

Defines config.json format for runc:

{
  "ociVersion": "1.0.0",
  "process": {
    "terminal": false,
    "user": { "uid": 0, "gid": 0 },
    "args": ["/bin/sh"],
    "env": ["PATH=/usr/bin:/bin"],
    "cwd": "/"
  },
  "root": {
    "path": "rootfs",
    "readonly": false
  },
  "linux": {
    "namespaces": [
      { "type": "pid" },
      { "type": "network" },
      { "type": "mount" }
    ],
    "resources": {
      "memory": { "limit": 536870912 }
    }
  }
}

You can run containers with just runc:

# Create OCI bundle
mkdir -p mycontainer/rootfs
cd mycontainer
docker export $(docker create alpine) | tar -C rootfs -xf -
runc spec  # Creates config.json

# Run container
sudo runc run mycontainer

Real-World Container Flow

What actually happens when you run docker run -d -p 8080:80 --memory=512m nginx:

1. Docker CLI → Docker daemon (REST API)
   POST /containers/create
   POST /containers/{id}/start

2. Docker daemon → containerd (gRPC)
   - Pull image if needed (registry → local storage)
   - Create container metadata
   - Create OCI bundle (config.json + rootfs)

3. containerd → runc (exec)
   - runc init: Set up namespaces, cgroups
   - runc create: Prepare container
   - runc start: Execute entrypoint

4. runc performs:
   a. clone(CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS | ...)
   b. mkdir /sys/fs/cgroup/docker/{id}
   c. echo 536870912 > memory.max
   d. mount overlay filesystem
   e. pivot_root to new root
   f. Create veth pair, attach to docker0
   g. Drop capabilities, apply seccomp
   h. execve("/docker-entrypoint.sh")

5. Docker daemon:
   - Add iptables DNAT rule for port 8080→80
   - Register container in internal DB
   - Return container ID to CLI

6. nginx is now running:
   - PID 1 in its namespace
   - eth0 with 172.17.0.x IP
   - 512MB memory limit
   - Isolated filesystem view
   - Accepting connections on port 80

Common Container Patterns

Init Process Problem

Container PID 1 has special responsibilities:

• Reap zombie processes
• Handle signals properly

Many applications aren't designed to be PID 1. Solution: use a proper init:

# Use tini as init
docker run --init nginx

# Or in Dockerfile
FROM nginx
RUN apt-get update && apt-get install -y tini
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["nginx", "-g", "daemon off;"]

Sidecar Containers

Share namespaces between containers:

# Main application
docker run -d --name app myapp

# Sidecar shares network namespace
docker run -d --network container:app logging-sidecar

Multi-stage Builds

Keep images small by separating build and runtime:

# Build stage
FROM golang:1.21 AS builder
COPY . /app
RUN go build -o /app/server

# Runtime stage (much smaller)
FROM alpine:3.18
COPY --from=builder /app/server /server
CMD ["/server"]

When containers are the right boundary (and when a VM or plain process is better)

Containers are the right answer when the workload needs process-level isolation with kernel sharing — different filesystem, network, and process view, but same kernel and same host hardware. They're the wrong answer when you need stronger isolation than namespaces can give you, or when the overhead of an image isn't justified by what you're isolating.

Use containers when:

• You need reproducible builds and deploys — an image bundles the application, its libraries, and a known filesystem snapshot. Same image, same behaviour on any host.
• You're packing many small services onto fewer hosts — each container costs a few MB of RSS plus its working set. Density is roughly 10× a VM.
• The workload is stateless or has clearly separated state — config, code, dependencies in the image; data in a mounted volume or a network store.
• You want fast startup — milliseconds to seconds versus the tens of seconds a VM takes to boot.
• You're integrating with Kubernetes, ECS, Nomad, or any orchestrator — these tools assume the container model.

Reach for a VM (or a hypervisor sandbox like Firecracker) when:

• You need kernel-level isolation — different kernel versions, different OS families, or a security model where one tenant escaping the kernel cannot reach another.
• You're running multi-tenant code from untrusted sources — function-as-a-service, online code execution, security CTFs. Container escapes via kernel bugs are real; VMs and microVMs are the production answer.
• The workload needs GPU, kernel-module, or low-level device access that's awkward to plumb through container runtimes.
• You're satisfying compliance requirements (PCI, HIPAA, FedRAMP) that mandate hypervisor-level separation between tenants.

Use a plain process when:

• The workload is a trusted single-tenant daemon on a machine you own — systemd already gives you cgroups, namespaces (partially), and resource accounting via unit files. Wrapping it in a container is a packaging choice, not an isolation choice.
• The deployment target doesn't have a container runtime and you don't want to introduce one — embedded systems, edge devices, tiny VMs.
• You need performance parity with the host down to the last percent — container networking, overlay filesystems, and seccomp filters all add cost that adds up under heavy load.

The honest decision rule: containers when you need process isolation plus reproducible packaging, VMs when the trust boundary must include the kernel, plain processes when you control all of the above.

Related concepts