Running Docker in LXC Containers

This guide explains how to run Docker inside LXC containers using the fuse-overlayfs storage driver. This is necessary because LXC containers cannot use Docker's default overlay2 storage driver due to kernel limitations.

Prerequisites

• LXC container with FUSE support enabled and nested containers allowed
• Root or sudo access inside the container
• Basic understanding of Docker and systemctl
• Docker 29.0+: Additional AppArmor configuration required (covered below)

Background: Why fuse-overlayfs?

Docker's default overlay2 storage driver requires kernel-level permissions that LXC containers don't have. When you try to use overlay2 inside an LXC container, Docker fails because:

1. LXC mounts its filesystem using overlay, and the kernel doesn't allow overlay-on-overlay mounting
2. The container lacks permissions to create kernel-level overlay filesystems

Without proper configuration, Docker falls back to the vfs driver, which creates full copies of each filesystem layer instead of using efficient copy-on-write. This can consume massive amounts of disk space.

Understanding Docker 29.0's Storage Driver Detection

Key Insight - Docker's "Prior Driver" Mechanism

Docker 29.0 introduced stricter storage driver validation. However, there's an important exception: if Docker detects that a storage driver was previously used (by checking existing data directories), it will skip the strict validation and continue using that driver. This is called the "prior storage driver" path.

This means: We DON'T explicitly configure the storage driver in daemon.json. Instead, we create the necessary directory structure that makes Docker think fuse-overlayfs was already in use.

Why this matters:

• Explicit configuration ("storage-driver": "fuse-overlayfs" in daemon.json) → triggers strict validation → fails in LXC
• Prior driver detection (directories exist) → skips strict validation → works perfectly

You'll see this log when it works: [graphdriver] using prior storage driver: fuse-overlayfs

Prerequisites Check

1. Verify FUSE Support

First, check if your container has FUSE support:

ls -la /dev/fuse

If /dev/fuse doesn't exist, contact your system administrator to enable FUSE support for your container.

Enabling FUSE in Proxmox

If using Proxmox, set features: fuse=1 in the LXC config or check "FUSE" under Options in the web interface. Restart the container after making this change.

2. Install Required Packages

First, install Docker following the official installation guide:

Docker Installation

Follow the official Docker Engine installation guide for Ubuntu to install Docker from the official repository. This ensures you get the latest stable version with proper support.

After installing Docker, install fuse-overlayfs:

sudo apt update
sudo apt install fuse-overlayfs

Initial Setup: Creating the "Prior Driver" Structure

Critical Step - Do NOT Skip

This step is essential for Docker 29.0+ in LXC containers. We create directory structure that makes Docker think fuse-overlayfs was previously used, allowing it to bypass strict validation.

For New Docker Installations

If you're setting up Docker for the first time, follow these steps before starting Docker for the first time:

# Stop Docker if it's running
sudo systemctl stop docker docker.socket 2>/dev/null || true

# Create the fuse-overlayfs driver directory structure
sudo mkdir -p /var/lib/docker/fuse-overlayfs/l

# Create image metadata directories
sudo mkdir -p /var/lib/docker/image/fuse-overlayfs/imagedb/content/sha256
sudo mkdir -p /var/lib/docker/image/fuse-overlayfs/imagedb/metadata/sha256
sudo mkdir -p /var/lib/docker/image/fuse-overlayfs/layerdb/sha256
sudo mkdir -p /var/lib/docker/image/fuse-overlayfs/layerdb/mounts
sudo mkdir -p /var/lib/docker/image/fuse-overlayfs/distribution

# Create empty repositories.json
echo '{"Repositories":{}}' | sudo tee /var/lib/docker/image/fuse-overlayfs/repositories.json > /dev/null

# Set correct permissions
sudo chmod 710 /var/lib/docker/fuse-overlayfs
sudo chmod 700 /var/lib/docker/fuse-overlayfs/l
sudo chmod -R 700 /var/lib/docker/image/fuse-overlayfs/imagedb
sudo chmod -R 755 /var/lib/docker/image/fuse-overlayfs/layerdb

What This Does

This creates the minimum directory structure that Docker's storage driver detection looks for. When Docker starts, it will:

1. Scan /var/lib/docker/ for existing driver directories
2. Find fuse-overlayfs/l/ (non-empty due to subdirectory)
3. Log: [graphdriver] using prior storage driver: fuse-overlayfs
4. Skip strict validation and use fuse-overlayfs successfully

For Existing Docker Installations

If you already have Docker installed with a different storage driver and want to switch to fuse-overlayfs:

Data Loss Warning

Switching storage drivers will make your existing images and containers inaccessible. Back up any important data before proceeding.

# Stop Docker
sudo systemctl stop docker docker.socket

# Backup existing data (optional but recommended)
sudo mv /var/lib/docker /var/lib/docker.backup

# Create the fuse-overlayfs structure (use the commands from above)
sudo mkdir -p /var/lib/docker/fuse-overlayfs/l
# ... (repeat all mkdir and echo commands from above)

# Start Docker
sudo systemctl start docker

Configuration: Keep daemon.json Minimal

Important - Storage Driver Configuration

Do NOT add storage-driver to daemon.json. Let Docker auto-detect from the directory structure we created above.

Important - AppArmor Configuration

Do NOT add default-security-opt to daemon.json in LXC containers. Docker 29.0's strict validation will fail even with this setting. Instead, specify AppArmor settings per-container in docker-compose.yml or docker run commands.

For most users, daemon.json should be empty or contain only non-storage, non-AppArmor settings:

# Option 1: No daemon.json at all (recommended for new installs)
# Don't create /etc/docker/daemon.json

# Option 2: Empty daemon.json
echo '{}' | sudo tee /etc/docker/daemon.json

NVIDIA Runtime Users Only

If you need NVIDIA container runtime, this is the ONLY thing that should be in daemon.json:

{
  "runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
  }
}

Do NOT add default-security-opt or storage-driver alongside this.

Starting Docker

Now start Docker and verify it's using fuse-overlayfs:

# Enable and start Docker
sudo systemctl daemon-reload
sudo systemctl enable docker
sudo systemctl start docker

# Verify the storage driver
sudo docker info | grep "Storage Driver"
# Should output: Storage Driver: fuse-overlayfs

# Check the logs for confirmation
sudo journalctl -u docker -n 50 | grep "storage driver"
# Should see: [graphdriver] using prior storage driver: fuse-overlayfs

Common Issues and Solutions

AppArmor Permission Denied (Docker 29.0+)

Common Issue with Docker 29.0+

Docker 29.0 introduced security changes (CVE-2025-52881 fix) that may cause permission denied errors in LXC containers. If you encounter these errors, follow the solutions below.

Symptoms:

# Docker service won't start
sudo systemctl status docker
# Shows: Failed to start Docker Application Container Engine

# Or containers fail with permission errors:
docker run hello-world
# Error: permission denied

Solution: Add AppArmor Override When Running Containers (Required)

You must add --security-opt apparmor=unconfined to every Docker container you run:

# Single container
docker run --rm --security-opt apparmor=unconfined hello-world

# With other options
docker run -d \
  --name myapp \
  --security-opt apparmor=unconfined \
  -p 3000:3000 \
  myimage:latest

For Docker Compose, add to your docker-compose.yml:

version: '3.8'
services:
  web:
    image: myimage
    security_opt:
      - apparmor=unconfined
    ports:
      - "3000:3000"

Why Not Use daemon.json for AppArmor?

You might be tempted to add "default-security-opt": ["apparmor=unconfined"] to daemon.json to avoid specifying it for every container. Don't do this in LXC. Docker 29.0's initialization will still trigger strict validation that fails in LXC environments. Always specify AppArmor settings per-container.

Security Note

Setting AppArmor to unconfined reduces container isolation. This is generally acceptable in LXC environments since the LXC container itself provides isolation. However, avoid running untrusted code without additional security measures.

If the above solutions don't work:

Contact your system administrator (RoseLab users: ziz244@ucsd.edu) to verify that your LXC container is configured for nested container support.

Permission Denied on Docker Socket

If you encounter:

permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

Solution: Run Docker commands with sudo or add your user to the docker group:

sudo usermod -aG docker $USER
# Log out and back in for changes to take effect

Service Failed to Start

To debug Docker service issues:

# Check service status
sudo systemctl status docker

# View detailed logs
sudo journalctl -xeu docker.service

# Check Docker daemon directly
sudo dockerd --debug

Common causes:

• Syntax errors in /etc/docker/daemon.json
• Missing fuse-overlayfs package
• FUSE not enabled in container

Docker Won't Start After Configuration Changes

Symptom: Docker fails to start after editing daemon.json

Common causes:

1. You added forbidden settings to daemon.json

   # Check your configuration
   cat /etc/docker/daemon.json
   
   # Remove these if present:
   # - "storage-driver": "fuse-overlayfs"      ← Triggers strict validation
   # - "default-security-opt": [...]           ← Causes validation failures in LXC
   
   # The correct configuration should be empty or only contain:
   # - NVIDIA runtime (if needed)
   # - Registry mirrors, log settings, etc.
   
   # Simplest fix: make it empty
   echo '{}' | sudo tee /etc/docker/daemon.json

2. Syntax error in daemon.json

   # Validate JSON syntax
   python3 -m json.tool /etc/docker/daemon.json
   # Should output formatted JSON if valid

3. Prior driver structure is missing

   # Verify the directories exist
   ls -la /var/lib/docker/fuse-overlayfs/l/
   ls -la /var/lib/docker/image/fuse-overlayfs/
   
   # If missing, recreate them (see Initial Setup section)

Multiple Storage Drivers Detected

Error: contains several valid graphdrivers: overlay2, fuse-overlayfs

Cause: Multiple non-empty driver directories exist in /var/lib/docker/

Solution:

# List all driver directories
ls -la /var/lib/docker/ | grep -E 'overlay|fuse'

# Keep only fuse-overlayfs, remove or rename others
sudo mv /var/lib/docker/overlay2 /var/lib/docker/overlay2.old

# Restart Docker
sudo systemctl restart docker

Performance Considerations

Performance Impact

fuse-overlayfs operates in userspace and has performance overhead compared to kernel-based overlay2. However, it's significantly more efficient than the vfs fallback:

• vfs: Creates full copies of filesystem layers (can use 3-4x more space)
• fuse-overlayfs: Uses copy-on-write like overlay2 but with ~10-20% performance overhead
• overlay2: Native kernel driver (not available in LXC)

Verification

Check Storage Driver

Verify Docker is using fuse-overlayfs via the "prior driver" mechanism:

# Check storage driver
sudo docker info | grep "Storage Driver"
# Should output: Storage Driver: fuse-overlayfs

# Verify it's using prior driver detection (not explicit config)
sudo journalctl -u docker --no-pager | grep "storage driver"
# Should see: [graphdriver] using prior storage driver: fuse-overlayfs

Test Container Execution

Test with a simple container:

# Without AppArmor override (may fail)
sudo docker run --rm hello-world

# With AppArmor override (should work)
sudo docker run --rm --security-opt apparmor=unconfined hello-world

If the second command works but the first doesn't, make sure you've configured default-security-opt in daemon.json as described above.

Advanced Configuration

Systemd Service Debugging

For persistent issues, create a systemd override:

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo vim /etc/systemd/system/docker.service.d/override.conf

Add debugging options:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --debug

About Rootless Docker

Rootless Docker: Possible but Not Recommended

Rootless Docker can work in LXC containers, but it requires the exact same "prior driver" directory structure setup as rootful Docker.

Why not use rootless?

• Originally, rootless Docker was expected to default to fuse-overlayfs automatically
• In reality, it does not default to fuse-overlayfs in LXC
• You still need to create the magic directory structure manually
• Since you need manual configuration anyway, rootful Docker is simpler and more straightforward

Recommendation: Use rootful Docker with the configuration described in this guide. The LXC container already provides process isolation, so running Docker as root inside the container is acceptable and avoids unnecessary complexity.

Alternative Storage Drivers

Not Recommended

While Docker supports other storage drivers (vfs, devicemapper), they are not recommended:

• vfs: Extremely inefficient (3-4x disk space usage)
• devicemapper: Requires complex setup and has performance issues

Stick with fuse-overlayfs for LXC containers.

Related Documentation

• Container Limitations - General limitations of nested containers
• Troubleshooting Guide - Additional Docker troubleshooting steps
• Security Considerations - Security implications of running Docker in LXC

Summary

Key Takeaways

Running Docker 29.0+ inside LXC containers requires a specific approach:

1. Storage Driver Setup:

   • Create directory structure that makes Docker auto-detect fuse-overlayfs
   • Never explicitly configure storage-driver in daemon.json
   • Look for log: [graphdriver] using prior storage driver: fuse-overlayfs

2. AppArmor Configuration:

   • Add --security-opt apparmor=unconfined to all docker run commands
   • Or specify security_opt: - apparmor=unconfined in docker-compose.yml
   • Do NOT add default-security-opt to daemon.json (will cause validation failures)
   • This resolves runc CVE-2025-52881 compatibility issues

3. What NOT to Do:

   • Don't explicitly set storage-driver (triggers strict validation → fails)
   • Don't use vfs or devicemapper (inefficient alternatives)
   • Rootless Docker works but needs same setup (no advantage, use rootful instead)

4. LXC Container Requirements:

   • security.nesting=true (for nested containers)
   • raw.lxc: lxc.apparmor.profile=unconfined (for AppArmor relaxation)
   • /dev/fuse device access (for fuse-overlayfs)

Quick Setup Checklist

# 1. Install prerequisites
sudo apt install docker.io fuse-overlayfs

# 2. Create "prior driver" structure
sudo mkdir -p /var/lib/docker/fuse-overlayfs/l
sudo mkdir -p /var/lib/docker/image/fuse-overlayfs/{imagedb,layerdb}/{content,metadata,sha256,mounts,distribution}/sha256
echo '{"Repositories":{}}' | sudo tee /var/lib/docker/image/fuse-overlayfs/repositories.json

# 3. Keep daemon.json empty or minimal
# Don't create daemon.json, or create empty: echo '{}' | sudo tee /etc/docker/daemon.json

# 4. Start Docker
sudo systemctl start docker

# 5. Verify and run containers
sudo docker info | grep "Storage Driver"  # Should show: fuse-overlayfs
sudo docker run --rm --security-opt apparmor=unconfined hello-world  # Note: requires --security-opt

While fuse-overlayfs has ~10-20% performance overhead compared to native overlay2, it's the only reliable solution for Docker in LXC containers and is vastly superior to the vfs fallback.