๐Ÿ–ฅ๏ธ System

Deep Dive into System Administration : Proxmox, Firewalls, Encrypted storage, etc.

Beyond the Basics

The Challenge: Take Proxmox from "it works" to "production-ready." Advanced networking with nftables. Nested containers running Docker. Full-disk encryption with LUKS. Automated container deployment via custom scripts.

This isn't a tutorial you follow. This is infrastructure you architect.

The goal? Build something that would actually survive in a real datacenter.

After mastering the basics of Proxmox (containers, VMs, basic networking), I wanted to push deeper. This TP covered everything from advanced firewall rules to encryption, from nested virtualization to automation scripting.

Here's what I builtโ€”and more importantly, why it matters.

7 Major Components
20h Total Work Time

Part 1: Advanced Firewall with nftables

The Problem: Default Proxmox firewall is fine for basic scenarios, but it's limited. nftables offers granular control, better performance, and modern syntax.

๐Ÿ”ฅ Why nftables over iptables?

Feature iptables (old) nftables (modern)
Syntax Complex, cryptic Clean, readable
Performance Rule-by-rule evaluation Optimized rulesets
IPv4/IPv6 Separate commands Unified framework
Atomicity No Yes (atomic updates)

โš™๏ธ nftables Configuration

Created a comprehensive ruleset for the Proxmox host:

# /etc/nftables.conf #!/usr/sbin/nft -f flush ruleset table inet filter { chain input { type filter hook input priority 0; policy drop; # Allow established/related connections ct state established,related accept # Allow loopback iif lo accept # Allow ICMP (ping) ip protocol icmp accept ip6 nexthdr icmpv6 accept # Allow SSH (rate limited) tcp dport 22 ct state new limit rate 3/minute accept # Allow Proxmox Web UI tcp dport 8006 accept # Log and drop everything else log prefix "nftables-drop: " drop } chain forward { type filter hook forward priority 0; policy accept; # Allow container traffic ct state established,related accept } chain output { type filter hook output priority 0; policy accept; } }

๐ŸŒ NAT Rules for Container Network

Separate NAT configuration file for cleaner management:

# /etc/nftables.d/nat_rules.conf table ip nat { chain prerouting { type nat hook prerouting priority -100; # Port forwarding example: HTTP to container iif "vmbr0" tcp dport 80 dnat to 10.0.0.100:80 # HTTPS to container iif "vmbr0" tcp dport 443 dnat to 10.0.0.100:443 } chain postrouting { type nat hook postrouting priority 100; # Masquerade outgoing traffic from containers oif "vmbr0" masquerade } }
# Enable and test nftables systemctl enable nftables systemctl start nftables # Verify rules nft list ruleset # Include NAT rules in main config echo 'include "/etc/nftables.d/nat_rules.conf"' >> /etc/nftables.conf nft -f /etc/nftables.conf

Pro Tip: Always test firewall rules on a console session (not SSH) before applying. Lock yourself out once, you'll never forget this advice.

Part 2: Storage - Mount Points & Partitions

Proxmox uses storage pools, but understanding underlying Linux storage is crucial.

๐Ÿ’พ Creating a Mount Point

# List available disks lsblk fdisk -l # Create partition (using fdisk or parted) fdisk /dev/sdb # n (new partition) # p (primary) # 1 (partition number) # [default start] # [default end - use full disk] # w (write changes) # Format partition as ext4 mkfs.ext4 /dev/sdb1 # Create mount point mkdir -p /mnt/data # Mount partition mount /dev/sdb1 /mnt/data # Make mount persistent (add to /etc/fstab) echo '/dev/sdb1 /mnt/data ext4 defaults 0 2' >> /etc/fstab # Verify df -h | grep /mnt/data

๐Ÿ”’ LUKS Encryption - Full Disk Encryption

For sensitive data, encryption isn't optional. LUKS (Linux Unified Key Setup) provides transparent disk encryption.

โš ๏ธ Warning: Encrypting a partition DESTROYS all existing data. Backup first!

# Install cryptsetup apt install cryptsetup -y # Create LUKS encrypted partition cryptsetup luksFormat /dev/sdb1 # Type 'YES' (uppercase) # Enter passphrase (STRONG password!) # Open encrypted partition cryptsetup luksOpen /dev/sdb1 encrypted_data # Enter passphrase # Format the encrypted partition mkfs.ext4 /dev/mapper/encrypted_data # Mount encrypted partition mkdir -p /mnt/encrypted mount /dev/mapper/encrypted_data /mnt/encrypted # Verify encryption cryptsetup status encrypted_data df -h | grep encrypted

๐Ÿ” Auto-unlock on Boot (Optional but Convenient)

# Create keyfile (more secure than password on boot) dd if=/dev/urandom of=/root/luks-keyfile bs=512 count=4 chmod 400 /root/luks-keyfile # Add keyfile to LUKS cryptsetup luksAddKey /dev/sdb1 /root/luks-keyfile # Configure crypttab for auto-unlock echo 'encrypted_data /dev/sdb1 /root/luks-keyfile luks' >> /etc/crypttab # Add to fstab echo '/dev/mapper/encrypted_data /mnt/encrypted ext4 defaults 0 2' >> /etc/fstab

Pro Tip: Store the LUKS header backup in a safe place. If the header corrupts, your data is gone forever.

# Backup LUKS header cryptsetup luksHeaderBackup /dev/sdb1 --header-backup-file /root/luks-header-backup-sdb1.img

Part 3: Containers vs VMs - Understanding the Difference

Proxmox supports both LXC containers and KVM virtual machines. Knowing when to use each is critical.

Feature LXC Container KVM Virtual Machine
Isolation Process-level (shared kernel) Hardware-level (full isolation)
Performance Near-native (very fast) Overhead from virtualization
Boot Time Seconds Minutes
Resource Usage Minimal (MBs of RAM) Higher (GBs of RAM)
OS Support Linux only Any OS (Windows, BSD, etc.)
Use Case Web servers, databases, services Different OS, legacy apps, testing

Key Insight: Containers are NOT just "lightweight VMs." They're fundamentally different. Containers share the host kernelโ€”fast but less isolated. VMs have their own kernelโ€”slower but fully independent.

Part 4: Nested Containers - Docker inside LXC

Running Docker inside a Proxmox LXC container (nested virtualization) unlocks powerful possibilities. Perfect for self-hosted apps like Immich (photo management).

๐Ÿณ Setting Up Docker in LXC

โš ๏ธ Important: Not all LXC templates support nesting. Must use an unprivileged container with nesting enabled.

1

Create Container with Nesting

In Proxmox UI: Create CT โ†’ Features tab โ†’ Check "Nesting"

# Or via CLI pct create 200 local:vztmpl/debian-12-standard_12.0-1_amd64.tar.zst \ --hostname docker-host \ --memory 4096 \ --net0 name=eth0,bridge=vmbr0,ip=dhcp \ --features nesting=1 \ --unprivileged 1
2

Install Docker

# Inside the container apt update && apt upgrade -y apt install curl -y # Install Docker using official script curl -fsSL https://get.docker.com -o get-docker.sh sh get-docker.sh # Install Docker Compose apt install docker-compose -y # Verify installation docker --version docker-compose --version
3

Deploy Immich with Docker Compose

Immich is a self-hosted Google Photos alternative. Perfect for testing Docker deployments.

# Create project directory mkdir -p /opt/immich cd /opt/immich # Download Immich docker-compose wget -O docker-compose.yml https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml wget -O .env https://github.com/immich-app/immich/releases/latest/download/example.env # Edit .env file (set upload location, timezone, etc.) nano .env # Deploy Immich docker-compose up -d # Check status docker-compose ps docker-compose logs -f

Why Nested Containers? Isolation + Portability. The LXC container isolates Immich from the host. Docker Compose makes the entire stack portable. Backup the container, restore anywhere.

๐Ÿ“ธ Immich Setup

After deployment, access Immich at http://CONTAINER_IP:2283. Create admin account, start uploading photos. Full Google Photos alternative running on your infrastructure.

๐Ÿ“ฑ
Mobile App Sync

Automatic photo backup from iOS/Android. Set it and forget itโ€”just like Google Photos.

๐Ÿ”
AI-Powered Search

Machine learning for object/face recognition. Search "beach" or "dog" and find relevant photos.

๐Ÿ—บ๏ธ
Location Tracking

View photos on a map based on GPS metadata. Perfect for travel memories.

๐Ÿ”’
Self-Hosted Privacy

Your photos stay on YOUR server. No cloud scanning. No data mining. Full control.

Part 5: Automation - CT-Creator Script

Manual container creation is tedious. Automation is essential for scale. Enter: CT-Creatorโ€”my Bash script for automated Proxmox container deployment.

CT-Creator

Automated LXC container creation script for Proxmox. Configure once, deploy instantly.

View on GitHub

๐Ÿš€ What CT-Creator Does

โš™๏ธ
Interactive Setup

Prompts for container ID, hostname, RAM, disk size, network config. No manual CLI syntax.

๐Ÿ”„
Template Selection

Choose from available OS templates (Debian, Ubuntu, etc.). Automatic download if missing.

๐ŸŒ
Network Auto-Config

DHCP or static IP. Auto-bridge selection. Firewall integration.

โœ…
Post-Deploy Tasks

Auto-start container, run initial updates, install essential packages.

๐Ÿ’ป Script Highlights

# Key features from CT-Creator #!/bin/bash # Validation function - ensures no duplicate CT IDs validate_ctid() { if pct status "$1" &>/dev/null; then echo "Error: Container ID $1 already exists!" exit 1 fi } # Auto-detect available templates list_templates() { pveam update pveam available | grep -E "(debian|ubuntu|alpine)" | awk '{print $2}' } # Create container with error handling create_container() { pct create "$CTID" "$TEMPLATE" \ --hostname "$HOSTNAME" \ --memory "$RAM" \ --swap "$SWAP" \ --disk "$DISK" \ --net0 name=eth0,bridge="$BRIDGE",ip="$IP" \ --features nesting="$NESTING" \ --unprivileged "$UNPRIVILEGED" \ --password || { echo "Container creation failed!" exit 1 } } # Post-creation tasks post_setup() { pct start "$CTID" sleep 5 pct exec "$CTID" -- bash -c "apt update && apt upgrade -y" echo "Container $CTID ($HOSTNAME) ready!" }

๐Ÿ“ Usage Example

# Clone the repository git clone https://github.com/TristanDivanach/CT-creator.git cd CT-creator # Make script executable chmod +x ct-creator.sh # Run interactive mode ./ct-creator.sh # Follow prompts: # - Enter CT ID: 300 # - Hostname: webserver # - RAM (MB): 2048 # - Disk (GB): 20 # - Template: debian-12 # - Network: DHCP or Static # - Enable nesting: yes/no # Script handles the rest!

Why Automation Matters: Deploying 10 containers manually = 30 minutes + errors. With CT-Creator = 2 minutes, zero errors. Reproducible infrastructure.

Integration: Putting It All Together

Here's how all these components work together in a real infrastructure:

๐Ÿ”ฅ
nftables Firewall

Protects Proxmox host. Rate-limits SSH. Forwards ports to containers. NAT for outbound traffic.

๐Ÿ’พ
Encrypted Storage

LUKS-encrypted partition stores sensitive data. Immich photos, backups, database dumpsโ€”all encrypted at rest.

๐Ÿ“ฆ
LXC Containers

Lightweight, fast, isolated environments. Each service gets its own container. Easy to backup/restore.

๐Ÿณ
Nested Docker

Docker inside LXC for complex apps. Immich runs as Docker Compose stack. Portable and maintainable.

๐Ÿค–
CT-Creator Automation

Deploy new containers in seconds. Consistent configuration. Version-controlled via Git.

๐Ÿ—‚๏ธ
Mount Points

Separate data storage from system. Easy to expand. Bind mounts into containers for shared storage.

Lessons Learned

๐Ÿ”’
Security is Layered

Firewall + encryption + isolation. One layer fails, others protect. Defense in depth isn't paranoiaโ€”it's professionalism.

๐Ÿ“š
Documentation Saves Time

Spent 2 hours writing CT-Creator. Saved 20+ hours on deployments. Good docs are force multipliers.

๐Ÿงฉ
Modularity Wins

Separate concerns. Firewall rules in one file. NAT in another. Storage isolated. Easier to maintain and troubleshoot.

โšก
Automation Compounds

Every manual task automated is time saved forever. Scripts improve over time. Manual work doesn't.

๐Ÿ”
Understand the Stack

Don't just copy commands. Know WHY nftables > iptables. WHY LUKS matters. WHY containers โ‰  VMs. Knowledge stacks.

๐Ÿ› ๏ธ
Break Things Safely

Proxmox snapshots are lifesavers. Test destructive changes (firewall rules, encryption) on snapshots first. Learn without fear.

Mistakes I Made (And How I Fixed Them)

Mistake #1: Applied nftables rules without testing SSH access first โ†’ Locked out

Fix: Used Proxmox web console. Added SSH rule BEFORE enabling firewall. Lesson learned.

Mistake #2: Forgot to enable nesting on LXC โ†’ Docker refused to start

Fix: Destroyed container, recreated with --features nesting=1. Now part of CT-Creator checklist.

Mistake #3: Encrypted partition without backing up LUKS header โ†’ Almost lost everything during testing

Fix: Fortunately it was test data. Now ALWAYS backup header: cryptsetup luksHeaderBackup

Mistake #4: Hard-coded values in CT-Creator script โ†’ Broke when storage pool changed

Fix: Made script fully interactive. Auto-detects available storage, templates, bridges. Much more robust.

What's Next?

This infrastructure is solid, but there's always room to grow:

๐Ÿ“Š
Monitoring Stack

Prometheus + Grafana for metrics. Loki for log aggregation. Alert on resource exhaustion.

๐Ÿ”„
Automated Backups

Proxmox Backup Server (PBS) integration. Scheduled snapshots. Offsite replication.

๐ŸŒ
Reverse Proxy

Nginx Proxy Manager or Traefik. SSL certificates via Let's Encrypt. Subdomain routing.

๐Ÿ”
SSO Integration

Authelia or Keycloak for centralized authentication. One login for all services.

The Bottom Line

This wasn't just a TP. This was a deep dive into production-grade infrastructure.

Every component serves a purpose. Every decision has a rationale. This is what real system administration looks like.

Build infrastructure that matters.
Automate what's repetitive.
Learn by doing, not just reading.

Want to Try CT-Creator?

The script is open source. Clone it, modify it, make it yours.

Star the repo if you find it useful. PRs welcome!

Clone on GitHub

Questions? Suggestions? Improvements for CT-Creator? Open an issue on GitHub or drop a comment below.

Infrastructure is code. Code is power. ๐Ÿ’ป

Help others level up their infrastructure skills

Retour en haut