Hello and welcome back to Architecting Zero Downtime Infrastructure! It’s fantastic to have you here for episode eleven. Today, we’re diving into a topic that I find is so often overlooked but holds the key to incredible gains: performance tuning Proxmox.

Let’s be honest, you’ve invested in some solid hardware. But are you truly getting every ounce of performance out of it? A standard Proxmox installation is a marvel of stability right out of the box. It’s designed to be a reliable workhorse for the widest possible range of hardware. Think of it like a new car straight from the factory—it runs beautifully, but it’s tuned for universal appeal, not for the specific track you’re racing on. The real power, the raw speed, and unshakeable stability that you paid for are often waiting just beneath that surface of default settings.

Our mission today is to move beyond those defaults and unlock that hidden potential. And this isn’t just for folks running massive enterprise clusters! These principles will enhance the speed and stability of any system, from a single-node homelab to a multi-rack deployment. After all, performance isn’t just about flashy benchmark numbers; it’s a cornerstone of reliability and a critical component of any serious zero-downtime philosophy.

So, let’s roll up our sleeves and get started!

The Heart of the System: CPU Tuning

We’ll begin with the very heart of your system: the CPU. When you configure a virtual machine, one of the most impactful settings you can touch is the CPU Type. By default, it’s set to something very generic for compatibility.

My unbreakable rule for any performance-sensitive workload is to change this setting from the default to host. It’s a simple, powerful change with a direct, tangible payoff. Setting the type to host essentially passes through your physical CPU’s exact feature set to the virtual machine. This allows your applications to leverage all the modern instruction sets they were designed for, like AVX. It’s like telling your VM, “Stop pretending to be a generic processor and use the real, high-performance engine we have right here!”

For mission-critical systems, we can go even further with CPU pinning and NUMA awareness:

• Pinning: This is where you assign a VM’s virtual cores to specific physical cores on your host. It prevents the hypervisor’s scheduler from bouncing the workload between different cores. Why does this matter? Every time a process moves, it can lose its cached data, forcing it to fetch from much slower main memory. Pinning ensures consistency and dramatically reduces latency. It’s a must for applications like databases or real-time processing.
• NUMA (Non-Uniform Memory Access): If you’re running a server with multiple physical CPU sockets, respecting NUMA is not just a tweak; it’s fundamental. It ensures that a VM running on Processor A uses the memory directly attached to Processor A, avoiding the much slower interconnect path to Processor B’s memory. For demanding databases, getting this right can be the difference between a system that flies and one that mysteriously crawls under load.

Don’t Let Your Memory Float Away

Next up, let’s talk about memory. One feature you’ll see enabled by default is memory ballooning. The idea is clever: the host can “reclaim” unused memory from a guest VM to give to another that needs it, allowing you to run more VMs on the same hardware. This is perfectly fine for development machines or less critical services.

However, for a production database, a critical application server, or anything where memory access latency matters, my strong recommendation is to turn ballooning off. Give the VM a fixed, guaranteed slice of RAM. This prevents the host from trying to negotiate memory back from the guest right when it’s under heavy load, which can introduce unpredictable latency spikes. You trade a little bit of density for a whole lot of predictable performance.

For a more advanced boost, I often look at enabling HugePages. Let me paint a vivid picture. Normally, the operating system manages memory in tiny 4-kilobyte chunks, like a warehouse manager tracking millions of individual small boxes. It’s a lot of overhead! HugePages allows the system to manage memory in much larger blocks—often 2 megabytes or even 1 gigabyte. It’s like switching from small boxes to massive shipping containers. The CPU has far less administrative work to do tracking memory locations, freeing it up to focus on your application’s actual work. For any workload with a large memory footprint, this is a change you will absolutely feel.

Taming the I/O Beast: Storage Optimization

If there’s one place where performance bottlenecks love to hide, it’s in storage I/O. For the underlying storage on a single host, I find ZFS is a fantastic choice for its incredible data integrity features and solid performance, while LVM-Thin is great for its speed and simplicity. (Ceph, of course, is the go-to for clustered environments, but that’s a whole other episode!)

But the real tuning often happens on the virtual disk itself. A critical choice you’ll make is the cache setting. You’ll see two main options: writethrough and writeback.

• writethrough is the safe default. It confirms a write only after the data is physically on the disk.
• writeback is where the speed is. It confirms a write as soon as it hits the host’s super-fast RAM cache. The risk? If your host loses power before that cache is flushed to disk, you lose that data.

My personal rule of thumb is to use writeback, but only if you have your host on a reliable battery backup (UPS). It’s a calculated risk for a massive reward in write performance. And while you’re in those settings, enable the modern io_uring option for I/O threading. It’s a genuine leap forward for asynchronous I/O and can really let your modern SSDs stretch their legs.

Building a Multi-Lane Highway for Your Network

Networking is an area where one simple choice can have an absolutely massive impact. When you create a VM, Proxmox has to give it a virtual network card. The default is often a fully emulated card, like an Intel e1000. It’s a marvel of compatibility—it will work with almost any guest OS, old or new. But it is slow because the host has to pretend to be a physical piece of hardware.

For any modern guest operating system (which is likely everything you’re running), you should be using VirtIO. This is a paravirtualized driver, meaning it was designed from the ground up for virtualization. The guest OS knows it’s in a VM and uses a highly efficient, direct communication path to the host. The performance difference isn’t just a few percentage points; it’s orders of magnitude faster. This should be your non-negotiable standard.

But here’s the pro-level trick: for any VM with multiple vCPUs, you must enable the Multi-Queue option. Without it, all network processing for that entire VM is crammed through a single virtual CPU core. A busy web server or firewall can easily overwhelm that one core, creating a completely artificial bottleneck. Enabling multi-queue allows the network workload to be distributed across several vCPUs in parallel. You’re effectively turning a single-lane country road into a multi-lane superhighway for your network traffic, preventing jams and dramatically increasing your total throughput. This change is transformative.

The Other Half of the Equation: Guest OS Tuning

Tuning the host is only half the battle. The performance you actually experience depends just as much on what’s happening inside the guest operating system.

1. Install the Guest Agent and Drivers: The very first thing I do on any new VM is to ensure the QEMU guest agent and the correct VirtIO drivers are installed and running. Without these, the host and guest can’t communicate efficiently. You simply won’t get the storage or network speeds we’ve worked so hard to enable.
2. Use noatime: This is a subtle but powerful tweak. In your guest’s filesystem settings (/etc/fstab on Linux), add the noatime mount option. By default, the OS performs a write operation every single time a file is read just to update its last-accessed timestamp. On a busy server, this can eliminate thousands of completely unnecessary I/O operations per second.
3. Check Partition Alignment: This is a small detail that can yield a very real improvement in disk throughput. Making sure your partitions are correctly aligned with the underlying storage blocks prevents the drive from having to do extra work on every read and write. Most modern installers handle this correctly, but it’s worth verifying.

Going Deeper: Tuning the Proxmox Host Kernel

For those who want to wring every last drop of performance from their hardware, we can go a layer deeper and tune the Proxmox host kernel itself.

• CPU Governor: The default governor, ondemand, saves power by scaling the CPU’s frequency up and down. This is great for a desktop, but it can introduce tiny bits of latency as the CPU ramps up. For systems where consistent, predictable speed is critical, I set the governor to performance. This pins the CPUs at their maximum frequency, ready to go at a moment’s notice.
• I/O Scheduler: With today’s lightning-fast NVMe SSDs, the traditional I/O schedulers can sometimes become a bottleneck themselves. What you’ll often find works best is to set the I/O scheduler to none or mq-deadline. This essentially gets out of the way and lets the high-performance drive’s own internal logic manage its I/O queue much more efficiently.

KVM vs. LXC: The Performance Trade-Off

No performance discussion is complete without touching on the choice between a full KVM virtual machine and a lightweight LXC container. The answer from a pure performance perspective is clear: containers are significantly faster.

Let me break it down. A KVM machine is like building a complete, separate house. It has its own foundation, its own walls, its own roof (the OS kernel). This creates a strong security boundary, but all that structure comes with overhead. An LXC container, on the other hand, is like a well-insulated apartment in a larger building. It shares the building’s foundation and structure (the host’s kernel) but has its own secure living space.

Because it shares the host kernel, a container’s performance is incredibly close to running on bare metal. For applications like web servers, databases, or microservices where I need maximum efficiency, containers are my go-to choice. But when iron-clad isolation is the top priority—for multi-tenant services or running untrusted code—the security boundary of a full VM is the only correct answer. It’s a classic trade-off: near-bare-metal speed versus hardened isolation.

You Can’t Improve What You Don’t Measure

All of these changes are powerful, but they mean very little if you can’t prove their impact. This is where my most important rule comes into play: adopt a scientific approach.

Before you change a single setting, establish a baseline. For storage, my go-to tool is fio, which can simulate real-world I/O patterns. For networking, iperf3 is perfect for measuring raw throughput.

My process is rigid, but it works:

1. Run the benchmark and document the baseline numbers.
2. Make one single change. Just one.
3. Run the exact same benchmark again.
4. Document the new result.

This discipline is what separates professional tuning from just guessing. It turns your assumptions into validated, quantifiable gains and ensures you’re actually making things better, not worse.

Bringing It All Together

Wow, we’ve covered a lot of ground today! We’ve walked through the full stack, from the host’s CPU and memory, down into the nitty-gritty of storage and networking, and finally into the guest OS itself.

What I truly hope you take away from this is that exceptional performance is never an accident. It’s the direct result of active, methodical tuning. It’s this deliberate process that transforms a standard setup into a truly high-performance, resilient system—one that you can count on when it matters most.

I hope you can join me next time for Episode 12, where we’ll be tackling a fascinating topic: “The Hidden Costs of ‘Free’: Calculating TCO for Open-Source Hypervisors”. We’ll dig into whether “free” software is truly free once you account for support, expertise, and operational effort. It’s a critical calculation for any business.

As always, thank you for your time and attention today. If you have any questions or your own favorite tuning tips, please share them in the comments below! I’d love to hear from you.

Hello and welcome back! It’s great to have you here for another episode of Architecting Zero Downtime.

Today, we’re diving deep into a topic that is absolutely critical but, I find, is often misunderstood:

Proxmox Storage Architecture

Specifically, we’re going to talk about why setting up compute high availability (HA) is only winning half the battle.

This is a big one, because getting it wrong doesn’t just reduce your resilience—it can create a false sense of security that’s arguably more dangerous than having no HA at all.

Let me paint a vivid picture, one I’ve seen play out more times than I can count.

A talented team builds a beautiful multi-node Proxmox cluster. They meticulously configure High Availability, run all the failover tests, and watch with satisfaction as a simulated node failure causes virtual machines to restart on another host in mere seconds.

It’s magic.

They feel secure.

They feel resilient.

They feel… done.

The problem is that they’ve often just shifted their single point of failure—not eliminated it.

By connecting this brilliant, redundant compute cluster to a single, non-redundant NAS or SAN, they’ve created an Achilles’ heel for their entire operation.

If that one storage box fails, the whole cluster goes dark.

This episode is about closing that gap.

We’re going to explore why storage is the other, equally critical, half of the high availability equation.

---

What Proxmox HA Actually Does (And What It Doesn’t)

First, let’s get crystal clear on what standard Proxmox HA actually does.

In a properly configured cluster, the system is like a vigilant flight controller, constantly monitoring the health of your virtual machines.

If one of the physical host servers suddenly fails—perhaps a power supply dies or a motherboard fails—the HA Manager detects this immediately.

It then automatically restarts those affected virtual machines on another healthy node in the cluster.

I like the analogy of having a hot standby pilot for an airplane.

If the primary pilot is incapacitated, the co-pilot seamlessly takes control.

The plane doesn’t fall out of the sky.

The journey continues.

But here’s the crucial distinction.

This process only protects the compute aspect of the VM.

It moves the running brain—the CPU and RAM state—to new hardware.

This is only possible if the new host can still access the VM’s virtual disk.

The storage—the VM’s long-term memory—must already be available from the new location.

This entire mechanism relies on mature, battle-tested technologies such as:

• Corosync for cluster communication
• Fencing to safely isolate failed nodes

It’s a fantastic system.

But it has one fundamental dependency:

Accessible shared storage.

---

The Classic Trap: The Single Box of Failure

This brings us to the design pattern responsible for many completely avoidable outages.

The classic mistake.

A team builds a beautiful Proxmox cluster…

…and then connects every node via NFS or iSCSI to a single standalone NAS or SAN.

At first glance this seems perfectly logical.

Centralised storage is easy to manage.

Unfortunately, this is where everything breaks.

If that single storage appliance fails:

• Controller failure
• Power supply failure
• Firmware issue
• Hardware fault

Every virtual disk immediately becomes unavailable to every node.

Proxmox HA becomes completely powerless.

It knows the VMs need restarting.

But their disks no longer exist.

It has the brain.

The body has disappeared.

What was supposed to be a high availability event becomes a complete cluster-wide outage—all because of one overlooked storage dependency.

---

The Local Storage Dead End

You might now be thinking:

“I’ll just use local storage on each Proxmox node.”

And you’re right.

For pure I/O performance, local storage such as:

• LVM
• ZFS

can be fantastic.

Unfortunately, it creates another fatal problem for High Availability.

I often describe this as putting your data in jail.

If a motherboard dies…

If a CPU fails…

The VM disk is physically trapped inside that failed server.

The remaining cluster cannot access it.

Proxmox HA is willing to restart the workload.

But the disk itself cannot move.

At this point you’ve transitioned from:

High Availability

to

Disaster Recovery.

Your only option becomes restoring from backups.

Which completely defeats the purpose of designing a zero-downtime platform.

---

The Two Paths to True Storage Resilience

So where do we go from here?

The answer is to stop thinking of High Availability as a feature…

…and instead design resilience into the architecture itself.

There are two primary approaches.

1. Resilient Shared Storage

Storage remains centralised…

…but the storage platform itself becomes highly available.

The storage is no longer a single point of failure.

2. Distributed Storage

This is the Hyper-Converged Infrastructure (HCI) approach.

Using software such as Ceph, the local disks inside every Proxmox node become one large distributed storage system.

Storage is no longer centralised.

Instead, it becomes self-healing and distributed across the cluster.

These are the two architectural roads available.

Either:

• Build an incredibly resilient storage platform.

or

• Remove the central storage platform entirely.

---

Path One: Fortifying the Citadel with Resilient Shared Storage

Instead of thinking about a storage appliance…

Think about a storage system.

Enterprise storage platforms eliminate every single point of failure.

Key design features include:

Dual Controllers

If one controller fails or is upgraded…

The second immediately takes over.

Redundant Power Supplies

Each controller has independent power.

Ideally supplied by separate PDUs and electrical circuits.

Redundant Connectivity

For iSCSI:

Always implement Multipath I/O (MPIO).

For NFS:

Use LACP link aggregation across multiple switches.

This ensures that one failed cable or switch never disconnects storage.

The good news is that this level of resilience is no longer limited to expensive enterprise hardware.

Solutions such as TrueNAS High Availability can provide enterprise-grade resilience on commodity servers.

The principle remains the same.

Your storage appliance is no longer one box.

It’s effectively its own highly available cluster.

---

Path Two: The Elegance of Distributed Storage with Ceph

Personally, this is the architecture I find most elegant within a Proxmox environment.

Rather than building a separate storage cluster…

Ceph integrates storage directly into the compute cluster.

It pools the local disks from every node into one distributed storage fabric.

One that is:

• Intelligent
• Distributed
• Self-healing

When data is written to Ceph…

It doesn’t create one copy.

It typically creates three replicas.

Each replica is stored on different disks in different physical hosts.

If an entire server fails…

Ceph doesn’t panic.

It simply recognises that replicas have disappeared…

…and automatically rebuilds them elsewhere.

The storage heals itself.

This is what enables:

• Live Migration
• True High Availability
• Continuous storage availability

for every node.

---

Choosing the Right Architecture

So which approach should you choose?

It depends on your:

• Budget
• Operational model
• Team expertise

My general rule of thumb is:

Dedicated HA SAN

Advantages

• Clear separation between compute and storage
• Familiar enterprise operational model
• Storage specialists manage storage

Disadvantages

• Higher capital cost
• Potential vendor lock-in

Ceph Hyper-Converged Infrastructure

Advantages

• Excellent scalability
• Lower hardware costs
• Compute and storage scale together
• No dedicated storage hardware

Disadvantages

• More operational complexity
• Requires an excellent network
• Overall cluster health directly affects storage performance

There isn’t one universally correct answer.

Only the architecture that best fits your organisation.

---

My Non-Negotiable Rule: The Network

Regardless of which storage architecture you choose…

Everything depends on the network.

This is my one non-negotiable rule.

A serious Proxmox deployment should always include:

• Dedicated storage network
• Redundant switching
• Minimum 10 Gigabit Ethernet
• Complete separation from management and VM traffic

Building high availability on an underpowered storage network is like constructing a skyscraper on sand.

Eventually…

It will fail.

---

Trust, But Verify

Building the solution is only half the job.

The other half is proving that it works.

The only way to truly trust a High Availability platform is through deliberate failure testing.

Pull power cables.

Disconnect storage links.

Simulate failed switches.

Watch what happens.

Verify that the environment behaves exactly as designed.

Because an untested HA platform isn’t really High Availability.

It’s simply a theory.

---

Final Thoughts

The key lesson from today’s discussion is simple.

Proxmox High Availability provides compute resilience.

It does not provide complete infrastructure resilience.

True zero downtime only exists when storage is designed with the same level of resilience as the compute layer itself.

Whether that means:

• A dedicated HA SAN
• Hyper-Converged Ceph

the principle never changes.

Your storage must be every bit as resilient as the workloads it supports.

---

Next Episode

In Episode 11 we’ll move from resilience to speed.

Join me for:

Performance Tuning Proxmox: Getting the Most from Your Hardware

We’ll explore how to unlock the full performance potential of your cluster—not just keep it online.

Until then…

Happy architecting!

And, as always, I’d love to hear your questions and experiences in the comments below.

Hello and welcome back to Architecting Zero Downtime Infrastructure!

Today we’re tackling one of those topics that generates endless forum debates and slightly heated Reddit threads: Proxmox VE versus Proxmox Backup Server.

Let me be upfront with you — that “versus” is completely wrong. It’s like asking whether your heart is better than your lungs. They don’t compete. They complete each other.

After years of designing resilient infrastructure, I’ve come to see the VE + PBS combination as one of the most elegant symbiotic relationships in modern IT. So instead of choosing sides, let’s explore how these two tools were designed to work together — and why understanding that partnership is the key to sleeping better at night.

The Fundamental Misunderstanding

The biggest mistake I see (and I see it constantly) is treating Proxmox Backup Server as just another feature of Proxmox VE. It’s not. It’s a specialized, purpose-built platform with its own mission.

Proxmox VE is your runtime environment. Its job is to keep your workloads running efficiently, day in and day out.
Proxmox Backup Server (PBS) is your dedicated data protection and disaster recovery system. Its entire existence is about making sure you can recover — cleanly, quickly, and confidently.

One keeps the lights on. The other makes sure the lights can be turned back on if everything goes dark.

This isn’t philosophical nitpicking. It’s architectural truth.

Proxmox VE: The High-Performance Engine

Let’s start with what most of us already love.

Proxmox VE is a complete open-source virtualization platform built on Debian. It gives you rock-solid KVM virtual machines and lightning-fast LXC containers. But it’s so much more than a hypervisor.

It includes built-in clustering for high availability, software-defined storage, networking, and an incredibly polished web interface that somehow never feels overwhelming. When I stand up a new Proxmox VE cluster, I know I’m getting an entire management fabric — not just a hypervisor.

Its mission is clear: run production workloads with maximum uptime.

That’s where its high availability features shine. If a node dies, the cluster can restart VMs on healthy nodes. Compute resilience? Handled.

But here’s what HA can’t protect you from — and this is where the story gets interesting.

Proxmox Backup Server: The Master Mechanic & Insurance Policy

I love using this analogy because it makes the relationship instantly clear.

Imagine your Proxmox VE cluster is the high-performance engine in a sports car. It’s all about speed, efficiency, and keeping you moving forward. It’s fantastic at what it does.

Now picture Proxmox Backup Server as the comprehensive insurance policy and the master mechanic’s repair shop rolled into one. It doesn’t make the car go faster. But when you crash — especially when the crash involves corrupted data, ransomware, or that “one bad update” that silently destroys your database — PBS is the only thing that can reliably put you back on the road with minimal scarring.

Proxmox VE’s HA handles compute failure.
PBS handles data integrity and recoverability.

They solve two completely different problems. And both problems are mission-critical.

The Magic Under PBS’s Hood

What makes PBS genuinely special is how intelligently it handles data.

The first backup is a full copy (as you’d expect). After that, everything becomes incremental — but with a brilliant twist. PBS uses block-level deduplication across your entire backup repository.

Every block of data gets a unique fingerprint. If PBS has seen that exact block before — from any VM, any container, any day — it doesn’t store it again. It just adds a pointer. The storage savings are massive, and the network efficiency is even better.

This is dramatically different from simple snapshots inside Proxmox VE. Snapshots are useful, don’t get me wrong. But they aren’t a backup strategy. They’re a safety net with limitations.

PBS is the real deal.

The Beautiful Operational Symbiosis

Here’s what makes me genuinely excited about this combination: it’s simple.

You add your PBS instance as a storage target directly in the Proxmox VE web interface you already know and love. That’s it. From that moment on, everything — scheduling, retention policies, browsing restore points — happens in the familiar GUI.

No complex scripts. No new dashboards to learn. Just clean, elegant integration.

And then there are the features that make you look like a wizard in front of stakeholders:

• Live Restore: Start the VM while the data is still streaming back from the backup server in the background. Your Recovery Time Objective shrinks from hours to minutes.
• Automatic Verification: PBS periodically reads back your backup data and verifies integrity. It’s quietly protecting you from bit rot while you sleep.
• Granular Recovery: Need to restore one deleted file from a 4TB file server? Just mount the backup and grab it. No full VM restore required.

I’ve used that last feature more times than I care to admit. The “oops, the intern deleted the entire customer database” panic turns into a five-minute task. It’s borderline therapeutic.

When the Real World Tests Your Architecture

Let me paint you a painfully common scenario.

You have a critical database VM. A routine update gets applied at 2 AM. Unknown to everyone, it contains a bug that begins silently corrupting data. The VM stays up. The cluster looks perfectly healthy. Proxmox VE’s High Availability features are working exactly as designed.

But the data is broken.

This is the moment where PBS proves its worth. From the Proxmox VE interface, you navigate to the VM’s backup tab, pick the restore point from before the bad update, and restore. Minutes later, you’re back to known-good data.

No drama. No heroic all-nighter. Just a documented incident instead of a potential business catastrophe.

That’s the difference between hoping for the best and knowing you’re protected.

The Cost Conversation (Yes, It’s Refreshing)

Both platforms are completely open source. The entire feature set is available for free.

However, in every production environment I build, I always include the optional subscription. Not because I need extra features — but because I want the stable enterprise repositories and, most importantly, professional support from the team that actually builds the software.

It’s one of the best value propositions in infrastructure today. You get enterprise-class resilience without the enterprise-class price tag.

The Big Takeaway

Here’s my unbreakable rule:

Proxmox VE runs your infrastructure. Proxmox Backup Server protects it.

True resilience is a two-part equation. High availability at the compute layer is necessary but incomplete. You also need an independent, purpose-built data protection strategy.

When you combine both, something magical happens. You stop being reactive and start being confident.

And in our world, confidence is the ultimate luxury.

---

In our next episode (Episode 10), we’re going to dive into the foundation that makes all of this possible: Proxmox Storage Architecture – Why Compute HA Isn’t Enough. We’ll explore ZFS, Ceph, and why your storage design is actually the most critical decision in your entire resilience strategy.

You won’t want to miss it.

In the meantime, I’d love to hear from you. Have you already paired Proxmox VE with PBS? Are you still on the fence about adding a dedicated backup server? Drop your thoughts in the comments — I read every single one.

Until next time, stay resilient.

— Your friendly infrastructure mentor

Hello and welcome back, friend!

Today in our Architecting Zero Downtime Infrastructure series, we’re tackling something that makes a lot of smart people suddenly get quiet: security.

I see it all the time. Someone spins up a shiny new Proxmox host, starts deploying VMs, and treats security like an optional side quest. But here’s the truth I want you to tattoo on your brain: a compromised system is a downed system. There’s no such thing as “high availability” when someone else is wearing your root account like a winter coat.

So consider today’s episode the non-negotiable foundation work. We’re going to take that fresh Proxmox installation and transform it, step by step, from “wide open Linux server” into a hardened platform you can actually build a business on.

Let’s get to work.

Step 1: Kill the Root Login (Your First Smart Move)

Every attacker on the planet knows the username “root” exists. That’s like putting a giant welcome mat at your front door that says “Key is under the mat.”

My very first action on any new Proxmox node is to create a dedicated administrative user and add it to the sudo group. Once I’ve tested that account, I immediately disable direct root SSH access.

You do this by editing /etc/ssh/sshd_config, finding the line PermitRootLogin, and changing it to:

PermitRootLogin no

One tiny change. Massive impact. Now any attacker needs to guess both a username and a password. We’ve just removed their head start.

Step 2: Isolate Your Management Network (The Front Door)

I like to say the management interface is the front door to your entire virtualization kingdom. In a perfect world, that door sits on its own physical network card, connected to a completely separate switch that nothing else touches.

Not always possible? Fine. But at minimum, that management traffic needs to live on its own dedicated VLAN. No exceptions.

This separation is pure gold. It stops lateral movement cold. If someone compromises a VM, they shouldn’t automatically get a clear path to your hypervisor. Think of it as building a moat.

Step 3: Wake Up the Proxmox Firewall (Most People Sleep on This)

One of my favorite things about Proxmox is that it ships with a perfectly good firewall that most people… completely ignore.

The firewall works at three levels: Datacenter, Host, and VM. For now, we’re focusing on the Host level because we want to protect the hypervisor itself.

Here’s my process:

1. Enable the firewall for the host
2. Set the default input policy to DROP (this is our “deny everything unless I specifically allow it” stance)
3. Create two allow rules: one for the web interface (port 8006) and one for SSH (port 22)
4. Critically, restrict the source to your management workstation’s IP (or a defined IP set)

Suddenly your management ports become invisible to the rest of the world. I love this. It’s simple, fast, and ridiculously effective.

Step 4: Enable Two-Factor Authentication (The Game Changer)

If I could force every Proxmox user to do just one thing from this entire episode, it would be enabling 2FA.

A stolen password becomes useless if the attacker also needs the device in your pocket. Proxmox supports both TOTP (Google Authenticator, Authy, etc.) and hardware keys. I personally run YubiKeys on all production systems and haven’t looked back.

Go to Datacenter ? Permissions ? Two Factor Authentication and set it up for your admin accounts. Takes five minutes. Worth its weight in gold.

Step 5: Make SSH Boring to Attackers

We can take SSH security much further than just disabling root login.

First, I change the SSH port to something non-standard (pick a high port you’ll remember). The automated bots scanning the internet are lazy—they mostly hit port 22. Make them work for it.

But the real move? Kill password authentication entirely.

My process:
– Generate an SSH key pair on my workstation
– Copy the public key to the server
– Test key-based login
– Only then do I set PasswordAuthentication no in sshd_config

The difference is night and day. Your logs become dramatically cleaner, and password-based attacks become impossible.

Step 6: Embrace Your Patching Discipline

Security isn’t a “set it and forget it” sport. It’s a lifestyle.

This is where Proxmox’s repositories matter. In production, I strongly prefer the Enterprise repository. The updates are more thoroughly tested and stable. For labs, the No-Subscription repo works fine, but understand what you’re getting.

Either way, make apt update && apt dist-upgrade part of your regular rhythm. You’re patching both Debian and Proxmox VE in one shot. Do it consistently.

Step 7: Give Your Server an Automated Bouncer (Meet Fail2Ban)

Some attacks are sophisticated. Most aren’t. Most are dumb, noisy, brute-force attempts from script-kiddie bots.

This is why I consider Fail2Ban essential on every Proxmox host.

It watches your log files in real time. When it sees someone failing to log in repeatedly, it automatically updates the firewall to block their IP for a period of time. It’s like having a bouncer who kicks out drunk people trying to force their way into the club.

I configure it to protect both SSH and the Proxmox web interface. Set it and sleep better.

Step 8: Practice Least Privilege Like Your Job Depends On It (It Does)

Finally, let’s talk about other people.

My unbreakable rule: Never give multiple people full administrator accounts. It’s sloppy and dangerous.

Proxmox’s Role-Based Access Control system is excellent here. I create custom roles like “VM Operator” that allow starting, stopping, and viewing consoles—but nothing that touches networking, storage, or cluster configuration.

Then I assign these roles to groups, add users to the groups, and give them access only to their specific pools of VMs. Containment and accountability in one elegant package.

The Foundation Is Now Solid

Let’s recap what we’ve built today:

• No more root SSH access
• Hardened and isolated management network
• Proxmox firewall with default-deny posture
• Two-factor authentication everywhere
• Key-only SSH on a non-standard port
• Regular patching discipline
• Fail2Ban standing guard
• Proper RBAC with least privilege

This isn’t theoretical security. This is practical, production-grade hardening that takes a relatively small amount of time but delivers enormous peace of mind.

Security isn’t a feature you add later. It’s the bedrock your entire high-availability strategy stands on.

Now that our host is secure, the next question becomes: how do we protect the data sitting on that host?

That’s exactly what we’re diving into next time in Episode 9: Proxmox VE vs. Proxmox Backup Server – A Symbiotic Relationship. I can’t wait to show you how these two tools work together to create something truly special.

Until then, go harden those hosts, my friend.

Drop a comment below and tell me: What’s the one security change that gave you the biggest “why didn’t I do this sooner” feeling? I read every single one.

See you in the next episode.

Stay secure out there,
— Your infrastructure mentor

Hello and welcome back!

If you’ve been riding along with me on this Architecting Zero Downtime Infrastructure journey, you know I’m not here to sell you dreams. I’m here to give you the map with all the cliffs, sinkholes, and surprise dragons clearly marked.

Today we’re talking about something close to my heart: open-source virtualization platforms like Proxmox and XCP-ng. I genuinely love these tools. They’re powerful, flexible, and can carry serious enterprise workloads without breaking a sweat. But the official documentation? It’s like that friend who only tells you the highlight reel of their road trip and conveniently forgets to mention the flat tires in the rain at 2 a.m.

This episode is about the stuff they don’t put in the manual—the real-world gotchas that separate “it works” from “it works reliably at 3 a.m. when the CFO is breathing down your neck.”

Let’s walk through the hidden realities together.

Hardware Compatibility: It’s a Spectrum, Not a Checkbox

Every project begins with the sacred Hardware Compatibility List (HCL). Sensible, right?

Here’s the first gotcha: compatibility isn’t binary. It’s more like a sliding scale with many shades of “technically works but might ruin your weekend.”

Your RAID controller might be on the list, but only stable with a firmware version released six months after the documentation was written. That shiny new 10GbE card? Works great—except its open-source driver doesn’t support SR-IOV, which you were counting on for performance.

My unbreakable rule: Before I buy anything, I search the community forums for that exact model number plus the words “production” and “pain.”

User-reported experience beats official documentation every single time. The forums are where you find out that certain Intel NICs start dropping packets under heavy load with ZFS, or that one particular Dell server model has a BIOS bug that breaks live migration every third Tuesday.

Think of the HCL as the trailer for a movie. The forums tell you whether the film is actually any good.

The “Free” Software Myth (and the Expertise Tax)

Here’s the misconception I see constantly: people think “free” means “zero cost.”

My friend, the license might be free, but your time is not.

The real currency in open-source virtualization is engineering hours. When something breaks at 3 a.m., there’s no support portal with a guaranteed SLA. There’s just you, a pot of coffee, and the cold realization that you’re the highest-paid person currently troubleshooting this issue.

This is what I call the Expertise Tax.

You have two honest choices:

1. Keep senior talent in-house and actually budget their time for maintenance and deep troubleshooting.
2. Buy an enterprise support subscription (which many vendors offer for Proxmox and XCP-ng).

I view these subscriptions as mandatory insurance for anything running production workloads. They’re not optional—they’re your safety net when the ARC cache decides to eat all your RAM at the worst possible moment.

Patching: High-Stakes Foundation Surgery

On a regular server, updating is boring. On a hypervisor, it’s like performing heart surgery while the patient is running a marathon.

A seemingly innocent kernel or QEMU update can quietly break ZFS kernel modules, GPU passthrough, or your carefully tuned Ceph configuration. I’ve seen it happen more times than I care to count.

My process is non-negotiable: Every serious deployment gets a non-production environment that’s architecturally identical to production (even if it’s smaller). Every single patch goes there first.

We test live migration, storage performance, backup/restore, and application behavior before anything touches the live cluster. This discipline turns a terrifying activity into a predictable, almost boring process. Boring is exactly what we want at 2 a.m.

Storage: Where Theory and Reality Diverge

The documentation makes creating a ZFS pool or Ceph cluster look almost too easy. Production has other ideas.

For ZFS, the classic gotcha is the ARC (Adaptive Replacement Cache). By default, it’s an aggressive beast that will happily consume most of your host’s RAM. Great for storage performance, terrible for VM performance when your virtual machines start swapping.

I always manually tune the ARC with hard limits. My rule of thumb: leave at least 2-4GB per VM core plus overhead. Your VMs deserve the memory you promised them.

Ceph is even more dramatic. The initial setup is well-documented. The moment you have a performance problem or need to recover from a string of degraded placement groups at 4 a.m.? That’s when you discover that the documentation only got you to base camp. The summit requires experience, deep knowledge, and occasionally some colorful language.

Networking: The Land of Subtle, Expensive Mistakes

Linux bridges are simple. Production-grade networking is where the pain lives.

The most common (and frustrating) culprit? MTU mismatches. Your physical switches are set for jumbo frames, but somewhere in the chain—virtual switch, VM, or container—someone is still running the default 1500. The result is mysterious, intermittent packet loss that makes you question your career choices.

Link aggregation follows the same pattern. I love LACP for both performance and redundancy, but the bonding configuration on your hypervisor must match the port-channel configuration on your physical switch with religious precision. Any mismatch creates the networking equivalent of a bar fight.

And VLANs? Every tag must be perfect as traffic flows from physical NIC through the virtual bridge and into the VM. These aren’t minor details—they’re the difference between a stable network fabric and a flaky, mysterious mess.

Backups: Don’t Forget the House

Your VM backup solution might be excellent. But what about the host running those VMs?

The painful gotcha here is the lack of a simple bare-metal recovery process for the hypervisor itself. Restoring your VMs is pointless if their home is gone.

My standard practice is regular backup of critical host configuration (on Proxmox, that’s the entire /etc/pve/ directory). But backups alone aren’t enough.

The most important thing is having a documented and tested procedure for rebuilding a failed node from scratch. You need to have practiced this when nothing is on fire so you can perform it calmly when everything is.

Clustering and HA: Quorum, Fencing, and Nightmares

The documentation makes High Availability sound like three simple clicks. Reality demands you understand quorum and, more importantly, fencing.

A split-brain scenario—where two nodes both think they’re in charge—can corrupt data faster than you can say “production incident.” Proper fencing (the ability to definitively power off a misbehaving node) is non-negotiable.

I also insist on a dedicated, low-latency network just for cluster communication. Corosync is extremely sensitive to latency, and shared networks have a nasty habit of getting congested exactly when you need them most.

The Most Important Gotcha: The Human Element

Here’s the biggest truth nobody puts in the docs:

The most sophisticated open-source tool is only as good as the person using it.

With commercial solutions, you’re a customer with a support contract. With open-source, you’re a community participant. That requires a fundamental mindset shift.

You need to learn how to debug effectively, gather clean logs, and ask intelligent questions. The difference between “someone please help” and “here’s what I’ve already tried, with logs and reproduction steps” is massive.

This soft skill might be the most valuable thing you’ll develop in your open-source virtualization journey.

The Takeaway

Success with open-source virtualization isn’t about avoiding all problems. It’s about knowing which problems are coming and having systems, processes, and mindsets in place before they arrive.

It means treating the documentation as a starting point rather than gospel. It means building test environments, tuning memory, obsessing over networking details, and—most importantly—becoming an active participant in the community that makes these incredible tools possible.

Now that we understand how to build the platform properly, the next logical step is making sure nobody unauthorized can touch it.

Episode 8 is coming soon: Securing Your New Proxmox Environment. We’ll dive into firewalls, proper user permissions, two-factor authentication, and the security model that turns your powerful platform into a fortress.

Until then, I’d love to hear from you.

What’s the biggest open-source virtualization gotcha that caught you by surprise? Drop it in the comments—I read every single one.

Keep building wisely, my friends.

Talk soon,
Your fellow infrastructure architect