Hello and welcome back to the workshop! I’m so glad you’re here. Today, we’re rolling up our sleeves and getting our hands dirty with the absolute core of ZFS, its foundational architecture: vdevs, Pools, and RAID-Z.

Now, one of the first hurdles I see people stumble over when they’re new to this world is thinking of ZFS as just another filesystem. That, my friends, is a fundamental misunderstanding that can lead to some painful mistakes down the road. After years of building and managing these systems, I’ve learned it’s far more accurate—and much more helpful—to think of ZFS as a combined volume manager and, critically, a tireless guardian of your data’s integrity.

Getting these foundational building blocks right from the very beginning is completely non-negotiable. If you want to build a storage solution that is genuinely resilient, performant, and scalable, your success is determined by the decisions you make at this level. Think of this as pouring the concrete foundation for a skyscraper. Everything we’ll cover in future episodes—all the advanced features and performance tuning—rests on the principles we’re about to discuss. So, let’s get to it!

The Bricks of the Build: What is a Vdev?

Let’s start with the absolute smallest component in our system: the vdev, which stands for Virtual Device.

I find the simplest way to explain it is to think of vdevs as the Lego blocks from which you construct your entire storage empire. They are the fundamental, indivisible units of your storage. Now, technically, a single disk can be a vdev, but for any system you actually care about, that’s a configuration I would never recommend. For our purposes, and for any resilient system, a vdev is a group of disks configured for redundancy.

This could be a two-disk mirror or a set of six disks in what ZFS calls RAID-Z. The crucial takeaway here is this: the type of vdev you create dictates the fault tolerance and the performance characteristics for that entire group of disks. It’s the foundational decision, and it’s one you make for each and every block you add to your build.

The Structure Itself: Combining Vdevs into a Pool

Once you have your vdevs—your resilient Lego blocks—the next step is to combine them into the structure itself. This is the ZFS pool, or ‘zpool‘ as you’ll see it called in the command line. This is the top-level container; it’s the actual storage space you’ll carve up and use.

A pool is created from one or more vdevs. And here we come to one of the most critical design rules in ZFS, something you absolutely must burn into your brain before you type a single command:

Once a vdev is added to a pool, it is permanent. You cannot remove it.

This isn’t a bug; it’s a core design feature tied to how ZFS achieves its incredible performance. ZFS stripes data across all the vdevs in the pool. The best analogy I’ve found is to think of a multi-lane highway. Each vdev you add is a brand new lane. When you add a second vdev to your pool, you’re not just adding capacity; you’re adding another lane to that highway, increasing the total data throughput of the entire system. All your data can now flow across all available lanes, which is what delivers that impressive aggregate performance. But you can’t just rip a lane out of the middle of a highway without causing a catastrophic failure, and the same is true here.

The Mirror Vdev: Built for Speed

Let’s examine our first, and simplest, redundant vdev: the mirror. In its most common form, this is a vdev made of two disks where data is written identically to both. Think of it as a perfect clone.

The pros here are incredibly compelling:

• Excellent Read Performance: ZFS can satisfy read requests from either disk in the mirror, effectively load-balancing and often doubling your read throughput compared to a single disk.
• Fast Rebuilds: If a disk fails, the resilver—ZFS’s term for a rebuild—is extremely fast. It’s just a straight, block-for-block copy from the surviving disk to the new one.

The con is obvious: storage efficiency. A two-way mirror provides only 50% of its raw capacity as usable space. That’s a trade-off, but in my experience, it’s one that is absolutely worth it for certain workloads. My personal gold standard is to use mirrors for boot drives, database storage, and virtual machine disks—any workload with heavy, random I/O where that blistering performance is paramount.

RAID-Z: The Capacity Champion with an Unbeatable Defense

While mirrors are fantastic for performance, when we need sheer capacity, we turn to RAID-Z. This is ZFS’s unique implementation of parity-based redundancy, an elegant evolution of the traditional RAID-5 and RAID-6 you may have used in the past.

Its defining feature, and frankly the reason I trust it implicitly with my data, is its immunity to the infamous RAID write hole. Let me paint a vivid, and painfully common, picture: with a traditional hardware RAID card, if you lose power right in the middle of a write operation, you can end up with inconsistent parity. The data block gets written, but the parity block doesn’t (or vice versa). Your array thinks it’s healthy, but your data and its protection are out of sync. This is a ticking time bomb of silent data corruption.

ZFS completely eliminates this risk through its copy-on-write nature. It never overwrites data in place. Instead, it writes new data and its corresponding parity to a completely new location on disk. Only when that entire transaction is confirmed as successful on the physical platters does it update the metadata to point to the new, correct blocks. If the power cuts out mid-way, the old, consistent data remains completely untouched. This transactional integrity is the bedrock of ZFS’s reliability.

With that peace of mind, let’s look at the family:

• RAID-Z1: Uses single parity, meaning the vdev can survive the failure of any one disk. It offers a good balance of capacity, but with today’s enormous multi-terabyte drives, I’ve grown very cautious about using it.
• RAID-Z2: Uses double parity and can withstand the failure of any two disks. When a 16TB drive is resilvering for hours (or days!), the stress on the other drives is immense. Having that second layer of protection is, in my view, essential. This is my recommended professional standard for almost any new capacity-focused build.
• RAID-Z3: Triple parity. It can survive three simultaneous disk failures. This is for massive arrays or truly mission-critical data where the cost of failure is astronomical.

Your choice here is a direct trade-off between usable capacity and your operational peace of mind.

Designing Your Vdevs: Practical Rules of the Road

Let’s talk design, because there are a few nuances here that separate a good ZFS setup from a great one.

First, unlike a mirror, the write performance of a single RAID-Z vdev is generally limited by the speed of a single disk. This is a consequence of the read-modify-write cycle and the parity calculations required for every write. It’s the trade-off you accept for that excellent storage efficiency.

Second, and this is a key best practice I build into all my designs: for optimal space efficiency, configure your RAID-Z vdevs so the number of data disks is a power of two. For example:

• A great RAID-Z1 vdev might have 5 disks (4 for data, 1 for parity).
• An excellent RAID-Z2 vdev might have 6 disks (4 for data, 2 for parity) or 10 disks (8 for data, 2 for parity).

This simple rule aligns with ZFS’s internal data structures and minimizes wasted space due to padding. It’s a small detail that pays dividends.

And finally, my unbreakable rule: use identically sized disks within any single vdev. If you mix a 4TB drive with a bunch of 8TB drives, ZFS will treat all of them as 4TB drives. You’re just throwing money and capacity away. Consistency is everything.

The Most Common ZFS ‘Gotcha’: How to Grow Your Pool

This brings us to a critical, operational question that trips up so many people. How do you grow your pool? Many people assume they can simply add one new disk to an existing RAID-Z vdev to increase its size. I need to be crystal clear:

You cannot expand a RAID-Z vdev by adding one disk at a time.

That is not how ZFS works. The correct procedure, and the one you must design for from day one, is to expand your pool by adding another, complete vdev. For example, if your pool currently has one six-disk RAID-Z2 vdev, you expand your storage by adding a second, identical six-disk RAID-Z2 vdev. ZFS will then see both vdevs and intelligently stripe all new data across them, giving you a boost in both capacity and performance.

This is why planning your server chassis layout and initial vdev size is so crucial. You aren’t just building for today; you are mapping out your expansion path for tomorrow.

A Rogues’ Gallery of Common Mistakes

Having seen many systems in the wild, I want to walk you through a few architectural mistakes that are painfully common but easy to avoid with a little foresight.

1. The Single, Massive Vdev: It’s tempting to create one giant RAID-Z2 vdev with, say, 24 disks. On paper, it seems efficient. In practice, should a disk fail, you’re facing a terrifyingly long resilver time, putting immense stress on the remaining drives and leaving yourself with no flexible path for expansion.
2. The ‘RAID-0’ Pool of Death: This is the most dangerous mistake, and something you should never, ever do. Creating a pool from multiple single-disk vdevs is just a stripe set. It has zero redundancy. The failure of any one of those disks guarantees the total and unrecoverable loss of the entire pool.
3. Using RAID-Z1 with Large Drives: I’ll say it again because it’s that important. I always avoid using RAID-Z1 with today’s multi-terabyte drives. The resilver window is simply too long. The risk of a second disk failing during that high-I/O process is one I’m not willing to take, which is why I see RAID-Z2 as the only responsible baseline for modern, capacity-focused arrays.

Tying It All Together

So, let’s bring this home. The key takeaway, if you remember nothing else from today, is this: vdevs are your foundational, redundant building blocks. The pools you create are the usable storage space, and their performance is a direct result of striping data across those vdevs.

I hope it’s clear now that the layout you choose on day one pre-determines everything that follows—your performance, your resilience, and, critically, your future expansion path. These aren’t small decisions, but by understanding these core principles, you are now equipped to make them wisely.

We’ve built a solid, safe foundation for our data. But how do we make it fly? In our next episode, we’ll explore exactly that by diving into ZFS’s incredibly intelligent caching layers. If today was about building a rock-solid foundation, next time is about strapping a rocket to it.

Join me for Episode 16: ‘The ZFS ARC, L2ARC, and SLOG Explained’—it’s where the real performance magic happens.

Until then, thank you for listening! As always, I’d love to hear your thoughts or questions in the comments below.