Hello and welcome back to Architecting Zero Downtime Infrastructure! This is episode twenty, and I’m so glad you’re here.

Today, we’re going to dive into something absolutely fundamental to getting the most from your storage: Performance Tuning ZFS. We’ll be focusing on two magic knobs—recordsize and compression—and how they relate directly to you and your data.

Now, ZFS is a powerhouse straight out of the box. Its default settings are a safe middle ground, designed to work reasonably well for almost everyone. But here’s the thing about compromises: they rarely deliver peak performance. A default ZFS setup is like an off-the-rack suit—it fits, but it wasn’t tailored for you. And in my experience, that’s where so much potential is left on the table.

If you’ve ever experienced that frustrating stutter during a video edit, or watched a database crawl to a halt despite running on incredibly powerful hardware, the root cause is often a simple storage misconfiguration. We’re going to fix that. Let’s get that suit tailored.

The Secret Language of ZFS: Copy-on-Write

Before we start turning any knobs, it’s critical we’re on the same page about how ZFS actually thinks about your data. At its very heart is a beautifully resilient mechanism called copy-on-write.

Let me paint a vivid picture for you. Most filesystems are like writers who use a pencil and an eraser. When you change a sentence, they erase the old words and write new ones in their place. ZFS, on the other hand, is like a meticulous accountant using permanent ink. It never overwrites data. When you modify a file, ZFS writes a completely new, updated version of that data to a free spot on the disk. Only then, once it’s safely written, does it update its internal pointers to say, “Hey, the real data is over here now.”

This is what makes snapshots and data integrity so incredible in ZFS. But the piece we need to laser-focus on today is the chunk of data it’s writing. ZFS writes data in blocks called ‘records’. This record is the fundamental unit, the atomic building block of your data on the disk. And the maximum size of that block… well, that’s precisely what the recordsize property controls.

Understanding this is the key that unlocks everything else. It’s the foundation for making smart, impactful tuning decisions.

Your Most Powerful Tuning Knob: recordsize

At its simplest, recordsize dictates the maximum size of a single chunk of data ZFS will write. The default on most systems is 128 kilobytes (128k). It’s a reasonable, all-purpose number, but as we said, it’s rarely optimal.

This is where we get to the core trade-off, and it all comes down to your workload. Think of it like choosing the right size of moving box.

If you’re dealing with large, sequential files—think massive video archives, scientific datasets, or disk images—you want the biggest moving box you can get. For these workloads, I love to see a much larger recordsize, often one megabyte (1M). This allows ZFS to write huge, efficient chunks of data at once, which maximizes throughput and minimizes the overhead of managing metadata. It’s like moving a whole library of giant encyclopedias using a giant pallet—fast and efficient.

But what happens if your workload is the exact opposite? Imagine a busy database server. This is a world of small, random I/O. Now, if you use that giant 1M moving box to change just a few bytes of data, ZFS has to perform a series of painful steps:

1. Read the entire 1-megabyte block from the disk.
2. Change those few tiny bytes in memory.
3. Write a brand new 1-megabyte block back to the disk.

This phenomenon is called read-modify-write amplification, and for random I/O workloads, it’s an absolute performance killer. It’s like using a massive moving box just to move a single screw. For these workloads, you need a much smaller, more precise tool. A recordsize of 16 kilobytes (16k), for example, is far more appropriate.

Matching the Tool to the Job

Let’s make this practical. My personal rule of thumb is to create different ZFS datasets for different kinds of data. Here are three common scenarios:

• Media Server: You’re storing huge video files and reading them from start to finish. This is the definition of sequential. My setting: recordsize=1M. The throughput is simply fantastic.
• Database Server: Precision is everything. An application like PostgreSQL or MySQL often thinks in 8k or 16k blocks. You want to match that. My setting: recordsize=16k. This aligns ZFS with the application, prevents write amplification, and keeps your transaction performance razor-sharp.
• Virtual Machine Host: This is the ultimate mixed workload. You have multiple guest operating systems all doing their own thing. My setting: A smaller recordsize of 64k often provides a very effective balance between decent throughput for large files and low latency for the constant random I/O of an OS.

The Big recordsize Gotcha

Now for a critical detail that trips up so many people. The recordsize property only applies to new data.

You cannot simply change the recordsize on an existing dataset and expect all your old files to magically re-form themselves. They won’t. The data that’s already on disk is stuck with whatever block size it was written with. This is a painfully common pitfall.

So what’s the fix? My process involves a clean migration. You create a new, empty dataset with the correct recordsize you want. Then, you use ZFS’s phenomenal native replication tools—zfs send and zfs receive—to stream a perfect, block-for-block copy from the old dataset to the new one. This process effectively rewrites the data, applying the new setting as it goes. It’s a powerful technique, but it really underscores the importance of planning your storage layouts from the start!

Compression: Your Free Performance Boost

Let’s talk about the second magic knob: compression. There’s an old, persistent myth that enabling compression on a live filesystem will slow everything down. Years ago, with the CPUs of the day, there was a kernel of truth to that. But today? With modern hardware, that thinking is completely obsolete.

In fact, ZFS inline compression is effectively a free performance boost.

Here’s how you should think about it. Your CPU is an F1 race car. Your disks, even your screaming-fast NVMe drives, are a winding country road by comparison. ZFS performs compression in memory, on that F1 racetrack, before the data ever has to get on the slow country road. The tiny fraction of a second your CPU spends making the data smaller is almost always less than the time you save by physically writing less data to the slow disk.

The net result is a win-win-win:

1. Faster Writes: You’re writing less data, so it finishes sooner.
2. Space Savings: This one’s obvious, but it’s a huge benefit.
3. More Effective Cache: Your ARC (ZFS’s main memory cache) can hold far more data in its compressed form, dramatically increasing the odds of a cache hit.

Which Compressor Should You Use?

ZFS gives you choices, and the one you pick matters. Here’s my breakdown:

• lz4: This is the modern standard and my default recommendation for the vast majority of active workloads. It is incredibly fast—so fast you are unlikely to ever notice the CPU overhead. Its compression ratio is very good, and this perfect balance of speed and efficiency makes it the right choice for almost all production systems. My unbreakable rule: compression=lz4 should be on for everything, unless you have a very specific reason to turn it off.
• zstd: If you’re dealing with data where storage density is the absolute top priority—think backups or large archives of cold data—then I look to zstd (Zstandard). It gives you a significantly better compression ratio, but it does ask for a little more from your CPU. For archival purposes, that’s a trade-off I’m very happy to make.
• gzip: You’ll see older options like gzip. While it can compress very tightly, it’s just too slow for most contemporary use cases. In the world of modern ZFS, it’s a bit of a relic. I don’t recommend it for anything active.

Putting It All Together: A Real-World Build

Let’s make this concrete. Imagine we’re provisioning a new server. It needs to host a high-transaction PostgreSQL database and serve as a long-term archive for large video files. These are two completely different workloads, but they can coexist perfectly on the same ZFS pool if we treat them as individuals.

Here’s what I would do. I’d create two separate datasets:

1. For the Database:

Bash

zfs create -o recordsize=16k -o compression=lz4 mypool/database

We’re explicitly setting the 16k recordsize to match PostgreSQL and using the blazingly fast lz4 compression for our active workload.

2. For the Video Archives:

Bash

zfs create -o recordsize=1M -o compression=zstd mypool/archives

Here, we’re setting a large 1M recordsize for that glorious sequential throughput and using the more aggressive zstd compression because space efficiency is paramount for this archival data.

With just two simple commands, we’ve created two specialized storage volumes on the exact same set of disks, each one perfectly tailored for its purpose. This is the real power of ZFS.

My Most Important Rule: Don’t Guess, Test!

Everything we’ve discussed is based on sound principles, but you must validate your assumptions for your specific hardware and workload. So please, don’t guess… test.

Before I change a single setting on a system, my process is always to establish a baseline. You have to know where you’re starting from to know if you’ve actually made an improvement. For this, my go-to tool is fio. It’s the industry standard for a reason. It lets you simulate complex workloads and gives you hard numbers for critical metrics like IOPS and throughput. For a simpler gut-check, you can use a tool like dd for a rough sense of sequential speed.

But the core principle is what matters: Measure first, make one change, then measure again. And always, always do this on non-production data to prove your tuning choices are the right ones before you roll them out.

Tying It All Together

So, to summarize our journey today, it really comes down to a few core ideas:

• Match your recordsize to your workload. It’s the single most impactful tuning you can make, and it costs nothing but a little planning.
• Enable lz4 compression by default. On almost every system I manage, this is a non-negotiable. The performance benefit is real.
• Plan your datasets in advance. Getting the recordsize right from the start saves you a migration headache later.
• And underpinning all of this is the golden rule: Benchmark your results.

Thank you for joining me today! I hope this helps you unlock that hidden potential in your own ZFS systems.

Next time, on Episode 21, we’ll explore ‘Homelab vs. Enterprise: Scaling ZFS for Production Workloads’. We’ll take these principles and see how they apply when you go from a small home server to petabyte-scale enterprise arrays, and we’ll dive into fascinating topics like vdev layouts and special vdevs.

Until then, happy architecting! I’d love to hear your questions or tuning stories in the comments below.