Hello and welcome back to Architecting Zero Downtime Infrastructure! I’m so glad you could join me for episode sixteen.

Today, we’re going to pull back the curtain on one of the biggest secrets to ZFS’s incredible performance. When people talk about ZFS, they often—and rightly so—focus on its world-class data integrity features. But its performance is equally brilliant, and that magic is delivered by an elegant, multi-layered caching system.

You’ve probably heard the terms thrown around: ARC, L2ARC, and SLOG. Let’s be honest, they can seem a bit cryptic at first. What I’ve found, time and time again, is that once you understand the distinct, vital role each one plays, tuning a ZFS system becomes a much more intuitive and rewarding process.

So, that’s our mission today. We’re going to break them down, one by one, in plain English. My goal is simple: to give you the confidence to configure these components correctly for your specific workload, unlocking the maximum performance and reliability your hardware has to offer. Alright, let’s dive in!

The ARC: Your Lightning-Fast RAM Cache

Let’s start with the heart of the system: the ARC, which stands for Adaptive Replacement Cache. This is your primary, first-level cache, and it lives entirely in your system’s RAM. To put it simply, this is the fastest part of the entire ZFS storage hierarchy. By far.

The key word here, and the source of its genius, is “adaptive”. Most traditional caching systems use a simple “Least Recently Used” (LRU) algorithm. The problem with LRU is that it’s… well, a bit naive. A single, massive read operation—like a weekly backup job kicking off—can completely pollute the cache, pushing out all the small, valuable bits of data your applications actually need from moment to moment. It’s a painfully common performance killer.

ZFS is much, much smarter. The ARC maintains a dynamic balance between data that was Most Recently Used (MRU) and data that is Most Frequently Used (MFU). This means it’s intelligent enough to distinguish between a one-time, multi-terabyte file read and a critical 8KB database block that gets hammered a thousand times a minute. It adapts to your workload in real-time.

And remember, it caches more than just read data! It also holds what we call “dirty” write data that’s patiently waiting its turn to be flushed down to your main storage pool.

So, how does it make these decisions? Under the hood, the ARC manages four lists. You have your active caches for MRU and MFU data, but the really clever part is that each of those has a corresponding “ghost list”. Think of a ghost list as a memory of data blocks that were recently evicted from the cache. If a program requests a block that’s currently on a ghost list, the ARC gets a tap on the shoulder. It learns, “Aha! I kicked that data out too soon!” and it adjusts its strategy to favor that type of data more in the future. It’s a self-tuning, self-correcting system.

My process for tuning the ARC:

The main lever you have is the zfs_arc_max parameter, which sets its upper memory limit. On a dedicated storage server, my rule is simple: I let ZFS have almost all the RAM. But if that server is also running virtual machines, containers, or a database, you absolutely must cap the ARC to leave enough memory for those applications to breathe. A common and safe practice is to reserve a few gigabytes for the OS and other critical services, and then give the rest to the ARC. You can easily verify your work with tools like arc_summary or zfs-stats, which let you see your cache hit ratio and confirm your tuning is effective.

The L2ARC: Your Massive Cache Extension

This all works beautifully, of course, but it leads to an obvious architectural question. What happens when your “hot” working data set—the files and blocks your applications need constantly—is simply too large for the amount of RAM you can reasonably install in a server? Your cache hit rate plummets, and performance follows it off a cliff.

This is the exact problem the Level Two ARC, or L2ARC, is designed to solve.

Think of the L2ARC as a vast, secondary read cache. It doesn’t live in your precious system RAM; instead, it lives on a fast, dedicated storage device. My personal go-to for this is a high-quality SSD or, even better, an NVMe drive. This creates a new, intermediate tier in the performance hierarchy. It is, of course, slower than the main ARC in RAM, but it’s an order of magnitude faster than going all the way back to your main pool of spinning disks. It’s an elegant way to bridge that performance gap.

The mechanics are quite direct. When a data block is being pushed out of the main ARC due to memory pressure, ZFS says, “Wait! Before you go, let me write a copy of you over here to the L2ARC device.” This is how the secondary cache gets populated.

But there is a critical trade-off here, a “cost of admission” you have to pay. To find anything in that massive L2ARC, ZFS has to keep a table of contents, and that table lives in your primary ARC… back in your RAM. This metadata consumes a small but non-trivial amount of your fastest resource, a cost you must account for.

My personal rule of thumb is this: only add an L2ARC if your ARC is already full and your ARC hit rate is lower than you’d like. Don’t add an L2ARC if you can just add more RAM!

Finally, a very practical note on reboots. In older ZFS versions, the L2ARC was wiped clean every time the system restarted, which was incredibly inefficient. Thankfully, modern ZFS now supports a persistent L2ARC, so it survives a restart and is immediately effective. This is a massive operational improvement!

The SLOG: Your Synchronous Write Accelerator

So far, we’ve focused entirely on read performance. Let’s shift our attention to the other side of the coin: writes. Because not all writes are created equal, and understanding this difference is fundamental to good design.

• Asynchronous Writes: Most of the time, an application writes data, and ZFS accepts it into the ARC (in RAM) and immediately tells the application, “Got it, job done!” This is incredibly fast because the application doesn’t have to wait for the data to hit the slow disks.
• Synchronous Writes: But there are critical workloads where that’s just not safe enough. Think of a database committing a transaction, an NFS server acknowledging a write, or a block update over an iSCSI LUN. These applications require an absolute guarantee that their data has been committed to stable, persistent storage before they will proceed. This is a synchronous write. And this is where you can hit a serious performance bottleneck if you’re not careful.

This is where the SLOG comes in. The mechanism ZFS uses for these high-integrity writes is called the ZFS Intent Log (ZIL). By default, the ZIL lives right alongside your data on your main storage pool. And if that pool is made of spinning rust, that’s your bottleneck. The application is stuck waiting for a slow, mechanical process.

To fix this, we introduce a Separate Log Device, or SLOG. For these sync-heavy workloads, I add a small, very fast device—typically a high-quality NVMe drive—and dedicate it entirely to hosting the ZIL. Its only job is to receive those urgent synchronous writes at incredible speed, allowing ZFS to immediately send that “committed to stable storage” acknowledgement back to the application. The data is then lazily flushed to the main pool later on.

For database servers, iSCSI targets, or NFS shares, adding a SLOG can be absolutely transformational. It is a targeted, surgical solution for a very specific, and very important, problem.

Let’s be absolutely clear on one thing: a SLOG will only accelerate synchronous writes. If your workload is purely asynchronous (like a large file archive), adding a SLOG will have zero impact on performance. I’ve seen too many people waste money on this!

When it comes to the device itself, here is my unbreakable rule: your SLOG device must have power-loss protection (PLP). This is non-negotiable. Enterprise-grade SSDs and NVMe drives have onboard supercapacitors that guarantee any write acknowledged is truly safe, even during a sudden power failure. A consumer drive without this feature can lie to ZFS, creating a terrifying window for data loss. It’s a risk you simply cannot afford to take. Because this device is so critical, I also always recommend mirroring your SLOG in production environments.

Tying It All Together: An Office Analogy

With all the pieces on the table, let’s trace the journey of a request in our perfectly tuned system. What I find helps is a simple office analogy:

• The ARC is the paperwork on your desk. It’s in RAM, giving you immediate, lightning-fast access.
• The L2ARC is the filing cabinet right behind you. It’s on a fast SSD. It takes a second to turn around and open the drawer, but it’s still very quick and holds way more than your desk.
• The Main Storage Pool is the off-site archive. Everything is there, and it’s vast, but it takes a significant amount of time to retrieve something.
• The SLOG? That’s the secure express courier waiting at the front door. It doesn’t store things long-term, but it provides a guaranteed, high-speed intake for your most urgent and critical documents (your synchronous writes).

When an application requests data, ZFS checks your desk (ARC), then the filing cabinet (L2ARC), and only then sends a request to the off-site archive (Pool).

When a critical document arrives, the courier (SLOG) signs for it instantly, providing guaranteed delivery, and then hands it off to you to place on your desk (ARC) before you eventually file it away in the archive (Pool).

Conclusion

So there you have it. Each of these components has a very distinct, well-defined role. The ARC is your primary cache for ultimate speed. The L2ARC is the massive secondary read cache for when your working set doesn’t fit in memory. And the SLOG is the specialized, high-speed intake built exclusively to handle the demands of synchronous writes.

Mastering the interplay between these three is, in my experience, the difference between a good ZFS system and a truly great one. It’s how you architect for both blistering performance and deep, unwavering resilience.

I really hope this deep dive has been helpful for you. Join me for the next episode where we’ll have a very special guest for an episode titled, Guest Interview: ZFS for the Enterprise with a Core Developer. It’s going to be a fascinating discussion you won’t want to miss.

Until then, thank you for reading, and as always, feel free to leave any questions or comments below!