Storage Disaggregation (Compute-Storage Separation)

Introduction

Traditional database and storage architectures couple compute and storage on the same node.
A query engine reads from locally attached disks, and scaling either resource means scaling both.
This tight coupling made sense when network bandwidth was the bottleneck, but modern datacenter networks (25–100 Gbps per host, with RDMA options) have shifted the economics.
Storage disaggregation separates compute and storage into independently scalable tiers, connected by a fast network fabric.

This architectural pattern now underpins systems like Amazon Aurora, Snowflake, Google BigQuery, Neon, and numerous object-store-backed analytics engines.
Understanding its mechanics, tradeoffs, and failure modes is essential for engineers designing or operating large-scale data systems.

Architecture Overview

In a disaggregated architecture, the system is split into at least two tiers:

1. Compute tier: Stateless (or soft-state) nodes that execute queries, run transactions, and manage in-memory caches. These nodes hold no durable data.
2. Storage tier: A shared, durable storage layer that persists data. This can be a purpose-built distributed storage service (Aurora's storage nodes, for example) or a commodity object store like S3.

A third tier often exists in practice: a metadata/coordination service that tracks data placement, schema, transaction state, and access control.

Key Architectural Properties

Independent elasticity. Compute can scale out for query load without provisioning additional storage, and storage can grow without adding CPU.
This decoupling is the primary economic motivation.

Stateless compute. Because compute nodes hold no durable state, they can be added, removed, or replaced without data migration.
Failure recovery is simplified: a crashed compute node loses its cache but no committed data.

Shared storage. Multiple compute nodes can read from the same storage layer, enabling multi-reader architectures.
Write coordination requires careful concurrency control, but reads scale naturally.

How It Works in Practice

Write Path

The write path in a disaggregated system typically follows one of two patterns.

Log-shipping (Aurora model). The compute node generates redo log records and ships them to the storage tier.
The storage tier applies these log records asynchronously to materialize data pages.
The database's write-ahead log (WAL) becomes the primary communication channel.
Aurora's key insight is that "the log is the database": the compute layer never writes data pages to storage, only log entries.
This reduces network I/O significantly compared to shipping full pages.
For typical OLTP workloads, log records are substantially smaller than the pages they modify, though the exact ratio is workload-dependent.

Stage-and-commit (Snowflake/Lakehouse model). The compute node writes output files (often in columnar formats like Parquet or ORC) to an object store, and then atomically registers them in a metadata catalog.
There is no traditional WAL shipped to storage.
Instead, immutable files are written and a metadata transaction records their existence.
This approach suits analytical workloads where data is ingested in bulk.

Read Path

Reads in a disaggregated system depend heavily on caching.
The network round-trip to remote storage (even at 100 Gbps) has higher latency than a local NVMe read.
Systems compensate with:

• Local page/block caches on compute nodes, often backed by local SSDs used purely as cache (not durable storage).
• Prefetching and read-ahead driven by query plans. The query optimizer can issue storage requests before data is needed.
• Tiered caching with memory, local SSD, and remote storage as three levels.

Cache hit rates above 95–99% are common for OLTP workloads with temporal locality.
When the working set fits in the compute tier's aggregate cache, remote storage latency is largely hidden.

Failure Handling

Compute node failure is straightforward: restart the node (or start a new one), and rebuild caches from storage.
No data is lost because no durable state resided on the failed node.

Storage tier failure requires the storage layer itself to be replicated and fault-tolerant.
Aurora organizes data into 10 GB protection group segments, each replicated across six storage nodes in three availability zones.
It uses a 4-of-6 write quorum and a 3-of-6 read quorum.
The quorum sizes are chosen so that write quorum + read quorum = 7 > 6, which by the pigeonhole principle guarantees that any read set of 3 nodes must overlap with any write set of 4 nodes by at least one node — ensuring at least one reader always has the latest acknowledged write.
In practice, Aurora does not issue parallel reads to 3 nodes for every page request; instead, it reads from a single storage node selected based on the system's consistency tracking (the Volume Complete LSN mechanism), falling back to additional nodes only when needed.
The 3-of-6 read quorum defines the fault-tolerance boundary, not the typical read fanout.
The compute tier is insulated from storage failures as long as the storage service meets its quorum requirements.

Walkthrough

The following walkthrough illustrates the lifecycle of a write and read operation in a log-shipping disaggregated database (modeled after Aurora).

Write Operation

1. Client sends: INSERT INTO orders VALUES (...)
2. Compute node acquires row lock, modifies page in buffer cache.
3. Compute node generates WAL record R for the modification.
4. Compute node sends R to storage tier (6 replicas across 3 AZs).
5. Storage nodes persist R to local durable log.
6. Once 4 of 6 storage nodes acknowledge, write is durable.
7. Compute node acknowledges commit to client.
8. Storage nodes asynchronously apply R to their copy of the
   data page, materializing the updated page in the background.

Note that step 8 happens lazily.
The compute node never sends the full data page over the network — only the WAL record.
For typical OLTP workloads, log records are substantially smaller than the pages they modify, making this a critical bandwidth optimization.

Read Operation (Cache Miss)

1. Query execution requests page P.
2. Compute node checks local buffer cache. Miss.
3. Compute node sends read request to storage tier for page P.
4. Storage node receives request. If page P has unapplied log
   records, it applies them now (on-demand materialization).
5. Storage node returns the up-to-date page P.
6. Compute node caches page P in buffer cache.
7. Query execution proceeds with page P.

On-demand materialization (step 4) ensures that even if background application of log records is behind, reads always return current data.

Tradeoffs and Challenges

Network as the New Bottleneck

Disaggregation trades local disk I/O for network I/O.
While modern networks are fast, they introduce new failure modes (congestion, packet loss, network partitions) into the storage path.
Tail latency at the P99 level is often dominated by network jitter rather than disk latency.

Cache Warmup and Cold Starts

A newly started compute node has an empty cache.
For OLTP workloads, this cold-start period can cause significant latency spikes.
Some systems (Aurora, for example) address this by pre-warming caches from a page log or by transferring cache state from a peer replica during failover.

Cost Model Complexity

Disaggregation changes the cost equation.
Object store reads are cheap but not free.
A poorly optimized query in Snowflake can scan terabytes from S3, incurring both latency and monetary cost.
Engineers must understand that "storage is cheap" does not mean "access is cheap."

Consistency and Coordination

When multiple compute nodes share the same storage layer, write coordination becomes complex.
Systems address this differently:

• Single-writer architectures (Aurora primary) avoid the problem by routing all writes through one node.
• Multi-writer architectures (Google Spanner, purpose-built disaggregated databases, or purpose-built disaggregated databases) require distributed transaction protocols (2PC, Paxos, or MVCC with centralized timestamp oracles).
• Append-only architectures (Snowflake, Delta Lake) sidestep page-level conflicts by writing immutable files and coordinating at the metadata/catalog level.

Storage Amplification

In log-shipping systems, the storage tier must maintain both the log and the materialized pages, creating storage amplification.
Garbage collection of old log records is necessary to bound space usage, adding operational complexity.

When to Use Disaggregation

Storage disaggregation is not universally optimal.
It is well-suited to:

• Cloud-native systems where elasticity and pay-per-use pricing are priorities.
• Bursty workloads that need to scale compute up temporarily without permanently provisioning storage.
• Multi-tenant platforms where many compute instances share a common data layer.
• Analytical workloads with large datasets and intermittent query patterns.

It is less suitable for ultra-low-latency workloads (sub-millisecond requirements) where any network hop is unacceptable, or for edge deployments with unreliable networks.

Key Points

• Storage disaggregation separates compute and storage into independently scalable tiers connected by a fast network, enabling elastic scaling of each resource independently.
• The write path can follow either a log-shipping model (sending WAL records to storage) or a stage-and-commit model (writing immutable files to an object store and updating metadata).
• Caching on the compute tier is critical to hiding remote storage latency, with local SSDs often serving as a non-durable cache layer.
• Compute node failures are simplified because no durable state is lost, but storage tier failures must be handled by replication within the storage service itself.
• Aurora's 4-of-6 write quorum and 3-of-6 read quorum guarantee overlap (4+3 > 6), so that reads always intersect the latest write set; in normal operation, Aurora reads from a single storage node rather than issuing parallel quorum reads.
• Network tail latency, cache cold starts, and I/O cost management are the primary operational challenges introduced by disaggregation.
• Multi-reader/multi-writer coordination over shared storage requires careful concurrency control, with most systems choosing single-writer or append-only designs to reduce complexity.
• The architecture is most effective in cloud environments with bursty, elastic workloads and less suitable for latency-critical edge deployments.

References

Verbitski, A., Gupta, A., Slichter, D., et al. "Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases." Proceedings of the 2017 ACM International Conference on Management of Data (SIGMOD), 2017.

Dageville, B., Cruanes, T., Zukowski, M., et al. "The Snowflake Elastic Data Warehouse." Proceedings of the 2016 ACM International Conference on Management of Data (SIGMOD), 2016.

Armbrust, M., Das, T., Sun, L., et al. "Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores." Proceedings of the VLDB Endowment, Vol. 13, No. 12, 2020.

Verbitski, A., Gupta, A., Slichter, D., et al. "Amazon Aurora: On Avoiding Distributed Consensus for I/Os, Commits, and Membership Changes." Proceedings of the 2018 ACM International Conference on Management of Data (SIGMOD), 2018.

Vuppalapati, M., et al. "Building An Elastic Query Engine on Disaggregated Storage." Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2020.