Skip to the stories
AI & ML

Kvrocks on RocksDB: How Redis-Compatible Data Moves to Disk

  • Martin Visser
  • 1,003 reads

Kvrocks keeps the Valkey and Redis command model while storing data in RocksDB. Cache hits, write amplification and collection scans shape its performance.

Kvrocks on RocksDB: How Redis-Compatible Data Moves to Disk

Kvrocks changes the cost and performance equation for Valkey and Redis by storing data in RocksDB instead of keeping the full dataset in memory. It still speaks the RESP protocol, so existing Valkey and Redis clients can connect. The trade-off is that disk access and background compaction now influence response times alongside the memory caches.

That design targets deployments where datasets have grown large enough that provisioning RAM dominates the infrastructure bill. Kvrocks is an Apache Software Foundation project, open source since 2019 under the Apache 2.0 license. The project lists production users including Baidu and Trip.com, particularly in Asia. Its compatibility makes it a candidate for workloads that rely on familiar commands and clients, though the fit depends on access patterns and latency needs.

The system is easiest to understand as a command layer over RocksDB. Kvrocks translates RESP commands into ordered byte-string keys and values; RocksDB handles persistence, caching, replication logs and the process of reorganizing data on disk. This architecture makes workload behavior central: a frequently accessed dataset may be served largely from memory, while cold reads reach storage and sustained writes create compaction work.

From commands to RocksDB rows

Valkey supports strings, hashes, lists, sets, sorted sets and other structures. RocksDB itself stores only ordered keys and values, so Kvrocks represents those higher-level types using two main patterns. A string is straightforward: its value is stored with its key. A collection, such as a hash, has a metadata row plus one row for each element.

For example, a hash called user:1278 has metadata describing its type, version and element count. If it has a time-to-live, that information is stored there too. Each field gets a separate RocksDB row, conceptually named user:1278|v1|first or user:1278|v1|email. The visible separators are illustrative; the real encoding uses length prefixes and adds a namespace and, in cluster mode, a slot identifier.

The version is generated when a collection is created. It ties its element rows to the corresponding metadata and makes large deletes inexpensive. To delete a hash with many fields, Kvrocks can remove its single metadata row rather than immediately writing a tombstone for every field. Since reads first consult metadata, the collection becomes inaccessible as soon as that row is gone.

If the same key is created again, its new metadata receives a different version. Its fields therefore use a new version prefix, leaving old rows unable to match the live collection. Those orphaned rows still occupy disk space until RocksDB compaction removes them. Kvrocks uses a compaction filter to discard subkey rows when their metadata is missing, their version no longer matches, or the collection has expired.

Lists use head and tail indices in metadata, which supports efficient pushes and pops at either end. Sorted sets keep an additional score index, allowing range-by-score queries to seek through ordered data instead of scanning every member. Multi-row changes are sent to RocksDB as a WriteBatch, which applies the batch atomically.

This deletion model differs from Valkey’s. Valkey removes a key from its keyspace and can free a large collection’s memory in a background thread; Kvrocks makes the collection logically absent at once, then reclaims its disk space during compaction. Applications that repeatedly delete and recreate large collections should account for the delay between logical deletion and physical cleanup.

Column families and the keyspace

Kvrocks divides data among RocksDB column families, each with its own memtables and sorted string table (SST) files. The metadata family holds key types, expiry information, versions and element counts; for strings, it also holds the value. The default family stores collection elements. Other families hold sorted-set scores, stream entries, search indexes for FT.* commands (including HNSW vector indexes), time-series label indexes, and Pub/Sub messages.

A further family stores Lua scripts, functions and namespace tokens, as well as commands that need to reach replicas but are not data writes. The separation lets RocksDB organize different kinds of records independently while keeping them within the same storage engine.

One consequential difference from Valkey is the absence of an in-memory dictionary containing every key. In Valkey, the keyspace itself has a RAM-resident hash table. In Kvrocks, the LSM tree is the keyspace: metadata rows can remain on disk, with memory allocated to the portions of the database that benefit from caching. A very large key count therefore does not require a matching number of in-memory dictionary entries.

Writes, logs and compaction

When a command such as HSET user:1278 first bob arrives, Kvrocks encodes it as metadata and subkey changes. RocksDB writes the batch to its shared write-ahead log (WAL), then places the records into the relevant column-family memtables. The command is acknowledged after that write path completes. The WAL also supplies the stream of changes sent to replicas.

The WAL is written by default without syncing every individual write to durable storage. In the source configuration, this is compared with Valkey’s appendfsync no: a Kvrocks process crash can leave the write recoverable, while a power failure may lose it. Setting rocksdb.write_options.sync yes requests a sync for each write, corresponding to Valkey’s appendfsync always behavior.

Replication is asynchronous. A write can be acknowledged before replicas receive it, so the primary’s response does not mean every replica has caught up. For applications that rely on replica freshness or need stronger durability, these settings and timing details matter.

When a memtable fills, RocksDB flushes it to an L0 SST file. Newer and smaller files start at L0; over time, background compaction merges data into lower levels. That process retains the latest version of each row and removes obsolete, deleted or expired records. It also means a row may be rewritten multiple times as it moves through the tree, a cost known as write amplification.

Write amplification is a key operational consideration, particularly for large values and heavy write workloads. Compaction consumes storage bandwidth and processing while application traffic continues. If that background work cannot keep up, write behavior and latency can suffer. The write rate, data size and compaction capacity all contribute to the result.

Reads depend on cache misses

For a point read such as HGET user:1278 first, Kvrocks first looks up metadata to check the type, version and expiry. It then constructs the subkey for the requested field. RocksDB checks memtables first; if the row is not there, it consults SST files, starting with the newest relevant data. Bloom filters can rule out files that definitely do not contain the key.

If a file might hold the record, RocksDB needs its index and data blocks. Those may already be in the block cache, or they may need to be read from the filesystem. An operating system page-cache hit can satisfy that read from RAM; otherwise, storage must serve it. The practical performance question is therefore how often requests miss the in-memory caches and how quickly the storage handles those misses. “On disk” does not mean memory stops mattering.

Range reads behave differently. A command such as HGETALL can scan a collection’s rows because their shared key and version place them next to each other in RocksDB’s ordering. The scan proceeds from the first matching row until it reaches the end of the collection. Bloom filters offer little help here: they test for exact keys, not whether a file contains a run of related rows.

Kvrocks also avoids adding blocks read during a scan to its own block cache, which protects frequently used blocks from being displaced by a large collection scan. The trade-off is that repeated scans rely more on the operating system’s page cache. By contrast, multi-field point reads such as HMGET act like batches of individual lookups. Two databases with equally sized datasets can therefore perform quite differently depending on whether clients fetch individual fields or scan whole collections.

Cluster topology and migration

Kvrocks follows the Valkey Cluster model of 16,384 hash slots distributed across shards, with replicas for each shard. Cluster-aware clients can use the familiar model, and multi-key operations require the keys to share a slot. Hash tags provide the same way to keep related keys together.

Topology management differs. Valkey Cluster nodes use gossip to share information with one another. Kvrocks does not use that protocol; instead, a Kvrocks controller owns the topology and pushes it to nodes through CLUSTERX SETNODES. The controller handles failover and stores its state in an external system such as etcd.

To add capacity, operators add a shard and migrate slots to it. Data is copied while writes continue. At the end, the system briefly pauses to apply final changes and transfer slot ownership. That pause is relevant for services with tight tail-latency targets during rebalancing.

There is also a format implication: cluster-mode data includes the slot identifier, while standalone data does not. Moving a standalone node into a cluster therefore requires a data migration. It is not simply a matter of changing the node’s topology settings.

What determines the fit

Kvrocks shifts the main capacity question from “how much RAM holds the whole dataset?” to “how much of the active workload can the caches serve, and what happens when they miss?” Read-heavy workloads with useful cache locality may avoid many storage reads. Workloads that frequently touch cold data will depend more directly on storage speed and the system’s ability to handle concurrent misses.

Writes bring a second constraint: each update can be rewritten as compaction moves records through the LSM tree. A high write rate creates more background work, and that work competes with application traffic for storage resources. Large collection scans add another pattern, since they bypass Kvrocks’ block cache and lean on the operating system’s page cache.

That’s why dataset size alone can’t predict performance. Access locality, the mix of point reads and range scans, write volume, storage speed and cache capacity all shape the outcome. Teams evaluating Kvrocks should measure those patterns against their own workload and include deletion cleanup and cluster migration in operational planning. The central trade is clear: less pressure to keep every key in RAM, in exchange for a storage system whose caches and compaction behavior become part of application performance.

Source: Martin Visser · www.percona.com

From the same set