Flash Disk - Support

[webclinic017]

Is there a planned support for flash disk / drive - based support - akin to redis / key db

https://github.com/szstonelee/redrock [ RedRock automatically converts data that exceeds memory to disk storage. In this way, hot data is in memory to ensure access speed, and cold and warm data is in disk and supports real-time reading and writing. RedRock does automatic hot and cold conversion for this purpose, thereby greatly saving hardware costs. ]

https://redis.com/redis-enterprise/technology/redis-on-flash/ [ Instead of keeping the entire dataset in DRAM, RoF holds only the keys and the Redis dictionary (the data structure behind the keys) in DRAM (the faster tier), along with the frequently accessed data or the “hot data” of the dataset (the working set). The “warm values,” the inactive data, are moved to the lower tier, the local Flash storage tier. ]

https://docs.keydb.dev/docs/coming-soon/#keydb-flash-support [ Support for KeyDB FLASH using the RocksDB engine will become available September 2022. ]

I think it will be a great addition to the feature set - by using RocksDB

[romange]

We have some thoughts about doing this, though we are planning to take a different approach to the aforementioned projects.
It's not gonna happen before September 2023 though :)

[JohnDotOwl]

It's September 2023, and keydb's flash storage solution still makes it an appealing option for some use cases. Even though keydb doesn't offer the ultra-low latency of some other databases, there are applications where somewhat higher latency is acceptable in order to take advantage of keydb's other benefits. As long as latency requirements aren't too demanding, keydb's flash storage can provide a cost-effective solution for the right workloads.

[gulldan]

any news here?

[doptime]

always use nvme 2.0 key-value storage.
It cut down disk io to 1/6, compared with other db according to samsung's early study.
It reduce flash write amplification to 1.0, while redis rdb has a write amplification ratio over 1000.
It's fast, typically, it can provide more than 2M+ iops with lowest latency. Some studies shows one nvme key-value disk can relace 50 xeon kernels that running rocksdb, using about 1/100 power and zero system memory.
nvme 2.0 kv disk can get LINEARY PERFORMANCE GROWTH while number of disk increase, while other db usually get a log-like performance growth using same single db interface.
it never lost data,once data send to disk.
It won't create a db file like rocksdb which always expanding its size, and awkward to shrink size.
there is RocksDB compatible driver of kv storage
It's cheap and easy available when time comes to 2023.

[stephenleo]

Are there still plans to build this feature?

[romange]

yes

[Athinivas]

HI, when can we expect the support for flash disk / drive - based support - akin to redis / keydb? could you share a possible timeline for same..

[bitactive]

@romange What is current state of Flash tiering? We have seen multiple commits about that feature, some of them incorporated to release code, some of them not. Tried running it, but with no luck:
ERROR: Unknown command line flag 'max_file_size'
ERROR: Unknown command line flag 'spill_file_prefix'

Will Flash Tiering support expire? We are considering using it for storing millions of web sessions where only few % will be hot and 90%+ will be cold data.

Do you have any timeline for this?

[romange]

@bitactive end of Q2

[Athinivas]

@romange do we have any update on the flash tiering?

[romange]

Most of the functionality is there, it's not prod ready yet but you can already try it out with --tiered_prefix=some_nvme_path .

Works with newer kernels - 5.19 or later.

[vandarkent100]

Can you please confirm that the work on Flash tiering support is on track to be completed by the end of Q2?

My team would benefit from full Flash tiering support compatible with RedisSearch and RedisJSON, allowing us to unload data from RAM and run the search queries involving indexed data stored on the disk without the need to load them to the RAM.

(KeyDB seems to have problems with GlobalLocksAcquired(), resulting in the assertion failure)

[dranikpg]

Tiering is currently in active development, see the previous reply about timelines

#269 (reply in thread)

would benefit from full Flash tiering support compatible with RedisSearch and RedisJSON

We currently do not plan to provide support for persisting search indices to disk. This means, that is most of the memory is used for storing the entries themselves with only ligh numeric indices, you will benefit from our future release. If instead your memory constraints are due to large string indices, then we sadly can't help you in the near future 🙂

[vandarkent100]

It appears there might be some confusion regarding your response.

We are utilizing RedisSearch and RedisJSON modules to store data in a JSON format, such as:

key:1 : {"name": "John", "month":"June"}

Subsequently, we index each JSON field and execute FT.SEARCH / FR.AGGREGATE queries to retrieve keys matching specific patterns.

Our objective is to offload a significant portion of the data onto disk while retaining the ability to execute search queries that access indexed data stored on disk, facilitated by auto-tiering flash storage.

If I interpret your statement "We currently do not plan to provide support for persisting search indices to disk" correctly, achieving this objective might not be feasible. I would appreciate clarification if my understanding is incorrect.

Could you please rephrase the following section of your message to enhance clarity?

"This implies that if the majority of memory usage pertains to storing the entries themselves with primarily lightweight numeric indices, you stand to benefit from our forthcoming release. Conversely, if your memory constraints are predominantly attributable to substantial string indices, regrettably, we are unable to offer assistance in the immediate future."

Thank you for your assistance!

[JohnDotOwl]

Can you please confirm that the work on Flash tiering support is on track to be completed by the end of Q2?

My team would benefit from full Flash tiering support compatible with RedisSearch and RedisJSON, allowing us to unload data from RAM and run the search queries involving indexed data stored on the disk without the need to load them to the RAM.

(KeyDB seems to have problems with GlobalLocksAcquired(), resulting in the assertion failure)

Snapchat killed keydb

[vandarkent100]

I have noticed, the project seems to be dead.

Is Dragonfly DB trying to fill the gap and become a production ready in memory database with the fully operational auto-tiering (flash support)?

[romange]

The SSD tiering support has been release in alpha. See https://github.com/dragonflydb/dragonfly/releases/tag/v1.21.1
Subscribe to our community connect tomorrow to hear more.