nimbusdb

WARNING: nimbusdb is in early stages of development; do not use this in production.

Persistent key-value store based on Bitcask paper.

nimbusdb is a fast, lightweight, and scalable key-value store based on Bitcask.

nimbusdb maintains an active datafile to which data is written. When it crosses a threshold, the datafile is made inactive, and a new datafile is created. As time passes, expired or deleted keys take up space that is not useful. Hence, a process called merge is done to remove all expired or deleted keys and free up space.

Features

Thread-Safe

All operations are thread-safe. Read and Write operations can handle multiple operations from multiple goroutines at the same time with consistency.

Portable

Data is extremely portable since it is only a bunch of files. All you have to do is move the folder and open an DB connection at that path.

Custom Expiry

Supports custom expiry for keys. Default expiry is 1 week.

Supports Merge

Supports `Merge` which can be called periodically to remove expired/deleted keys from disk and free-up space.

Supports Batch operations

Batch operations can be performed and committed to save to disk or rollbacked to discard the batch. Operations cannot be performed once the batch is closed.

Single disk-seek write

Writes are just one disk seek since we're appending to the file.

Block cache for faster reads.

Blocks are cached for faster reads. Default size of an Block is 32KB.

Usage

Initialize db connection

d, err := nimbusdb.Open(&nimbusdb.Options{Path: "/path/to/data/directory"})
if err != nil {
  // handle error
}
defer d.Close()

Set

key, err := d.Set([]byte("key"), []byte("value"))
if err != nil {
  // handle error
}

Set with ttl

key, err := d.SetWithTTL([]byte("key"), []byte("value"), time.Second * 10)
if err != nil {
  // handle error
}

Get

value, err := d.Get([]byte("key"))
if err != nil {
  // handle error
}

Delete

key, err := d.Delete([]byte("key"))
if err != nil {
  // handle error
}

Compaction

TODO

• Merge
• Hintfiles

err := d.RunCompaction() // runs Merge
if err != nil {
  // handle error
}

Batch Operations

d, err := nimbusdb.Open(&nimbusdb.Options{Path: "/path/to/data/directory"})
if err != nil {
  // handle error
}
defer d.Close()
b, err := d.NewBatch()
if err != nil {
  // handle error
}
defer b.Close()

key, err = b.Set([]byte("key"), []byte("value")) // not written to disk yet.
if err != nil {
  // handle error
}

key, err := b.Get([]byte("key"))
if err != nil {
  // handle error
}

key, err = b.Delete([]byte("key"))
if err != nil {
  // handle error
}

exists, err := b.Exists([]byte("key"))
if err != nil {
  // handle error
}

b.Commit() // write all pending writes to disk
b.Rollback() // discard all pending writes

Watch keys

func watchKeyChange(ch chan nimbusdb.WatcherEvent) {
  for event := range ch {
    switch event.EventType {
      case "CREATE":
        // Handle create key event
        break

      case "UPDATE":
        // Handle update key event
        break

      case "DELETE":
        // Handle delete key event
        break
    }
  }
}

func main() {
  d, err := nimbusdb.Open(&nimbusdb.Options{Path: "/path/to/data/directory", ShouldWatch: true})
  if err != nil {
    // handle error
  }
  defer d.Close()
  defer d.CloseWatch() // optional
  
  watchChannel, err := d.Watch()
  if err != nil {
    // handle error
  }
  
  go watchEvents(watchChannel)

  key, err := d.Set([]byte("key"), []byte("value")) // will trigger an CREATE event
  if err != nil {
    // handle error
  }

  key, err := d.Set([]byte("key"), []byte("value")) // will trigger an UPDATE event
  if err != nil {
    // handle error
  }

  key, err = d.Delete([]byte("key")) // will trigger an DELETE event
  if err != nil {
    // handle error
  }
}

Progress Board | Streams | godoc