Skip to content

Commit

Permalink
docs: pebbleds profile
Browse files Browse the repository at this point in the history
  • Loading branch information
lidel committed Oct 2, 2024
1 parent 6a5dae8 commit a868e08
Show file tree
Hide file tree
Showing 4 changed files with 63 additions and 26 deletions.
26 changes: 20 additions & 6 deletions config/profile.go
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,11 @@ You should use this datastore if:
* You want to minimize memory usage.
* You are ok with the default speed of data import, or prefer to use --nocopy.
This profile may only be applied when first initializing the node.
See configuration documentation at:
https://github.com/ipfs/kubo/blob/master/docs/datastores.md#flatfs
NOTE: This profile may only be applied when first initializing node at IPFS_PATH
via 'ipfs init --profile flatfs'
`,

InitOnly: true,
Expand All @@ -153,11 +157,16 @@ You should use this datastore if:
- You need a datastore that is focused on performance.
- You need reliability by default, but may choose to disable WAL for maximum performance when reliability is not critical.
- This datastore is good for multi-terrabyte data sets.
- May benefit from tuning depeending on read/write patterns and throughput.
- Performance is helped significnatly by running on a system with plenty of memory.
- This datastore is good for multi-terabyte data sets.
- May benefit from tuning depending on read/write patterns and throughput.
- Performance is helped significantly by running on a system with plenty of memory.
See configuration documentation at:
https://github.com/ipfs/kubo/blob/master/docs/datastores.md#pebbleds
This profile may only be applied when first initializing the node.`,
NOTE: This profile may only be applied when first initializing node at IPFS_PATH
via 'ipfs init --profile pebbleds'
`,

InitOnly: true,
Transform: func(c *Config) error {
Expand All @@ -181,7 +190,12 @@ Other caveats:
* Good for medium-size datastores, but may run into performance issues
if your dataset is bigger than a terabyte.
This profile may only be applied when first initializing the node.`,
See configuration documentation at:
https://github.com/ipfs/kubo/blob/master/docs/datastores.md#badgerds
NOTE: This profile may only be applied when first initializing node at IPFS_PATH
via 'ipfs init --profile badgerds'
`,

InitOnly: true,
Transform: func(c *Config) error {
Expand Down
11 changes: 6 additions & 5 deletions docs/changelogs/v0.31.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,22 @@

- [Overview](#overview)
- [🔦 Highlights](#-highlights)
- [Include pebble as built-in plugin](#pebble-plugin)
- [Experimental Pebble Datastore](#experimental-pebble-datastore)
- [📝 Changelog](#-changelog)
- [👨‍👩‍👧‍👦 Contributors](#-contributors)

### Overview

### 🔦 Highlights

#### Pebble Plugin
#### Experimental Pebble Datastore

Pebble provides a high-performance alternative to leveldb as the datastore, and will serve as a replacement for badger1.
[Pebble](https://github.com/ipfs/kubo/blob/master/docs/config.md#pebbleds-profile) provides a high-performance alternative to leveldb as the datastore, and provides a modern replacement for [legacy badgerv1](https://github.com/ipfs/kubo/blob/master/docs/config.md#badgerds-profile).

There are a number of tuning parameters available for tuning pebble's performance to your specific needs. Default values are used for any parameters that are not configured or are set to their zero-value.
A fresh Kubo node can be initialized with [`pebbleds` profile](https://github.com/ipfs/kubo/blob/master/docs/config.md#pebbleds-profile) via `ipfs init --profile pebbleds`.

For a description of the available tuning parameters, see: https://github.com/ipfs/kubo/blob/master/docs/datastores.md#pebbleds
There are a number of parameters available for tuning pebble's performance to your specific needs. Default values are used for any parameters that are not configured or are set to their zero-value.
For a description of the available tuning parameters, see [kubo/docs/datastores.md#pebbleds](https://github.com/ipfs/kubo/blob/master/docs/datastores.md#pebbleds).

### 📝 Changelog

Expand Down
26 changes: 19 additions & 7 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -523,8 +523,8 @@ Spec defines the structure of the ipfs datastore. It is a composable structure,
where each datastore is represented by a json object. Datastores can wrap other
datastores to provide extra functionality (eg metrics, logging, or caching).

For more information on possible values for this configuration option, see
[docs/datastores.md](datastores.md)
> [!NOTE]
> For more information on possible values for this configuration option, see [`kubo/docs/datastores.md`](datastores.md)
Default:
```
Expand Down Expand Up @@ -2397,9 +2397,9 @@ Inverse profile of the test profile.

### `flatfs` profile

Configures the node to use the flatfs datastore. Flatfs is the default datastore.
Configures the node to use the flatfs datastore.
Flatfs is the default, most battle-tested and reliable datastore.

This is the most battle-tested and reliable datastore.
You should use this datastore if:

- You need a very simple and very reliable datastore, and you trust your
Expand All @@ -2410,7 +2410,11 @@ You should use this datastore if:
- You want to minimize memory usage.
- You are ok with the default speed of data import, or prefer to use `--nocopy`.

This profile may only be applied when first initializing the node.
> [!WARNING]
> This profile may only be applied when first initializing the node via `ipfs init --profile flatfs`

> [!NOTE]
> See caveats and configuration options at [`datastores.md#flatfs`](datastores.md#flatfs)

### `pebbleds` profile

Expand All @@ -2425,7 +2429,11 @@ You should use this datastore if:
- May benefit from tuning depeending on read/write patterns and throughput.
- Performance is helped significnatly by running on a system with plenty of memory.

This profile may only be applied when first initializing the node.
> [!WARNING]
> This profile may only be applied when first initializing the node via `ipfs init --profile pebbleds`

> [!NOTE]
> See other caveats and configuration options at [`datastores.md#pebbleds`](datastores.md#pebbleds)

### `badgerds` profile

Expand All @@ -2446,7 +2454,11 @@ Also, be aware that:
- Good for medium-size datastores, but may run into performance issues if your dataset is bigger than a terabyte.
- The current implementation is based on old badger 1.x which is no longer supported by the upstream team.

This profile may only be applied when first initializing the node.
> [!WARNING]
> This profile may only be applied when first initializing the node via `ipfs init --profile badgerds`

> [!NOTE]
> See other caveats and configuration options at [`datastores.md#pebbleds`](datastores.md#pebbleds)

### `lowpower` profile

Expand Down
26 changes: 18 additions & 8 deletions docs/datastores.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@
This document describes the different possible values for the `Datastore.Spec`
field in the ipfs configuration file.

- [flatfs](#flatfs)
- [levelds](#levelds)
- [pebbleds](#pebbleds)
- [badgerds](#badgerds)
- [mount](#mount)
- [measure](#measure)

## flatfs

Stores each key value pair as a file on the filesystem.
Expand Down Expand Up @@ -39,7 +46,15 @@ Uses a leveldb database to store key value pairs.

Uses [pebble](https://github.com/cockroachdb/pebble) as a key value store.

#### The following options are availble for tuning pebble. If they are not configured (or assigned their zero-valued), then default values are used. Start using pebble with only default values and configure tuning items are needed for your needs. For a more complete description of these values, see: https://pkg.go.dev/github.com/cockroachdb/pebble@v1.1.2#Options
```json
{
"type": "pebbleds",
"path": "<location of pebble inside repo>",
}
```

The following options are availble for tuning pebble.
If they are not configured (or assigned their zero-valued), then default values are used.

* `bytesPerSync`: int, Sync sstables periodically in order to smooth out writes to disk. (default: 512KB)
* `bisableWAL`: true|false, Disable the write-ahead log (WAL) at expense of prohibiting crash recovery. (default: false)
Expand All @@ -53,13 +68,8 @@ Uses [pebble](https://github.com/cockroachdb/pebble) as a key value store.
* `walBytesPerSync`: int: Sets the number of bytes to write to a WAL before calling Sync on it in the background. (default: 0, no background syncing)
* `walMinSyncSeconds`: int: Sets the minimum duration between syncs of the WAL. (default: 0)


```json
{
"type": "pebbleds",
"path": "<location of pebble inside repo>",
}
```
> [!TIP]
> Start using pebble with only default values and configure tuning items are needed for your needs. For a more complete description of these values, see: `https://pkg.go.dev/github.com/cockroachdb/pebble@vA.B.C#Options` (where `A.B.C` is pebble version from Kubo's `go.mod`).
## badgerds

Expand Down

0 comments on commit a868e08

Please sign in to comment.