-
-
Notifications
You must be signed in to change notification settings - Fork 176
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
7d64a47
commit 729a59b
Showing
42 changed files
with
2,939 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
name: "mkdocs" | ||
on: | ||
push: | ||
branches: | ||
- latest-release | ||
permissions: | ||
contents: write | ||
jobs: | ||
deploy: | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Configure Git Credentials | ||
run: | | ||
git config user.name github-actions[bot] | ||
git config user.email 41898282+github-actions[bot]@users.noreply.github.com | ||
- uses: actions/setup-python@v5 | ||
with: | ||
python-version: 3.x | ||
- run: pip install -r requirements.txt | ||
working-directory: ./mkdocs | ||
- run: mkdocs gh-deploy --force | ||
working-directory: ./mkdocs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -23,3 +23,8 @@ debian/changelog | |
|
||
# RPM files | ||
rpmbuild/ | ||
|
||
# mkdocs files | ||
mkdocs/env | ||
mkdocs/.env | ||
__pycache__/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# mkdocs | ||
|
||
## Getting started | ||
|
||
```bash | ||
python3 -m venv env | ||
source env/bin/activate | ||
pip3 install -r requirements.txt | ||
mkdocs serve | ||
``` | ||
|
||
## References | ||
|
||
- https://squidfunk.github.io/mkdocs-material/ | ||
- https://www.mkdocs.org/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
# mergerfs - a featureful union filesystem | ||
|
||
## DESCRIPTION | ||
|
||
**mergerfs** is a union filesystem geared towards simplifying storage | ||
and management of files across numerous commodity storage devices. It | ||
is similar to **mhddfs**, **unionfs**, and **aufs**. | ||
|
||
## FEATURES | ||
|
||
- Configurable behaviors / file placement | ||
- Ability to add or remove filesystems at will | ||
- Resistance to individual filesystem failure | ||
- Support for extended attributes (xattrs) | ||
- Support for file attributes (chattr) | ||
- Runtime configurable (via xattrs) | ||
- Works with heterogeneous filesystem types | ||
- Moving of file when filesystem runs out of space while writing | ||
- Ignore read-only filesystems when creating files | ||
- Turn read-only files into symlinks to underlying file | ||
- Hard link copy-on-write / CoW | ||
- Support for POSIX ACLs | ||
- Misc other things | ||
|
||
## SYNOPSIS | ||
|
||
mergerfs -o<options> <branches> <mountpoint> | ||
|
||
## DOCUMENTATION | ||
|
||
- [https://oregonpillow.github.io/](https://oregonpillow.github.io/) | ||
|
||
## TOOLS | ||
|
||
- [mergerfs tools](https://github.com/trapexit/mergerfs-tools) |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
# BASIC SETUP | ||
|
||
If you don't already know that you have a special use case then just | ||
start with one of the following option sets. | ||
|
||
#### You need `mmap` (used by rtorrent and many sqlite3 base software) | ||
|
||
`cache.files=auto-full,dropcacheonclose=true,category.create=mfs` | ||
|
||
or if you are on a Linux kernel >= 6.6.x mergerfs will enable a mode | ||
that allows shared mmap when `cache.files=off`. To be sure of the best | ||
performance between `cache.files=off` and `cache.files=auto-full` | ||
you'll need to do your own benchmarking but often `off` is faster. | ||
|
||
#### You don't need `mmap` | ||
|
||
`cache.files=off,dropcacheonclose=true,category.create=mfs` | ||
|
||
### Command Line | ||
|
||
`mergerfs -o cache.files=auto-full,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media` | ||
|
||
### /etc/fstab | ||
|
||
`/mnt/hdd0:/mnt/hdd1 /media mergerfs cache.files=auto-full,dropcacheonclose=true,category.create=mfs 0 0` | ||
|
||
### systemd mount | ||
|
||
https://github.com/trapexit/mergerfs/wiki/systemd | ||
|
||
``` | ||
[Unit] | ||
Description=mergerfs service | ||
[Service] | ||
Type=simple | ||
KillMode=none | ||
ExecStart=/usr/bin/mergerfs \ | ||
-f \ | ||
-o cache.files=auto-full \ | ||
-o dropcacheonclose=true \ | ||
-o category.create=mfs \ | ||
/mnt/hdd0:/mnt/hdd1 \ | ||
/media | ||
ExecStop=/bin/fusermount -uz /media | ||
Restart=on-failure | ||
[Install] | ||
WantedBy=default.target | ||
``` | ||
|
||
See the mergerfs [wiki for real world | ||
deployments](https://github.com/trapexit/mergerfs/wiki/Real-World-Deployments) | ||
for comparisons / ideas. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
# BENCHMARKING | ||
|
||
Filesystems are complicated. They do many things and many of those are | ||
interconnected. Additionally, the OS, drivers, hardware, etc. can all | ||
impact performance. Therefore, when benchmarking, it is **necessary** | ||
that the test focuses as narrowly as possible. | ||
|
||
For most throughput is the key benchmark. To test throughput `dd` is | ||
useful but **must** be used with the correct settings in order to | ||
ensure the filesystem or device is actually being tested. The OS can | ||
and will cache data. Without forcing synchronous reads and writes | ||
and/or disabling caching the values returned will not be | ||
representative of the device's true performance. | ||
|
||
When benchmarking through mergerfs ensure you only use 1 branch to | ||
remove any possibility of the policies complicating the | ||
situation. Benchmark the underlying filesystem first and then mount | ||
mergerfs over it and test again. If you're experiencing speeds below | ||
your expectation you will need to narrow down precisely which | ||
component is leading to the slowdown. Preferably test the following in | ||
the order listed (but not combined). | ||
|
||
1. Enable `nullrw` mode with `nullrw=true`. This will effectively make | ||
reads and writes no-ops. Removing the underlying device / | ||
filesystem from the equation. This will give us the top theoretical | ||
speeds. | ||
2. Mount mergerfs over `tmpfs`. `tmpfs` is a RAM disk. Extremely high | ||
speed and very low latency. This is a more realistic best case | ||
scenario. Example: `mount -t tmpfs -o size=2G tmpfs /tmp/tmpfs` | ||
3. Mount mergerfs over a local device. NVMe, SSD, HDD, etc. If you | ||
have more than one I'd suggest testing each of them as drives | ||
and/or controllers (their drivers) could impact performance. | ||
4. Finally, if you intend to use mergerfs with a network filesystem, | ||
either as the source of data or to combine with another through | ||
mergerfs, test each of those alone as above. | ||
|
||
Once you find the component which has the performance issue you can do | ||
further testing with different options to see if they impact | ||
performance. For reads and writes the most relevant would be: | ||
`cache.files`, `async_read`. Less likely but relevant when using NFS | ||
or with certain filesystems would be `security_capability`, `xattr`, | ||
and `posix_acl`. If you find a specific system, device, filesystem, | ||
controller, etc. that performs poorly contact trapexit so he may | ||
investigate further. | ||
|
||
Sometimes the problem is really the application accessing or writing | ||
data through mergerfs. Some software use small buffer sizes which can | ||
lead to more requests and therefore greater overhead. You can test | ||
this out yourself by replacing `bs=1M` in the examples below with `ibs` | ||
or `obs` and using a size of `512` instead of `1M`. In one example | ||
test using `nullrw` the write speed dropped from 4.9GB/s to 69.7MB/s | ||
when moving from `1M` to `512`. Similar results were had when testing | ||
reads. Small writes overhead may be improved by leveraging a write | ||
cache but in casual tests little gain was found. More tests will need | ||
to be done before this feature would become available. If you have an | ||
app that appears slow with mergerfs it could be due to this. Contact | ||
trapexit so he may investigate further. | ||
|
||
### write benchmark | ||
|
||
``` | ||
$ dd if=/dev/zero of=/mnt/mergerfs/1GB.file bs=1M count=1024 oflag=nocache conv=fdatasync status=progress | ||
``` | ||
|
||
### read benchmark | ||
|
||
``` | ||
$ dd if=/mnt/mergerfs/1GB.file of=/dev/null bs=1M count=1024 iflag=nocache conv=fdatasync status=progress | ||
``` | ||
|
||
### other benchmarks | ||
|
||
If you are attempting to benchmark other behaviors you must ensure you | ||
clear kernel caches before runs. In fact it would be a good deal to | ||
run before the read and write benchmarks as well just in case. | ||
|
||
``` | ||
sync | ||
echo 3 | sudo tee /proc/sys/vm/drop_caches | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
# BUILD | ||
|
||
**NOTE:** Prebuilt packages can be found at and recommended for most | ||
users: https://github.com/trapexit/mergerfs/releases | ||
|
||
**NOTE:** Only tagged releases are supported. `master` and other | ||
branches should be considered works in progress. | ||
|
||
First, get the code from [github](https://github.com/trapexit/mergerfs). | ||
|
||
``` | ||
$ git clone https://github.com/trapexit/mergerfs.git | ||
$ # or | ||
$ wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.tar.gz | ||
``` | ||
|
||
#### Debian / Ubuntu | ||
|
||
``` | ||
$ cd mergerfs | ||
$ sudo tools/install-build-pkgs | ||
$ make deb | ||
$ sudo dpkg -i ../mergerfs_<version>_<arch>.deb | ||
``` | ||
|
||
#### RHEL / CentOS / Rocky / Fedora | ||
|
||
``` | ||
$ su - | ||
``` |
Oops, something went wrong.