Repository manifest is limited to 20 MiB size, then borg create fails #8532
Comments
|
Hmm, that shouldn't happen. The msgpack unpacker is configured with some limits (see In the case seen above, the client sends a packed str value to the server which would exceed MAX_OBJECT_SIZE. You are the first borg user reporting this although these limits are like that since long. So, is there anything unusual with your repository? How many archives are in that repository? Does the crash happen at the end of |
|
"I noticed that with every transaction the size of not very small chunks grows monotonic towards the 20 MiB chunk size limit." It's not clear what you mean by that. How did you notice that and what chunk(s) do you mean (what is the contents of these chunks)? |
ThomasWaldmann
changed the title
|
I don't want to waste your precious time too much, if you think the use case is unusual, lets close it, it's probably smarter to divide the repository by at least years and maybe even fine by other categories.
Borg info is still running, I'll try to report later, I estimate it's around 400000 - 500000 (3 years * 366 days * 4 * 100+).
I am not sure enough, will try to test after borg info completed. I will try to determine that with verbose output.
I mean files in the folders of the data folder: For example the 2nd last created folder is like this: And the last created folder in data is like this: If you don't look at the outlier 1617504 which is 100+ MB and the small ones, you see it's steadily growing the chunks in size, the last one it could create is only 5 byte bellow 20 MiB and it always grows comapred to the previous one, The problem first occoured in the night between 5th and 6th November. Before the Night from 8th to 9th I removed the transactions from the last backup batch where it failed the first time according to the borg manual for append-only mode (The server can only append, only another machine can compact) and rebuild the cache and deleted that timestamp file. |
|
I saw you are on key.openpg.org too. But to be honest I am starting to think the use case I have is a bit not normal / not smart by me and it might be wasting your time. |
|
borg info manual borg create --debug: Edit: forgot to count archives, will edit it in, counting ... Edit2: number of archives in repo: Edit3: Considering this I should probably start a new repo either way: Lines 55 to 58 in 958c5a2 |
|
I think this might be my fault for coming 130 archives close to the MAX_ARCHIVES limit, is it ok if I close the issue or you prefer for it to stay open? |
|
OK, looks like your repository manifest (== the list of all archives) is growing to >20MiB. The manifest is stored as a single chunk, so it shouldn't be bigger than 20MiB. Considering you have >300k archives and one entry might be 50-100 bytes, that is expected. Thanks for scalability testing! :-) Nobody had that yet, because usually people use That's not only good to keep the amount of archives in bounds, but also for the runtime of some borg commands. For your repository, I guess it is close to impossible to run a full Instead of starting a new repo, you could carefully and slowly reduce the amount of archives you have in that repo. Make sure you don't try to reduce from 300k to 1k in a single step, that might also take rather long. Rather start slowly, reducing the archive count by 100 or 1000 per To free space, you need to run borg compact now and then. It is not needed to run it after each prune. Just check that its runtime isn't getting too long (see also the |
ThomasWaldmann
changed the title
|
I'll keep the issue open, but maybe only to improve the error message or adjust MAX_ARCHIVES. It would be good to have a more clear indication of what the problem is. As I explained above, there are a lot of reasons usually to stay far below that limit, so the limit itself is usually a non-issue. |
|
Side note: borg2 already does this very differently. It stores "pointers" to each archive as separate directory entries, so there is no limit in borg for that. Depending on the scalability of the borgstore backend used for that, directories with a huge amount of entries at some time might get slow to deal with. Thus, the fundamental issue seen above is fixed in borg2. |
|
Thank you for your time and replies, much appreciated :-) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The obvious workaround is to start a new repository after a while, before it grows too big, so the issue (if any) is of low priority.
My question is if this problem is already known or even fixed in newer versions, because I wasn't aware of this problem.
Have you checked borgbackup docs, FAQ, and open GitHub issues?
Yes
Is this a BUG / ISSUE report or a QUESTION?
Question
System information. For client/server mode post info for both machines.
client: Hetzner VPS
server: Hetzner storage box
Your borg version (borg -V).
client: 1.2.4
server: 1.2.8
Operating system (distribution) and version.
client: Debian 12
server: unknown
Hardware / network configuration, and filesystems used.
client: 8 shared vcpu, 32 GB ram, x86_64, ext4
server: unknown
How much data is handled by borg?
The repository is about 600 - 900 GB (depends on the borg compact and sometimes I delete some archives) in size and has more than 1617800 transactions logged
The borg info is very slow, if needed I will try to supply that info later when it finished.
Full borg commandline that lead to the problem (leave away excludes and passwords)
Any borg create command.
Describe the problem you're observing.
Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
I noticed that with every transaction the size of not very small chunks grows monotonic towards the 20 MiB chunk size limit.
Eventually it surpasses that limit and borg serve fails with an exception upon borg create by the client.
Include any warning/errors/backtraces from the system logs
The text was updated successfully, but these errors were encountered: