Memory leak #10461
Comments
RubenKelevra
added
kind/bug
|
Two minutes after restarting the service, ipfs uses 1% of 32 GB memory in my usecase. |
|
It's really bad. I increased my IPFS VM from 8GB to 12GB Ram, but with AcceleratedDHT on, it can't even make it past 24 hrs. |
|
Thanks for confirming @Rashkae2 |
|
@RubenKelevra can you give a pprof dump |
|
@aschmahmann sure, do I need to censor anything in the dump to protect my private key or the private keys of ipns? |
|
I'm also having the same memory leak on latest version 0.29.0, this is my server memory over the last 15 days
and then after restarting ipfs/kubo |
|
I also got this warning when I shut ipfs down. 128 provides in 22 minutes is an atrocious rate. Wtf? This server got 2.5 Gbit/s an NVMe and uses 500–600 connections. The number should be a couple of magnitudes higher. |
Hey @aschmahmann, don't bother. I've started ipfs now with a fresh key and no keystore in the server to provide a full dump without any concerns. But it would be nice to know for the future on how to do this safely, maybe with an howto? |
|
@RubenKelevra good news, privacy notice exists under If you could share profile .zip (here or privately via message to https://discuss.ipfs.tech/u/lidel/), that would be helpful. FYSA there will be 0.30.0-rc1 next week, which includes some fixes (#10436) which might help, or narrow down the number of existing leaks. |
|
@lidel thanks for the info! Will do ASAP :) |
|
btw: if you want to improve the provide speed without running accelerated dht client, you may also experiment with https://github.com/ipfs/kubo/blob/master/docs/experimental-features.md#optimistic-provide |
|
@lidel wrote:
Thanks, but I think this may be more related to the memory leak issue. 474 sec for a single provide feels a bit too high. ;) As soon as the issue is gone I'll look into that. Filelink is out via PM. |
|
The pprof data you provided indicated that the memory consumption is primarily due to quic connections. There was at least one quic resource issue that has been fixed in a later version of libp2p then you version of kubo is using. That, and the settings in your config my be responsible for this memory use. In you config you have "GracePeriod": "3m0s",
"HighWater": 600,
"LowWater": 500,The It would be informative to see if using values closer to the defaults would help significantly. Results may also improve with the next version of kubo using a newer go-libp2p that has fixes that may affect this. |
|
Hey @gammazero, I've adjusted the default settings, but it seems like they are more suited for a client application, right? I'm running a server with 2.5 Gbit/s network card, 10 cores, and 32 GB of memory. Its only task is to seed into the IPFS network. Given this setup, the current configuration feels a bit conservative rather than excessive. Do you know what settings ipfs.io infrastructure uses for their connection manager? @gammazero wrote:
I don't think that's the issue. I've been using these settings for 3 years without any memory problems until now. It seems unlikely that the settings are the cause, especially since the memory usage increases steadily over 18 days, rather than spiking within an hour. |
I was thinking that the 3 minute grace period was the setting that may have the most effect.
OK, that is a hint that it may be a libp2p/quic issue. Lets keep this issue open and see what it looks like when we a kubo RC with new libp2p and quic. |
|
@gammazero the idea behind using 3 minutes was, to not kill useful long term connections due to an influx of single request connections which end up stale afterwards. Not sure how kubo has improved in the meantime, but I had a lot of "stalls" while downloading from the server in the beginning, if it did other stuff. The switch from 20 seconds to 3 minutes fixed that. |
|
@gammazero wrote:
@gammazero wrote:
Just started 749a61b, I guess this should contain the fix, right? I'll report back after a day or two if this issue persists or not. If it persists I would be happy to run a bisect to find what broke it. :) |
|
749a61b runs for 4 days straight now and still uses just 2% memory. I call this fixed. Thanks @gammazero @lidel and @aschmahmann! |
|
Great news, thank you for reporting and testing @RubenKelevra ❤️ |
Checklist
Installation method
built from source
Version
Config
{ "API": { "HTTPHeaders": {} }, "Addresses": { "API": "/ip4/127.0.0.1/tcp/5001", "Announce": null, "AppendAnnounce": null, "Gateway": "/ip4/127.0.0.1/tcp/8081", "NoAnnounce": [ "/ip4/10.0.0.0/ipcidr/8", "/ip4/100.64.0.0/ipcidr/10", "/ip4/169.254.0.0/ipcidr/16", "/ip4/172.16.0.0/ipcidr/12", "/ip4/192.0.0.0/ipcidr/24", "/ip4/192.0.0.0/ipcidr/29", "/ip4/192.0.0.8/ipcidr/32", "/ip4/192.0.0.170/ipcidr/32", "/ip4/192.0.0.171/ipcidr/32", "/ip4/192.0.2.0/ipcidr/24", "/ip4/192.168.0.0/ipcidr/16", "/ip4/198.18.0.0/ipcidr/15", "/ip4/198.51.100.0/ipcidr/24", "/ip4/203.0.113.0/ipcidr/24", "/ip4/240.0.0.0/ipcidr/4", "/ip6/100::/ipcidr/64", "/ip6/2001:2::/ipcidr/48", "/ip6/2001:db8::/ipcidr/32", "/ip6/fc00::/ipcidr/7", "/ip6/fe80::/ipcidr/10" ], "Swarm": [ "/ip4/0.0.0.0/tcp/443", "/ip6/::/tcp/443", "/ip4/0.0.0.0/udp/443/quic-v1", "/ip4/0.0.0.0/udp/443/quic-v1/webtransport", "/ip6/::/udp/443/quic-v1", "/ip6/::/udp/443/quic-v1/webtransport" ] }, "AutoNAT": {}, "Bootstrap": [ "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt", "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/ip4/104.131.131.82/udp/4001/quic-v1/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ" ], "DNS": { "Resolvers": null }, "Datastore": { "BloomFilterSize": 0, "GCPeriod": "48h", "HashOnRead": false, "Spec": { "mounts": [ { "child": { "path": "blocks", "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2", "sync": false, "type": "flatfs" }, "mountpoint": "/blocks", "prefix": "flatfs.datastore", "type": "measure" }, { "child": { "compression": "none", "path": "datastore", "type": "levelds" }, "mountpoint": "/", "prefix": "leveldb.datastore", "type": "measure" } ], "type": "mount" }, "StorageGCWatermark": 90, "StorageMax": "500GB" }, "Discovery": { "MDNS": { "Enabled": false, "Interval": 10 } }, "Experimental": { "FilestoreEnabled": true, "GraphsyncEnabled": false, "Libp2pStreamMounting": false, "P2pHttpProxy": false, "StrategicProviding": false, "UrlstoreEnabled": false }, "Gateway": { "APICommands": [], "HTTPHeaders": {}, "NoDNSLink": false, "NoFetch": false, "PathPrefixes": [], "PublicGateways": null, "RootRedirect": "", "Writable": false }, "Identity": { "PeerID": "x" }, "Internal": { "Bitswap": { "EngineBlockstoreWorkerCount": 32, "EngineTaskWorkerCount": 128, "MaxOutstandingBytesPerPeer": null, "ProviderSearchDelay": null, "TaskWorkerCount": 128 } }, "Ipns": { "RecordLifetime": "4h", "RepublishPeriod": "1h", "ResolveCacheSize": 2048, "UsePubsub": true }, "Migration": { "DownloadSources": null, "Keep": "" }, "Mounts": { "FuseAllowOther": false, "IPFS": "/ipfs", "IPNS": "/ipns" }, "Peering": { "Peers": null }, "Pinning": {}, "Plugins": { "Plugins": null }, "Provider": { "Strategy": "" }, "Pubsub": { "DisableSigning": false, "Enabled": true, "Router": "gossipsub" }, "Reprovider": {}, "Routing": { "AcceleratedDHTClient": false, "Methods": null, "Routers": null }, "Swarm": { "AddrFilters": [ "/ip4/10.0.0.0/ipcidr/8", "/ip4/100.64.0.0/ipcidr/10", "/ip4/169.254.0.0/ipcidr/16", "/ip4/172.16.0.0/ipcidr/12", "/ip4/192.0.0.0/ipcidr/24", "/ip4/192.0.0.0/ipcidr/29", "/ip4/192.0.0.8/ipcidr/32", "/ip4/192.0.0.170/ipcidr/32", "/ip4/192.0.0.171/ipcidr/32", "/ip4/192.0.2.0/ipcidr/24", "/ip4/192.168.0.0/ipcidr/16", "/ip4/198.18.0.0/ipcidr/15", "/ip4/198.51.100.0/ipcidr/24", "/ip4/203.0.113.0/ipcidr/24", "/ip4/240.0.0.0/ipcidr/4", "/ip6/100::/ipcidr/64", "/ip6/2001:2::/ipcidr/48", "/ip6/2001:db8::/ipcidr/32", "/ip6/fc00::/ipcidr/7", "/ip6/fe80::/ipcidr/10" ], "ConnMgr": { "GracePeriod": "3m0s", "HighWater": 600, "LowWater": 500, "Type": "basic" }, "DisableBandwidthMetrics": true, "DisableNatPortMap": true, "RelayClient": {}, "RelayService": {}, "ResourceMgr": { "Limits": {} }, "Transports": { "Multiplexers": {}, "Security": {} } } }Description
ipfs's memory usage increased over the uptime of the server (11 days, 16 hours, 40 minutes) until it reached 69% of my 32 GB memory:
The text was updated successfully, but these errors were encountered: