RocksDBException: While opendir: xxx: Too many open files on Mac

[halibobo1205]

System information

OS: Mac OS X
JVM: Oracle Corporation 1.8.0_321 x86_64
Version: 4.7.3.1

ulimit -a

-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8176
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       2666
-n: file descriptors                256

 lsof -p pid | wc -l 

output: 10287

Expected behaviour

Normal synchronization of blocks on Mac using RocksDB.

Actual behaviour

After running for a while, the block stops synchronizing after an error of Too many open files.

Steps to reproduce the behaviour

Use RocksDB on Mac to start the node.

Backtrace

16:55:00.757 ERROR [sync-handle-block] [DB](RocksDbDataSourceImpl.java:273) Open Database checkpoint/1706604900706 failed
org.rocksdb.RocksDBException: While opendir: output-directory/database/checkpoint/1706604900706: Too many open files
	at org.rocksdb.RocksDB.open(Native Method)
	at org.rocksdb.RocksDB.open(RocksDB.java:231)
	at org.tron.common.storage.rocksdb.RocksDbDataSourceImpl.initDB(RocksDbDataSourceImpl.java:267)
	at org.tron.common.storage.rocksdb.RocksDbDataSourceImpl.initDB(RocksDbDataSourceImpl.java:199)
	at org.tron.common.storage.rocksdb.RocksDbDataSourceImpl.<init>(RocksDbDataSourceImpl.java:72)
	at org.tron.core.db.TronDatabase.<init>(TronDatabase.java:55)
	at org.tron.core.store.CheckPointV2Store.<init>(CheckPointV2Store.java:18)
	at org.tron.core.db2.core.SnapshotManager.getCheckpointDB(SnapshotManager.java:430)
	at org.tron.core.db2.core.SnapshotManager.createCheckpoint(SnapshotManager.java:408)
	at org.tron.core.db2.core.SnapshotManager.flush(SnapshotManager.java:357)
	at org.tron.core.db2.core.SnapshotManager.buildSession(SnapshotManager.java:144)
	at org.tron.core.db2.core.SnapshotManager.buildSession(SnapshotManager.java:127)
	at org.tron.core.db.Manager.pushBlock(Manager.java:1310)

There isn't a configuration to change rocksdb's maxOpenFiles, which is currently fixed at 5000. It is recommended to add the configuration.

[lxcmyf]

@halibobo1205
Is this optimization aimed at improving the operating experience under Mac OS?

[halibobo1205]

@lxcmyf Not sure how it performs under other OS.

[tomatoishealthy]

Seems it will occur when a fullnode that adopts the RocksDB as its storage starts on an OS with a low ulimit configuration .

I think we can do a check before the fullnode starts to remind the OP to adjust the OS configuration.

[tomatoishealthy]

This issue is not a high priority and will be notified here when the follow-up begins.

[halibobo1205]

No change plan for now.

[aguinaldok4]

#5778