expose chainbase's new "reclaimable" memory in db_size_api

[spoonincode]

As of Leap 3.1, chainbase has not just the boost interprocess allocator (segment_manager) but also chainbase_node_allocator was is layered on top of the segment_manager. Both of these allocators maintain their own free list. But since chainbase_node_allocator sits on top of segment_manager, the memory tied up in chainbase_node_allocator's free list is considered used memory from the perspective of segment_manager.

The chainbase_node_allocator isn't used for everything, for example it's not used for the CoW strings. So various bits of code interface with the top layer but then other bits of code the bottom layer.

And the db_size_api is looking at free memory in the bottom layer. That is, when db_size_api reports the free bytes it's reporting from the perspective of the bottom allocator even though the top allocator may have an extensive free list that would be used first for allocations against it. This can create some confusing and/or deceptive reports of memory usage.

To help provide some visibility to this "used from perspective of bottom layer but free from perspective of top layer" memory, add a new field to db_size_api's response which is the total amount of memory in the top layer's free list. I've called this 'reclaimable' memory as a differentiator.

Some other options I considered:

Don't have a new field indb_size_api's response but instead consider the top allocator's free list as part of the existing free/used counts. The problem with this is that the bottom allocator's free memory is what governs nodeos falling over. So it would be deceptive to return, in an extreme example, 4GB of free memory but really only 256MB of that is free from the perspective of the bottom allocator (since some allocations do go directly to the bottom allocator).

Don't allow chainbase_node_allocator's free list to grow limitless as it can now. This was my first preference. However chainbase_node_allocator allocates chunks of 64 items from the bottom allocator which means those chunks would need to be freed as a whole. Pretty much impossible. So instead chainbase_node_allocator would need to allocate 1-by-1 from the bottom allocator, but I'm unsure the time and space penalty for doing that. Without further research it would not be a change I'd blindly make.

Needs AntelopeIO/chainbase#51

[greg7mdp]

It would be possible to improve the allocator so that fully free chunks are returned to the bottom allocator, with a small memory cost. I implemented something like that a while back.

[spoonincode]

I was actually thinking the other direction: the chunks should be larger than 64. The largest of these types is 192 bytes but most are in the 40-64 range. It seems entirely reasonable to be allocating on the scale of 64KB or larger per chunk given our scale.

[greg7mdp]

What I was thinking was something like:

• say chunk_size is 128
• allocate 100 chunks. Waste at most 1 chunk out of 100 so chunks are aligned modulo chunk_size * sizeof(T).
• each chunk wastes 1 item to store a header at the beginning, containing a struct { uint32_t num_free; void *free_list; void *next_chunk; void *prev_chunk; };
• one free_list per chunk. When num_free == 128 the chunk memory can be returned to the bottom allocator. num_free is initialized not to 0, but to how many Ts are used to store the header (likely 1 or 2).
• instead of a top-level free_list, we have a list of chunks.
• when deallocating an item, we can find the chunk header because they are aligned modulo chunk_size * sizeof(T).

Actually instead of void * they probably would be bip::offset_ptr or something, and we can store 3 offset_ptr into 128 bits :-).

[ericpassmore]

Note:start
group: STABILITY
category: INTERNALS
summary: Improve reporting of chainbase's db size by including a reclaimable size.
Note:end