The effect of switching to TCMalloc on RocksDB memory use

Glibc malloc is unsuitable for multithreaded workloads because it needs to lock.
8 (configurable) arenas per core
Allocation for a thread occurs on the last arena it used, and other arenas are tried sequentially if that one is locked.
Crucially allocations can’t move between arenas, and memory is only reclaimed when it isn’t bounded by used regions.
malloc_trim can be used to reclaim memory that’s bounded by used sections, but needs to be called manually.
allocations above M_MMAP_THRESHOLD use mmap instead of the arena strategy.
TCMalloc and jemalloc are designed for multithreaded workloads and they improved rocksdb memory usage from 25G to 15G