How I improved the memory usage of my web server

Core Development & Architecture
1

When I saw someone wanted to replace Caddy with Ferron 3, because they said Caddy was too heavy. I checked the memory usage while benchmarking Ferron 3 and Caddy, and it turned out Ferron 3 had higher memory usage, and in addition it leaked memory… :smiling_face_with_tear:

So I investigated why did this happen. And ended up finding out about memory leak caused by this (function called when connection is about to be return to the connection pool by the way):

/// Creates a new `PoolReturnInfo` from a pool item and wrapper.
    ///
    /// This consumes the item without running its Drop impl (via `ManuallyDrop`),
    /// allowing the wrapper to be stored separately and returned later.
    #[inline]
    pub fn from_item(item: PooledConnection, wrapper: SendRequestWrapper, is_unix: bool) -> Self {
        // Prevent item's Drop from running (we'll handle return manually)
        let item = std::mem::ManuallyDrop::new(item);

        Self {
            key: item.key().cloned(),
            wrapper: Some(wrapper),
            local_limit_key: item.local_limit_key().cloned(),
            is_unix,
        }
    }

I replaced it with this:

/// Creates a new `PoolReturnInfo` from a pool item and wrapper.
    ///
    /// This disarms the item's Drop accounting (via `disarm`) and moves its
    /// key out, allowing the wrapper to be stored separately and returned
    /// later without leaking the item's `Rc` pool reference and `Arc`
    /// upstream key.
    #[inline]
    pub fn from_item(
        mut item: PooledConnection,
        wrapper: SendRequestWrapper,
        is_unix: bool,
    ) -> Self {
        // Disarm accounting; `item` still drops normally (freeing its `Rc`),
        // but no longer touches `outstanding` counts (handled on return).
        let (key, local_limit_key) = item.disarm();

        Self {
            key,
            wrapper: Some(wrapper),
            local_limit_key,
            is_unix,
        }
    }

The previous code caused a memory leak, because manually dropped value had heap allocations, which would cause a memory leak if Drop impl wasn’t executed (it could be String, which is heap-allocated, and Drop impl would deallocate memory from the heap).

But memory usage improvements haven’t stopped there.

I was also adjusting jemalloc settings by the way. I chose jemalloc, because after I got many allocators tested, jemalloc turned out to be an optimal choice.

Static file 16k.bin (HTTPS/H2):

Allocator req/s GB/s median ms p95 ms p99 ms max ms
glibc (no crate) 193012.7 2.95 7.28 30.66 39.12 87.11
tikv-jemallocator 0.7 (tikv-jemallocator::Jemalloc ) 196934.3 3.01 8.33 26.86 36.90 67.43
mimalloc 0.1.52 (mimalloc::MiMalloc) 173863.6 2.66 9.58 28.84 35.38 57.79
mimalloc secure (features=["secure"]) 149922.3 2.29 11.54 29.27 38.68 73.76
tcmalloc-better 0.1.19 (tcmalloc_better::TCMalloc) 191960.8 2.93 8.85 25.70 31.43 53.44
snmalloc-rs 0.7.5 (snmalloc_rs::SnMalloc) 191199.2 2.92 7.59 31.38 46.87 77.60
rpmalloc custom FFI 184794.4 2.82 8.94 27.19 34.57 64.24
hoard custom FFI 181335.9 2.77 9.24 22.52 27.55 92.95

Reverse proxy http://127.0.0.1:3000/16k (HTTPS/H2 front, H1 backend):

Allocator req/s GB/s median ms p95 ms p99 ms max ms
glibc 103017.2 1.57 13.49 77.83 110.47 167.66
tikv-jemallocator 116227.2 1.78 15.20 35.30 48.04 1130.0
mimalloc 109045.4 1.67 16.75 36.51 50.34 1090.0
mimalloc secure 99697.8 1.52 16.31 47.08 67.84 1160.0
tcmalloc-better 112870.9 1.72 15.78 36.93 47.93 91.05
snmalloc-rs 112127.9 1.71 16.16 35.06 46.66 1100.0
rpmalloc custom FFI 113986.7 1.74 15.41 37.42 49.60 1090.0
hoard custom FFI 102450.1 1.57 18.02 37.01 49.05 1080.0

All runs: 0 failed, ~1.9M static / ~1.0-1.1M proxy requests in 10.03-10.05s, TLS_AES_128_GCM_SHA256, X25519, h2.

I noticed jemalloc consumed more memory by default than glibc malloc when used in the same program. So I searched for jemalloc fine-tuning resources, and came up with this:

export JEMALLOC_SYS_WITH_MALLOC_CONF='narenas:2,percpu_arena:phycpu,metadata_thp:auto,dirty_decay_ms:100,muzzy_decay_ms:100'

Explanation:

  • narenas:2 reduces amount of independent arenas to 2, preventing memory fragmentation. Since Ferron 3 is mostly share-nothing, thread-per-core architecture, tcache would be doing heavy lifting here.
  • percpu_arena:phycpu pins arenas to physical (not hyper-threaded) CPUs, reducing hyper-threading overhead and improving cache locality
  • metadata_thp:autoturns on transparent huge pages when necessary, reducing TLB misses
  • dirty_decay_ms:100 and muzzy_decay_ms:100 makes the allocator free memory faster, without much performance penalty, which leads to lower memory usage.

This also helped reduce memory usage overall, but I wanted even more.

I ended up finding out that observability async channels (used for sending events from modules to observability sinks) were initialized even if unused, and consumed some memory (because they were bounded channels, which would use pre-allocated array in kanal).

So I ended up getting the code refactored to initialize the channels only when needed.

And I was so satisfied with Ferron 3 becoming more lightweight! :grinning_face_with_smiling_eyes:

Here are the results by the way (from a Ferron announcement post for new version)

Workload / benchmark Protocol / tool NGINX Caddy Ferron 3
Reverse proxy (peak RAM) HTTP/3 (tquic_client) N/A ≥ 1,000 MB ~600 MB
Static serving (peak RAM) HTTP/2 multiplex N/A N/A ~90 MB (down from 250MB)
Static serving (peak RAM) HTTP/1.x (wrk) ~300 MB ~125 MB ~100 MB
Reverse proxy throughput HTTP/2 (h2load) N/A Lower rps Higher rps (equal RAM)

By the way, I also released it as a new Ferron 3 version, so Ferron 3 becomes more lightweight for everyone.

1 Like