# MinIO MemKV RELEASE.2026-07-08T07-15-37Z Released: 2026-07-08 This release fixes a ~40–50 second stall on the first LMCache retrieve after a restart by removing the retrieve-staging lane and moving the stack to LMCache v0.5.x with its lazy CPU allocator. The TCP data path now exports Prometheus metrics, and the RDMA latency histograms are renamed to match the client operation (put/get) — a breaking metric-name change. --- ## Downloads ### Server Binary | Platform | Architecture | Download | | -------- | ------------ | -------- | | Linux | amd64 | [memkv](https://dl.min.io/aistor/memkv/release/linux-amd64/memkv) | | Linux | arm64 | [memkv](https://dl.min.io/aistor/memkv/release/linux-arm64/memkv) | ### NIXL Plugin (for Dynamo / KVBM integrations) | Platform | Architecture | Download | | -------- | ------------ | -------- | | Linux | amd64 | [libplugin_MEMKV.so](https://dl.min.io/aistor/memkv/release/linux-amd64/libplugin_MEMKV.so) | | Linux | arm64 | [libplugin_MEMKV.so](https://dl.min.io/aistor/memkv/release/linux-arm64/libplugin_MEMKV.so) | ### LD_PRELOAD Shim (for MLPerf-Storage kvcache workloads) | Platform | Architecture | Download | | -------- | ------------ | -------- | | Linux | amd64 | [libmemkv_preload.so](https://dl.min.io/aistor/memkv/release/linux-amd64/libmemkv_preload.so) | | Linux | arm64 | [libmemkv_preload.so](https://dl.min.io/aistor/memkv/release/linux-arm64/libmemkv_preload.so) | ### Packages `.deb`, `.rpm`, and `.apk` packages bundle the server + both `.so` sidecars + the LMCache and sglang Python wheels into a single per-arch install. | Format | Architecture | Download | | ------ | ------------ | -------- | | DEB | amd64 | [memkv\_20260708071537.0.0_amd64.deb](https://dl.min.io/aistor/memkv/release/linux-amd64/memkv_20260708071537.0.0_amd64.deb) | | DEB | arm64 | [memkv\_20260708071537.0.0_arm64.deb](https://dl.min.io/aistor/memkv/release/linux-arm64/memkv_20260708071537.0.0_arm64.deb) | | RPM | amd64 | [memkv-20260708071537.0.0-1.x86_64.rpm](https://dl.min.io/aistor/memkv/release/linux-amd64/memkv-20260708071537.0.0-1.x86_64.rpm) | | RPM | arm64 | [memkv-20260708071537.0.0-1.aarch64.rpm](https://dl.min.io/aistor/memkv/release/linux-arm64/memkv-20260708071537.0.0-1.aarch64.rpm) | | APK | amd64 | [memkv\_20260708071537.0.0_x86_64.apk](https://dl.min.io/aistor/memkv/release/linux-amd64/memkv_20260708071537.0.0_x86_64.apk) | | APK | arm64 | [memkv\_20260708071537.0.0_aarch64.apk](https://dl.min.io/aistor/memkv/release/linux-arm64/memkv_20260708071537.0.0_aarch64.apk) | After installing the deb/rpm, the Python plugin wheels land at `/usr/share/memkv/wheels/`: ```bash pip install /usr/share/memkv/wheels/memkv_lmcache-*.whl pip install /usr/share/memkv/wheels/memkv_sglang-*.whl ``` The NIXL plugin is auto-symlinked to `/opt/nvidia/nvda_nixl/lib/plugins/` when that directory exists (postinstall hook). ### Python Plugins (PyPI) The LMCache and sglang plugins are also published to PyPI, so they install without the deb/rpm: ```bash pip install memkv-lmcache memkv-sglang ``` Install **1.0.3 or later** — earlier wheels predate cross-instance KV reuse (1.0.1), RC-transport batching on non-mlx5 NICs (1.0.2), and the retrieve-staging lane removal in this release (1.0.3). The wheels `dlopen` the RDMA libraries at runtime; the transport needs host `rdma-core`, otherwise it falls back to TCP. ### Container Image ```bash docker pull quay.io/minio/memkv:RELEASE.2026-07-08T07-15-37Z docker pull quay.io/minio/memkv:latest ``` Container ships the server + the NIXL plugin (under `/usr/local/lib/plugins/`). The LD_PRELOAD shim and Python wheels are not included in the container image — use the deb/rpm or PyPI for those. ### Verification Each binary is signed with both minisign (preferred) and GPG; sha256sums are published alongside. ```bash # minisign minisign -Vm memkv -P RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGav # sha256 sha256sum -c memkv.sha256sum ``` --- ## Changes since RELEASE.2026-07-07T15-55-04Z ### Bug Fixes - **The first LMCache retrieve after a restart no longer stalls for ~40–50 seconds.** The retrieve-staging lane (`MEMKV_RETRIEVE_LANE_GB`) page-locked its entire pinned CPU arena in a single allocation on the first retrieve — a host-side cost that hit DC and RC alike (42.5 s and 49.5 s observed, against 24 ms warm). The lane is removed; retrieves stage through the shared LMCache CPU pool, so size `max_local_cpu_size` to at least ~2.5× peak in-flight staging (see `deploy/lmcache.yaml`). ### Improvements - **The TCP data path now exports Prometheus metrics.** Six new series mirror the RDMA ones — `memkv_tcp_put_total` / `memkv_tcp_get_total`, matching `_bytes_total` counters, and `memkv_tcp_put_latency_us` / `memkv_tcp_get_latency_us` — covering inline, memory-only, and chunked transfers. Latency measures server-side handling (storage + encode), not the socket write. All three bundled Grafana dashboards gained TCP throughput and latency panels. ### Breaking Changes - **RDMA latency histograms renamed to match the client operation.** `memkv_rdma_read_latency_us` is now `memkv_rdma_put_latency_us`, and `memkv_rdma_write_latency_us` is now `memkv_rdma_get_latency_us`. The old names followed the wire verb: a client put makes the server RDMA-read from client memory, so "write latency" only moved on gets. Update any external dashboards or alerts on the old names; the bundled dashboards are already updated. - **The LMCache plugin now requires LMCache v0.5.x** (`lmcache>=0.5,<0.6`), and the `MEMKV_RETRIEVE_LANE_GB` environment variable is removed. Enable `enable_lazy_memory_allocator` in the LMCache config so a large CPU pool pins incrementally in the background instead of eagerly at startup; the segpin monkeypatch workaround for driver pin caps is obsolete. --- ## Documentation - Hosted docs: - Embedded docs (in the binary): `memkv doc` serves the same site locally. ## Support - Security disclosures: security@min.io