BentoML's adaptive-batching guide describes server-side batching and separate size and latency controls. A BentoML service should not assume batching is enabled or beneficial for every workload.
Measure sparse traffic as well as sustained load. Higher throughput can conceal an unacceptable wait for isolated requests.
Akousa