Google engineers Ashok Chandrasekar and Jason Kramberger, behind the open-source inference-perf project, warned on the AI Engineer podcast that most LLM inference benchmarks are unreliable because the client-side benchmark harness silently fails under pressure. In a controlled experiment, harnesses were asked to generate 200 QPS against a model server: a single-process harness with small shard/core allocation achieved 38 QPS, a single-process harness with larger allocation capped at roughly 170 QPS, and only a multi-process harness delivered the load as requested. Chandrasekar noted that even when 200 QPS is requested, only 38 QPS may be achieved, and the harness still reports it ran. The failure stems from Python's global interpreter lock, which makes execution effectively single-threaded, preventing a CPU-bound Python harness from using all cores. Client-side thrashing also inflated latency by up to 58 seconds, a delay normally blamed on the server. The proposed fix is a multi-process load generator with self-observability, declarative configuration files, and a shared catalog of standardized workloads. The commercial takeaway: before trusting a vendor's benchmark, ask whether it actually delivered the load it claims.