MongoDB Atlas Vector Search for Large-Scale eKYC Deduplication: Lessons from a 10M-Vector POC

Vector search is often discussed as an AI retrieval capability, but some workloads place very different demands on it. Identity deduplication is one example. In a 1:N eKYC matching workflow, a new identity is represented as a vector and compared against a large existing population to identify possible duplicates or fraud. In this type of system, vector retrieval is not simply a recommendation step. Search quality can directly affect a business decision. That changes how latency, recall, index architecture and memory optimization should be evaluated.

The workload: 10 million vectors with production-oriented latency requirements

The evaluation focused on a dataset of 10 million vectors, each representing a 1024-dimensional facial embedding. The primary performance target was at least 100 requests per second while maintaining P95 query latency below 10 milliseconds. Separate test environments were used for performance and accuracy. The larger environment measured latency, throughput, stability and index behavior at 10-million-vector scale, while a smaller environment was used to evaluate recall and quantization with controlled datasets.

What the performance tests showed

At concurrency 10, the 10-million-vector environment reached 1,255 QPS at P95 9.85 ms. The original requirement was at least 100 requests per second below 10 ms P95, so the tested configuration maintained substantial throughput headroom before crossing the latency threshold. At concurrency 20, throughput increased further to 1,697 QPS, but P95 latency rose to 19.91 ms. This is an important reminder that maximum throughput and latency SLOs should be evaluated together rather than independently.

Sustained performance matters more than a short benchmark

A fast benchmark over a few minutes does not establish production readiness. Vector indexes, memory behavior and search processes need to remain stable under sustained traffic. The continuous-load test ran for 20 hours and 16 minutes, processing 98,111,145 requests with zero recorded timeouts or errors. Average throughput was 1,344 QPS, while average P50, P95 and P99 latencies were 7.23 ms, 9.34 ms and 11.16 ms respectively. Search-process memory remained stable and no throttling was observed.

For identity matching, recall cannot be treated as a secondary metric

Approximate nearest-neighbor search trades exhaustive comparison for dramatically lower query cost. The relevant engineering question is therefore not simply “Is it fast?” but “How much retrieval quality is lost to achieve that speed?” Using exact nearest-neighbor search as the reference, HNSW achieved 98.96% Recall@10 with 100 candidates on the 10-million-vector dataset. Increasing numCandidates to 200 raised Recall@10 to 99.42%. On the controlled 1-million-vector dataset, recall remained around 99.3% across 512-, 768- and 1024-dimensional vectors.

Why exhaustive search stops being practical at scale

Exact nearest-neighbor search becomes increasingly expensive as vector populations grow because each query must compare against the full search space. In the POC, exhaustive search required approximately 182 ms per query at one million vectors and 480 ms at ten million vectors. HNSW remained near 8 ms at both scales, making it approximately 24 times faster at one million vectors and around 60 times faster at ten million vectors.

Memory optimization can become an accuracy problem

Quantization can substantially reduce vector-index memory, but the trade-off must be evaluated against the role search results play in the application. In the test, scalar quantization reduced index memory from 4.51 GiB to 1.36 GiB. However, Recall@10 fell from 99.37% to 92.83%, while Exact Match@10 fell from 94.88% to 41.22%. For a recommendation workload, some ranking variation may be acceptable. For 1:N identity deduplication, where the vector-search result can directly influence whether a duplicate identity is detected, that loss of exact-match quality was not considered acceptable.

Production readiness also includes index lifecycle behavior

One target that was not met was full index rebuild time. Rebuilding the 10-million-vector index took approximately 41 minutes, above the initial 15-minute estimate. That result needs context. Full rebuilds are uncommon during normal operation because newly inserted vectors are indexed incrementally. The practical production question is therefore not only how long a complete rebuild takes, but also how incremental indexing behaves under normal write volume and how recovery procedures account for rare full-rebuild scenarios.

Five lessons for production vector-search architecture

1. Start from the decision being made, not from the vector database. Define what a false negative or ranking change means to the business before choosing search parameters.
2. Measure latency and recall together. Lower query latency has little value if retrieval quality falls below the acceptable threshold.
3. Isolate search workloads where possible. Dedicated search resources reduce contention between database writes, indexing and query execution.
4. Test sustained load, not only peak benchmarks. Memory behavior, throttling and long-running stability can expose problems that short tests miss.
5. Treat index lifecycle as part of production architecture. Incremental indexing, rebuild scenarios and recovery procedures need explicit operational planning.

Vector search is an engineering trade-off, not a single benchmark number

The POC demonstrated that MongoDB Atlas Vector Search can support low-latency, high-throughput retrieval at 10-million-vector scale while maintaining high recall for the tested eKYC deduplication workload. More importantly, the evaluation showed why production vector-search design should consider latency, accuracy, memory, index lifecycle and workload isolation together. The best configuration is not necessarily the one with the lowest latency or smallest memory footprint; it is the one whose trade-offs remain acceptable for the decision the system is responsible for making.