Three assumptions that expired before I shipped

I built a CRM to learn vector search properly: semantic search over deals, contacts and activities, so that "customer worried about renewal" finds the call notes where someone hedged about their contract, without those words appearing anywhere in the text.

It worked on my laptop for months. Then I went to put it on a public URL, and found three things that had been true when I wrote them and had quietly stopped being true since.

None of them were exotic. That's rather the point.

What actually gets embedded

Worth establishing first, because it matters later.

There's no magic to what goes into the index. Each entity is flattened into a short string, and that string is what gets embedded:

static contactText(c) {
  return [c.firstName, c.lastName, c.title, c.email].filter(Boolean).join(' ');
}
 
static dealText(d) {
  return [d.title, d.stage].filter(Boolean).join(' ');
}
 
static activityText(a) {
  return [a.type, a.title, a.description].filter(Boolean).join(' ');
}

A Python service embeds that string with all-MiniLM-L6-v2 into a 384-dimensional vector and stores it in ChromaDB, one collection per entity type. A search embeds the query the same way and runs cosine similarity across all three collections, then drops anything scoring below 0.3 so that a query with no real matches returns nothing rather than noise.

That floor turns out to matter a great deal.

1. Every API endpoint was public

The auth architecture was real: Clerk middleware protecting pages at the edge, a global ClerkAuthGuard on the API verifying JWTs, a RolesGuard for role checks. Correctly wired.

And then, at the top of five of six controllers:

@Public()
@Controller('deals')
export class DealsController {

@Public() makes the global guard return true immediately. Every route underneath it (GET, POST, PATCH, DELETE) was reachable by anyone who could find the origin, with no credential.

This wasn't an accident. It was a decision I'd made months earlier and written down in my project notes: there was no User table yet, so I marked the domain controllers public to keep moving, with a note explaining why. The note was still there. The reason had expired long before I noticed.

The fix was deleting six decorators. Making sure it stayed fixed was more interesting.

A test that reads the routing table

My first attempt listed every controller and asserted none were public. It passed. It was also wrong: I'd forgotten AuthController, so GET /me was never checked.

A list you have to remember to update is exactly the failure mode the test existed to prevent.

So it walks the filesystem instead. It finds every *.controller.ts, imports it, reads Nest's own route metadata, and asserts that the only publicly reachable handler in the entire application is the health check. A new controller is covered the moment it exists.

Then three more things I'd otherwise have missed:

That the guards are still registered. Delete one line from AuthModule and every route becomes unauthenticated while the test above stays green; "no @Public() anywhere" and "no guard at all" look identical from the metadata. So the test asserts both guards are registered as APP_GUARDs.

That no controller hides outside the naming convention. Discovery keys off the filename. A @Controller() in a file named something else would be invisible to the audit. So the test greps the source for @Controller( and asserts that set matches the set it discovered.

That inherited routes count. Object.getOwnPropertyNames misses a method inherited from a base class. Nest's router doesn't; it walks the prototype chain. So the test uses Nest's own MetadataScanner, the same enumeration the framework uses to build its routing table.

For each of these I introduced the regression deliberately and watched the test fail before trusting it. A security test you have never seen fail is a security test you don't know works.

2. The search returned nothing

I wanted real latency figures rather than ones I half-remembered, so I wrote a script to measure the deployed service: five warmup requests, fifty measured, percentiles rather than a mean.

It reported 35ms per query. And zero results. Every query, zero results.

The index was populated: 50 contacts, 20 deals, 102 activities, all embedded. The service was healthy. It simply never matched anything.

So I dumped the raw scores with the 0.3 floor bypassed:

-- deal top 3 --
  score=0.180  Recycled Aluminum Chips — engineer global architectures
  score=0.175  Incredible Aluminum Tuna — brand revolutionary web services
-- activity top 3 --
  score=0.153  CALL Suspendo commodo sono. Conicio appono alius tripudio

My seed data was faker output. Random product nouns and Latin filler. The best match anywhere in the corpus scored 0.18, and the floor was correct to reject it: there is no semantic relationship between "customer worried about renewal" and "Suspendo commodo sono", because the second phrase doesn't mean anything.

The tempting fix is one character. Drop the floor to 0.05 and everything "works": every query returns ten results, all nonsense. That demos worse than returning nothing, because nothing is at least honest.

The actual fix was writing the data. Twenty-five hand-written activity notes about renewal risk, pricing objections, contracts stuck in legal review, churn signals, expansion conversations. Realistic deal titles. And job titles on contacts, which, per that contactText above, are part of the embedded string, and which I had never populated. All 50 contacts were indexed as a bare name and email address. They scored 0.09 against everything, which is roughly what a name and an email address mean to an embedding model.

Same 170 documents. Every query now returns matches.

The general lesson, which I keep relearning: for anything semantic, test data is not scaffolding. A fixture only needs the right shape for a SQL query to pass. An embedding model reads it. If your fixtures are meaningless, your search is correctly telling you so.

3. The seed wasn't deterministic

Same file, found while rewriting it. At the top:

faker.seed(42);

Which makes faker reproducible. Every selection underneath it went through Math.random():

function pick<T>(arr: T[]): T {
  return arr[Math.floor(Math.random() * arr.length)];
}

So two runs produced different data, and had done for as long as the file existed. Nothing depended on it, until I added an hourly job that reseeds the public demo, at which point "the demo always looks the same" became a property I actually wanted. Selection now runs through a seeded PRNG.

What the latency number turned out to be

With real data: 35ms p50, 44ms p95, measured on a 2-core VPS.

What matters more than the figure is what's inside it. With 170 documents, the vector lookup is microseconds: cosine similarity over 170 × 384 floats is nothing. Essentially all of the 35ms is a single forward pass through MiniLM to embed the query.

So the number measures embedding cost, not vector database performance. Corpus size barely moves it. If I wanted it faster, the levers are a smaller model, batching, or ONNX and quantisation, not a different store. Knowing which number you have actually measured seems worth more than the number itself.

The shape of all three

Each failure was the same: something true when written, quietly false later.

The @Public() decorators were correct until the auth work landed. The latency figure was probably correct on the machine where I first ran it. The seed was fine until the thing consuming it started caring about meaning rather than shape. The Math.random() didn't matter until it did.

Code doesn't announce when its assumptions expire. The only ones I caught were those where something eventually failed loudly. So the fix is to write assumptions down somewhere a machine checks them, rather than to be more careful.

The demo is at vectro.luisv.dev: one click, no signup. Seeded data, regenerated hourly.