Putting a Language Model Behind a Node.js API

A Node service is built on a set of assumptions about what it calls. Dependencies respond in tens of milliseconds. The same input produces the same output. A retry is free. A failure is an exception with a status code.

A language model violates all four, and a service that was not designed with that in mind degrades in ways that are hard to diagnose, because nothing actually errors.

Latency stops being an outlier and becomes the design

A database call takes 5ms. A model call takes two to thirty seconds, and the variance is not noise — it is a function of input length, output length and how busy the provider is.

The immediate consequence is that request-scoped work no longer fits inside a request. If an HTTP handler awaits a model call, a burst of twenty users occupies twenty connections for the better part of a minute. Node will not fall over, but the event loop backs up behind everything else those handlers are doing, and unrelated endpoints start timing out.

The pattern that works is to stop treating it as a call and start treating it as a job. Accept the request, return an identifier immediately, run the work outside the request lifecycle, and stream or poll for the result. Streaming is worth the extra work: a first token in 400ms changes how long a twenty-second response feels, and perceived latency is what users actually complain about.

One warning specific to serverless. On platforms that freeze the container when the response is sent, any promise still in flight is killed rather than completed. Fire-and-forget after responding does not work there — the model call simply vanishes, silently, some of the time. If you need work to continue past the response, it has to go to a queue or a durable worker, not a dangling promise.

Retries are no longer free

Retrying a failed database write costs a few milliseconds. Retrying a model call costs seconds and real money, and if your retry policy is the default three attempts with backoff, a provider slowdown turns into a bill and a queue at the same time.

Worse, the usual retry trigger is wrong. Model failures are frequently not exceptions. A 200 response containing a refusal, an empty completion, or valid JSON with the wrong shape is a failure your HTTP client will report as success. Validate the body, not the status.

Budget explicitly: a deadline for the whole operation, a hard cap on attempts, and a decision about what to return when the budget is exhausted. That last part is the one usually missed. There has to be an answer for "we could not get a response in time", and it should be a defined outcome rather than a 500.

Non-determinism breaks your test suite

The same prompt does not produce the same output twice, and the standard backend testing approach — call it, assert on the result — does not survive contact with that.

What works is to separate the parts you can pin from the parts you cannot. Everything around the model is deterministic and should be tested normally: how context is assembled, how the response is parsed, what happens on a malformed body, what the tool-calling loop does when the model asks for a tool that does not exist. Mock the model and test all of that properly.

For the model itself, you are testing properties rather than values. Does the output parse. Does it contain a citation when one was required. Does it stay within the allowed set of actions. Does it decline when the context does not support an answer. Run those against a fixed set of cases on every change, accept that they are probabilistic, and track the rate rather than demanding a pass.

State outlives the request

Conversations, tool-call chains and multi-step agent runs all need memory across calls, and process memory is the wrong place for it — it disappears on deploy, and it is not shared across instances.

This is where a lot of Node AI services acquire their worst bug: the thing works perfectly on one instance in development and behaves erratically behind a load balancer, because half the state is in a Map somewhere. Put it in a store, key it properly, and give it a TTL, because most of it is worthless after an hour and you do not want to pay to keep it.

Watch cost like a latency metric

An endpoint that is slightly too generous with context is not a bug you will find in code review. It is a line on an invoice at the end of the month.

Log the token counts, per route and per user, alongside your latency numbers. Set a spend ceiling at the provider. Then look at the distribution rather than the average — the top 1% of requests by token count is usually somebody pasting a document into a field intended for a sentence, and it is usually most of the bill.

What actually changes

None of this is a rewrite. It is the same service, with model calls treated as expensive, slow, unreliable, non-deterministic dependencies rather than function calls that happen to be a bit slower. Designed that way from the start, the result behaves well. Bolted on afterwards, you get a service that works in development and is mysteriously fragile in production, which is a much harder problem to describe than to prevent.