Research note
One envelope, many upstreams
Every upstream API fails differently and says so in its own dialect. Notes from normalising a dozen of them behind a single response shape.
The premise of the v1 API Kit is small enough to state in a sentence: one account, one key, one response shape, over a set of upstream services that agree on none of those things. The interesting engineering is entirely in the gap between that sentence and what the upstreams actually do.
Success is easy. Failure is the product.
Normalising a successful response is mostly mapping fields, and it is the part that looks like the work. It is not. Where a gateway earns its keep is in failure, because that is where the dialects diverge most violently.
A survey of what "this request was bad" looks like across upstreams we front:
- HTTP 400 with a machine-readable error code, which is the good case.
- HTTP 200 with
{"error": "..."}in the body, which is success at the transport layer and failure at the application layer. - HTTP 200 with an empty result set that means "not found" and is indistinguishable from "found nothing".
- HTTP 403 that means the key is wrong, and HTTP 403 that means the quota is exhausted, with nothing in the response separating them.
If a caller has to know which upstream is behind an endpoint in order to interpret an error, the gateway has not actually abstracted anything. So the envelope commits to a small, closed set of error classes and every upstream failure gets mapped into one of them, with the upstream's own message preserved in a detail field, because throwing it away makes debugging impossible.
The rule we settled on: the envelope's status is about the caller's request. If the caller did nothing wrong and the upstream is broken, that is our failure to report as such, not the caller's error to handle.
Credentials never leave the server
This is the security property the whole design exists to provide, and it is worth being explicit about what it means. A customer holds a key with an ak_live_ prefix. That key authenticates them to us. It has no relationship to the credentials we hold for upstream providers, which live server-side, are never included in any response, and are not reachable from any customer-facing route.
The consequence for design is that we cannot offer a passthrough mode. A "just proxy my request unchanged" escape hatch is the single most requested feature on any gateway and it quietly reintroduces every upstream's dialect into the contract, and, depending on how it is built, the upstream's own error responses, which can leak more than intended. We have not built it.
Rate limiting has two jobs
The obvious job is protecting us from a customer. The less obvious and more important one is protecting the upstream from all of our customers at once. These need different mechanisms, and conflating them was our first real mistake.
Per-customer limits are about fairness and billing, they are enforced on the customer's key, and exceeding one is the caller's problem to handle, via a clear rate-limit error with a retry hint. Per-upstream limits are about capacity we bought and do not control. Exceeding one is our problem, and the right response is usually a short queue or a served cache entry, not an error handed to a caller who did nothing wrong.
We use a circuit breaker per upstream so that a provider having a bad day degrades one endpoint instead of consuming the connection pool and taking the rest down with it. The first version of this was a single global pool, which meant one slow provider could stall every unrelated request in the system. That failure taught us more than the design docs did.
Caching is a correctness decision
Some upstream data changes by the second and some does not change at all. Currency rates and weather have short, meaningful lifetimes; a book's metadata is effectively static. Applying one cache policy across all of them means either serving stale rates or hammering providers for data that has not changed since 1998.
So cache lifetime is a property of the endpoint, declared alongside it, rather than a global setting. Two things we would do the same way again:
- Say when the data was fetched. The envelope carries the age of what it served, so a caller can make their own decision about whether it is fresh enough.
- Serve stale on upstream failure, and label it. A slightly old rate is almost always more useful than an error, but only if the caller can tell that is what happened.
What this is really for
API Kit is a product, and it is also a testbed. Authentication, rate limiting, failure isolation, honest error semantics: those are concerns in every other system we run, including the ones carrying live phone calls. Having a place to get them wrong where the blast radius is a weather lookup rather than a customer's sales line has been worth considerably more than the endpoint catalogue.