The Five Dots That Slow Down Your Cluster

Somewhere in your cluster, a pod is asking a question ten times to get one answer. It has been doing this since the day you deployed it. Nobody noticed, because it works — just slowly, and occasionally not at all.

The culprit is five characters in a config file you have probably never opened: ndots:5.

This one is worth understanding not because it’s exotic, but because it’s the default. Every pod, every cluster, out of the box. And the symptom it produces — the infamous five-second hang — looks nothing like a DNS problem when you’re staring at it.

First, a post office

When your app asks for api.stripe.com, something has to turn that name into an address. That’s DNS, and it’s the postal service of the internet: you give it a name, it gives you a location.

Inside Kubernetes there’s a wrinkle. Pods need to talk to each other too, and they do it with short nicknames — just payments instead of some long official address. Very convenient.

But now the post office has a problem. When you hand it payments, that’s a nickname needing expansion. When you hand it api.stripe.com, that’s a real address needing no help at all.

How does it tell the difference? It counts the dots.

The rule in one line: fewer dots than the threshold means "expand me", more means "I'm already complete."

The threshold is ndots. Kubernetes sets it to 5.

Now count the dots in api.stripe.com. Two. Which is fewer than five. So Kubernetes looks at that perfectly valid, fully-formed internet address and concludes: that must be a nickname. Let me help.

It is about to be extremely unhelpful.

What “helping” actually looks like

Every pod gets a small list of suffixes to try — its neighbourhood, its building, its city, roughly. Faced with a name it thinks is a nickname, the resolver works down that list, trying each one in turn.

So api.stripe.com becomes a sequence:

Four guaranteed failures, then the answer that was correct from the start. The last line is the kicker: everything doubles for IPv6.

Four failures, then success. And because the resolver asks for IPv4 and IPv6 addresses simultaneously, every one of those lines is really two queries.

Ten DNS queries. One answer. Every single time.

Not cached, not amortised — this repeats for every connection your app opens to that external service.

Scenario 1: the payment API that got slow

A checkout service calls Stripe. Each call needs a name lookup first, and each lookup is ten queries instead of one.

On a quiet cluster nobody notices; the DNS server is fast and 9 wasted queries cost maybe a millisecond. Then traffic triples for a sale, and suddenly the DNS pods are pinned at 100% CPU while the actual application servers sit at 20%.

The investigation goes exactly where you’d expect, and exactly where the problem isn’t: scale up DNS, tune the app, blame Stripe. Nobody looks at the resolver config, because the resolver config is a default — and defaults feel like they’ve already been thought about by someone smarter.

What’s really happening: the cluster isn’t handling more traffic. It’s handling the same traffic, multiplied by ten, in one specific subsystem.

The tell: your DNS service is the busiest thing in the cluster, and the ratio of DNS queries to actual requests is wildly above 1:1.

Scenario 2: the five-second hang

This is the one that burns a week.

A service works fine. Then occasionally — maybe one request in two hundred — it takes exactly five seconds. Not four, not six. Five. Then it works fine again.

Exactness like that is a gift, because nothing in your application is that punctual. Your code doesn’t produce round numbers; timeouts do. Five seconds is the resolver’s default “I got no reply, let me retry” delay.

Here’s the mechanism, and it’s a genuinely elegant piece of bad luck:

Two questions leave through the same door at the same moment, the kernel loses track of one, and your app pays five seconds for it.

The IPv4 and IPv6 questions go out simultaneously through the same network socket. The kernel tracks connections by their source and destination, and two packets leaving at the same instant from the same place can collide in that bookkeeping. One gets dropped. Silently.

Your app sits there waiting for a reply that no longer exists, times out after five seconds, retries, and succeeds immediately.

This is where the two problems meet. The collision is rare. But ndots:5 makes you roll that dice ten times per lookup instead of twice. Rare multiplied by ten, multiplied by every request your service makes, becomes “a few times an hour” — which is exactly often enough to be infuriating and rare enough to never reproduce on demand.

The tell: suspiciously round latency. Exactly 5s, sometimes 10s. Round numbers mean timeouts, and timeouts mean something is waiting for a reply that isn’t coming.

Scenario 3: the logging bill

A background worker ships logs to an external collector. It doesn’t reuse connections — every batch opens a fresh one, so every batch does a fresh lookup.

Ten thousand batches an hour becomes a hundred thousand DNS queries an hour, ninety thousand of which are guaranteed to fail before the real one succeeds.

The visible costs arrive from directions that make the root cause nearly invisible:

the DNS pods need more memory, so somebody raises the limit
cloud DNS charges per query, so the bill creeps up
DNS latency graphs look terrible, so somebody adds another replica

Each fix is locally sensible and none of them touch the actual problem. The cluster is working perfectly — it’s just answering a question nobody meant to ask, ninety thousand times an hour.

The tell: your DNS failure graph is enormous and nothing is actually broken. Those aren’t errors. That’s the system doing exactly what you configured it to do.

How to fix it

Three options, cheapest first.

Fix 1: add a dot

The fastest fix in this entire post. Put a trailing dot on external names in your config:

- STRIPE_API=https://api.stripe.com
+ STRIPE_API=https://api.stripe.com.

That dot means “this is complete, don’t help me.” Ten queries become two, instantly.

The catch is that it’s easy to miss one, and some libraries and TLS certificate checks get grumpy about the trailing dot. Test it rather than rolling it out everywhere on faith.

Fix 2: lower the threshold

The real fix. Tell your pods to stop being so eager:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

Now anything with 2+ dots is treated as a complete address. api.stripe.com resolves in one step; payments still expands as before, so internal nicknames keep working.

Why 2 and not 1? Because one-dot internal names like payments.default are common, and ndots:1 would break them. 2 is the sweet spot for most clusters — but check how your services actually address each other before committing.

The trade-off is honest: this is per-pod config, so it needs to land in every deployment. Most teams apply it to the handful of services that talk to the outside world constantly, and leave the rest alone.

Fix 3: put a cache on every node

Run a small DNS cache on each node so lookups are answered locally instead of crossing the network.

This doesn’t reduce the number of queries — the nine useless ones still happen — but it makes them nearly free, and it sidesteps the dropped-packet problem by changing how queries travel. It’s the sturdiest option and the one to reach for at scale.

Not mutually exclusive — plenty of clusters run the cache and set ndots on their chattiest services.

Why the default is like this

It’s tempting to call ndots:5 a mistake. It isn’t, quite.

Kubernetes has an internal address format that genuinely contains four dots — service.namespace.svc.cluster.local. For partial versions of that name to expand correctly, the threshold has to be high. Five makes internal service discovery work beautifully.

It just does so by assuming the thing you’re looking up is probably inside the cluster. In 2015, that was a fair bet. Today your pods talk to payment APIs, object storage, managed databases and half a dozen SaaS vendors — mostly names with two dots.

The default optimises for a world that stopped existing a decade ago. It’s still the default because changing it would break clusters that depend on the old behaviour, which is the most ordinary reason anything stays the way it is.

The short version

Kubernetes counts dots to decide if a name needs help. Fewer than five means “expand it.”
api.stripe.com has two dots, so it gets expanded — four guaranteed failures before the right answer.
Double it for IPv6. Ten queries, one answer, every time.
Exactly 5 seconds of latency is never your code. It’s a dropped DNS packet waiting on a timeout, and ndots gives it ten chances to happen instead of two.
The fix is three lines of YAML. ndots: 2 on your outward-facing services, or a trailing dot if you need something today.
Check the ratio. If DNS queries vastly outnumber actual requests, you’re already paying this tax.

Go look at a pod that calls an external API. Count how many questions it asks to get one answer. If the number is ten, it has been ten this whole time.