Kubernetes networking is one of the most misunderstood parts of running containerized workloads. A pod can reach another pod by IP — but why does that stop working after a deployment? A service exists and resolves in DNS — but traffic isn't arriving at the application. An Ingress resource is configured — but requests return 502. These puzzles are common and they stem from the same root: Kubernetes networking has several distinct layers, each solving a different problem, and it's easy to conflate them.
This article walks through how Kubernetes networking actually works at each layer — from pod networking to services to Ingress to network policy — so the next time something breaks, you have a mental model to reason from.
The fundamental promise: flat pod networking
Kubernetes makes one core promise about networking: every pod can communicate directly with every other pod in the cluster without NAT. Every pod gets a real IP address from the cluster's pod CIDR range, and those IPs are routable between pods regardless of which node they're running on.
This is not something Kubernetes itself implements. It's a contract that every Kubernetes-conformant CNI (Container Network Interface) plugin must fulfill. When you install Calico, Cilium, Flannel, Weave, or any other CNI, you're installing the component that actually creates this flat network. The mechanism varies — Flannel uses VXLAN overlays, Calico can use BGP for direct routing, Cilium uses eBPF — but the result is the same: pod-to-pod communication without NAT.
Here's what a pod's network namespace looks like:
$ kubectl exec -it my-pod -- ip addr
1: lo: <LOOPBACK> ...
3: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
inet 10.244.1.15/24 brd 10.244.1.255 scope global eth0
$ kubectl exec -it my-pod -- ip route
default via 10.244.1.1 dev eth0
10.244.0.0/16 via 10.244.1.1 dev eth0
The pod has an IP (10.244.1.15) on a /24 subnet. The node this pod runs on has an IP from the same range — or a different /24 within the same /16. Traffic from this pod to 10.244.2.8 (a pod on another node) goes through the CNI-managed overlay or routed network.
Why you shouldn't use pod IPs directly
If pods can communicate directly, why not just use pod IPs? Two reasons.
First, pod IPs are ephemeral. When a pod is killed and rescheduled — whether from a node failure, a rolling deployment, or the scheduler moving it — it gets a new IP. Any hardcoded reference to the old IP breaks immediately.
Second, horizontal scaling creates multiple pod instances. If you have three replicas of your API service, you need something that routes traffic across all three, not just the first one.
Services solve both problems.
Services: stable DNS names and load balancing
A Kubernetes Service is a stable virtual endpoint that fronts a set of pods. It has a cluster-scoped DNS name (my-service.my-namespace.svc.cluster.local) and a virtual IP (ClusterIP) that doesn't change. The Service uses a label selector to determine which pods are in its backend pool:
apiVersion: v1
kind: Service
metadata:
name: api
namespace: backend
spec:
selector:
app: api
tier: backend
ports:
- port: 80
targetPort: 8080
Kubernetes continuously watches pods with labels app=api, tier=backend. When pods matching the selector are created, their IP:port is added to an Endpoints object. When they're deleted, they're removed. The Service IP stays the same; the backing Endpoints change as pods come and go.
Traffic routing to services is handled by kube-proxy (or Cilium/eBPF in modern clusters). kube-proxy watches for Endpoints changes and programs iptables rules (or IPVS rules) that intercept traffic to the ClusterIP and distribute it across the current pod set.
ClusterIP, NodePort, and LoadBalancer
Services have a type field that controls how they're exposed:
ClusterIP (default): The service is accessible only within the cluster. Other pods can reach it at api.backend.svc.cluster.local:80. External traffic cannot reach it. This is the right type for internal services — databases, caches, backend APIs called only by other services.
NodePort: Opens a port on every node (in the 30000–32767 range by default) and routes traffic to the service. You can reach the service at <any-node-ip>:<node-port>. NodePort services are rarely used in production for external traffic because they require knowing node IPs and using non-standard ports. They're sometimes used for development or for specific cases where an external load balancer handles the stable IP layer.
LoadBalancer: Provisions an external load balancer from the cloud provider (an AWS ALB/NLB, a GCP load balancer, an Azure LB). The load balancer gets a stable external IP and routes to the NodePort service underneath. This is the straightforward way to expose a single service externally, but it creates one load balancer per service — expensive at scale.
ExternalName and headless services
Two less-common service types worth knowing:
ExternalName: Returns a CNAME record to an external DNS name. Useful for giving a cluster-internal alias to an external database or API: db.backend.svc.cluster.local resolves to my-db.rds.amazonaws.com. Services can reference external dependencies by internal name without embedding external DNS names in application code.
Headless services (clusterIP: None): Instead of returning the virtual ClusterIP, DNS returns the actual pod IPs. Each DNS query returns the IPs of all pods matching the selector. This is used for stateful applications (StatefulSets) where clients need to connect to specific pod instances, not a load-balanced virtual IP. MongoDB, Cassandra, and other distributed databases use headless services so each pod is individually addressable.
DNS in Kubernetes
Kubernetes runs CoreDNS (or kube-dns in older clusters) as a cluster DNS resolver. Every pod is configured with a resolv.conf that points to CoreDNS:
$ kubectl exec -it my-pod -- cat /etc/resolv.conf nameserver 10.96.0.10 search backend.svc.cluster.local svc.cluster.local cluster.local options ndots:5
The search entries mean that short service names like api are automatically expanded. A lookup for api from within the backend namespace first tries api.backend.svc.cluster.local, finds the service, and returns. A lookup from a different namespace would need api.backend or the full FQDN.
The ndots:5 setting means names with fewer than 5 dots are treated as relative and go through the search path before being treated as absolute. This can cause unexpected DNS latency if every external hostname lookup triggers multiple failed cluster-internal lookups first. Setting ndots: 2 or using fully-qualified hostnames (with a trailing dot) for external lookups can reduce DNS lookup latency in high-throughput services.
Ingress: HTTP routing at the cluster edge
LoadBalancer services expose one service per load balancer. In practice, a production cluster might have 20 HTTP services that should all be accessible under a single external IP. Ingress handles this with HTTP-level routing — one external load balancer, multiple services, routing by hostname and path.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: api.mycompany.com
http:
paths:
- path: /v1
pathType: Prefix
backend:
service:
name: api-v1
port:
number: 80
- path: /v2
pathType: Prefix
backend:
service:
name: api-v2
port:
number: 80
- host: admin.mycompany.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: admin
port:
number: 80
An Ingress object is just configuration. The actual traffic routing is done by an Ingress controller — a deployment running in your cluster that watches Ingress resources and programs an actual reverse proxy accordingly. Common controllers:
- nginx-ingress-controller: The most widely deployed. Configures nginx based on Ingress resources. Mature, well-documented, broad feature support via annotations.
- AWS Load Balancer Controller: For EKS, provisions AWS ALBs instead of nginx. Routes HTTP/HTTPS at the ALB layer, with support for target groups, WAF integration, and certificate management via ACM.
- Traefik: Auto-discovers Kubernetes services, has a dashboard, good support for middleware (rate limiting, auth, header rewriting).
- Istio Gateway: Part of the Istio service mesh. More complex but enables traffic management, mTLS, and observability beyond what a simple Ingress controller provides.
The Ingress spec is intentionally minimal — just host/path rules and backend services. Advanced features (SSL termination, authentication, rate limiting, sticky sessions) are handled via controller-specific annotations. This means Ingress resources are somewhat controller-specific despite being a standard API object.
For clusters on Kubernetes 1.19+, ingressClassName replaces the older kubernetes.io/ingress.class annotation and allows multiple Ingress controllers to coexist in the same cluster, each handling different Ingress objects.
Network Policies: restricting traffic between pods
By default, Kubernetes networking is fully permissive. Any pod can reach any other pod, any service, and any external IP. In a shared cluster, this means a compromised pod can attempt to reach databases, other teams' services, or cluster control plane components.
NetworkPolicy resources define firewall rules for pod-to-pod and pod-to-external traffic:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-network-policy
namespace: backend
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
- namespaceSelector:
matchLabels:
name: monitoring
ports:
- protocol: TCP
port: 8080
egress:
- to:
- podSelector:
matchLabels:
app: database
ports:
- protocol: TCP
port: 5432
- to: # allow DNS
- namespaceSelector: {}
ports:
- protocol: UDP
port: 53
This policy says: pods labeled app=api in the backend namespace accept ingress on port 8080 only from pods labeled app=frontend or from any pod in namespaces labeled name=monitoring. They can only egress to pods labeled app=database on port 5432, plus DNS (port 53 UDP to any namespace).
Two things to know about NetworkPolicy:
NetworkPolicy requires a compatible CNI. Not every CNI plugin enforces NetworkPolicy rules. Flannel does not enforce NetworkPolicy without a companion policy controller. Calico, Cilium, and Weave enforce NetworkPolicy natively. If your cluster's CNI doesn't support it, NetworkPolicy objects exist in the API but have no effect — silently.
Default-deny is not automatic. A NetworkPolicy that selects a pod switches that pod from the default-allow to the deny-unless-explicitly-allowed model. But you have to write the NetworkPolicy for it to apply. A common pattern is a default-deny policy that selects all pods in a namespace, then explicit allow policies for specific traffic:
# Default deny all ingress and egress for all pods in namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: backend
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
Common networking problems and how to debug them
Pod can't reach a service by DNS name. Check that CoreDNS is running: kubectl get pods -n kube-system -l k8s-app=kube-dns. Check the service exists: kubectl get svc. Check the Endpoints object has pods: kubectl get endpoints my-service. If Endpoints is empty, the service's selector isn't matching any pods — check pod labels match the service selector exactly.
Service exists but traffic isn't reaching pods. Check the Endpoints: kubectl describe endpoints my-service. If there are endpoints, the problem is between kube-proxy and the pods. Run kubectl exec -it debug-pod -- curl http://<pod-ip>:<port> to check direct pod connectivity. Also check that the pod's containerPort matches the service's targetPort.
Ingress returns 502. The Ingress controller reached the backend service, but the service is returning an error or not listening. Check that the backend service exists and its Endpoints are populated. Check pod logs for errors. A 502 at the Ingress layer almost always means the backend is reachable but failing, not an Ingress configuration problem.
NetworkPolicy broke everything. If you applied a NetworkPolicy and traffic stopped working, check what DNS looks like — often the first symptom is DNS failures because the default-deny policy blocks UDP/53 egress. Make sure your network policy allows egress to the DNS port, or tests will fail mysteriously on service name resolution before you get to the actual policy issue you were trying to configure.
Kubernetes manifests describe all of these components — pods, services, Ingress, NetworkPolicy — in text form. Visualizing how they connect can be useful when debugging or reviewing changes to a cluster's configuration. InfraSketch parses Kubernetes YAML and generates diagrams that show which services expose which pods and how Ingress rules route between them.