[23/24] B is for Best Practices: Building Secure and Reliable Apps

This is Post #23 in the Kubernetes A-to-Z Series

Reading Order: Previous: GitOps | Next: Etcd

Series Progress: 23/24 complete | Difficulty: Intermediate | Time: 20 min | Part 6/6: Security & Production

Welcome to the second post in our Kubernetes A-to-Z Series! Now that you have a grasp of the basic architecture, let’s talk about how to use Kubernetes correctly. It’s easy to deploy a container, but deploying it securely and reliably requires following some golden rules.

In this post, we’ll cover the “B” of Kubernetes: Best Practices.

1. Resource Requests and Limits

One of the most common mistakes is deploying pods without defining resource requests and limits.

Why it matters

  • Reliability: Without limits, a single buggy pod can consume all CPU/Memory on a node, crashing other pods (the “noisy neighbor” problem).
  • Scheduling: The scheduler uses requests to decide where to place pods. Without them, it’s flying blind.

The Golden Rule

Always set requests and limits for both CPU and Memory.

resources:
  requests:
    memory: "64Mi"  # Guaranteed minimum
    cpu: "250m"     # 0.25 core guaranteed
  limits:
    memory: "128Mi" # Hard cap (OOMKilled if exceeded)
    cpu: "500m"     # Throttled if exceeded

Pro Tip: Keep requests close to actual usage. Keep limits for memory close to requests to avoid OOM kills on overcommitted nodes. CPU limits can be higher to allow bursting.

2. Health Probes (Liveness & Readiness)

Kubernetes needs to know if your application is running and if it’s ready to accept traffic.

Liveness vs. Readiness

  • Liveness Probe: “Is the app dead?” If this fails, Kubernetes restarts the container.
  • Readiness Probe: “Is the app busy?” If this fails, Kubernetes stops sending traffic to the pod (removes it from the Service load balancer).

Example Configuration

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 3
  periodSeconds: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

3. Security Contexts & Non-Root Users

By default, many containers run as root. This is a major security risk. If an attacker compromises the container, they might gain root access to the node.

The Golden Rule

Never run containers as root.

securityContext:
  runAsUser: 1000
  runAsGroup: 3000
  fsGroup: 2000
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true

4. Use Namespaces

Don’t dump everything into the default namespace.

Why?

  • Isolation: Separate teams or environments (dev, staging, prod).
  • Resource Quotas: Limit how much resource a team can use.
  • Access Control: easier to apply RBAC policies.
kubectl create namespace production
kubectl create namespace development

5. The 12-Factor App in Kubernetes

Kubernetes was designed with the 12-Factor App methodology in mind.

  1. Codebase: One repo, many deploys.
  2. Dependencies: Explicitly declare (Docker image).
  3. Config: Store config in the environment (ConfigMaps/Secrets).
  4. Backing Services: Treat databases/queues as attached resources.
  5. Build, Release, Run: Strictly separate stages (CI/CD).
  6. Processes: Execute the app as one or more stateless processes.
  7. Port Binding: Export services via port binding.
  8. Concurrency: Scale out via the process model (Replicas).
  9. Disposability: Fast startup and graceful shutdown.
  10. Dev/Prod Parity: Keep development, staging, and production as similar as possible.
  11. Logs: Treat logs as event streams (stdout/stderr).
  12. Admin Processes: Run admin/management tasks as one-off processes (Jobs).

Summary Checklist

Before you deploy to production, check these boxes:

  • Resources: Requests and limits are set.
  • Probes: Liveness and Readiness probes are configured.
  • Security: Running as non-root, read-only filesystem where possible.
  • Config: Configuration is externalized (ConfigMaps/Secrets).
  • Labels: Proper labels for selection and organization.
  • Replicas: More than 1 replica for high availability.

Next Steps

Now that you know the rules of the road, let’s dive into the brain of the operation. Next up is E is for Etcd, where we’ll look at where all this state is stored.

Series Navigation: