Welcome to the twenty-fifth post in our Kubernetes A-to-Z Series. Kubernetes is famously small at its core. The API server, scheduler, controller manager, kubelet, and kube-proxy do not actually know about cert-manager, Istio, Argo CD, Prometheus, or any of the dozens of platform tools you probably run in production. So how does an out-of-the-box cluster grow into a full platform without forking the upstream code?

The answer is eXtensibility. Kubernetes was built from day one to be extended, not modified. Every popular add-on you have ever installed plugs into one of a small number of well-defined extension points. In this post we walk through the four big ones: Custom Resource Definitions (CRDs), admission webhooks, the API aggregation layer, and kubectl plugins.

Why Kubernetes is Extensible by Design

The API server is the only component every other piece talks to. It is a thin layer over a strongly-typed, declarative object store. That architecture has two consequences that make extension natural.

┌─────────────────────────────────────────────────┐
│  Kubernetes Control Plane                       │
│                                                 │
│   kubectl, controllers, operators, dashboards   │
│              │                                  │
│              ▼                                  │
│  ┌─────────────────────────────────────────┐    │
│  │           API Server                    │    │
│  │  Authentication                         │    │
│  │  Authorization                          │    │
│  │  Mutating  admission                    │    │
│  │  Schema    validation                   │    │
│  │  Validating admission                   │    │
│  │  Persist to etcd                        │    │
│  └─────────────────────────────────────────┘    │
│              │                                  │
│              ▼                                  │
│           etcd (state)                          │
└─────────────────────────────────────────────────┘

First, every object is a typed resource with a spec (desired state) and a status (observed state). If you can teach the API server about a new type, the rest of the ecosystem (kubectl, RBAC, audit logs, garbage collection, owner references, finalizers) works for free. That is what CRDs do.

Second, every write request passes through a request pipeline. If you can plug into that pipeline at the right moment, you can validate, mutate, or reject requests without recompiling Kubernetes. That is what admission webhooks do.

Third, when even a CRD is not enough (you need a custom storage backend, custom subresources, or aggregated streaming), you can register your own API server and have the main API server forward requests to it. That is the API aggregation layer.

Finally, kubectl itself is extensible via plugins, so the operator experience can grow with the platform.

Together these four mechanisms let teams ship cert-manager, service meshes, GitOps controllers, policy engines, and serverless frameworks on top of a vanilla cluster.

Custom Resource Definitions

A CustomResourceDefinition (CRD) registers a new resource type with the API server. Once it is installed, users can kubectl get, kubectl describe, RBAC-restrict, and audit the new resource exactly like a built-in Pod or Deployment.

Anatomy of a CRD

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.platform.example.com
spec:
  group: platform.example.com
  scope: Namespaced
  names:
    plural: backups
    singular: backup
    kind: Backup
    shortNames:
    - bk
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        required: [spec]
        properties:
          spec:
            type: object
            required: [source, schedule]
            properties:
              source:
                type: string
                description: "PVC name to back up"
              schedule:
                type: string
                pattern: '^(\*|[0-9,\-\*/]+) (\*|[0-9,\-\*/]+) (\*|[0-9,\-\*/]+) (\*|[0-9,\-\*/]+) (\*|[0-9,\-\*/]+)$'
              retention:
                type: integer
                minimum: 1
                maximum: 365
                default: 7
          status:
            type: object
            properties:
              lastBackupTime:
                type: string
                format: date-time
              phase:
                type: string
                enum: [Pending, Running, Succeeded, Failed]
              conditions:
                type: array
                items:
                  type: object
                  properties:
                    type: { type: string }
                    status: { type: string }
                    reason: { type: string }
                    message: { type: string }
    subresources:
      status: {}
      scale:
        specReplicasPath: .spec.replicas
        statusReplicasPath: .status.replicas
    additionalPrinterColumns:
    - name: Source
      type: string
      jsonPath: .spec.source
    - name: Schedule
      type: string
      jsonPath: .spec.schedule
    - name: Phase
      type: string
      jsonPath: .status.phase
    - name: Age
      type: date
      jsonPath: .metadata.creationTimestamp

Key fields worth understanding:

  • group, versions, names: how the resource is addressed. Together they form the GVK (Group / Version / Kind), the unit of identity in Kubernetes.
  • schema.openAPIV3Schema: the validation contract. The API server rejects any object that does not match. Without this, your CRD effectively accepts arbitrary YAML, which is rarely what you want.
  • subresources.status: enables /status as a separate endpoint. Controllers update status, users update spec, and RBAC can split the two. This is critical for the controller pattern.
  • subresources.scale: lets kubectl scale work against your custom resource, which is how HorizontalPodAutoscaler ends up scaling third-party workloads.
  • additionalPrinterColumns: what shows in kubectl get backups. Investing five minutes here pays off every day.

Creating and Using a Custom Resource

kubectl apply -f backup-crd.yaml
kubectl get crd backups.platform.example.com

cat <<'EOF' | kubectl apply -f -
apiVersion: platform.example.com/v1
kind: Backup
metadata:
  name: nightly-postgres
  namespace: production
spec:
  source: postgres-data
  schedule: "0 2 * * *"
  retention: 14
EOF

kubectl get backups
kubectl get bk -n production
kubectl describe backup nightly-postgres -n production

The API server now stores the object in etcd, enforces the OpenAPI schema, and serves it on /apis/platform.example.com/v1/namespaces/production/backups/nightly-postgres. RBAC works (kubectl auth can-i create backups), audit logs work, and kubectl explain backups.spec describes the schema.

The Status Subresource

The single most-overlooked CRD feature is the status subresource. When enabled, the API server splits writes into two endpoints.

PUT /apis/.../backups/nightly-postgres          updates .spec
PUT /apis/.../backups/nightly-postgres/status   updates .status

This matters because the controller pattern assumes the controller owns status and the user owns spec. Without the subresource, every user-level edit can clobber controller state and every controller reconcile can clobber user edits. The split also lets you grant update on the resource without granting update on /status, so users cannot fake “Succeeded” on their own Backup objects.

CRD Versioning and Conversion

Every CRD lives forever, which means schema evolution is a real problem. The recommended pattern:

  1. Add v1beta1 first, mark served: true, storage: true.
  2. When the schema stabilizes, add v1 alongside v1beta1. Set storage: true on v1 only.
  3. Provide a conversion webhook if the schemas are not bitwise compatible. Kubernetes will call your webhook to translate objects between versions on read and write.
  4. After clients migrate, set served: false on v1beta1 and eventually remove it.

Skipping the conversion webhook is fine when the two versions only add optional fields. The moment you rename, retype, or restructure anything, you need a conversion webhook. Plan for this before you ship v1beta1 to real users.

Controllers and Operators

A CRD by itself does nothing. It is data sitting in etcd. To make a Backup actually back something up, you need a controller: a process that watches the resource and reconciles desired state with reality.

An Operator is the pattern of packaging one or more CRDs together with the controller that understands them. cert-manager defines Certificate, Issuer, and ClusterIssuer; its controller watches those and reconciles real TLS certificates via ACME. Prometheus Operator defines ServiceMonitor, Prometheus, and Alertmanager; its controller reconciles real Prometheus deployments.

We covered the operator pattern, the control loop, the Operator SDK, and OLM in detail in our earlier post on O is for Operators. The two posts are complementary: O focuses on the controller side, X focuses on the broader extension surface (CRD plus webhooks plus aggregation plus kubectl). If you have not read O yet, do that next.

The one rule that ties both posts together: CRDs are useless without a controller. If you ever find yourself shipping a CRD with no reconciler, you are using Kubernetes as a YAML database, which works but rarely justifies the operational cost.

Admission Webhooks

CRDs extend the type system. Admission webhooks extend the request pipeline. They run inside the API server’s write path after authentication and authorization, and before persistence to etcd.

Request lifecycle for any write:

  kubectl apply ─►  API server
                       │
                       ├─ Authentication  (who are you?)
                       ├─ Authorization   (can you do this?)
                       ├─ Mutating  admission webhooks   ─► may modify the object
                       ├─ Schema validation              ─► OpenAPI / CRD schema
                       ├─ Validating admission webhooks  ─► may reject the request
                       └─ Persist to etcd

There are two kinds, and the order matters.

MutatingAdmissionWebhook

A MutatingAdmissionWebhook can modify the incoming object before it is stored. Common uses:

  • Sidecar injection. Istio’s istio-sidecar-injector adds an Envoy container to every pod in a labeled namespace. The user never wrote that container in their Deployment; the webhook injects it during admission.
  • Default values. Filling in imagePullPolicy, default resource limits, or organization-required labels.
  • Identity stamping. Adding annotations such as created-by, tenant-id, or cost-center for downstream tooling.
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
- name: inject.platform.example.com
  clientConfig:
    service:
      name: sidecar-injector
      namespace: platform
      path: "/mutate"
    caBundle: <base64 PEM>
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  namespaceSelector:
    matchLabels:
      sidecar-injection: enabled
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Ignore
  timeoutSeconds: 5

ValidatingAdmissionWebhook

A ValidatingAdmissionWebhook can only accept or reject. It runs after mutating webhooks and after schema validation, so it sees the final object that will be persisted. Common uses:

  • Policy enforcement. OPA Gatekeeper and Kyverno are validating webhooks. They reject pods without resource limits, deployments with :latest images, ingresses with wildcard hosts, and so on.
  • Cross-object invariants. Rejecting a Service whose selector matches no Pods, or a NetworkPolicy that contradicts an existing policy.
  • Naming conventions. Enforcing prefixes, namespaces, or label schemas the project requires.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: enforce-resource-limits
webhooks:
- name: limits.policy.example.com
  clientConfig:
    service:
      name: policy-controller
      namespace: policy-system
      path: "/validate"
    caBundle: <base64 PEM>
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["apps"]
    apiVersions: ["v1"]
    resources: ["deployments", "statefulsets"]
  failurePolicy: Fail
  sideEffects: None
  admissionReviewVersions: ["v1"]
  timeoutSeconds: 5

Webhook Reliability: The Bootstrap Problem

Webhooks have one famous failure mode that bites every team eventually. The webhook itself runs as a Pod on the cluster. If that Pod is unhealthy, your API server cannot reach it, and depending on failurePolicy, every matching request fails.

Pick failurePolicy carefully:

failurePolicyBehavior on webhook timeoutWhen to use
FailReject the requestSecurity-critical policies that must not be bypassed
IgnoreAllow the request throughConvenience features (sidecar injection, defaults) that should not break the cluster

And always exclude the webhook’s own namespace from its own scope. The classic outage is a validating webhook on Pods, with failurePolicy: Fail, whose namespaceSelector includes kube-system. When the webhook Pod restarts, it cannot come back up because admitting its own replacement requires the webhook itself. The cluster is bricked until you kubectl delete validatingwebhookconfiguration manually.

Other reliability rules:

  • Set short timeouts. The default is 10 seconds; 1 to 3 seconds is usually enough. The API server adds the webhook latency to every matching write.
  • Run at least two replicas of the webhook with anti-affinity across nodes.
  • Use objectSelector and namespaceSelector to limit scope. A webhook that runs on every Pod everywhere is a single point of failure.
  • Monitor webhook latency and error rate. The apiserver_admission_webhook_admission_duration_seconds metric exposes this directly.

Validating Admission Policy (CEL)

Since Kubernetes 1.30, ValidatingAdmissionPolicy offers a webhook-free alternative for simple rules using CEL expressions. The check runs inside the API server, so there is no network call and no bootstrap dependency.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-resource-limits
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  validations:
  - expression: "object.spec.template.spec.containers.all(c, has(c.resources.limits))"
    message: "Every container must declare resource limits"

For policies expressible in CEL, this is strictly better than a webhook. Reach for webhooks only when you need network calls, external data, or logic that does not fit CEL.

The API Aggregation Layer

CRDs cover roughly ninety percent of extension needs. The remaining ten percent push into territory where CRDs cannot follow: custom storage backends, subresources beyond status and scale, custom output formats, streaming responses, or protocols outside HTTP+JSON.

For that, Kubernetes offers the API aggregation layer. You run your own API server as a Pod, register it with the main API server via an APIService object, and the main API server transparently proxies matching requests to yours.

                    ┌──────────────────┐
  kubectl  ────────►│  kube-apiserver  │
                    │                  │
                    │   /api/v1/...    │  served locally
                    │   /apis/...      │
                    │                  │
                    │   /apis/metrics  │  proxied via APIService
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ metrics-server   │  your aggregated API server
                    │ (in-cluster Pod) │
                    └──────────────────┘

The canonical example is metrics-server, which serves metrics.k8s.io. Pod metrics are too high-volume and time-sensitive to live in etcd, so metrics-server keeps them in memory and serves them through the aggregation layer. To kubectl and to HorizontalPodAutoscaler, this is invisible: it looks exactly like a built-in API.

Registering an Aggregated API

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
    port: 443
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: false
  caBundle: <base64 PEM>
  groupPriorityMinimum: 100
  versionPriority: 100

CRD or Aggregated API: How to Choose

FactorCRDAggregated API
Setup complexityLow (one YAML)High (run your own API server)
Storageetcd, automaticYou choose (in-memory, SQL, etc.)
Schema validationOpenAPI v3 onlyAnything you implement
Custom subresourcesstatus, scale onlyAny number you want
Watch and listFreeYou implement
RBAC, audit, kubectlFreeFree (proxied through main API server)
Right choice forConfiguration objects, declarative stateHigh-volume metrics, custom protocols, non-etcd storage

Start with a CRD. Only move to aggregation when you hit a wall you cannot climb with a CRD plus a controller.

kubectl Plugins and krew

The fourth extension point is the operator interface itself. Any executable on your PATH named kubectl-foo becomes invokable as kubectl foo. That is the entire plugin protocol.

cat > /usr/local/bin/kubectl-whoami <<'EOF'
#!/usr/bin/env bash
kubectl config view --minify -o jsonpath='{.contexts[0].context.user}'
echo
EOF
chmod +x /usr/local/bin/kubectl-whoami

kubectl whoami

For discovery and version management, the community maintains krew, a plugin manager analogous to Homebrew or apt.

kubectl krew install ctx ns tree neat who-can stern
kubectl ctx production
kubectl ns kube-system
kubectl tree deployment nginx
kubectl neat get pod nginx -oyaml
kubectl who-can create pods
kubectl stern -l app=nginx

Plugins are the lowest-ceremony extension point in Kubernetes. They cannot change cluster behavior, but they can dramatically improve the day-to-day experience of working with one. If your team keeps writing the same shell snippet, package it as a plugin.

Real-World Examples

Three production projects illustrate how these mechanisms combine.

cert-manager is a textbook CRD plus controller. It ships Certificate, Issuer, ClusterIssuer, CertificateRequest, Order, and Challenge CRDs, all with status subresources. A single controller watches all of them and reconciles real ACME flows against Let’s Encrypt, Venafi, HashiCorp Vault, or a self-signed CA. No webhooks, no aggregation. Pure CRD plus reconciler. The lesson: most platform problems do not need anything fancier than this.

Istio is a hybrid. It defines CRDs (VirtualService, DestinationRule, Gateway) and a controller (istiod) that pushes configuration to Envoy sidecars. But it also runs a mutating admission webhook for sidecar injection: when a Pod is created in a labeled namespace, Istio’s webhook adds the Envoy container during admission. Without the webhook, users would have to manually edit every Deployment. With the webhook, the mesh is transparent.

Argo CD is heavy on CRDs. Application, ApplicationSet, AppProject, and Repository are all CRDs with rich status subresources. Argo CD’s controller reconciles cluster state against Git, and its UI is a separate process that reads the CRDs through the API server. The result is a GitOps system that is itself fully declarative: you can manage Argo CD with Argo CD, because everything Argo CD knows about is a Kubernetes object.

Pitfalls

A short list of pitfalls every platform team eventually hits.

Webhook bootstrap deadlocks. Already covered above. The failurePolicy: Fail plus self-targeting namespace combination kills clusters. Always exclude kube-system and the webhook’s own namespace.

CRD version migrations. Once a CRD is in production, removing fields, renaming fields, or changing types requires a conversion webhook. Decide on v1beta1 versus v1 versus v1alpha1 deliberately. Anything marked v1 is an implicit forever-promise.

Schema permissiveness. A CRD without required fields, enum constraints, or pattern validation accepts almost anything. Users will then find creative ways to break your controller. Tighten the schema before you ship.

Controllers that fight users. If your reconciler always overwrites spec, users cannot edit the resource. The status subresource exists precisely so the controller can update state without touching user input. Use it.

Webhook latency. Every webhook adds milliseconds to every matching request. Cluster-wide Pod webhooks are particularly dangerous: a slow webhook can make kubectl get pods feel sluggish across the whole cluster.

Plugin proliferation. Krew makes it easy to install dozens of plugins. Few teams audit what they install. Treat plugins like any other dependency: pin versions, review code for sensitive operations, and prefer well-maintained projects.

Forgetting RBAC. A CRD by itself is invisible to RBAC. You must define Role and ClusterRole rules over the new resource, or only cluster-admins will be able to use it.

Extensibility Cheatsheet

MechanismUse whenComplexityReal example
CRDYou need a new declarative resource typeLowCertificate (cert-manager)
CRD plus controllerYou want behavior, not just dataMediumArgo CD Application
MutatingAdmissionWebhookYou need to modify objects on writeMediumIstio sidecar injection
ValidatingAdmissionWebhookYou need to enforce a policyMediumOPA Gatekeeper
ValidatingAdmissionPolicy (CEL)The policy fits in a CEL expressionLowRequire resource limits
API aggregation layerYou need custom storage or protocolsHighmetrics-server
kubectl pluginYou want better operator UXVery lowkubectl ctx, kubectl tree
Conversion webhookA CRD schema must evolve incompatiblyMediumAny long-lived CRD v1beta1 to v1

Key Takeaways

  • Kubernetes is intentionally small at the core and intentionally extensible at the edges. Every major add-on plugs into one of four extension points.
  • CRDs add new resource types. They get RBAC, audit, kubectl, and watch semantics for free. Use OpenAPI schemas and the status subresource from the start.
  • Admission webhooks plug into the write pipeline. Mutating webhooks modify, validating webhooks reject. Pick failurePolicy carefully, set short timeouts, and never let a webhook gate its own bootstrap.
  • The API aggregation layer is for cases where CRDs cannot reach, such as custom storage or protocols. Start with CRDs, escalate only when needed.
  • kubectl plugins are the lowest-ceremony way to improve the operator experience. Krew is the package manager.
  • Real-world systems combine these mechanisms. cert-manager is CRD plus controller; Istio adds a mutating webhook; Argo CD is mostly CRDs with a polished UI.

Next Steps

You have now seen every extension point Kubernetes exposes. Pair this post with O is for Operators for the controller side, K is for Kubernetes Basics and Architecture for the API server context, and B is for Best Practices for production reliability patterns that apply to the controllers and webhooks you ship.

Resources for Further Learning