Six practices for implementing Kubernetes on GCP

Summary

This article outlines our recommendations for implementing and managing Kubernetes on GCP with GKE, based on our experience with the system across multiple large scale production platforms and environments. Below, we discuss our recommendations and implementation examples for:

1. Local development
2. Building containers
3. Container runtime
4. Running a Kubernetes cluster
5. Ingress node pools
6. Deployments

1. Local development

Minikube

Run Kubernetes on all developer machines. Use Minikube to achieve this. Ensure that you’re using the same version of Kubernetes locally as in production. GKE default is 1.24 at time of writing.

Skaffold

Use Skaffold for iterative development and deployment to Kubernetes clusters. This tool watches for changes to your local environment and rebuilds the container and deploys it to the cluster.

2. Building containers

Public Images

Don’t trust or use the containers published in public registries unless 100% certain of their authenticity. There have been documented cases of malware/rootkits installed in popular images:

When in doubt, use only for guidance and rebuild from source.

Base Image

Restrict the set of base images that can be used for all deployed containers. Ideally just use one. Suggestions:

• alpine:3.17
• debian:bullseye
• distroless

Using the same base container allows control over when base packages get updated.

Pid 1

Unless you’re using a Distroless image, it’s generally recommended to use a lightweight init system within your container. For example, install Tini into the base image and use it as the entrypoint for all containers to ensure signals are handled properly and zombie processes are properly reaped.

Minimize Image Size

Don’t include non-runtime dependencies in the built image. This includes toolchain dependencies like JDK, gcc, etc. Use Docker layer caching aggressively.

Advantages:

• Faster builds
• Faster image pulls
• Less storage
• Potentially less attack surface

See multi-stage builds and Building Minimal Docker Containers for more details.

Vulnerability Scanning

OS dependencies within Docker containers are rarely updated. Use an image scanner to detect and report vulnerabilities so that they can be updated as appropriate. Images need to be periodically rebuilt to ensure patches are applied.

3. Container runtime

Single Process

Each container should have one responsibility and one process to perform that responsibility. Let Kubernetes manage the orchestration of pods/services. If multiple processes are required to work together, use the pod concept.

Fail Fast

If a fatal error occurs within the process in the container, don’t try to recover — fail fast. This allows Kubernetes to detect which pods/services are healthy and direct traffic accordingly.

Health Checks

Each container running in the cluster must have a liveness and readiness probe defined. See here for more on liveness, readiness, and startup probes.

Combined with “Single Process”, this gives Kubernetes a fine-grained understanding into the health of the pods running within the cluster.

Use a non-root user inside the container

Containers provide a sandbox, not true isolation. Applying the defense in depth principle, create a user in the base image and run as this user within the container. Ensure this by setting a security context within the Kubernetes manifest.

Read-only file system

Container files systems should be read-only by default. Don’t use log rotation techniques within a container, this is the job of Kubernetes.

See here for more security best practices.

Output All Runtime Data to stdout/stderr

Use the structured logs technique and direct log output to stdout/stderr. GKE natively integrates with Cloud Logging to ensure that application logs written to stdout/stderr are automatically collected and sent to Cloud Logging in addition to cluster audit and worker node logs.

Seccomp and AppArmor Profiles

If in the unlikely event a container is breached (via remote shell or similar), limiting the capabilities to only what is necessary to complete the task is important. In the Kubernetes manifest:

apiVersion: v1
kind: Pod 
metadata: 
  name: my-app 
  annotations: 
    seccomp.security.alpha.kubernetes.io/pod: docker/default
spec: 
  containers:
    - name: my-app 
      securityContext: 
        allowPrivilegeEscalation: false
        capabilities: 
          drop: 
            - all 
          add:
            - NET_BIND_SERVICE 
            - ...

Resource requests and limits

Set both “requests” and “limits” for all containers running within the cluster. This will dramatically help with scheduling and auto-scaling. To start with, these limits should be set aggressively low and increased over time if the need presents itself. Audit the Kubernetes event history regularly (either via kubectl or Cloud Logging) to see if Kubernetes is restarting pods due to resource constraints. Performing load tests will help a great deal.

See Resource Management for Pods and Containers for more information.

Workload Identity

Use Workload Identity to allow workloads running in GKE to access GCP services rather than injecting service account keys into containers or using the node’s service account. Workload Identity allows a Kubernetes service account in your GKE cluster to act as a GCP IAM service account. This allows you to implement fine-grained permissions for applications running in GKE.

4. Running a Kubernetes cluster

Pods are ephemeral

Kubernetes pods are mortal. The service should be designed with this in mind. Keep pods as stateless as possible and use liveness/readiness probes to signal their health.

Namespace per product

When starting out with Kubernetes, the default namespace is, well, default. As more services are deployed and more teams start deploying, service name collisions can occur. Kubernetes supports namespaces to provide multi-tenancy. Namespaces are also useful for GKE Usage Metering to track resource usage for cluster workloads.

We recommend creating product-specific namespaces to deploy related services.

Environments

Separate prod/non-prod environments into separate GCP projects. Use Kubernetes RBAC controls to limit who can access which clusters. Use Resource Quotas to ensure that costs don’t get out of control and GKE Usage Metering to accurately attribute costs to tenants/namespaces.

Dedicate VPC to cluster

Create the GKE clusters in a separate VPC to the project’s “default”.

Infrastructure as Code

Use Google Deployment Manager or Terraform to automate the creation and maintenance of all GCP resources. Use standard SDLC around this code.

GKE

• Use regional cluster
• Minimum 3 zones. Use zone-aware deployments in Kubernetes for high availability
• Enable automatic node repair

5. Ingress node pool

Traffic-based autoscaling by GCLB is done at the node level. Create a separate ingress node pool to allow for scaling based on network traffic.

GCLB

Use Google Cloud Load Balancer for all ingress into the cluster. Combine with Cloud Armor for a powerful WAF.

6. Deployments

Automate

All deployments should be automated. Use a tool like Argo or Flux to automate this deployment. Use immutable infrastructure principal, all changes to service should be recreated rather than updated in place.

Design for rollbacks

Production issues are a fact of life. Design with that in mind. The important part is how fast you can detect an issue and rollback to a known good version.

Latest tag

The default tag on Docker images is “latest”. Docker tags are mutable but should not be used as such. See this post for a good description of the problem and some useful tips.

Labels

Use semantic labeling for all Kubernetes objects defined in the cluster. Example for the “myproduct-web” service:

app: myproduct 
service: api 
version: v1.2.5 
deployment: v5 
env: dev
tier: frontend
role: canary

Configuration management

• Use environment variables by default
• Revision config and Kubernetes manifests into a code repository. Use SDLC practices for change management.
• Use GCP’s Secret Manager, Hashicorp Vault or Bitnami Sealed Secrets to manage secrets. Using a git repository to manage the Kubernetes manifests and configuration for each environment allows you to control updates to the cluster in a fine-grained manner. This post describes the GitOps model in more detail.

Want to know more?

We have extensive experience with Kubernetes at scale in a variety of environments, for a range of different use-cases. If there are specific topics you’d like to discuss in depth, we’d love to hear from you. These emails come directly to us, and we respond to every one.