Kubernetes Security: Common Issues and Best Practices

There’s a myth out there that Kubernetes is not secure. In fact, Kubernetes supplies a good number of security controls you can use to make your clusters, workloads and containers as secure as you want.

Kubernetes Security Issues

There are three real Kubernetes security issues:

  1. If you deploy Kubernetes yourself from open source none of these controls are configured, leaving it up to you to figure out how they work and how to configure them.
  2. Whether you use a Kubernetes distribution that has security controls pre-configured, or you build it all yourself, developers and application teams that may not be familiar with all the ins and outs of Kubernetes might find it frustrating to figure out how best to deploy their workloads in your cluster that will pass a security audit.
  3. Many of the controls in Kubernetes are access controls and options geared towards helping you create a secure cluster. But there is nothing built-in that will ensure the containers and code running on the cluster are safe.

Kubernetes Security Solutions

It takes additional Kubernetes security solutions beyond the built-in features to provide the complete picture. Fortunately, the Kubernetes security solutions ecosystem is robust so there are no shortage of choices. Some areas to consider:

  • Workload security: the majority of Kubernetes workloads are containers running on Docker engines. You might use CRI-O or containerd in some circumstances as well, but no matter which engine is running underneath the covers, you still run the same containers. The code and other packages in those containers should be free from vulnerabilities.
  • Workload configuration: whether you use Kubernetes YAML, Helm Charts, or templating tools, the configuration for deploying your applications in Kubernetes is typically done in code. That configuration code affects the Kubernetes security controls that determine how a workload runs, and what can or cannot happen in the event of a breach.
  • Cluster configuration: There are a number of tools that will help run Kubernetes security assessments against your running clusters to check for Kubernetes security best practices and compare against benchmarks like CIS.
  • Kubernetes networking: Networking in Kubernetes is a big topic and securing the network is an important consideration. You should consider pod communications, ingress, egress, service discovery, and, if you determine you need one, service meshes
  • Infrastructure security: Kubernetes is essentially a distributed application that you will run across many servers along with the requisite physical and virtual networking and storage. If someone can breach your infrastructure they may be able to find everything they need to get into your Kubernetes cluster and your applications. You need to secure the infrastructure, and particularly the Kubernetes master nodes, databases, and certificates.
kubernetes security tool demo
Kubernetes Workload Configuration Scanning Result

Kubernetes Container Security

The fundamental unit of work in Kubernetes is the pod. In most cases, a Kubernetes pod is just a container but it could be multiple containers. Kubernetes security can control how that pod operates but it does not do any inspection of the containers to ensure they are safe and approved to run. That’s left to you, the user, and whatever tooling you choose to add.

We have provided an in-depth container security best practices guide but here are four key highlights:

  1. Secure your code and dependencies: Snyk Open Source performs a full dependency analysis for code to uncover your open source dependencies and any vulnerabilities they contain, then helps developers fix them automatically.
  2. Secure your container base image: Your workload inherits everything that comes with the base image you choose to build upon, so select minimal base images, add only what you require, and scan them for vulnerabilities.
    Ideally, the tool you choose for this function should be aware of your base image choice and alert you when there is an updated version of your base image or if there are alternate base images that can reduce vulnerabilities.
  3. Secure the layers you add: containers are built up in layers, typically spelled out in a Dockerfile, but vulnerabilities are often found in libraries deep in the dependency tree of the tools you install.
    Your container vulnerability assessment tool should uncover vulnerabilities and relate them to the layers in the container image, and help you track dependencies so you can determine which of the tools you have installed are introducing vulnerabilities.
  4. Manage your configuration: there are two layers of configuration security to be aware of: (1) the configuration of the container engine; and, (2) the configuration of your workload.

    The Docker Engine generally comes with sensible defaults, and if you’re using a Kubernetes platform distribution, like OpenShift, VMware Tanzu/PKS, AKS, EKS or GKE, they will already have the container runtime locked down.

    But Kubernetes security for the workload configuration is up to the user. At a minimum, you should have policies for workload security and resource controls that are agreed upon by developers, operators, and security teams. Ideally, since the workload configuration is code, it would be tested in your pipelines, just like any other code.

Kubernetes security best practices

As you can tell, there is a large number of Kubernetes security considerations. Given all of this, what is the best way to get started and stay secure? Here are 3 tips to ensure Kubernetes security:

1. People and process are critical

Most of what we have described here deals with technology, but your people and process are just as important. Running containers and Kubernetes will affect developers, security, infrastructure and operations teams – basically everyone in IT and development.

So start small and build your knowledge base and core experts across disciplines, and don’t try to do it all on your own. Get help from the Kubernetes community and ecosystem and from vendors and service partners with expertise in rolling out Kubernetes and all that entails. These partners can also provide ongoing Kubernetes security assessments to ensure you keep up with the latest best practices.

2. Use a supported Kubernetes distribution from a vendor you trust

You are almost always better off using a supported Kubernetes distribution from a vendor you trust, rather than trying to setup Kubernetes yourself for a production environment. There are over 90 certified conformant Kubernetes distributions, and with a distribution you can make sure that platform security is built-in for things like role-based access control; but, there should be at least some security for the network and possibly admission controllers and pod security policies for workloads as well. This sets you on the right path for security, but does not absolve you of the need to check for Kubernetes and container security vulnerabilities or misconfigurations.

3. Layer on the tools to secure the workload and containers

Kubernetes is an orchestrator and a set of APIs that can be used to build and run all sorts of workloads. But it’s not a complete solution as-is for most production environments. Layer on the tools you need to complete the picture: secure the workload and containers, as detailed above, and consider how you will monitor running workloads:

  • Behavioral analysis and network monitoring are worth considering, though they are not trivial to implement.
  • Logging and monitoring work different in containers and Kubernetes and you will likely want tools and process that are built for these environments.
  • Networking and storage are not built-in, rather they are handled by plugins. Your distribution will have defaults but you can likely choose others and if you find you do need a service mesh at some point there are many options for that as well.

4. Find and fix vulnerabilities and secure your running workloads

Reduce your blast radius by eliminating security vulnerabilities in your application code, dependencies, and containers and securing your running workloads. Many of these security issues are going to relate back to code of some sort: your applications, the container build files, or the workload configurations. That means you need more than just a list of vulnerabilities and issues – you need to make sure the developer and DevOps teams responsible for fixing these issues know what to do. A few specific tips:

  • Don’t run containers with root and avoid running privileged pods in Kubernetes: if somebody gets into the container you want to limit access to the rest of the system
  • Set limits on container resources: if a container is breached you can prevent denial of service attacks
  • Secure your pod configuration YAML and use Kubernetes pod security policies: this may seem redundant, after all, if your pod security policy says containers cannot run as root, then why bother explicitly setting this in the workload configuration? Two key reasons:
  • Kubernetes pod security policies are applied at runtime: if you’re a developer and you go to deploy a workload and it gets all the way to production before failing the pod security policy it’s frustrating. Enforcing this in the workload configuration means that the policy is enforced anywhere the workload is run, even a single node setup on the developer’s workstation.
  • Similarly, you may grow to the point in your Kubernetes usage where you’re using multiple Kubernetes distributions and each distribution has different built-in settings. You will hopefully be able to set pod security policies everywhere, but having the security built-in to the configuration of the workload provides a layer of insurance.

Test your application for vulnerabilities

By submitting this form you consent to us emailing you occasionally about our products and services.
You can unsubscribe from emails at any time, and we will never pass your email onto third parties. Privacy Policy

Jim Armstrong
July 26, 2020
| By Jim Armstrong