Best practices for container image in K8s

With the adoption of microservice and the Kubernetes ecosystem, container image creation and maintenance become one of important factors for application reliability and security. Based on my experience from fixing and investigating several costly incidents, I can recommend some of the best practices for container image creation and maintenance. However, there is no “one-size-fits-all” solution, and some of the following practices may not exactly fit with the ecosystem of your company due to the choice of tooling or the workflow followed for creating and deploying applications in the production. Nevertheless, the overall idea may help you to increase the security and reliability of your application and infrastructure.

Some of the best practices you may also adopt in creating and maintaining container images are

• Distroless as final base image: Using the Distroless image can drastically reduce the overhead in your container image. I have already explained in this post how Distroless images can enhance the security and reliability of your system and also reduce cost and network overhead on your cloud network.
• No dependency on third-party registries: A lot of third-party applications usually run in a Kubernetes cluster that facilitates the reliable, secure, and efficient operation of a cluster. However, when you rely on those third-party applications for the reliable operation of your clusters and those applications rely on third-party image registries(for example- quay.io, index.docker.io), you partially hand over the reliability of your clusters to those third-party image registries over which you have no control. All the images/applications on which your cluster reliability depends should be always maintained in private image registries to avoid issues due to rate limiting or unavailability of those image registries.
• Globally unique image naming: The name of a container image should not reveal just an application name but it should reveal enough information to identify the owner of the image in your company. The following naming convention would be helpful to identify the owner and also to know some basic information about the application.

[Team]-[Sub-team]-[App name] : [last 15 Characters from git commit SHA]

For example, a commit with hash 881c2d2cffa9375 for an application named MY_APP from Team A and Sub-team B should be named “A-B-MY_APP:881c2d2cffa9375”

The commit SHA as tag makes the container image traceable(and unique) i.e. an image can be mapped to a git commit. The above naming may not fit your company but the main goal should be to make a container image globally unique inside your company regardless of how many Kubernetes clusters or teams are in your company. It should be easy to map a container image to a team, sub-team, and even to a commit in a repository.

• Avoid repetitive image pulling: The image pull policy determines when a container image will be pulled while creating a container in the Kubernetes cluster. For all applications in your Kubernetes clusters, you can use IfNotPresent as the default imagePullPolicy and avoid unnecessary image pulling in the cluster if you could make your image name globally unique in your company. This will reduce unnecessary image pulling from the image registries and also reduce network load. The above container image naming convention will ensure that an app image name is globally unique in your company.
• CI only image push: The capability to PUSH an image to your internal image registries(at least production registries) should be limited to CI only. This will help you enforce any kind of policy and standard for all the container images that are being pushed to your image registries. In case you want to allow only a specific set of base images(ex- distroless images) for internal apps, apply security patches, or update private CA certificates in all internal application images then it will be easier to enforce this kind of policy and standards in a single place via the CI.
• No container exec in prod: It is a security best practice to not allow access to containers in a production environment to avoid any manual interference in the production environment. As distroless images come with no shell by default, it is obvious that if you can establish the distroless image as standard, you can block the exec access to the pods/container in the production clusters. You can also disable the following Kubernetes API resources access for the users accessing the cluster via the RBAC permission.
  - “pods/attach”
  - “pods/exec”

Also, with proper observability in place for your production environment, application developers should be able to debug their application and investigate an incident from telemetry signals(which should be the standard practice) without having access to the container or pod.

• Single app per container: This one ensures that when a container starts, so should the app, and when an app stops so should the container. The following image shows examples of a good and bad way of handling PID in a container.

Source: https://cloud.google.com/static/architecture/images/bp-containers-processes-inside.svg

With multiple apps or parent processes in the same container, Kubernetes will fail to restart the container when one of the processes crashes or is unresponsive. Sometimes it is observed that a bash script is used as an entrypoint in the container to spawn multiple processes in a container which is not considered a good practice.

• Continuous image scanning: Continuously scan the container images for vulnerabilities. Most of the cloud and managed container registries has built-in image scanning feature. Continuously discover vulnerabilities and update images with patches to keep your container images secure.

Having a set of best practices for creating and maintaining container images as a prerequisite for production applications will increase the awareness and knowledge level of your whole Organization. These are some of the best practices that will reduce the attack surface, and increase the security and reliability of the applications as well as your Kubernetes cluster. As mentioned at the beginning, some of the best practices may not be suitable for your infrastructure or you may have a different strategy to achieve almost similar type of outcome. Please feel free to let me know about any good practices you follow for your container image creation or maintenance that you think can be adopted globally as well.