by: Bob Peterson
Recently, someone came to me with a new challenge – “Our team was already working on a technology strategy around containers, Docker, and orchestration with Kubernetes. I knew we needed to make sure we address the security aspects of the technology from the onset. So, this was a perfect time to get started on some formal guidance.
I was now on a new journey. I had minimal Docker experience and needed a place to start. My initial Google searches turned up a lot good information from multiple sources. I ultimately leaned on The NIST Special Publication 800-190 Application Container Security Guide and The Center for Internet Security (CIS) Docker Benchmark as the basis for my framework. Vendor and industry guidance filled in the gaps.
Through my research, I settled on the following topics to address in our guidance:
- Basic container security
- Amazon Web Services
- Incident Response and Forensics
- Tooling Capabilities
Let’s dive a bit into each of these topics and look some areas to address.
Basic Container Security
The NIST 800-190 gives us this overview of the architecture tiers, components, and life-cycle phases for containerized applications. This helps us look at how security is applied at different points.
The NIST 800-190 breaks security risks into the following component areas:
- Image risks – security of what is in the container images
- Registry risks – security of how images are stored
- Orchestrator risks – security of how containers are run and managed
- Container risks – security of the running containers
- Host risks – security of the hosts running the containers
The CIS Benchmark for Docker provides us a set of definitive controls to implement that are aligned with the NIST 800-190 recommendations. Recognizing that some of the CIS controls may not be necessary for every environment, I aligned which controls were required by what I am calling the “security level” of the environment. The security levels are broken down into Low, Medium, and High. We determine the security level based on factors such as data classification, exposure to the Internet, and use-case of the environment. Your environment may be different, and the security controls should be selected based on your needs.
We are looking at Kubernetes as our primary container orchestration system. My initial guidance is to follow the CIS Benchmark for Kubernetes. More detailed guidance is currently in progress. Stay tuned for more details
Amazon Web Services
The use of Amazon Web Services’ (AWS) container offerings do not change the security requirements. We just may end up implementing some of the controls differently. There are also some AWS specific implementation aspects that may add or remove controls. Like Kubernetes, we will be working on more detailed guidance on using the AWS services.
Logs will likely be the primary historical evidence of happened in your container environment from a security and operation point of view. We see that logging becomes pretty important. Here are some of the logs that help provide this audit trail:
- Application logs – details logged by your application
- Container logs – messages sent to STDOUT and STDERR
- Orchestrator logs – a history of who did what and when they did it.
- Process logs – what is happening at the OS, process, and network layer within the container
How you collect and manage the logs will be covered in a later discussion. You will need to determine how to do this in a way that meets your organization’s logging requirements.
Incident Response and Forensics
Incident response and forensics brings a new set of challenges. Events unfold faster than before, and container systems can generate a lot of information very quickly. You may also have to piece together information from multiple different sources. To top it off, containers can be spread across different hosts based on how you have your environment orchestrated.
The differences in architecture between containers and traditional operating systems or virtual machines makes collecting “host” evidence more challenging. Currently, there is no “snapshot” function available to us to give us forensics copy of the container and its memory, file systems, and related components. to find other ways to capture this data.
Another challenge is the short life-span and transient nature of containers, especially in larger environments. By the time a problem is detected, the compromised container may have been shut down and replaced. You still have a vulnerable container, but you may not have the one with the evidence you need.
You must then capture all this evidence with as little disruption as possible in a way that is forensically sound. There are different techniques and tools that are emerging for capturing these components that can be used as part of your incident response and forensics processes. In the end, you need a process that works for your environment and your incident response team needs to be well versed in how to tackle an investigation.
As I was going through this, I started looking at the different tooling capabilities to help us with our container journey. Let’s explore a few of possible tool categories:
- Image scanning:
These are tools to assess the security of the components that make up the image. Imaging scanning tools are like the traditional operating system vulnerability scan tools we have used for years. There are many different methods, vendors, and points in the lifecycle of an image where this can occur. Some tools work at image creation, others scan images within the registries, and others when the containers are pulled for launch.
- Registry services:
We need a place to store our containers after we create them. Many of us will be relying on some type of a Registry service to store and manage access to our container images. Whatever one you choose needs to be able to provide the security and control based on your needs.
- Configuration assessment tools:
Part of a good security process is making sure everything is configured correctly. Manual checks do not scale well and can’t be automated. Open source tools like Docker-bench-security (Docker) and kube-bench (Aqua Security) can help with this. The commercial tools focusing on Docker and Kubernetes have some type of assessment capabilities as well.
- Process monitoring:
Being able to monitor and track what is happening inside of a running container helps us with troubleshooting, support, and security forensics. We can monitor what operating system calls are being made, network activity, and what processes start and stop in a running container. There are several commercial vendors in this space that can help with task.
For me, my next steps are to refine the guidance and help expand our processes around the 5 topic areas discussed here. Improving our tool integration is also on the top of my list.
Like all technology, containerization is a constantly evolving technology space. We must continue to evolve our knowledge and skills to keep on top of the shifting landscape.