A cost-effective, and less sophisticated alternative to Virtual Machines, Containers, have revolutionized the application delivery approach. They have dramatically reduced the intake of IT labor and resources in managing application infrastructure. Yet, while securing containers and containerized ecosystems software teams are met with many roadblocks. Especially for enterprise teams accustomed to more traditional network security processes and strategies. We’ve usually preached that containers offer better security given the fact they isolate the application from the host system and also from each other. Somewhere we have made it sound like they’re inherently secure and almost impenetrable to threats. But how far-fetched is this idea? Let’s dive right into it!
Let’s get a high-level view of what the market looks like –
Take a look at this survey conducted by Business wire, the global Container Security Market size is projected to reach $3.9 billion by 2027, that’s a 23.5% CAGR.
Pretty much like any software, containerized applications can fall prey to security vulnerabilities that include bugs, inadequate authentication and authorization, and misconfiguration.
Container Threat Model
Let’s understand the container threat model.
Source – Container Security by Liz Rice
Listing down all the possible vulnerable factors in a container.
- External attackers ( from outside) trying to access one of our deployments (Remember when Tesla got crypto-hacked!)
- Internal attackers are those who have some level of access to the prod environment (not necessarily admin)
- Malicious internal factors are the privileged internal users like developers and administrators who have some good access to the deployment
- Inadvertent internal factors who might accidentally cause problems like carelessly storing some secret key or certificate in the container image
- In the name of making the customer experience better, by introducing some new service or reducing the waiting time, companies tend to open some ports in their servers or firewall. Now, this if not thoroughly guarded might become the passage of hackers.
Hence there can be several routes that can compromise your container security (and the above image pretty much summarizes so). Let’s discuss some of these factors precisely.
Challenge 1: The Fault in Container Images
It’s not all about having malicious software. Poorly configured container image can be the reason for introducing vulnerabilities. The problem begins when people think they can spin up their own image or download it from the cloud and start using it straight. Well, we should be aware that every day new vulnerabilities are introduced on the cloud. Every container image needs to be scanned individually to prove it is secure.
Some known cases to avoid:
- What if the image launches an extraneous daemon or service that allows unauthorized network access?
- What if it’s configured to work with more user privileges than are necessary?
- Another danger to look out for is – if any secret key or credential is stored within the image.
Note: From all the above cases we notice that docker will always give priority to its own network above your local network.
Recommendations:
- Pulling images from trusted container registries. Trusted container registries are not poorly configured. Usually referred to private registries, but not necessarily, unless they are encrypted and have authenticated connections. This should include credentials federated with existing network security controls.
- The container registry should undergo frequent maintenance tests to keep it devoid of any stale images with lingering vulnerabilities.
- Software teams need to frame shared practices with blue-green deployments or rollbacks of the container changes before pushing them into production.
Challenge 2: Watch out for your Orchestration Security!
Popular orchestration tools – Kubernetes is unmissable while addressing security issues. It has become the prime attack surface.
If referred to the recent “State of API Security 2022” report approximately 34% of the organizations have absolutely no API security strategy in place. Adding to that 27% say that they have just a basic strategy that involves minimal scanning and manual reviews of API security status and no controls over them.
When Kubernetes is handling multiple containers, it is in a way exposing a large surface area for attack. Following industry-wide practiced field-level tokenization is not enough, when we are not securing the orchestrator’s ecosystem. Because it’s just a matter of time before the sensitive information gets decoded and exposed.
Recommendations:
- Ensuring the administrative interface of the orchestrator is properly encrypted which can include two-factor authentication and at-rest encryption of data.
- Separation of network traffic into discrete virtual networks. This segregation should be done on the basis of the sensitivity of the traffic being transmitted.
For example, public-facing web apps can be categorized as low-sensitivity workloads and something like tax reporting software as high-sensitivity workloads – and separating them. The idea is to ensure each host runs containers of a certain security level. - The best practice can be to adhere to end-to-end encryption of all network traffic between cluster nodes and also includes authenticated network connections between cluster members.
- We should aim to introduce nodes securely into clusters, maintaining persistent identity for each node throughout its lifecycle. Isolate/Remove compromised nodes without affecting the cluster’s security.
Challenge 3: Prevent the “Container Escape scenario”!
Popular container runtimes such as containerd, CRI-O, and rkt might have hardened their security policies over time, however, there’s still very much possible that they might contain bugs. This is an important aspect to consider because they can allow mischievous code to run inside the “container escape” out onto the host.
If you remember back in 2019, a vulnerability was discovered in runC called Runscape.
This bug (CVE-2019-5736 ) had the potential to enable hackers to break away from the sandbox environment and gave root access to the host servers. Well, this led to compromising an entire infrastructure. At first, they assumed that it could be a malicious docker image as there has to be a malicious process inside. After all tests, they realized it was a bug in runC.
There’s no shortcut – Security needs to Shift Left
While dealing with microservices-based environment the best practice says to bring in automated deployments at every step. We are not agile if we are still performing deployments manually, following a weekly or monthly cadence. To actually shift left in application delivery we need to create a modern toolchain of security plugins and their extensions throughout the pipeline.
How that’s supposed to work is – if there’s any vulnerability present in the image, the process should stop right there in the build stage. Periodic audits should be conducted on RBACs to monitor all access levels. Also, all the tools and processes should align with the CIS benchmarks.
A good approach will be to adopt security-as-code practices to write the security manifest for Kubernetes-native YAML files as custom resource definitions. These are human-readable and declare the security state for the application at runtime. Now, this can be pushed into the prod environment and protected with the zero-trust model. Thus, there are never any changes to the code outside the pipeline.
Wrapping Up!
It’s time to wrap our thoughts around containerization and container security handling! This hasn’t been a comprehensive blog, but we tried to highlight some of the easily doable yet highly neglected zones while practicing containerization. And for all those who want to focus more on business and least on these technical aspects – the market has heard you!
There are many container management platforms that can take care of all these heavy liftings, and leave you with the most secure containers where you are always ahead in the race. This is because automated security processes across CI/CD pipelines and declarative zero-trust security policies are the need of the hour and they enhance Developer productivity and are also a part of DevOps best practices.
With all that said, thanks for keeping up with this long blog.
Do share your comments – we’ll be happy to make things better. Also, if you have any business cases – feel free to contact us.
