DevSecOps: Taking a DevOps approach to security

More organisations are embracing DevOps and automation to realise compelling business benefits, such as more frequent feature releases, increased application stability, and more productive resource utilization. However, many security and compliance monitoring tools have not kept up. In fact, they often represent the largest single remaining barrier to continuous delivery.

By working with the DevOps team, you can ensure that the production environment is more predictable, auditable and more secure than before. The key is to integrate your security requirements into the DevOps pipeline; however as part of that integration you will need to change the way you work. A normal approach of checklists, templates, manual processes etc will not scale. With the speed of cloud deployments, you will need to automate and use tools and scripts. This will allow you to move as fast as the DevOps team needs you to.

You will hear the concept of ‘Infrastructure as Code’ within DevOps. This is where the platforms infrastructure is stored as a set of scripts that can be executed in a repeatable way. Security needs to be looked at in the same way, with moving to ‘Security as Code’ or ‘Software Defined Security’. By moving from a legacy procedure in a Word document to a set of scripts, we can automate that document which means that it can be executed in a repeated and predictable way - it can be included into the DevOps pipeline.

The benefits will start to show immediately. By automating many of the tasks, not only can you check that the DevOps pipeline has the security and compliance controls that you need, but you can also run those scripts across production to ensure that the environment has not drifted. In an automated world, you can run those checks multiple times a day.

For security professionals it is key to understand that instead of validating the end solution you need to validate the pipeline. If you are happy that the pipeline is building the solution in a way that meets you security goals you can be confident that this will be repeated every time a developer needs to get source code into production.

10 Practical Security Tips for DevOps

1. Architecture and Design

During architecture and design the development teams will be attempting to rapidly iterate against the requirements whilst building out the Cloud infrastructure. It is at this point that security teams need to get involved to understand the scope of what teams are looking at, different elements of the infrastructure need protection in different ways. Learn and understand the shared security model - Amazon S3 is very different to protecting storage on IaaS instances, the barriers between IaaS and PaaS are rapidly breaking down, and each has a different security paradigm.

Threat Modelling can be done against the different components. This will allow security teams to define the threats against the different components, and what elements are going to be needed further up the DevOps pipeline to secure them.

ACTION: Work with the architecture to understand the cloud components being used, and the security controls required for each. Take this further by using techniques like Threat Modelling

2. Static code analysis + Code Reviews

Code reviews are a common part of DevOps, Security team should educate colleagues on secure coding techniques so that they can include this into their secure code reviews. However, many of these items can be validated by using Static Code analysis. This is where the source code or a partially compiled version of the source code can be checked for potential vulnerabilities. Many potential security vulnerabilities can be picked up at this point, and if they fail the checks - it breaks the build. Developers will quickly change coding techniques to meet the requirements.

ACTION: Understand what the current code review process is and ensure that there are security elements within that. Likewise investigate what Static Code Analysis tools are available and if they can be used

3. Audit of Chef Scripts / CloudFormation Scripts

You will hear the phrase ‘infrastructure as code’ a lot in the DevOps world. This is where the infrastructure is built in a highly automated way using scripts and configuration files. The advantage of this for a security professional is that automated checks can be run against these scripts. If a developer creates an infrastructure script to create a storage bucket with public access to the internet, this can raise an error. Combine this with the threat modelling where you have identified potential issues (like marking a storage bucket as public) and you have a very powerful tool to validate the infrastructure every time a developer makes a change.

ACTION: Use the automation tools to ensure that the infrastructure is being built to meet the security standards

4. Security Testing post build

Automated builds and unit tests running after check-in are a core part of DevOps. This is where security teams can add-in security testing tools to automate the validation of the build (https://www.owasp.org/index.php/Appendix_A:_Testing_Tools ). The reason why automated build and testing is so key in DevOps is that the shorter the time between a developer checking in code and a test failing - the less time it will take for the developer to fix the issue. The same holds true for security vulnerabilities, running testing at the end of the project can inject significant delays as developers struggle to identify the issues and fix the bug. Identifying the issue within minutes of a developer checking the code in reduces the time taken to identify and fix the issue.

ACTION: Investigate automated security testing tools and integrate into the build process

5. Harden your OS deployment

Let’s move from ‘Infrastructure as Code’ to ‘Secure Infrastructure as Code’. If you are creating and building servers via scripting, lets also add in the scripts to lockdown the box as well. The risk of applying OS hardening at the end of the project is that the application stops working. If it is applied at the beginning of the project, firstly issues are identified up front, and secondly if the hardening has to be relaxed, it can be identified early, and security teams can work with the developer to potentially find another way of performing the function. If it occurs late in the project, the development team will force the issue through.

ACTION: Review the automation scripts to ensure that the OS is being deployed in a secure way and any changes to this standard are controlled. Use resources like SANS Linux Security Checklist or CIS Benchmark

6. Harden your Cloud Deployment (standard AMIs, Security Groups, IAM roles, MFA tokens)

Cloud services can deliver incredibly secure infrastructures ‘if’ done correctly. However it is also very quick and easy to open up significant security holes. You need to review how your company is using the cloud, this includes the segregation of roles – do your developers have the rights to change the production environment? If so why? I am sure you do not let your server administrators walk around using Domain Admin accounts; so why should people have root access in the AWS Console. You need to review everything from the development environment through to production.

ACTION: Review how teams are accessing the console and what permissions that they have. People should only have the permission they need to do their job, and if they have significant permissions they should be using two factor authentication.

7. Deployment of security tools

Once you get to deploying applications to production, are you going to be able to keep up with multiple teams deploying multiple applications to production? In the same way you can use automation to ensure that security is as you require it, you can ensure that your security tools are deployed at the same time.

You should be looking at deploying network detection for threats on your network, monitoring of HTTP for attacks as well as monitoring log files. There are solutions that allow you to monitor these three different feeds as well and at the same time have a 24x7 SOC investigate the threats and escalate if required.

ACTION: Script the deployment of your security tools so that all environments have a baseline coverage

8. Vulnerability scanning of OS and applications

One of the most common attack vectors is for people to exploit the vulnerabilities in the OS or applications that are running on the servers. As part of a DevOps pipeline, servers can be checked for vulnerabilities, this ensures that you know what state your servers are at any point. In addition you can use a solution which has an analytics engine. Then this information would feed into an analytics engine allowing potential attacks to be rated with the additional data of what software you are using; this will help reduce false positives.

ACTIONS: Run regular vulnerability scans against the environments and remediate any vulnerabilities

9. Phoenix Upgrades

Instead of deploying patches to production, you should be burning and redeploying servers as required. This not only increases your agility to roll out new versions, but also increases your ability to rapidly respond to security issues. You can deploy a new patched version across your entire cloud environment rapidly and safely; and with the phoenix upgrade strategy you also reduce the risk of technical debt and configuration drift.

ACTIONS: Work with the DevOps team to support those using Phoenix Upgrades and ensure this gives you the ability to patch security issues and roll them out.

10. On-going and real time audit of production environment

Visibility post deployment is often down to the level of auditing that has been put in place. You should have standard auditing levels across different Server roles and Applications. Your goal is to get a level of auditing that can be fed into a security tool to give the data that is needed, but not swamp your servers with too much auditing.

Once all of these elements are in place, it will allow you to audit production to ensure that at any point in time you understand what state production is in, and if it has drifted from it defined security profile. The cloud is often referred to as a programmable datacentre - Developers can use this to create huge IT systems in very short timeframes - you can use this same power to audit these systems multiple times a day.

ACTION: Work with the development team to set logging levels and use a tool like Chef to ensure that your configuration does not drift

The evolution of DevOps should be extended to embrace Security – providing speed and agility to securing critical applications, assets and services in a more predictable, auditable and secure way.