Kubernetes & container security tips

Nice talk on modern K8s and container security tips by Google's Samuel Davidson:

(Backup link)

The highlight for me: 

Using “distro-less” base images seems like a recommendation that's easy to do and would help make post-exploitation harder.

Samuel has put all his recommendations in this doc.

Other resources I've come across lately:

• Sysdig's SELinux, Seccomp, Sysdig Falco, and you: A technical discussion article.
• Jessie's Setting the Record Straight: containers vs. Zones vs. Jails vs. VMs article.
• Jessie's Security in a containerized world presentation.
• Microsoft's Threat matrix for Kubernetes article.

Running product security

Julian Cohen and Michal Zalewski share their opinions on building a product security program. I find them incredibly on-point:

• Getting product security engineering right
• A Product Security Framework

And here's another page that ties up maturity and DevSecOps and looks interesting:

• Lean security

Happy reading!

Thank you to my colleague Luis for sharing these.

Tooling: Testing IaC for security flaws

TL;DR

I made a Github Action that enables you to easily scan infrastructure code for security mistakes. Grab it on Github's Marketplace!

Overview

So, static code analysis for infrastructure code is hot! A few tools have appeared recently, and Checkov looks pretty good.

I tested it locally (as a command line tool on my laptop) against my aws-student-lab project. This terraform code just sets up a simple student lab on AWS, and I like to use it for testing new tools, since it's both simple and realistic. Checkov found no problems, and showed a nice green "PASSED":

The next step after local analysis, of course, is continuous integration, so I used Github Actions to run Checkov each time there's a commit. This involved writing a small yaml snippet.


To avoid having to do this setup on each repository, I decided to make a new Github Action and publish it on the Github Marketplace. This way, other people can easily integrate Checkov in their CI with minimal configuration!

Deep-dive test-drive

To test out Checkov and my new Github Action, I added the definition for an S3 bucket in my terraform code and gradually removed security features, to see if Checkov complains.

Starting with a nice secure bucket:

This bucket is pretty ok: it has encryption, versioning, logging and MFA confirmation for deletions.

As expected, Checkov checks everything and is happy, so the build passes:

Then, I decided to remove most of the nice features and options. Checkov breaks the build and complains:


The output is very helpful in determining how to fix this:

• Ensure all data stored in the S3 bucket is securely encrypted at rest
• Ensure the S3 bucket has access logging enabled
• Ensure all data stored in the S3 bucket have versioning enabled
• Ensure S3 bucket has MFA delete enabled

Nice! Working backwards, someone who sees these errors can look up cloud provider (AWS in this case) and terraform documentation and enable the missing features, reaching a more secure configuration.

Overall I'm happy and will be using this on all my future cloud code.

Pull requests are welcome!

Asynchronous communication

For years now I've lived with email notifications turned off. Both at work and home accounts. It's never caused a problem, since people know how to reach me for emergencies, and I believe it has helped me be calmer and more productive.

Doist explains why asynchronous communication leads to increased productivity.
The article is great, and here are some highlights:

Real-time communication comes with a few downsides:

• Leads to constant interruptions,
• prioritizes being connected over being productive,
• creates unnecessary stress,
• leads to lower quality discussions and suboptimal solutions.

Having a more asynchronous way of working is better because of:

• Control over the workday = happier and more productive employees,
• high-quality communication versus knee-jerk responses,
• better planning leads to less stress,
• deep work becomes the default,
• automatic documentation and greater transparency,
• time zone equality.

And here's how to get there:

• Plan ahead to give people time to consider your message,
• after meetings, document discussions, and outcomes,
• turn off notifications,
• evaluate people based on their output and results, not how responsive they are or the number of hours they work,
• emphasize trust, organization, independence, and accountability,
• adopt a Direct Responsible Individual (DRI) model for management and decision-making,
• make transparency a priority,
• use tools that promote transparency, deep work, and async communication.

Meta: Managing engineers

Dear engineering managers, here’s an excellent 17' talk of a Github manager on “How to grow emotionally intelligent engineering teams”.
It’s really to the point imho. Highly recommended to watch!

Practical AWS security setup

2021 Update: Since I wrote this article in 2019, I've found this PDF guide on the same topic. It is much more elaborate and complete than my own notes. I highly recommend going through it :)

AWS security on the cheap

So, you're on a budget (say, you are voluntarily building a workshop for students in Ghana) and you want a secure AWS environment to allow people to learn on.

How do you setup and secure the account?
You use infrastructure-as-code and a few free tools. I'll discuss the setup and give code in my next blog post.

In this post I'll show something about two of the tools:
How does AWS's Trusted Advisor (free version) compare to NCC group's ScoutSuite?

Well, on a relatively simple AWS account, Trusted Advisor finds no issues:

But ScoutSuite has findings: 

Many of these findings are worth looking into and getting fixed. The rest can be marked as false positives, so they don't show up in future scans.

Please note that getting a paid AWS support plan would enable extra Trusted Advisor features that might produce similar results. But I like to be able to do it for free :D.

NSA gives defense advice

The NSA’s Tailored Access Operations unit gives a talk about how they attack the rest of the world. Very interesting for us defenders. 

My take-aways to make these guys' (and other nation state attackers') lives harder: 

• They get an edge by knowing the network better than the people running it. 
• 0-days are not such a huge tool for them (!). They can breach most corp networks using known vulnerabilities. So use recent software! 
• User awareness can only help up to a point. Beyond that (for advanced persistent threats), the users can always be tricked into clicking. You must have technical measures to defend against those. 
• Older protocols are usually targeted because they are easy to sniff/decode. There is no use in upgrading your apps to use latest (e.g.) SSO technologies if one still uses legacy. 
• One of their worst nightmares: an out-of-band network tap, monitoring for anomalous behavior. 
• Don’t use old-school AVs which rely only on signatures, use something with reputation checking.

AWS security tool collections

I've come across a few of them lately:

• Jason Chan's list of AWS security resources
• Toniblyx's mega collection of AWS security tools
• Something that's missing from there: Brian Warehime's guide to Easy to use GuardDuty alerting with the help of Terraform and Slack

Edit 20-Oct-19: Configuring GuardDuty on multiple accounts is a PITA, so this article might help.

Being a tech lead & project management

I found the guidelines that Webflow issues to their Tech Leads useful and inspiring. Webflow is a tool for building websites, but it doesn't really matter, it could be any cool software company that issued this.

The whole article is worth a read for people in such roles (and their managers), but here are three highlights:

On crunch time:
"When people operate at their peak performance, where they engage in the flow state 2–4 hours a day, they are incapable of more work without drastic consequences. They should already be operating at peak efficiency, and asking more of them has severe diminishing returns and a detrimental impact to them personally, and to Webflow as a company."

On moving deadlines:
"Too much is at stake when we attempt to hit an unrealistic deadline, and among them are team burnout, poor product quality, reduced morale, and more.

The important idea here is not to lose sight of a delivery date. That’s all that matters. Projects will fall into limbo when a missed deadline stays (ahem) dead and the project careens toward the unknown. This is worse than moving the deadline, so move it!"

On the 80-20 principle, the last 20% of a project takes a lot of time, and is usually needed. If you want to stop at 80%, consider this: you want to buy a piano for your living room. How much would you pay for one that plays fine [80%] but its wood is still untreated [20%]?
Webflow says "Just treat the 80% point in your project as the halfway marker. That will align expectations against the added effort nuance prescribes."

Here's a very mature culture I admire

https://jobs.netflix.com/culture
Pure. gold.
This text changed my perception. It also changed my standards for a good employer, forever (I'm afraid). Takes time to read but is absolutely worth it.

One point that is especially relevant to security:

In general, freedom and rapid recovery is better than trying to prevent error. We are in a creative business, not a safety-critical business. Our big threat over time is lack of innovation, so we should be relatively error tolerant. Rapid recovery is possible if people have great judgment. The seduction is that error prevention just sounds so good, even if it is often ineffective. We are always on guard if too much error prevention hinders inventive, creative work.

If you want to see some of this in a talk, here's what Devops means for them. Watch at least until 16'. The job of a manager is explained around 13'.

Scaling the security team

This. so much this.

(Source)

DevSecOps intro

A couple of resources I found interesting lately:

• Pretty good introduction to DevSecOps, in the first half hour of this video.
• Short article on why we want to shift security left.

And a few thoughts and quotes I stole..ehm..gathered from people that are smarter than me:

• Teams that are doing peer reviews for all their code, are usually doing real devops (this was used to distinguish real devops against just using devopsie tools and calling it a day. DevOps is not just about tools!).
• Every time I do an architecture review (from a security perspective), I should ask myself: how do I make sure that next time I don't need to perform this? Automate it or teach a developer to do it, because security engineers can't scale to all the dev teams of any company.
• If we have embrace CI/CD, there is no option for a security gate/audit/review. Everything has to be automated, short-circled and repeatable!

Phishing is "improving"

This reddit thread warns us that in case we get a call claiming to be from our bank or other organization, we should not give out any of our private info, but request that we end the call and call them back on a verified official number, even if the call we received originates from it.

The reason is that the traditional calling system is vulnerable to number spoofing, so caller ID cannot be trusted. So, this is one more thing to remember, on top of SMS being easy to intercept.

Good luck explaining thisattack & defense to elderly laymen 😨.

Meta: Management and employee retention

A few articles that I like, explaining how employee motivation works.
Great for all managers, pretty interesting for the rest of us.

• must-read: Why your employees are losing motivation,
• should-read: A bit over-sensational, but this guy is right, remove the toxic people from your company! 
• could-read: A view of a millennial, catching some important points nicely (although I don't agree with everything).

DevSecCon London 2018 takeaways

I recently visited DevSecCon London 2018 and kept notes of my takeaways. Read about them in this Google Doc and feel free to comment on it.

IoT security

Here's the best recent article/opinion I've read on IoT security, written by Bruce Schneier.

It touches upon the market dynamics and covers the inevitable government regulation.

Very well put! Highly recommended read.

Handy tool to check CSP

https://blog.thomasorlita.cz/vulns/google-csp-evaluator/

Useful to defenders and attackers (bug bounty hunters?) alike.

Sec tools: Should you buy or build?

Are you contemplating buying a security blinky box that will solve your problem? Could the team build something similar from scratch or re-using open source components? Valid dilemma.

This should help. Especially the analysis of steps #2, #3 and #4.

Lessons on implementing SAST

Long but great reading on how Google does SAST. Contains things that didn't work for them and what they did to end up with working solutions.

The Lessons and Conclusion chapters are must read! Especially for anyone that tries to implement SAST in an organization.