How to make your security training fun? Think like a deviant

Secure code training is one of the first things Chief Technology Officer Rob Zuber asked me to handle when I started as CircleCI’s first security engineer a couple years ago. He wanted it to be fun. He wanted it to create tension in people’s chests.

A few years earlier, he’d taken part in a security training event at Google. During an exercise at the event, he discovered a vulnerability that was wide open on his service. His heart started pounding as he raced to patch it. That pulse racing experience changed the way he thought about software. He wanted all of our developers to experience it.

We looked at every training imaginable, from online courses to flying the entire department to a security conference as an offsite. None of them promised that same visceral experience.

Teaching security

As an organizer and MC of the Bay Area OWASP MeetUps, I get pitched lots of presentation ideas. One came from a small company based out of Hungary called Avatao, which offers secure code training. This was hands-on breaking things, with some guidance and felt different than manual static code analysis games and similar things I’d been aware of. 

The training platform is designed by three security researchers who, not coincidentally, have been finalists multiple times at the DefCon CTF. It includes a couple hundred modules on everything from binary code exploitation to SQL injection to language-specific matters. And it’s extensible. If we wanted something they didn’t have, they’d build a module or help us write one. 

The perfect opportunity

While my security training research was starting to bear fruit, our Engineering and Product teams were busy planning a week-long offsite in Las Vegas. I was given four hours of prime time after lunch on the second day to run a game, and knew this would be a great opportunity to invite Avatao to come work face-to-face with our team. 

Together with a few others from our security team, we selected a list of 12 modules focused on the topics most relevant to our engineers’ daily lives, things like Docker secrets, OWASP Top 10 exploits like Cross Site Request Forgeries, a Vault tutorial and default passwords. For good measure, we threw in replications of a couple of real world hacks like the Facebook Imagekick.

It’s showtime

After lunch on that second day of the offsite in Las Vegas, the Avatao folks installed two large scoreboards at the head of the room and kicked off the competition with all participants breaking into pairs. Pair programming is a big part of the culture at CircleCI so we utilized that for the event, too. As a twist, we intentionally paired engineers with people and teams they didn’t typically work with and organized them based on experience level.

Half an hour in and I knew it was a success. Every engineer was focused on their screen, not a single person was sitting back and chatting, and there was a sense of competition in the room. Blessedly, the two scoreboards on projection screens showed every team was making progress. 

At the end of two hours, we handed out prizes, conducted a Q&A with the Avatao folks and then broke into groups of six engineers so they could discuss both what each person learned as well as what was the most applicable to our internal processes. Finally, everyone moved to an adjoining room for cocktails, coffee, and lock picking exercises.

Lessons learned, securely

For me, there were four key lessons I took away from this event:

• Focus on the modules your audience uses every day. Just because the security team is interested in an esoteric exploit doesn’t mean everyone else will be.
• Keep the exercises short for quick wins. It’s important that people feel like they can do security rather than driving home the idea that security is hard and should be left to specialists.
• Assemble a group of engineers from across the department to shape the curriculum and try out in advance. This not only aligns the modules with needs, it creates teaching assistants during the event who can triage problems.
• Security is more than just code. Add things on both sides of the programming like lock picking to keep the fun quotient high.

For others, the event took away some of the mystery around how specific issues like cache poisoning happen. Reverse engineering real-world hacks like Imagetrick are fascinating. The modules were challenging, but not discouraging. 

“Capture the flag was cool,” said Software Engineer Breon Knight, who paired with a Principal Software Engineer. “It was interesting to see it from a principal level engineer’s mindset.” And our VP of Platform, Michael Stahnke, commented: “Who knew security training wasn’t just ’90s clip art with the bad guys wearing ski masks while typing?” 

Software Engineer Jacqueline Garcia said it was interesting to see security bugs firsthand and to spend two hours discussing them with a teammate. That made them think differently about implementing security into coding practices.

“What I enjoyed the most was the collaboration involved,” she said.

For teams looking to increase their security competency or skills, I would highly consider putting together a hands on event such as this one. The time it takes to plan and assemble a hackathon of this kind is well worth the ROI, and is more interesting than the standard training you hear about. It brings teams closer and improves communication and processes in the event of a real security threat. And if you need any ideas, or at the least a great playlist to jam out to during the event, I have a few to share with you.