7 steps to building a stellar engineering competency matrix

As a manager, I view it as my responsibility to provide every engineer with a realistic and achievable growth plan they can execute against. In that responsibility, I try to be fair and objective, ensuring each individual is being held to the same set of standards at each level of engineering growth.

This objectivity is important in removing any potential bias from both promotion and compensation conversations. And because the growth of an engineer is a complex thing, I like to have a tool to utilize in that pursuit. I call that tool a “competency matrix.”

At the beginning of 2018, I was working at CircleCI as an Engineering Manager, where we had 32 individual contributors and a plan to double that year. The company already had a competency matrix, but it was outdated and not current with the skills we had grown to value. This made performance and growth conversations extremely difficult, as the tool we had to guide these conversations had to be used with a heavy set of asterisks.

We had a problem, and I wanted to be a part of the solution, so I set out to build a new competency matrix for the company. This was a lengthy process, spanning eight months from inception to organizational adoption.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

Along the way I learned a lot, including what steps we took were wasteful and which were useful. I believe this is a powerful tool in enabling organizations to provide their employees with clear growth paths, and I would like to reduce the barriers to companies developing their own.

So here I present to you the seven essential steps you need in building your own competency matrix. If you want to provide your employees and reports with a clear, agreed-upon, and well-defined path for growth within your organization, then this is for you.

Step 1: Make this someone’s top priority

In retrospect, this was the biggest factor in our lengthy redesign process. I had initially taken on this project as one of my many side projects. The only time I had to dedicate to the matrix were early mornings, late nights, and weekends. This was a passion project for me, and I loved working on it, but I was not able to give it the care it needed.

When Lena Reinhard, our new Director of Engineering, joined, she took this on as her first big project. We worked together closely after this, but the fact that she now “owned” it made it immediately obvious to me how much my limited availability had been holding this project back. If I had continued on driving this during my free time, I don’t know if we would have seen it completed in 2018.

If you are taking this project on, give it the attention it deserves by giving it to someone as their #1 job.

Step 2: Agree on what you are building

Until all stakeholders agree on what your goals are, every attempt at implementation will stall.

A competency matrix is a powerful tool to set a cultural tone and direction, so in designing your matrix, you’ll have many impactful choices to make. Each one has consequences, and you’ll only get through them if the team is aligned.

One of the questions that came up repeatedly for us was: is this codification of the status quo or is this aspirational? (It took us about halfway through the process to explicitly agree it was a blend of both.)

Another key point of agreement is: who is going to be affected by this? In our case this question hinged on whether or not our new matrix would affect our Site Reliability Engineers (we decided it would).

Maybe you are building a matrix for your whole company, or maybe you are building a matrix for your front-end development team. The breadth of roles affected will greatly change the competencies you choose to codify, and the level of abstraction you need to use. So ask these questions, and get explicit agreement.

Finally, we needed to agree on the goals of the matrix. We decided the primary goal was performance evaluation and growth planning with individual engineers. Secondarily, we wanted to use it to influence our hiring process and communicate externally what being an engineer at various levels meant. Knowing the potential uses, and the priority of those, guided certain decisions.

Step 3: Guide by your values

This, for me, was the most fun part of this entire process. This is the part where you sit down and debate “what matters to us?” We had some help from our excellent Head of HR, David Mann, who came in with note cards detailing out ~100 behavior traits that are valuable to have as a professional. These were not engineering-specific, and ranged from communication to political awareness. We also utilized other publicized competency matrices to seed ideas of what could be in the running.

We also enjoyed coming up with our own values as a team. The one that sticks out most in my mind is “economic thinking,” something the management team continually discussed as being one of the key skills that differentiates good developers from great ones. We didn’t borrow that competency from any HR cliff notes or other matrices we consulted, but from the get go this was a key competency we knew would be in the final cut.

Start wide, dream big! It’s better to cast a wide net up front and merge and cut later.

Once you have a list, doing a few passes to merge similar competencies is a good idea, as it will save you time later in the process. For example, we merged pragmatism and economic thinking, as we realized the manifestation of these competencies resulted in the same behaviors.

Step 4: Define your levels

Before you start to actually fill out the content, it’s good to define the x-axis of the matrix: the growth in terms of title and responsibility. At our company, we already had the existing set of titles we wanted to use, (E1 / Associate Software Engineer to E6 / Principal Software Engineer). Then we explicitly agreed upon on generalized responsibilities for each title.

We broke it down into two categories, both focusing on individual contributors: E1, E2, and E3 focus on mastering the skill of software engineering and becoming a highly effective IC. E4, E5, and E6 focus on utilizing those skills to scale impact and create leverage across larger and larger groups of people.

Once we had this high-level guidance, we broke it down further, assigning scope to each of those execution levels. Before we started creating content, we had the following guiding principles:

• E1 – E3: Execution of Work
  • E1: Within task
  • E2: Within project
  • E3: Within team
• E4 – E6: Utilizing skills to scale and generate leverage
  • E4: To team
  • E5: To set of related teams
  • E6: To engineering department

Of course, these are guidelines, not rules. For example, we expect E4s, E5s, and E6s to contribute to organizational practices and processes at differing frequencies. Career development, like all human pursuits, is messier in practice than in theory. However, by setting clear guidelines up front, you can make the low resistance paths easy, and use your time focusing on those harder, messier bits.

Step 5: Create the content

This step is definitely the most laborious. However, if you have followed steps 1-4, it should make this step much easier for you than it was for us. I tried to tackle this step multiple times throughout the process, but without the correct framework to shape my approach, I often felt aimless and scattered. Once the framework was in place, filling out the content became much more straightforward.

Given the meatiness of this step, I’m going to break it down further into sub-steps. Following this sequencing should minimize any wasted work.

• 5.1: Define a single level for all competencies
• 5.2: Look for opportunities to merge competencies
• 5.3: Fill out the rest of the levels
• 5.4: Wordsmith

Step 5.1: Define a single level for all competencies

The first step I would strongly recommend is to define one level, such as your Senior Software Engineer, for all competencies. Rather than trying to wordsmith the nuanced differences between what it means to exhibit “delivering feedback” as an E2 vs an E3, reifying what you mean by each competency by defining a single level is a very enlightening and aligning process.

We approached this slightly differently, by defining two tiles: E3 (top level individual contributor) and E4 (first position of scale and leverage). Given our bifurcated approach to executing vs scaling, this gave us a better picture of what growth looked like through a pivotal step in a developer’s career at our company.

Step 5.2: Look for opportunities to merge competencies

An amazing thing will happen while you are defining that single level for every competency: you’ll realize some of the values point at the same behaviors. This is an opportunity to merge!

After we had defined single levels, we noticed that we had two tiles that were eerily similar: self starting and delivery accountability. Although these competencies had appeared different, the behaviors we had codified were similar enough that we merged them into a single tile: reliability and delivery accountability.

Early on, we made the explicit agreement that we wanted our matrix to be a collectively exhaustive definition of what growth looked like for a developer at CircleCI while also being as simple as possible. Ultimately, we ended up with 27 competencies. However when we started 5.1 we had almost 50. Defining the behaviors for each showed us where we had opportunities to consolidate.

Step 5.3: Fill out the rest of the levels and wordsmith

Now that you know you have the behaviors and competencies you want in your matrix, it’s time to fill out the rest of the cells! This step is a lot of work, but an important step as it is the core of the matrix.

During this step would be the best time to start wordsmithing. While you are in the throes of defining the nitty gritty details of what it means to scale economic thinking to one team vs many is the best time to be critical about whether or not the language you are using properly conveys the intent.

I would strongly advise against wordsmithing before this, but this would be a great time to revisit the tiles you defined in your first pass.

Step 5.4: Polish and consistency

I have no doubt during your run through defining each individual tile you will learn things that you want to apply to tiles you have already defined. About halfway through the process, Lena and I discovered that we used “often” and “usually” to mean the same thing at about a 50/50 rate. Since consistency was a big focus of ours, we decided on “usually” and put it in a proverbial parking lot to apply later.

I would suggest a similar approach. As you find opportunities for polish (such as replacing all occurrences of “often” with “usually”), write them down, and use that as your to-do list for a final pass. That way if you have learnings that contradict each other, you can resolve those conflicts before application, minimizing churn and ensuring greater consistency.

Step 6: Involve the people it will affect

Great, you have a competency matrix, you’re done! Actually, not quite. To us the idea that Lena and I would sit down, write up a bunch of values and behaviors, get it signed off by our VP of Engineering, and then present it to the world as final was a laughable proposition.

This was going to affect our culture, our hiring, and most importantly, shape career growth for all the people we work with and whose careers we care about. We had been testing different versions and aspects of the matrix throughout the months it took to design it, but now it was time to really put it to the test. With our first pass done, we set out to garner feedback.

Lena orchestrated an excellent feedback session in the format of a focus group, made up of individual contributors of all levels, managers, and HR, that had both asynchronous and synchronous portions. There are many ways to gather feedback, such as having a diverse portion of individual contributors complete mock self evaluations with their manager. Regardless of how you do it, the important thing is ensuring you gather feedback from the people this will affect.

The feedback we received was extremely valuable for two reasons: it surfaced good questions, which led us to tweak the matrix so it conveyed what we intended. This process also ensured we had codified the values of the organization, not just a few managers, and that the value progression was representative of how our engineers saw their career growing.

This step was extremely important in generating confidence that what we had developed represented what we wanted, and would be well received.

Step 7: Ship it!

The best step of all – ship it! You have a finished product that has been wordsmithed, stress-tested, and has buy in from respected individuals in your organization. However, even after you following a thorough process, it will never be perfect. As people start to use and test what you have created they will find slight inconsistencies and opportunities for improvement. Be ready for feedback now, and on an ongoing basis.

There are two mechanisms you should put in place now that you have your matrix: a way to gather feedback and an agreed-upon timeline for addressing it. Constantly iterating on your competency matrix would be a bad idea, because it means constantly moving the goal posts for career development at your organization. However, pretending that it is perfect and will never change is foolhardy and unrealistic.

We’ve developed a mechanism for collecting feedback (spoiler alert: it’s a Google Doc) and will be revisiting the matrix at the six month mark to address what we’ve learned. This provides a level of stability that is important, while also allowing for iteration and evolution over time. I suggest you provide a mechanism to achieve the same goals.

If you have your own thoughts, experiences, or questions about building a competency matrix, I’d love to hear them! Feel free to reach out to me via email (justin@circleci.com) or via Twitter (@JustinC474).