Engineering Alignment | Engineering Alignment with Reviews
Dylan Etkin
November 8th, 2024
Guide to Monthly Engineering Reviews
Engineering efforts are complex. But once you have larger than two-pizza teams working on an engineering problem, coordination, accountability, and alignment become a challenge.
This is when many teams fall into a trap, hoping meetings will solve their challenges. But traditional meetings waste time, focusing on status updates instead of data and outcomes.
Monthly engineering reviews offer a better way. They treat your meetings as reviews by bringing together the right people, the right data, and informed data interpretation to ensure you’re adjusting early and often to work efficiently. You just have to know how to conduct them effectively.
Move from status updates to outcomes
Unlike tactical engineering-focused sprint planning or weekly progress reviews, monthly engineering reviews zoom out to solve higher-level strategic concerns.
They focus on data-driven, actionable outcomes, allowing teams to evaluate and adjust how they are executing on big-picture priorities like delivery activity, performance and efficiency, resource allocation, operational excellence, and business-engineering alignment.
Here are some examples of great outcomes driven by a monthly engineering review:
| Data | Outcome |
|---|---|
| Operational excellence data shows your incident rate increasing | You reallocate some of your engineering work toward KTLO and bug fixes. This will also likely require an adjustment to feature delivery, which can be communicated upward with the reasons why and with plenty of notice. |
| Resource allocation data shows you are spending more time and resources on a project that doesn’t align with your highest business priority | You adjust your allocations to put more resources on the higher priority. Or, you stop working on the project and focus on the business priority instead of spreading your team too thinly. |
| Performance and efficiency data shows that you have bottlenecks in your deployment pipeline | You prioritize work to improve your deployment pipeline, understanding that this work will allow other work to move quicker. |
| Delivery activity data shows that your critical project, despite completing work each week, isn’t making real progress because you’re adding issues at the same rate you complete them. | You adjust the scope of the project so you can realistically deliver to your goal, knowing you’ll iterate toward your full spec if it’s still a business priority. |
| Sometimes, everything is working well, but data and discussion allow you to identify something you’re worried about. | Task a team or individual to investigate so you have enough commentary and understanding to take an action in your next review. |
Include the right people to succeed
Monthly engineering reviews are versatile, working across org sizes. In larger orgs, the review will scale with the org structure. For example, if you have one Engineering Manager (EM) looking after three Team Leads (TLs), that EM can run a monthly engineering review. Equally, if that EM reports to a senior EM who has three to four other EMs, that same group can run another review.
Each step up the reporting hierarchy means you’ll need to zoom out another 10,000 feet. You may need to adjust the resolution of the data accordingly, but the same principles hold at each level.
Similarly, the same set of three actors is involved in the process regardless of the level where the review is conducted:
| Actor | Description | Role |
|---|---|---|
| The doer | Team Leads or Engineering Managers who hold or have access to first-hand knowledge. | Gather and interpret the right data |
| The owner | Manager who leads the group being reviewed. The interface between the doer and the “why” behind the work being done. | Own the review process and craft the overall narrative. Review data and commentary with the doers until both parties agree the narrative focuses on what matters. |
| The authorities | VP of Engineering, Senior Engineering Manager, Project Manager, or Executive Sponsor. Often a peer or direct manager of the owner. | Help assess the review. Provide high-level direction. Ask for high-level adjustments. Approve impactful decisions. |
With these three levels of people and perspective involved in the review, you have the right ingredients to:
- Collect the right data and understand the why behind it
- Build a narrative about what’s actually going on and shape the conversation to talk about what’s important, not just status and data
- Quickly make strategic and impactful adjustments
The anatomy of a monthly engineering review
Here’s how you can execute a well-run monthly review.
- Identify and invite your team of doers, owners, and authorities to your monthly meeting. Clearly state what will be expected of each member on meeting day – i.e., that you will move through your collected information and narrative to discuss and create new goals and outcomes.
- The owner builds a document in a system of record to capture qualitative and quantitative data and add commentary. Ideally, this will allow for asynchronous collaboration via comments and workflow.
- The owners and doers work together to identify the “right” data to gather for this review. This may include requesting additional data from the authorities from a previous review. We provide suggestions below for what data to start with.
- Preferably a week before your review, task the doers with gathering and adding commentary to the data. This may be easy or hard to gather, depending on your team’s maturity and tooling. This is a critical step, because without data you will be working on sentiment alone, which is rarely enough to make informed decisions.
- The owner will asynchronously review the data and commentary, going back and forth with the doers until the owner is happy with the overall narrative and feels ready to share upwards with the authorities.
- The owner shares the document with the authorities, giving them time to digest, understand, and focus on the important information to be discussed. They’ll appreciate receiving enough data and commentary to have an informed point of view during the review.
- When you meet, spend the first portion of the discussion reviewing accountability for previous outcomes. Then, discuss and debate the current important data and capture new outcomes.
- Rinse and repeat!
Data for monthly engineering reviews
The data you include in monthly engineering reviews will be somewhat specific to your organization and its goals, but this list of common data can drive discussion, decisions, and outcomes to get you started.
Delivery activity data
- Burn down and percentage completeness of projects/initiatives
- Breakdown of percentage of the types of work being done in a period, e.g. KTLO, bugs, new features, support escalations, etc.
- Amount of unplanned work and the amount of rework included in a period
- Engineering priorities from the period and their on track/behind/not started status
Performance & efficiency data
- DORA metrics – deployment frequency, change lead time, failure rate, and MTTR
- KPIs – performance metrics from metrics systems like DataDog or sales systems that show key metrics for your projects and initiatives
- Employee sentiment data – surveys that give you a sense of how your team is doing and if they feel blocked or empowered
Resource allocation data
- Projects/initiatives being worked on
- Number of hours spent on different projects/initiatives
- How many issues/pull requests are shipping against a project/initiative
- Overall dollar cost invested in a project/initiative in the period
- The connection between a project/initiative and a business goal for engineering
Operational excellence data
- Number of incidents
- Number of bugs
- Percentage of rework in a period
- Number of support escalations
Get started with monthly engineering reviews
If you want your engineering teams to stay focused and deliver at a world-class pace, try doing monthly engineering reviews regularly! Bring together the right people with the right data and commentary, and use this effective process to streamline your Engineering organization.
