The post-mortem on Engineering dashboards

Why dashboards are dead

You’ve probably looked at a dashboard today. Every team has at least a few dashboards built over time. Even though we all use dashboards daily, are they actually useful?

As in, are they helping us move in the right direction? Make the right decisions? Or, are they simply something you use to alert you when things are bad?

The truth is, we never think about dashboards that much. We go through the motions: we check some numbers, take screenshots, paste them in some reports, and forget about them until the next time we need to do it again.

Considering the time and energy spent to build them, and how important they seem to management, this feels like such a waste.

How did we get here?

It’s how we’ve always done it

Your team just spent hours planning projects and goals for the next quarter, chose metrics to track progress, and built a new dashboard to display the progress in real time.

The product team begins every sprint retrospective reviewing and discussing the quarterly goals dashboard. The leadership team gets monthly reports with screenshots from the dashboards. Some teams even use the metrics daily.

The quarter ends and everyone celebrates as the dashboards show the team met the quarterly goals — success!

But eventually, new goals arrive, and with them a new set of metrics and dashboards to track them. What you actually end up with is something akin to the “Standards” XKCD comic — just replace “standards” with “dashboards:

You’re left with a dashboard graveyard that only the most savvy of data analysts and BI team members know how to hack through to get to the maintained dashboards.

It’s a painful reality, and I see it daily as I coach engineering organizations on how to implement and drive action through metrics. There are three main reasons why dashboards die.

Cause of death #1: Misuse of metrics

In the early days of any initiative, tracking metrics and reviewing them on dashboards works well enough to keep everyone focused. But eventually teams start focusing on ways to improve the metric, ignoring what it originally was meant to represent.

It leads teams to focus only on whether the metrics are “good” (green) or “bad” (red), which encourages gamifying ways to keep them in the green on the dashboard. The metric then loses its effectiveness.

It’s an example of Goodhart’s Law which states: “When a measure becomes a target, it ceases to be a good measure.”

Without consistently focusing on why metrics were chosen and what goals they’re supposed to represent, we end up chasing things that don’t matter.

Cause of death #2: Static views

If we have the perfect dashboard showing all the right metrics, and we have not fallen into the traps of misuse, it can still be difficult to understand how to use dashboards to drive change.

Many dashboards are static and prevent users from drilling down into the dataset to gain useful insights. They display metrics in a visually appealing way but don’t offer much else.

At best, we extract the raw data as rows and columns into a spreadsheet, which becomes sliced-and-diced views of the data from the original dashboard. The original dashboard is merely a catalyst to look for answers elsewhere — other tools, services, analytics, and colleagues.

In a previous life, my colleagues and I were responsible for understanding a dashboard representing the previous quarter’s sales. At quarter end, we waited for a BI analyst to tell us the dashboard was updated with the metrics. A mad dash ensued to look at the data for our products.

We compared quarter over quarter, snagged screenshots, and added them to slide decks. We spent weeks comparing and contrasting the data, looking at other dashboards, and talking with the sales team and customers. We gathered our findings and presented them in a “this is what we learned” meeting.

We spent all that time trying to gain insight because the original dashboards didn’t allow us to seek answers from within it.

Cause of death #3: What’s found is now lost

Let’s say you somehow gained insight from a dashboard you use regularly. How did you share the insights and make the action items visible to the rest of your team? How did you track progress on the actions?

It’s likely the findings weren’t tracked anywhere. At best, you discussed them in a meeting and people walked away with their own to-do lists. But were those to-dos tracked in a central location and connected to the original data that provided the insight?

The insights are interesting one day but easily forgotten the next if they're not captured properly and made visible to the team. We also struggle to understand why an action item is important if it’s not connected to the original insight, which is connected to metrics in a dashboard. We need to keep insights close to their source so that desired outcomes don’t get missed.

Dashboards are dead, long live dashboards!

We live by dashboards, but so many of them are dead — they’re misused or serve simply as a vehicle for pasting content into spreadsheets and slide decks.

Visualizing metrics can be useful, but adding a dashboard to a long list of already existing dashboards is ineffective.

We need a constant focus on why metrics were pulled together and how the metrics should be used. Otherwise, the dashboard just becomes another measure.

The next time you open a dashboard, think about why and how you’re about to use the metrics within it. Chances are you’re staring at something bound for the graveyard.