Outcome based metrics

What is an outcome based metric

Outcome based metrics are a way to quantitatively tell whether you’ve delivered a particular outcome. They should measure something that has meaning to your organization’s customers or something of relevance to your organization that gives an indication that you are meeting your customer’s needs. It’s important to identify outcome based metrics to avoid measuring progress and success based solely on output.

An outcome is the change you’re trying to make in the world. It’s the need of the customer that you’re trying to satisfy, or it’s a change in your organization. It’s ultimately why you create or change your products (internal or external).

An output is something you create in order to make an outcome happen. Outputs can include, but are not limited to:

Software
Requirements
Tests
Documentation

Teams typically measure progress and gauge success based on measures of output such as story points, and velocity.

Describing outcome based metrics

When you identify an outcome based metric, you start with the actual outcome you’re trying to accomplish. You can think of these as goals, which are described by the BABOKv3 as “A state or condition that an organization is seeking to establish and maintain, usually expressed qualitatively rather than quantitatively”.

An outcome based metric is the objective that lines up with that goal. BABOKv3 defines an objective as “A measurable result to indicate that a goal has been achieved”

When defining an outcome based metric, it’s helpful to identify specific attributes for that metric. I’ve found the following attributes that Tom Gilb suggested in Competitive Engineering to be particularly helpful:

Name

Unique name for the objective

Be precise about what you call the metric. Make sure it’s unique from all other metrics and that you have a shared understanding between everyone involved with your product about what that metric means.

Units (Gilb refers to this as scale)

Be specific about what you’re going to measure. This describes the specific things you are going to count and the things you are going to ignore. In many cases, this is a definition of the name.

Method (Gilb refers to this as meter)

Explain specifically how you’re going to measure the metric. Provide as many specifics as possible to prevent any confusion about what the metric is.

Target

Level you want the metric to get to consider the desired outcome achieved. This is your measure of success.

Constraint

This is a value of the metric that you’re trying to avoid.

The value of the constraint will vary depending on the nature of the metric and the situation in which you work. The constraint may be equal to the baseline, which means that you want to see a change in the metric but you don’t want to get worse than the current baseline.

The constraint may be between the baseline and target which means you want to see some change in the baseline in order to not consider your effort a failure.

In a case where you want the metric to increase from baseline to target, your constraint may be less than the baseline which indicates that you’re willing to see a decrease in the metric temporarily as you explore different options to eventually get to the target.

Baseline

This is the current value of the metric. You may find that you have to do a little research to determine the current value of the metric, or you may not be currently tracking the metric so you don’t have an established baseline.

An Example

Outcome: Increase the number of practitioners engaging with a member association.

Attribute	Example
Name	New and renewed memberships/month
Units	Individual Members
Method	Sum of new memberships and renewed memberships within the month
Target	300 members/month
Constraints	200 members/month
Baseline	250 members/month

In this particular case, the association want to get to a total of 300 new memberships and membership renewals within a given month from their current value of 250 members/month. They realize that they may try some things that will cause that number to decrease in a given month, and they are willing to live with that as long as the number does not decrease below 200.

When to use outcome based metrics

Establish an outcome based metric at the beginning of any effort to define success. This gives you a clear indication of when you can or should stop work on one outcome and move to a different one.

Use outcome based metrics to measure progress toward delivering a particular outcome and to measure how successfully you’ve been at delivering the desired outcome.

Why use outcome based metrics

You want to establish outcome based metrics so you have a quantifiable way of knowing that your efforts are moving you in the right direction to delivering the outcome you seek. This allows you to run effective experiments to find the best solution (or combination of solutions) and it helps you to determine when to stop working on one outcome so that you can switch your efforts to address a different outcome.

In this way, outcome based metrics provide decision filters you can use to determine whether you should continue with your current efforts, or move on to a different effort.

You want to use outcome based metrics to gauge progress rather than output based metrics such as velocity because output metrics don’t provide any context. They may indicate that your team is doing a lot of work and delivering a lot of output, but they don’t provide any context that tell you that you’re delivering the right things.

How to use outcome based metrics

Determining outcome based metrics

Gather the right people together. This includes your team and the key stakeholders for the product.
Confirm what outcome that you’re looking to accomplish. You can use approaches such as the problem statement or internal product opportunity assessment to determine what that outcome is.
Decide what a good measure would be to know when you’ve reached that outcome. Avoid metrics that are attached to delivering something (such as velocity, or percent complete) and instead consider a metric that is directly related to the outcome you want to accomplish. This may be some measurement of revenue, adoption, customer service, or organization performance. Ideally the metric you identify is directly to changes in customer behavior, but it may need to be a metric that looks at some measurable aspect of your business that indirectly relates to customers.
As part of that discussion, determine a name that is meaningful to everyone involved with the product and is unique from other metrics that you may be using. As a way of gaining agreement on what the metric is, you’ll also want to establish units and method. During this discussion, it’s also helpful to determine who is going to own determining the value of the metric and for what frequency.
As a group, identify the baseline for the metric. This may be relatively straight forward if it’s a metric your organization currently tracks. If it’s a new metric, you may need to do some research to figure out what your baseline is, or you may not have a baseline if you’re introducing a brand new product or process.
As a group, identify the target for the metric, which represents the value you want to get to.
As a group, discuss what value you don’t want the metric to get to as a result of any work that you do. This is the constraint on the metric.
Make sure everyone involved with the product understands the metric, what it means, and how it’s determined.

Using outcome based metrics

Your metric is a measuring stick you use to determine the effectiveness of the possible solutions you identify.

Select a particular output, deliver that output, give it enough time to have some effect and then compare the current value of the metric with your target and constraint. Did you reach your target? Then you can stop working on that particular outcome and move to something else.

Did you make progress toward the target, but didn’t reach it? Identify the next possible solution you could try in addition to the one you just delivered.

Did you hit the constraint level? Back out the solution you just tried and try something else, or stop work on the effort.

Caveats and Considerations

If your outcome based metric is new, you may need to incorporate some work in your backlog to track the metric on an ongoing basis.

When you have discussions to establish your outcome based metrics, the discussions themselves allow your team to get a clearer understanding of what success looks like.

You only want one or two outcome based metrics for any effort. You don’t want a whole bunch because then the metrics might start working counter to each other. Y

In some cases the metrics you identify are very easy to measure and provide results immediately. Those types of metrics are very helpful because they help you to have short feedback cycles. In other cases there are metrics that you have to wait for a few weeks or a month to see the impact of changes, you may want to identify a different metric.

Additional Resources

Beyond Requirements: Analysis with An Agile Mindset by Kent McDonald

Analysis Techniques for Product Owners Live Lessons by Kent McDonald

Empower Product Teams with Product Outcomes, Not Business Outcomes by Hope Gurion
The key point here is that your team should focus on product outcomes which measure changes in human behavior, is within your team’s control, and provides insight into whether you’re having the desired impact on your business outcome. This was a key insight I was somehow missing before seeing this post.

Doing Discovery Well: How to Measure and Guide Your Team by Teresa Torres

Teresa discuss how a team should measure and track their performance to an outcome on an ongoing basis.

What characteristics make a good metric

Picking the right metric is hard. But there is a systematic way to evaluate whether the metric you’ve ideated is a good. Shaw Li suggests the following guiding principles to evaluate your metrics:

Is the metric easy to understand and interpreted by people without training?
Will increases or decreases in the metric always have the same interpretation?
Is your metric a comparison (i.e., ratio or rate)?
Is it easy to measure and calculate?
Does it drive the behaviors you want?
Do you have ≤ 7 metrics?