Scrum, day 35: choosing evidence of value

Today's objective: propose a measure of a user outcome, explain its limitations, and identify the decision it should inform.

Read alongside: Mastering Professional Scrum, chapter 4, focusing on measuring value. Allow roughly five minutes for this companion, plus the book section.

Begin with the decision, then choose the measure

Elena wants to know whether the fictional ShiftSwap team should keep improving the manager handover view or direct its attention to another problem. Counting visits to the page would be easy. It would not tell her whether incoming supervisors can identify and resolve pending requests more reliably.

The decision concerns usefulness, so the team needs evidence about the handover. It might observe whether supervisors identify the relevant requests without asking the previous supervisor to reconstruct the list. It might also investigate where they still need help.

A measure becomes useful through the decision it can influence. Collecting a large dashboard first and searching for a favorable story afterward makes it easier to confuse available data with relevant evidence.

Define an observation precisely enough to repeat

The team proposes tracking handovers in which the incoming supervisor correctly identifies the pending requests using ShiftSwap without reconstruction by the outgoing supervisor. It needs to clarify several terms before the resulting count means much.

Which handovers are included? Does a handover with no pending requests count? What qualifies as reconstruction? How will the team establish that the list identified was correct? Does ordinary discussion about an unusual request count as failure?

For an initial small study, Elena chooses handovers with at least one pending request and records a short description of any help needed. This keeps the observation connected to the problem. It also makes clear that the results apply to those observed handovers, not every possible use of the product.

Keep the denominator visible

Suppose six of eight observed handovers meet the team's criterion. Saying “six successful handovers” omits the opportunities that did not succeed. Saying “75 percent success” without mentioning eight observations can create an impression of precision the evidence does not justify.

The team reports the count, the setting, and the remaining difficulties. It explains that the observations are limited and may reflect the managers who were available, the types of requests present, or the team's presence during the handover.

These limitations do not make the evidence useless. They help people choose an appropriately modest next step, such as investigating the two difficult cases before expanding the feature to more cafés.

Pair a number with an explanation

One unsuccessful handover involved an old request with unclear shift context. The other involved a supervisor who did not know the new view existed. Combining them into a single failure count hides two different interventions.

The first may require a product change. The second may require a clearer introduction or a better route to the capability. A conversation about each case helps the team understand what the number can and cannot explain.

Qualitative observations also need care. A vivid story from one manager should not automatically dominate every other observation. The team can use stories to generate explanations and then look for further evidence that supports or challenges them.

Watch for a misleading improvement

The average time spent in the view falls. That could mean managers find decisions more easily. It could also mean they stop using the view and return to chat. The same number supports incompatible interpretations until the team examines the behavior behind it.

Similarly, fewer support requests could indicate clearer information or a less accessible support route. More completed swaps could reflect a busier staffing period rather than a better product.

Before adopting a measure, ask how it could improve while the user's situation worsened. Add a complementary observation where it helps distinguish those cases. Avoid responding by collecting everything; choose the information that addresses a real interpretive gap.

Separate comparison from attribution

If handovers improve after a release, the release may have contributed. Other changes may also matter. The cafés could have introduced a new supervisor checklist, changed staffing, or experienced a quieter week.

ShiftSwap records relevant context and compares similar situations where possible. It avoids claiming that the feature alone caused an improvement unless the evidence supports that stronger conclusion. A small product team can make useful decisions without pretending every observation is a controlled experiment.

The practical question may be whether the feature appears helpful enough to retain and where to improve it next. That decision often requires less certainty than a broad claim about a guaranteed effect across an entire organization.

Collect only what serves the question

The team does not need to record private employee conversations to understand whether managers can identify pending requests. It can use appropriately limited observations and aggregate findings, keeping unnecessary personal details out of the learning record.

This is also a product design choice. Measurement should not create an intrusive experience or consume more attention than the decision warrants. Elena can explain what the team is trying to learn and why the chosen observations are proportionate.

The Evidence-Based Management Guide provides a broader reference for using evidence and experiments to improve outcomes. Here, the team's small observation plan is an original application to a specific fictional decision.

Design one useful measure

Write the decision you need to make, the outcome you care about, and the observation you propose. Define the numerator, denominator, time window, and relevant setting if you use a rate or percentage.

Add one limitation, one possible misleading interpretation, and one complementary question. Decide in advance what different results would lead you to investigate or change. If every possible result supports the same action, reconsider whether you are collecting evidence or decorating a predetermined decision.

Reflection: could someone unfamiliar with your team explain what the measure actually represents? Which missing definition would most change their interpretation?

Next, day 36 examines what to do when stakeholders describe different needs.