The Confession
Early in my career, I measured my own effectiveness by the quality of what I shipped. Did the screens look good? Was the interaction model clean? Did the engineers build it close to spec? Those were my metrics. They felt like the right ones.
I was wrong. Execution matters, but I was measuring the work, not what it changed. And those are different things in ways that took me years to fully understand.
Here is what I mean.
Output: What You Made. Outcome: What Changed.
Output is the design artifact. The wireframe. The component. The flow. The shipped feature. Output is what you can put in a portfolio and point to.
Outcome is what happened because of the artifact. Did onboarding time drop? Did conversion increase? Did fraud analysts process more cases? Did students complete applications at a higher rate? Outcome is what the business cares about. It is also what users experience, though they usually do not have language for it.
The confusion between these two things is institutional. Design teams are typically resourced around output. You are hired to produce screens. You are evaluated on design quality, delivery speed, and stakeholder satisfaction with the artifact. The connection between that artifact and a measurable downstream change in user behavior is often nobody’s explicit job to track.
This is how you end up with beautiful work that changed nothing.
The Netflix Lesson: Trust Nothing, Including Instinct
I was at Netflix during the international expansion, working within the Netflix XP experimentation platform. The thing that changed my thinking most was not any particular test result. It was the posture of the organization toward design decisions.
At Netflix, the assumption was that you did not know whether your design was better until you measured it against a population of real users. Not a user research panel. Not a usability test with eight participants. Millions of users, in market, choosing between version A and version B with their actual behavior.
This was initially uncomfortable. I had opinions about which designs were better. Some of those opinions were informed by years of experience and pattern recognition. Some of them were just wrong. The data told me which was which, and it was not always the answer I expected.
The principle I internalized: animations and transitions should only exist if you can prove via A/B testing that user engagement increased to a degree of statistical significance. If not, it is noise distracting the user from achieving their goal. This sounds obvious. It is a principle that most design organizations never operationalize.
The experimentation culture Netflix built during that period is still in use today. That is the outcome of that work. Not the individual screens that were designed and tested and revised. The cultural and operational infrastructure for continuous measurement.
The Appily.com Lesson: The Number Tells You When You’re Right
At Appily.com, A/B and multivariate tests were routine. We had a lead statistician. Decisions about design changes went through hypothesis formation, test design, statistical analysis, and a decision about whether the result was significant enough to ship.
The Universal College Application achieving a 47% completion rate against a 20–35% industry benchmark did not feel like a win when I was in the middle of building it. It felt like a series of iterations, some of which worked and many of which did not. The college selection interface failed usability testing twice before we got it right. Deadline visibility was a problem we did not identify until mid-process testing revealed students were missing important information.
The number at the end validated the process, but it also revealed something more important: we hit 47% because we measured everything. We tracked completion rates per step, not just overall. We knew exactly where students dropped off and why. We iterated against specific failures, not general impressions.
Design without measurement and you ship something that looks right and works well enough. You never learn whether it is as good as it should be.
The TeleSign Lesson: Design Is Infrastructure
At TeleSign, the product processed 21 billion annual transactions. That number changes the stakes of design decisions in a way that is difficult to fully internalize until you are inside it.
A large share of the six-digit verification codes on your phone come from TeleSign APIs. When you get a login code from your bank, a ride confirmation from Uber, or a package notification from Amazon, there is a significant probability that a UI I designed was part of the infrastructure that made it happen. Not visible to you. Not something you would ever attribute to a product designer. But present.
This taught me that outcome at scale is almost never visible to the user. The outcome of the fraud prevention interface is that millions of legitimate users were approved instantly without knowing they were being evaluated. The outcome of the self-service portal: Fortune 500 teams went live in 35 days instead of 67, and same-day for simpler products. They never knew 67 was the baseline.
Good infrastructure is invisible. Good design at scale is the same way. The measure of success is not how often users notice the design. It is how rarely they need to.
What Changes When You Measure Outcome
I want to be specific about what actually changes when a design organization shifts from measuring output to measuring outcome.
- Prioritization becomes objective. When you can connect a design change to a measurable user behavior, you can make a case for building it that does not depend on stakeholder intuition. The TeleSign portal moved from proposal to funding in part because I could quantify the cost of the status quo: 40% of CS team time on tasks that could be automated, multiplied by fully-loaded cost, divided by the story points required to build the alternative.
- You stop defending the wrong work. When you measure outcome, you find out faster that a direction is not working. This is uncomfortable in the short term and valuable in every other way. The directions I abandoned at Appily.com after failed tests freed up time for the ones that worked.
- The conversation with engineering changes. When you show up to a sprint planning meeting with a hypothesis attached to the design request, you are a different partner than someone showing up with a mockup and a preference. Engineers respond differently to “we think this will reduce drop-off by 15% based on heatmap analysis and the two previous tests” than they do to “the current design feels confusing.”
- Your portfolio writes itself. If you measured outcome, you have the numbers. The case study headline is already there. You just have to tell the story that connects the design decision to the result.
The Honest Caveat
Not every design decision can be measured. Some things matter and are not measurable. Brand coherence. Accessibility for users who are not large enough in your user population to show up in a statistical test. The experience of a user who encounters an error state once in five years. These things are real. They do not always produce a number.
If you cannot say what changes when the work succeeds, you do not have a problem yet. Go back and sharpen it.