(with Jennifer Gormley)
Product roadmap meetings — here’s the ideal: You and four other people cram into a small conference room that’s also stuffed with Post-it notes, whiteboards, and maybe even a slide deck. The scent of dry erase markers hangs in the air. It’s the fragrance of promise, a bouquet of hope. The idea is to come up with a plan for what the team is going to build in the next couple of quarters. A clean, prioritized list with strong consensus behind it. Ideas flow freely and fall into place like they were sent down from upon high.
Now, here’s the reality: You say, “We’ve got 37 great feature ideas to run through and plot out against our dev resources, user needs, and product vision. We need to explain to the board why we’re doing what we’re doing and stick to the plan. So I think the best way to…” and then everyone in the room interrupts you at the same time:
The Engineer: “Uhhh, yeah. We don’t really have any resources available with all these bugs the designers are flagging. Seriously, I can’t put it on a half pixel.”
The Designer: “If you’re going to tell me again we can’t do Feature 19 — that animated kitten gif for the users who get on the leaderboard — then I don’t really need to be here. And you all know I like blue, so make all the new features blue.”
The CFO: “He’s right. We don’t have any resources — since they are mostly playing
foosball in the lobby…”
The CEO: “Well, I’ve already sold Features 35, 36, 12, and eight to our new client in my last meeting with them, so those go to the top of the list…. And I have 2 more to add that their secretary’s cousin thought would be cool…. But of course we can do those after the four I just listed get finished in Q1. I can be reasonable.”
The onslaught draws a sigh out of you as you wonder if it is okay to drink at 10:17 am. As you uncap a marker and pass the sticky notes around, it’s clear that this effort failed before you all even squeezed into the room.
Product Design is fraught with best guesses, good ideas nestled among complex bad ones, and tenuous relationships with distant, sometimes theoretical users. Product roadmapping sessions are a strange mix of passion, interpersonal influence, poor planning, and misdirected energies. We’ve been in what must be hundreds of product meetings, and come up with one universal truth: Product Design = Hard. There’s no getting around that. Creating product roadmaps, however, well that can be easy. Seriously. Easy.
Recently, we faced the challenge of road mapping exactly 37 features with a new team. We didn’t want to make the same mistakes we had each made with previous teams, which almost always went like the example above. With this new team, we had just launched v1.0 of the product together. And remarkably, we all still liked each other. We even felt good about moving to the next phase; our product was in the market and feedback was coming in, mostly positive.
We looked at the blank sheet of paper upon where next year’s plan was about to unfold, and we thought: instead of tackling things the usual way, let’s take the personal passions out of it. Questions immediately clouded the way. This time, can we make — with a new, unsullied team — a less contentious, more productive, and more confident (team wide) roadmap? Could we filter out personal biases and focus only on goals (for our company, for our users, for our bottom line)? What basic math would result in a non-controversial, prioritized list to tackle with available resources? And could we use the list to make smart choices about spending more on developers, with quantifiable market impact?
Hell yeah, we could — and it actually works.
We considered 5 key perspectives:
- Market Perception: Are we a leader? Do we have feature parity with others?
- Cost Savings: Does the feature make things more efficient for internal ops? Solve technical debt that is becoming costly?
- Key Vertical Differentiator: Does the feature clearly distinguish us among competitors?
- End User UX Perception: How great is the UX for the bulk of our users?
- Administrator UX Perception: How great is the UX for our purchase decision makers?
…and one overarching key driver:
- Revenue Generation Potential: Obvs.
We began by asking the senior subject matter expert in relevant departments to rate each of the 37 features from the perspective of their team. These ratings were done based only on the title and brief description of the feature, because we didn’t want to spend a lot of time building specs or wireframes at this early stage. Ratings were done on a scale of 1–5 (with one being little to no potential impact, and five being high to “we better do it or else” potential impact). They did this in a vacuum. That is, no one knew how the others were rating things.
The benefit of this blind rating, done only in one’s area of focus, is that no one is influenced by the others’ numbers. Groupthink becomes negligible, and features which might otherwise overlap on a feature/value matrix begin to come into stark contrast with one another. When we reassembled to look at the features and their ratings as a group, it was clear something remarkable was happening. We’d cleared away all the extraneous debate and laid bare the priorities our collective brain deemed most critical to work on. Magic — or is it math?
Our Mathematics of Roadmapping:
For each feature we calculate a “Feature Score” (FS) as:
FS = (average of perspective scores) x (revenue generation score/5)
The first multiplier is a simple average of all the perspectives, each equally weighted. The second term overlays revenue on a normalized basis. By “normalize,” we mean we divide the revenue rating by the number of possible points so it’s value is never greater than 1. In this way, revenue can significantly boost or limit the FS but its impact doesn’t overshadow the average of the feature perspectives. Let’s take this for a spin.
- Feature 1: “Improve User Registration” had an average score of 2.6 and a revenue score of 5, giving it a 2.6 FS
- Feature 2: “Real-time Chat” had an average score of 1.2 and a revenue score of 1, giving it a 0.24 FS
- Feature 3: “Offline Sync for Native Mobile Apps” had an average score of 2 and a revenue score of 3, giving it a 1.2 FS
By normalizing the revenue generation score (n/5) we can see the subtle variations in priority among all the features. Although the CTO wouldn’t have rated user registration improvements as a high-priority project, a balanced perspective shows it to be very important to take on. We have seen that natural patterns and affinities begin to emerge, like a focus on collaboration tool features, or on internal content authoring efficiencies. These roll up into easier market discussions, client pitches, and board reviews of the company vision.
This also works well when thinking about “themed” releases. If, for example, the next major market release has a “social” theme, the team can filter out features that don’t fit “social,” and then conduct the exercise. The result will be a prioritized set of projects that accomplish the theme.
Our approach has worked well for small teams, particularly so for startups. They tend to be constantly responding to changing priorities and moving targets, yet they need a roadmap to guide 3–6 months worth of development. For larger teams, it’s likely that more complex math, a different approach to averages, or laying in additional team perspectives would be more suitable. The inputs are not one-size-fits-all, but we think this technique is, and we urge you to try it. This alone isn’t a recipe for product success, but it might just save your team’s sanity, and get you a roadmap everyone believes in.
So, instead of the contentious clamor described earlier, imagine a quiet afternoon plugging in the data from your team. They provide accurate scores from their perspectives quickly and easily. You save hours avoiding the hair-pulling, passionate, but unproductive debates that devolve into eye-rolls over the designer’s animated cat gif. You come together to review the early plotting with a sense of order already on the table, making roadmapping actually easy.
When it comes to imagining and building great products, we wouldn’t trade the passion of startup teams for anything. And any tool or process that converts those strong feelings into a viable plan — or working code — is a great thing. So, in your next product roadmap session, give this math a chance. It takes the passion out of the planning and puts it where it belongs — into the work. Now stop sniffing those whiteboard markers and get to it.
I don’t often promote conferences and events here or generally, but there’s one coming up here in NYC that has me pretty jazzed. My friend Evan Nisselson and LDV Capital are putting up the first annual LDV Vision Summit in June and I am pleased to be helping out a bit on the sidelines. The cast of participants is stellar and the price is right.
There’s an old joke that says calling something the “first annual” is just wishful thinking. But I am quite certain the First Annual LDV Vision Summit will be the first of many.
It takes a peculiar and elusive kind of alchemy to assemble a successful technology event. Evan and his compatriots have lightning in a bottle on this one, so I’m recommending it. For a hefty discount, use code ‘PARSONS’ when you sign up and I will see you there.
The NY Times piece examining the perilously flawed rollout of HealthCare.gov is balanced and detailed. It is worth a read. It deconstructs the mad scramble to fix a problem that never should have happened given the resources applied to this big, but solvable problem.
Some details are in the NY Times article and countless blogs, and of course only those engineers involved know the real story. And despite the many educated guesses about the real net cost of the project, perhaps we can all agree a lot of the budget is waste blown on consultants and processes that didn’t make an iota of positive impact. In fact, a lot of the cash likely introduced risk by adding layers of management and friction that tend to dilute engineering focus.
I’m basing this on years of observations of large companies, agencies, consultancies and quite a ton of startups. It comes down to how cash and people are deployed, paced and exhausted. In large teams with virtually unlimited resources, waste creeps in before you can say “SCRUM” or “lean” or “iterate.” This is why “Lean” and “Agile” consultancies can charge a premium to swoop in and make big organizations more efficient. Don’t take the quotation marks as sarcasm- they undoubtedly can and do build effeciency where is is most needed. When resources are limited, one uses them efficiently out of the gate and knows when they are too costly to sustain. And most importantly, one spends the scarce dollars to ensure measurable progress and feasibility of the end goal. It seems to me the massive Healthcare.gov teams did neither.
The Healthcare.gov software cannot have been complex enough to justify even the smallest of the expenditure estimates I’ve seen. Let’s call it an even $100,000,000.00 before the rescue missions began in November. Now let’s refine that with a gross oversimplification: staff the project with 200 engineers at a cost of $500,000.00 each. With these folks, I’m quite sure a v1 of nearly anything can be built, launched and adequately monitored. I’ll further guarantee that this team can build a Healthcare.gov that supports at least hundreds of simultaneous enrollments and displays useful error messages beyond its capacity. What happened instead is shameful; embarrassing to the administration and inflammatory to we taxpayers who shelled out for it.
Technology can be hard. Coupled with government regulations and supercharged politics it must be really hard. But necessity (scarce resources) is the mother of invention. Invention yields systems that work. Predictably. Healthcare.gov was given virtually unlimited resources, and judging by the HTTP 500 error I saw only yesterday, its architects still have quite a bit of inventing to do.
Recently I had the good fortune of connecting with Dušan Omerčević, VP of Engineering for Zemanta who visited NYC from their engineering center in Slovenia.
Dušan wrote a terrific post inspired by our conversation here, and it is worth a read.
We talked a lot about how important patience and pacing are in an engineering organization, especially when building yours from scratch or pivoting an existing one with new hires, new processes, and new focus. Our tendency as engineers is to fix what we can fast and get things moving. We want to focus on code, and through code right everything and release fast. But there is a discipline to learn around letting some things run their course while very precisely controlling others. In mentoring team members, for example, it is often essential to stand on the sidelines and allow some hard but harmless lessons to be learned, while at other times one has to steer a less experienced developer away from failure with words and examples.
It was fascinating to compare engineering cultures and practices, and I came away feeling once again that common sense, a foundation in solid engineering process, and indeed patience are the basic ingredients for a winning productive development team. No matter what it’s size or location.
The Web is abuzz this week with anticipation of the upcoming Apple Television rollout and how it’s going to revolutionize the way we interact with television programming from the couch.
While I agree that’s going to happen, I think the reasons are not as intuitive as the analysts believe. Apple isn’t going to do something we haven’t seen before, at least not yet. And, the living room TV content revolution has largely already happened.
Cable TV is a giant set of brotherly monopolies in the US. As Henry Blodgett points out, there is a history lesson to be taken from the fate of the newspaper industry. But that’s already an old story- what the market wants is on-demand content with a great user experience (UX). It’s sad and infuriating that there is no competition in unified set top boxes or a way to accomplish delivery of all TV content. I’d pay for this, and so might you. For years innovation in television UX has taken a back seat to everything else in television while every other manner of consumer device has benefitted from faster hardware and smart interaction design. But how revolutionary can entrepreneurs be when the very core of the content ecosystem is locked? Just ask Boxee, Logitech, Roku and many others. Consumers have a binary choice: deal with crappy cable boxes or “cut the cord.” More and more are cutting the cord. And getting everything they need from their screens.
So Apple can’t solve this. But- if they have partnered with cable providers as the swirling rumor mill supposes, Apple will be able to bring a decent user experience to television, in a way that the $99 Apple TV in my living room fails to do. Google failed at this marginally, too. Cable company set top boxes fail at it spectacularly.
One has to wonder who is responsible for cable box UX in OEM deals with Cablevision and Time Warner. Are they given shoddy second-hand hardware to work with? Terrible design teams? A mandate to frustrate their end users? Perhaps a crash quota of 2 kernel panics per week? Craziness, right? And yet, this is how much cable company television user interfaces suck.
If it takes the magical shimmer of Apple’s Television to break free, so be it. Apple doesn’t need to do more than provide a decent, modern touch-friendly experience to push the cable guys out of the UI business forever. But let’s hope and do our best to ensure that we don’t trade a bunch of regional monopolies for an international one. Give TV consumers a choice of great UX’s from which to choose, and I think they will continue to pay for cable subscriptions and suffer the advertising that makes the whole think tick. At least for a while.
Bring on great living room experiences powered by innovators who care about responsiveness, intuitive design, and have studied the art of user experience. Give me UX, not SUX.
I spent much of last week soaking in all the inspiring developments in the Python community at Pycon 2012. A huge thanks is due Jesse Noller @jessenoller and his team for organizing an engaging, well-paced conference and for executing it flawlessly. Impressive.
I played a small part with a Poster Session on Sunday, the final session day. I talked about how and why we arrived at a Python codebase for my new project, Happify. My poster, below, is an unscientific look at various startup-friendly languages across axes like hiring, readability, produtivity, and performance. I spoke about the state of the NYC startup scene, how Python is on the rise, and why it’s a particularly good fit for small teams that require each member of the team to contribute at every layer of the stack. It was very well-received, sparking some vibrant post-session conversations on everything from Positive Psychology to creative use of Python decorators. My short intro video is here. You’ll see clearly I’d had insufficient sleep ond more-that-sufficient caffeine.
But the best aspect of Pycon for me was getting to know the community of folks that make the language thrive. For years I have been intrigued by the differences between various self-organizing open-source technical communities, with first-hand knowledge of the groups driving Ruby, Python, Scala, Java, and PostgreSQL. I have in mind a long post which will explore their similarities and differences, but I will pen a few words about Python here.
Python has a long history as a utilitarian “get things done” language. It’s wide adoption by academic and scientific programmers, as well as the web application startup scene has proven that its readability, lack of magic, and productivity are important to anyone writing code. Perhaps even more important than out-of-the-gate performance and concurrency. This cross-cutting eye toward utility is very clear at Pycon. Talks and Posters surrounding my own included explorations of computer vision, artificial intelligence, asynchronous evened programming, web frameworks, and even my favorite “Militarizing Your Backyard with Python: Computer Vision and the Squirrel Hordes.” Get the language out of your way, and you can do anything from shooting rodents to making people happier through gaming.
Anyway, I’m excited to be a small part of the Python community.
And a PDF version: PyCon 2012 Poster
I am excited to be in Santa Clara this week for company business and for PyCon 2012.
In recent years I have seen a tremendous lift in the size of the Python community and the innovation coming out of it, and I am thrilled to be a part of it. On Sunday I’m giving a “poster” presentation on this very topic; I’ll be engaging with the community here on why a Python-based stack is the right fit for our efforts at my new startup, Happify.
Beyond the obvious energy and creativity coming out of Python, my thesis is that NYC is becoming a center of great Python work in startups. Thanks to open-source frameworks like Brubeck.io, a growing talent pool of xooglers and Bay-area expats, and the sheer number of funded early-stage companies embracing Python (for it’s readability, productivity/fun, and good-enough performance), it is a natural and solid choice for NYC startup efforts. I am jazzed about becoming more involved with the NYC Python scene and contributing wherever I can.
If you are at PyCon, please say hi. I’ll post about my experience here when I am back in NYC.