←
→

How do you prioritize research?

Posted: August 28, 2012 | Author: | Filed under: blog | Tags: , | 14 Comments »

One of the most fun and challenging parts of my job is setting bitly’s research agenda. We’re a startup, so this means prioritizing the set of questions we look into in the context of what will be most beneficial for the rest of the business, for the short and long-term, by creating opportunity and opening up potential futures. We work on a wide variety of projects, from pure research to press collaborations to infrastructure and experimental products.

We always have a list of research questions way longer than we have time and resources to pursue, so we developed a process for evaluating whether a given question is worth pursuing at a particular time.

This is the kind of process that I’ve only discussed with several people over whisky (thanks!), but not seen written up. I initially had a much longer list of questions but have decided to keep it as simple as possible, to frame a discussion but not dictate or burden it. I hope it’s helpful and I would love to hear about other appproaches.

For each research question that we might look into, we ask the following:

  1. State the research question.
  2. How do we know when we’ve won?
  3. Assume we’ve solved this question perfectly. What are the first things that we’ll build with it?
  4. If everyone in the world uses this, how does it change human behavior?
  5. What’s the most evil thing that can be done with this?

State the research question.

It’s important to state the question in language that everyone can understand. The bitly team comes from a variety of scientific and business backgrounds, and we’ve developed some of our own common vocabulary, but it still takes a bit of effort to make sure that everyone understand the fundamental challenge and why it’s interesting.

How do we know when we’ve won?

Here we define the metrics that we’ll use to measure our success. For some questions, this is obvious, and for others it’s impossible to define — we can at least acknowledge that ahead of time.

Assume we’ve solved this question perfectly. What are the first things that we’ll build with it?

This question allow us to assess the potential business and product impact. What capabilities will we have with this that we don’t have now? It allows us to keep the long-term research vision in mind while still optimizing for shorter-term opportunities.

If everyone in the world uses this, how does it change human behavior?

What’s the maximum potential impact of this work? If it’s not inspiring, is it worth pursuing at all?

What’s the most evil thing that can be done with this?

I don’t ask this question to encourage evil (>:]) but as a creative tool for expanding how we think about validity, impact, and potential applications of the research. The label evil is so ridiculous that it permits people share their craziest ideas. Plus, it’s always a fun conversation to have.

Finally

I’m always revising this list, and I would love to hear how you think about prioritizing your work.

  • Anonymous

    I think an interesting question to ask is, “Do we know enough to have a starting point to answer the question?”, as a form of risk management.  There are lots of interesting research topics that either take too long or for which you don’t know enough to really tackle, and it’s good if you can identify and shelve those topics (or get the right help if you really need the answers).  I guess, in the software world, we have a tendency to think about implementation too early, but it also seems like having a basic idea about how you will accomplish your goal is at least a part of the prioritization process.

  • Anonymous

    Do you get much input about research direction from the management/business side? How do you determine what will be most beneficial from a business perspective? What are your deciding factors when trying to balance those business needs with wanting to push the research envelope? 

    • http://www.hilarymason.com Hilary Mason

      These are really good questions. We have a weekly leadership and I sit in our product and engineering prioritization meetings, but there’s no formal process for me to follow.

      • Luiz Aoqui

         And how do they usually respond? Do they see the importance of research?

  • http://twitter.com/fkschmidt Ken Schmidt

    This is a nice list. Reminds me of George Heilmeier’s criteria for a great DARPA proposal: http://www.csee.umbc.edu/~finin/home/heilmeyerCatechism.html

  • http://twitter.com/fkschmidt Ken Schmidt

    This is a nice list. Reminds me of George Heilmeier’s criteria for a great DARPA proposal: http://www.csee.umbc.edu/~finin/home/heilmeyerCatechism.html 

  • Pingback: » How do you prioritize research? hilarymason.com - 逆读网 IT Needle

  • Kevin Mote

    I think this is a great list of questions to help focus and refine the directions of your research. But I’m afraid many companies err on the other side of the spectrum: dropping research altogether for the sake of Getting Things Done. Nose too close to the grindstone to see the sparks, if you get my meaning. I think the healthiest companies encourage “ambient” research: it should be part of the atmosphere. Every day employees should be learning something outside of their normal realm; every week they should be exposed to new ideas and new paradigms. “TED Talk” moments should be encouraged and cultivated and nurtured. I think you have a real privilege to be working for a company that seems to get this.

  • Lehmann

    The main purposes of research is to reduce risk by defining the limitations and opportunities.  Since most companies are not as big as IBM or the universities, pure research – just follow possibilities – is not really an option.  So the first criteria is to focus on a business need, and preferably a customer need, that points to a product that will generate income.  At this stage the costs and benefits (income) will be guesses.  If you double your costs and half you income and the project still works, then it’s past the first stage.

    The next stage is to look at your windows – time and cash flow.  Could the problem or opportunity be investigated within the time a competitor will fill the gap and within the cash that’s available.  If the answer is no, put the project on the back-burner.Of the remaining items on your idea list, find the critical knowledge needed to determine if the whole idea can go forward.  If this knowledge/technique won’t work, or is too expensive, you need to find this out quickly and before assigning more resources to the overall project.  If the critical knowledge doesn’t work, then cancel the project and move onto something else.  A large part of research management is knowing when to stop.  Sometimes this is worded as “fail often and fail early”.  In a sense, it’s the corollary of your “how do we know we’ve won?: how do we know we’ve lost?  Again, this is risk management: most likely determining whether the critical factor works or not will remove 80% of the risk of the entire project.  This is news that investors like to hear.

  • http://twitter.com/AmanQA Aman Ahuja

    Your questions address the _scope_ and _impact_ of the proposed research. One might also consider the _investment_ in resources and time required, and perhaps allocate and _set limits_ on these costs. All these considerations and similar, like _risk_, are typical in project management strategy, and you probably give them due consideration already. 

    Thee following questions are hopefully more in the spirit of your original five: 
     - How much fun will we have working on this problem? Quick projects can be worth it for fun’s sake. Long projects that are significantly stressful and boring can drain a team’s energy and cohesion. 
     - Have other people worked on similar problems? Existing research (or the lack thereof) may influence decisions. 
     - How similar is this proposal to active projects (and in what ways)? The balance between complementary and diverse tasks in a team’s to-do list can be an important consideration. 

  • Pingback: When to recommend a research project to a client | TechRepublic

  • Pingback: When to recommend a research project to a client | Software Digest

  • Pingback: איך מתעדפים מחקר? | ליקוטי שיבולים

  • http://www.facebook.com/phillip.middleton2 Phillip Middleton

    This kind of problem is generally rampant. The business has a question, but doesn’t know how to clarify it. And an analyst/scientist (at least in title) may not have developed strong enough a skill in refining the business thought into a valid research question, set of questions, or hierarchy of questions. The statistical community sees this all too well. I often see people committing these errors rather flagrantly, without even a gross sense of humility that they might be way off base (ahem….dangerous IMHO).

    For the data mining community, this is where the VAST majority of data defined projects die.  One of my mantras is “You can do something the right way the first time, else you can always do it again”. While blind alleys and failures during the research process are equally as important as the successes, in business enveloped research, many of the things which confound one’s work can be solved in the beginning, *before* an experiment of any sort is conducted. Which is why the data science/statistics/etc community are not, and should not be considered merely as ‘data crunchers’ (common misnomer, mostly due to our own lack of establishing this understanding to consumers of our work).

    A number of process frameworks in our arena have been created. Notably SEMMA (from SAS), and CRISP-DM. I think that the latter framework is a good place to start. I have written a whitepaper that applies this framework and to both traditional and non-traditional research domains (for the private sector of course). It gets quite extensive (not in the amount of labor, but rather in the checkpoints you use to test your ‘creation’), but here are the super HIGH LEVEL basic steps in the process:

    Business Understanding – here comes the dialogue on developing the research question and what you hope to achieve (deploy) with it
    Data Understanding – exploration, initially evaluating feasibility of your research aims
    Modeling – distill the complex universe into something meaningful
    Evaluation – determine what of your output makes the most sense to deploy
    Deployment – the production process of your delivery
    Monitoring – if measurable, how well is your final product working? did it help achieve an objective? etc etc

    Some of these things are naturally cyclical as, at any step, the process may promote a novel understanding.

    Some folks are catching on to this idea (my bet is, generally if they are in a more regulated environment), while others continue to scoff at the idea (some reasons of which are honorable, many reasons of which are not).

    Some feel that a regimen with some of the scientific discipline we’re discussing stifles creativity. Issued with an overly heavy hand, this may have some truth to it. However without some sense of this discipline (i.e. a Vision *with* a Strategy), chaos can indeed reign.

←
→