Kate Mills’s job is to assess ideas. As an editor at Orion Publishing, she receives fiction manuscripts sent to her by literary agents and determines whether a market will buy them. The authors believe that they have created something that other people will enjoy reading. Their agents believe that they’re representing a winner. Kate Mills, like other editors has to decide whether they’re right.
In 2013, she thought that one agent and their author had got it wrong. The detective novel she had been sent “was well-written but quiet,” she told The Independent. “It didn’t stand out for me and new crime novels are hard to launch right now.”
She rejected it. The author, though, was working under a pen name. “Robert Galbraith” was actually J.K. Rowling, and by the end of 2015, The Cuckoo’s Calling had sold more than a million copies and was being adapted into a television series.
Mills shrugged off her decision. You have to love a book to take it on, she said, and this one wasn’t for her. Nonetheless, missing what turned out to be a very good book idea cost her company millions in lost revenue.
It’s a problem that’s not unique to the publishing industry. Companies constantly have to assess new ideas. Technology firms have to decide which features to develop. Product manufacturers have to find new ways to compete with rival companies. Advertising firms rely on creatives to generate new brand stories for their clients, then rely on art directors and clients to be able to choose which ideas will resonate most.
So what’s the best way to judge ideas and how can a company reduce the risk that they’ll miss a good, valuable concept?
An Idea-Picking Machine
One option is to use a tool. The Darwinator is a program for running innovation tournaments. Developed by business school professors Karl Ulrich, Karan Girotra, and Christian Terwiesch, the program acts first as a submission box. Participants can pitch their ideas to which administrators can apply rating mechanisms. Other participants can then judge the ideas, and the highest rating suggestions move on to the next round. Each round consists of four stages: submission, discussion, evaluation, and reporting so that at the end of the tournament, the tournament administrators will not only have a winner but can see how each idea fared during the evaluation phase.
The company behind the Darwinator claims that the tool has been used by a large healthcare system with multiple hospitals to crowdsource ideas and judge innovations for clinical care coordination and patient experience. Any employee, including doctors, transporters, nurses, and senior executives could submit and judge ideas. It’s also been used by the R&D department of a biotech company to direct clinical development capacity and by business schools to judge start-up ideas and select courses.
But while the case studies describe how some organizations have used the Darwinator to receive and evaluate ideas, those case studies don’t describe the results. They don’t provide evidence that the ideas that received the highest evaluations went on to perform well in practice or reveal whether rejected ideas were picked up successfully by a competitor. The Darwinator is one way to adjudicate ideas but it’s not clear that it’s the best way or that it can ensure that good ideas aren’t lost.
What it does have working in its favor though is the nature of its crowdsourcing. Administrators can allow anyone in the organization to contribute ideas, and they can also allow anyone to evaluate them. That’s not how idea judging is usually performed at companies. At most firms, someone first pitches an idea to a supervisor who either shoots it down or agrees to think about it. It’s managers who judge ideas and it’s the managers who have the responsibility for making the final decisions.
According to some studies, though, managers are not the right people to determine whether an idea is good or not.
Choose the Right Idea Judges
In 2016, Justin Berg, a professor at Stanford Graduate School of Business, conducted an experiment. Working with James Tanabe, a former creative director for Cirque du Soleil, and Lena Gutschank, a circus artist, he collected more than 150 videos of circus acts around the world. He showed ten videos to 339 circus professionals, and asked them to predict which of the acts audiences would like, share, and pay for. He then tested those predictions by surveying more than 13,000 audience members.
The circus professionals contained a mixture of people in two types of roles. Some were creators. They were the people who designed and performed the acts shown in the videos. Others were managers. They were former creators who had been promoted to roles with more responsibility.
What Berg and his team found was that the creators weren’t very good at predicting the success of their own ideas. They tended to think that the creative performance ideas that they had produced would do much better than they actually did. However, they were better than the managers at forecasting the success of the ideas of other creators.
“Typically, creators are asked to evaluate only their own ideas, and managers are the sole gatekeepers in deciding which ideas reach the audience,” he writes. “Yet my results suggest that creators are bad at forecasting the success of their own ideas but better than managers at forecasting about others’ ideas.”
Berg suggests the reason managers struggle to predict the success of creative ideas is related to the difference between the kind of thinking required to generate new ideas and the kind of thinking needed to evaluate them.
Creators, he says, engage in divergent thinking. They look for connections between unrelated concepts—between juggling and chainsaws, for example. When managers evaluate ideas, however, they engage in convergent thinking. They look for solutions to a specific problem: what would an audience enjoy?
Divergent thinking can offer a wide range of different options: adding rain to trapeze acts; bringing treadmills to hoop-diving. But convergent thinking can only look towards what has solved the problem before and assess whether an idea contains similar characteristics. “Convergent thinking alone is dangerous because you’re just relying on the past,” Berg told the Stanford Business blog. “What will succeed in the future may not resemble what succeeded in the past. Divergent thinking helps people stay more open-minded about new ways ideas may succeed.”
One way to decrease the chances that you’ll miss a good idea or endorse a bad idea then is to change the responsibilities associated with company roles. Instead of creating a division between creators and managers or between producers and supervisors, companies should combine the roles. Managers might still have responsibility for choosing the ideas that are presented to clients or passed on to developers, but they should also be responsible for generating at least some of those ideas. Because the managers are still also creating, they’ll be able to avoid relying on the divergent thinking that only assesses what has worked in the past.
But because even supervisors will be bad at judging their own ideas, team members should also have enough psychological safety to give even their managers’ ideas a thumbs down.
It’s The Way You Sell Them
In practice, though, idea assessment can take place in a number of different ways. In an environment in which managers are also creators, and team members have the psychological safety to criticize a supervisor’s ideas frankly, those assessments are likely to be informal. A team leader might shoot out an idea during a meeting or around the watercooler. Other members of the team will point out the flaws or suggest improvements and the conversation will move on.
Sometimes though, an idea has to be pitched in a more formal way and to people who are neither creators nor supervisors. Advertising agencies have to persuade clients that their ideas are both innovative and will be effective. Start-up founders have to show investors that their ideas will find an audience and can make money. Small business owners need to talk banks and other lenders into handing over the funds they need to scale.
For those kinds of assessments, it’s not just the person doing the judging that matters but the skill of the person presenting the idea. In one study, a group of researchers gave employees of a large advertising company a one-page brief and asked them to design advertisements for a household insecticide brand. Participants first wrote down a list of ideas then selected the best idea from the list and used a two-page spread to communicate it. Finally, they completed a self-assessment that rated the originality of the idea and also its appropriateness, the degree to which it matched the advertising context.
The researchers presented the ideas to other advertising professionals who completed a similar assessment and who also rated the quality of the presentation.
Both judges and idea-generators were able to recognize the originality of the ideas. But idea generators tended to give their own ideas a higher appropriateness score than the judges did. Judges were more likely to see original ideas as “bizarre” than appropriate, a difference the researchers put down to the inability of judges to see the thought process that went into the creation of the idea. “Advertising creative personnel are recognized as exceptional creative thinkers with the ability to connect distant domains,” the authors write. “Rather than a linear approach, this work involves leaps of insight. External judges who see only the idea, not the process, may have trouble following along the path of connection.”
The authors argue that the “fundamental issue” for creative professionals is to get others to see the appropriateness of their creative ideas when those ideas have been built on significant conceptual leaps. The more original and strange an idea, the better the creative has to be at presenting it. Correlating originality scores with expression quality showed that the more an idea generator struggled to present an idea, the more a judge struggled to see that idea’s creative value.
It’s possible that relying on a two-page spread to pitch an idea, instead of a Powerpoint presentation or an in-person talk, might not have been the best way to show an idea’s originality. But the studies do show two of the most important aspects that affect the selection of a good idea.
The first is making sure that the judges are capable of assessing the concept. They need to know how the idea-generator created the idea and what they were trying to do. They need to be able to think like a creative. Judges who can’t think like creatives struggle to think like audiences.
That’s not too difficult to do inside a company: the managers who assess the team’s ideas can also be part of a team that creates ideas.
Companies that also have to pitch ideas outside the company though, need to make sure that they know how to sell them. They have to invest in presentation training so that they’re not only producing original and appropriate ideas but can also show the value of those ideas to others.
Even if you’re creating ideas as well as judging them though, and even if the people presenting them know exactly how to make a good pitch, it’s inevitable that you’re going to make some bad decisions. You’re going to choose ideas that don’t work and overlook good ideas that do… then see those ideas taken up by rivals. Just tell yourself that the concept didn’t stand out for you, that you didn’t love it, and that ideas like that are difficult to launch right now.
Recent Comments