There is little about creativity that seems logical. And yet, we continue to chase a reproducible path to creative insight. It is that insight, after all, that is the source of innovation, and innovation is the key to business success.

Models, concepts, and frameworks offer approaches to innovation, with varying success, and those who have pursued innovation attribute their successes and failures to varying levers. Is it possible, then, to get to the heart of what spurs creativity and innovation in a company?

We think it is, but what is there is anything but logical. We have observed 10 factors that play a significant role in driving innovation and suggest that what these factors have in common is what we refer to as the “three Ps”: paradox, pain, and problems.

The three Ps are not, however, harms to overcome; rather, they are the breeding ground for insights. Embrace the chaos and mine the insights, and innovation is sure to follow.

Abandoned Ideas. Have you ever seen rag-pickers trying to find utility in items strewn in a garbage dump? How is it that a pile of items, dumped by the world, becomes relevant for these rag-pickers? When your survival depends on finding new utility in old junk, you have no choice, but in a growth period, we usually overlook this option. Yet, innovation often involves finding new uses for old-junk ideas, failed as well as successful. Austin Kleon, author of the bestseller Steal like an Artist, echoes the concept, declaring, “Every new idea is just a mashup or a remix of one or more previous ideas.”

The fabled trip Steve Jobs and his engineers took in 1979 to Xerox PARC was a game-changer for Apple when, as the story goes, Jobs and his team discovered the elements of graphical user interface abandoned in Xerox’s archived research. Visiting knowledge junkyards, product dump yards, idea observatories, and industrial architectures often proves to be decisive in revealing something that has happened in one context and can be replicated for a different outcome in another. This is how innovations span companies, industries, and even time.

Frustration. To a large extent, frustration drives people to innovate. The issue is whether they innovate inside the company and setting that is responsible for the frustration or attempt to resolve the problem in another setting that is open and receptive.

Time and again, people pushed beyond the limits of frustration have found new horizons that unleashed their innovative pangs and talent. The Swiss scientists who discovered piezoelectricity—the energy-generating property of quartz crystal—were convinced of a new phenomenon in the world of watchmaking. However, they were driven to take their talents to Seiko because the Swiss watchmakers just couldn’t see any paradigm beyond the sophistication of gears.

Do not let bureaucracy or politics frustrate your innovators beyond the lines of disenchantment. Respond to the frustration of your innovative people or they will be lapped up by your competitors.

Rejection. Innovators are lucky when their initial ideas are rejected by the sponsors and stakeholders bestowed with the powers to bet on ideas. Rejection acts as a rebound in the natural process of rejection–rebound–reinforcement–reassertion, spurring the zest and zeal of innovators to find another avenue to express their ideas. Those not prepared to undertake this journey will fall short of their expectations for innovation. Keep rejection fair, impartial, and informative, and innovators will learn from the process and make changes.

Steven Sasson, the Kodak engineer who invented the first digital camera in 1975, was indulgently dismissed when he brought his invention to management. It was filmless photography and Kodak made money from film. According to media reports, Sasson described management’s reaction as, “That’s cute, but don’t tell anyone about it.” Sasson took the rejection in stride and went on to make the first digital camera, which changed the world of photography. Kodak made money from it, but it embraced the innovation too late: Kodak filed for bankruptcy in 2012.

Mavericks. At the end of the value chain, there is someone who will, for the first time, express skepticism about the way things are. That person notices that the shelf life of an existing product, practice, model, or approach is long gone. The maverick doesn’t know what cannot be done, has little regard for the past, and is willing to take challenges head on. Mavericks are blissfully unaware of limitations unless forced to conform, much as the prisoners in the movie The Shawshank Redemption were. When one of the longtime prisoners commits suicide, unable to cope with life out of prison, Red, another of the longtime prisoners, comments: “These walls are funny. First you hate ‘em, then you get used to ‘em. Enough time passes, you get so you depend on them.”

Give mavericks room and resources to experiment, and innovation is likely to be the outcome. Western militaries, usually a bastion of institutionalized conformity, are learning to heed the mavericks within their highly structured organizations. Soldiers in low to middle ranks often have to improvise in the moment when tactics developed by senior officers fail to address the situation on the ground. Those improvisations are often sound manoeuvres that can inform training and change the structure for future tactics.

Obscene Goals. Steve Jobs famously said, “We’re here to put a dent in the universe. Otherwise, why else even be here?” Many innovators are inspired by and strive for such obscenely bold goals. The Wright brothers gave us flying; Google made driverless cars a reality; GE’s efforts with the jet engine made it possible to fly Transatlantic without refuelling; and Amazon brought us shopping without visiting a store (it’s now working on enabling us to store shop without checkout lines). All these goals were once considered obscene, unimaginable, and impossible, but were eventually conquered.

Leaders need to be able to conjure goals that have the potential to overwhelm their company’s intrinsic paradigm—or be able to recognize when others offer those goals and get out of their way. Self-limiting orientations take the wind out of momentum and limit innovation.

Customers’ Irritation and Apathy. Irritated customers and apathetic customers are both good reasons for an entrepreneurial company or culture to reinvent itself and become relevant to existing and future customers. If you want to grow and ensure the sustenance of growth, keep finding newer and smarter ways of being relevant to someone. It is not easy. Gaining proximity, demonstrating empathy, and embracing customers’ pain allow us those critical insights that plant the seeds of innovation. The alternative is merely lip service—the superficial touch that merely nods towards innovation. Customers will detect the apathy and walk away.

Zappos understands customer service. According to CEO Tony Hsieh, “Zappos is a customer service company that just happens to sell shoes” (and many other things). Stories about going to extra lengths to support customers, sending flowers of condolence, and chatting with customers at length are not urban myths. Nor are they exceptional; rather, exceptional customer service is the norm at Zappos, whose call centre handles thousands of calls daily, answering each in under a minute. Understanding and responding to customers’ needs has earned Zappos a loyal following: 75 per cent of its orders are from repeat customers.

Diversity. VIBGYOR—the array of violet, indigo, blue, green, yellow, orange, and red—is nature’s own wonderful innovative interplay manifesting as a rainbow. When diverse people are brought together in a controlled environment, they each bring and contribute their individual perspectives in a new context. Andrew Hargadon and Robert I. Sutton, in the famous Harvard Business Review article “Building an Innovation Factory,” postulated that “taking an idea that’s commonplace in one area and moving it to a context where it isn’t common at all is a powerful and effective way to spark creativity.”

People naturally prefer homogeneity, not because they dislike foreign individuals or cultures but because diversity brings mental chaos that can be overwhelming. But it is in this overwhelming nature of chaos that the seeds of novelty lie hidden, the useful bits of information brought together in new ways.

Artists and novelists have famously moved abroad to heighten their creative powers. Companies that understand the value of interdisciplinary thinking create innovation days that bring together employees who might not otherwise interact. Many companies have embarked on active recruitment campaigns for a diverse workforce, recognizing the value of doing business from different perspectives. Mastercard bridged generational divides in its staff by having young professionals offer “social media reverse mentoring” to older employees. The result was not just social media savvy almost-retirees, but also a greater acceptance of technology in the workplace, better communication among staff, and an opportunity to transfer knowledge and experience between generations.

Disciplined Experimentation. Innovation is not just how fast a game-changing idea is adopted, but also how fast an idea is dumped after swift and disciplined experimentation. In the business of innovation, it is better to fail early and cheaply. To sustain series of innovation cycles, organizations need robust systems and processes to quickly put an idea forward, evaluate it, then dismiss it if futile or irrelevant.

Visual artists refer to the adage that the difference between drawing and making art is that artists know which mistakes to keep. The point is that they keep drawing and testing their techniques, willing to toss out what doesn’t work while keeping an eye for what does work, sometimes surprisingly so. Louis Pasteur, credited for revolutionizing science with his scientific approach to experimentation, launched his work on micro-organisms when he attempted to help an alcohol producer resolve his problem with spoiled, fermented beer.

Brokering Technology. Technology infuses new charm into old affairs, regardless of the industry landscape or business canvas. A few years ago, patients could often not get a timely diagnosis because precision medical diagnostic equipment was confined to plush hospitals. GE and Phillips changed that with their portable ultrasound and ECG equipment—technical innovations rooted in micro-technologies borrowed from many different industries. Great companies are good at scouting new technologies as well as recovering established technologies from other industries and sectors, giving them a new lease with efficacy in a different context.

Brokering technology is a powerful driver of change and innovation. Companies that sincerely invest in scouting and brokering technology invariably find themselves in sweet spots of innovation and are able to leverage an early-mover advantage. Finding a technological sweet spot has the power to attract investment, attention, and aspiration. In 2015, Paul Binun and his co-founder, Morgan Gustavsson, who were not happy with the experience of shaving with blades, discovered what seems to be a perfect solution: shaving hair with a laser, not a blade. They called their device the Skarp Razor and put the project to crowd-funding: they attracted US$4 million in an initial fundraising drive.

Knowledge Transfer and Networking. Tacit knowledge left stored is of no utility. In fact, it is stale junk, occupying unwarranted space and critical bandwidth. To grow and make a difference, knowledge (and money) must remain in circulation. New ideas are most often developed from existing ideas applied in a new context, and companies that recognize that principle are engaging in a process of open innovation: sharing internal research and development with outsiders.

The lightbulb—the icon of new ideas—was not a new invention when Thomas Edison patented it in 1879. Rather, it was the culmination of tinkering and experimenting carried out by a number of scientists over 40 years. The scientists who joined Edison at Menlo Park came with prior knowledge of these experiments and with knowledge of electromagnetic powers. Together, their toying resulted in advances in lighting, telephony, and mining.

Leading innovation has been and will remain one of the primary challenges of the human race. It is an art perfected by few and unknown to many, but these ten determinants give us practical clues for handling the psychology of innovation. If you want to innovate, embrace the chaos—the paradox, pain, and problems—and use structure to demystify and bring clarity to what was found. Those who take on the challenge, despite the overwhelming and conflicting nature of the situation, will usually find creative insight for innovation.

For a greater conceptual understanding of the three Ps, see the book Leading Innovation by Vijay Govindarajan—which takes readers through the journey of innovation from its philosophy to practice, providing a simple construct for innovation.