In recent weeks, energy analysts have highlighted the limits of existing clean energy technologies.

Even as their share grows, variable renewables face serious economic challenges at higher levels of integration. The latest iteration of Gen3+ nuclear reactors has been beset by cost overruns and construction delays. And while the recent progress in storage technology is worth celebrating, scalable utility-scale storage remains a distant dream.

Increasingly, analysts describe current designs as "bridging technologies," stepping stones toward more advanced clean energy, whether in the form of perovskite solar cells or modular nuclear reactors. To become commercially viable, however, these emergent technologies will need all the policy support they can get.

Thankfully, the U.S. government is rather good at driving radical innovation. The decades-long struggle to make fracking viable is one of the DOE’s great success stories, where public and private groups worked together to break the shale code. As such, the shale gas revolution offers important lessons for energy innovation policy.

1. The government must gamble

Predictions for the Next Big Thing are almost invariably wrong. Fossil energy bets, whether public or private, were no different.

When the U.S. government stepped up its R&D efforts following the 1973 oil embargo, many assumed underground coal gasification and synthetic fuels would power the future. Even within the niche of unconventional gas, most experts believed that tight sands gas would play a larger role than shale ever would.

Today, there is very little consensus around what the next wave of clean technology will be -- all the more reason for governments and policymakers to avoid technological tribalism. As shale shows us, the technologies with little early promise are often those with the biggest long-term payoff.

2. Public and private must cozy up

Because radical innovation is so complex, and requires such different types of expertise, multi-group collaboration is invaluable. Against widespread concerns about the corrupt "revolving door" between government and industry, fracking is actually an example of why it’s essential for government agencies to work closely with the private sector.

In the case of shale, the Department of Energy and the Gas Research Institute were extremely successful in recycling national labs' research to meet industry needs. As they have no skin in the game, these public or semi-public bodies also acted as intermediaries between potential rivals. None of this could have happened if public and private were kept away from each other.

3. Pork barrel spending has its upsides  

One of the most maligned aspects of industrial policy is also one of its strengths. Even the best-designed energy policy is useless if it cannot gain political support. By also providing a local stimulus, pork barrel spending can make large research programs and tests beds politically saleable.

In 1976, Senator Robert Byrd (D-WV) lobbied to get a major shale research project, the Eastern Gas Shales Project, funded in his home state. It was three years after the oil embargo, and Byrd wanted to direct some of that spending for jobs in his constituency. The result was not simply pork. The EGSP established that there was a lot of natural gas in shales, and helped spark the interest of Mitchell Energy.

4. Firms should come in all shapes and sizes

Innovation scholarship has long debated the optimal industry structure and the merits of small versus big, new versus old. But what really comes through in the shale gas case is that a diverse network of firms is optimal. 

Mitchell Energy, rightly credited for pioneering shale gas fracking, was a medium-size firm in the oil and gas industry: big enough to invest substantial funds into R&D, but small enough as to have few alternatives other than to bet on shale.

The majors, by contrast, focused on the less-risky option of offshore drilling, and yet they still played an important role in developing early iterations of horizontal drilling. Small service firms were important vehicles through which science (and most often scientists) could make its way from the national lab to the field.

5. Investors must play the long game

Energy transitions are not a game of immediate gratification. By the time Mitchell and Devon Energy combined to “crack the Barnett” in 2002, work on unconventional gas extraction had been underway for decades, tracing back to the oil embargo and even before that to the nuclear tests of the 1960s. The fact that we are still waiting on the next generation of solar, nuclear and batteries suggest the next breakthroughs may take just as long. With these timelines in mind, policy must be oriented toward the long term and provide the patient capital that is often lacking in the private sector.

Innovation is hard work. The kinds of breakthroughs needed to fully decarbonize the economy are likely to be harder still. To stand any chance of success, policymakers and business leaders alike must move beyond old ideas about what public and private can and can’t do. When it comes to innovation, neither can thrive without the other. 

***

Loren King is a senior analyst at the Breakthrough Institute. His interests lie at the intersection of innovation studies, energy policy and economic growth. He has published reports on international innovation, the origins of the shale gas revolution, and the role of general-purpose technologies in sustaining long-term growth. The Breakthrough Institute's report can be found here.