One of the biggest, baddest smart grid data sets out there describes how more than 50 million people lost power in the northeastern U.S. and Canada on Thursday, Aug. 14, 2003. That hot afternoon, a series of overheated power lines sagged into overgrown trees and shut down, triggering a cascading collapse of the grid across eight northeastern states and Canada’s Ontario province. The two-day blackout that cost about $6 billion in lost productivity -- and played a part in at least a dozen deaths.
Faulty equipment and human error were both to blame, a 2004 Department of Energy task force found. In 2006, the DOE and the Federal Energy Regulatory Commission (FERC) recommended that utilities and grid operators install synchrophasor-based transmission monitoring systems to collect the real-time data needed to predict and manage these kinds of problems, in close to real time, to catch the initial errors and fix them before they lead to disaster.
That involves a lot of new data -- up to 6.2 billion data points per day at a size of up to 60 gigabytes with 100 synchrophasors. Increase that to 1,000 synchrophasors, and you’re talking up to 41.5 billion data points, or 402 gigabytes of data, per day -- and a lot of that data is flowing into back-end IT systems at the microsecond speed.
The statistics come from a recent study by the Electric Power Research Institute (EPRI) and NoSQL database vendor Versant. The Redwood City, Calif.-based company joined EPRI’s IntelliGrid research program in February, becoming the only object-oriented data management solution provider in a long list of major smart grid vendors working with the utility-funded research group.
The study, one of the first fruits of the EPRI/Versant partnership, “shows, in a very simple way, that if you have measurement units in place, they would have given [grid operators] plenty of time to respond in an effective way,” Bert Taube, Versant’s director of business development, said in an interview last week.
But that response only starts with the billions of data points coming in from grid sensors like synchrophasors. Actually doing something with that data will require a new level of integration across discrete back-end IT systems, as well as flexibility to adapt to constantly changing sets of real-world conditions -- essentially, constantly rewriting the data flows for the grid in near-real time. And that task, Taube said, could simply be too much data, moving too fast, for traditional relational database management systems to handle.
That could open up the utility sector as a new growth market for so-called NoSQL, or object-oriented, database technology. Consider it yet one more example of how cutting-edge developments in the information technology space are slowly spreading to the smart grid, from the terabyte-scale data management platforms being built to manage the millions of smart meters being deployed across the world, to new service-oriented architectures and cloud-based analytics platforms to tackle pan-enterprise integration challenges.
Greentech Media’s Soft Grid 2012 conference this week in San Francisco will be covering topics like these in detail. In the meantime, here’s a precursor to how one company is trying to bring the NoSQL revolution to bear in the smart grid.
NoSQL vs. SQL: The Importance of Objects for Smart Grid
Relational databases, or RDBMS, are the table-based databases using the standard SQL language, as deployed by giants like Oracle, Microsoft, IBM and the other heavyweights in the field, using architecture created by IBM researchers in the 1970s.
Object-oriented databases, on the other hand, can offer increased speed and flexibility compared to their relational cousins, particularly when you’re dealing with thousands and thousands of discrete endpoints that relate to each other in complex and changing ways.
Google, Amazon, LinkedIn, Facebook, Twitter and other Internet giants have turned to NoSQL in a big way, and that’s led to big growth for the upstart database technology, even if relational databases still holds the vast majority of the world’s market. NoSQL startups like 10gen, DataStax, Hbase and others are taking up new technologies like MongoDB, Cassandra, CouchDB and Dynamo to deliver cloud-based, open-source data tools for industries from e-commerce to telecommunications.
So far, the slow-moving utility industry hasn’t jumped on the NoSQL bandwagon in the same way. But with the rise of the smart grid, utilities are going to need to upgrade their database technology to handle the millions of communicating, intelligent end points like smart meters, grid sensors, power control devices and the like they’re deploying -- and Taube believed NoSQL is a perfect match.
Relational databases think of each of these objects as a set of data collected from different tables. Thus, creating an “object” requires calling up all the data that describes it. That can take a lot of time, and involves complicated rules that define each object in terms of data that’s constantly changing.
Object-oriented databases, on the other hand, treat these objects in real life as objects in the IT sense, with each one storing its own data, with characteristics that don’t have to be looked up each time the software wants to “think” about that object, so to speak.
Here’s one metaphor Taube offered: while relational databases store goods in a warehouse, data objects that need to be accessed quickly are more like cars you park in a garage -- and for that kind of storage project, “you don’t take the car apart,” he said.
Relational databases can be beefed up to manage high-complexity, high-speed tasks, but at some cost. Microsoft, Oracle, IBM and other big RDBMS vendors are managing the challenge today by sending in “armies of technicians” or dedicating large numbers of servers to the task, he said.
“We have shown that object models are just the right way to go after network structures and get that connectivity into place,” he said. “If you put all that into an integrated solution, as Versant does, you can do a lot in the database, that’s very fast, very reliable. It’s the perfect way to deal with power quality issues -- and that’s what we’re talking about” with the smart grid.
From Telecoms and Building Automation to Grid Applications
Versant, a publicly traded company founded in 1988, has about 150,000 installations across thousands of global accounts, including big telcos like Alcatel-Lucent, EADS, Erickson, Alcatel, Intel, Verizon and Siemens. Many of these are deeply involved in the smart grid, Taube noted, though he didn’t say just how those partnerships might lead to more smart grid work for the company.
Since 1994 or so, Versant has also provided key tools for Echelon’s LonWorks platform, one of three prominent technology standards for building automation networks. Specifically, LonWorks has built Versant’s FastObjects database technology into its LNS network management system, which allows a variety of suppliers to build control networks that can install, configure and manage thousands of “objects” in a building. Echelon estimated that going with Versant saved several “developer years” on building the LNS, as well as making ongoing development work easier and faster, in comparison to RDBMS technologies.
In the world of the smart grid, Versant is targeting similar opportunities with a set of International Electrotechnical Commission (IEC) standards known as Common Information Model -- the set of standards used to manage the flow of power across regional and continent-wide grids around the world.
Three European independent system operators (ISOs) -- RTE in France, National Grid in the U.K. and Elia in Belgium and Germany -- are using Versant’s technology as part of a project that coordinates the prediction of region-wide grid stresses and strains, Taube said. Versant is doing something similar as part of a R&D project with EPRI working on wide-area monitoring, protection and control, he added.
More research projects, involving “situational awareness” of grid assets from generation to consumption, and “similar day” approaches that use historical data to predict ongoing grid operations, are underway, he said. In all of these cases, NoSQL databases are being applied as integral building blocks to larger IT frameworks.
Opening the Smart Grid NoSQL World to Third-Party Developers
In the meantime, Versant has been making a big push to open up the still relatively little-known world of NoSQL programming languages to a wider audience, via its June launch of Versant JPA for the Cloud. That’s a Java client for Versant’s Object Database (VOD) engine, certified for use on the Amazon’s Elastic Compute Cloud platform -- in other words, a tool for programmers to write open-source code to run Versant’s technology across a wide array of applications.
While utilities are still just an emerging market for Versant, Taube said that the company is planning to build a fully integrated solution development platform to better integrate NoSQL technology into a variety of industry verticals, including energy.
“I think in today’s world, you better support open-source approaches, because the utilities are really not interested in continuing a life where they have to face vendor lock-in,” he said. We’ve certainly seen a growing push from smart grid vendors to offer combined hardware-software solutions that are simpler to install and maintain -- sometimes via hosting the heavy IT lifting in the cloud -- to open the market to the masses of smaller utilities that can’t afford to hire the likes of IBM or Accenture as their smart grid IT project manager.
In the long run, Taube sees Versant’s role in the utility industry as being a “silo-buster,” helping break down the walls that are keeping today’s utility IT systems isolated from one another. As he put it, “If we don’t bring IT and OT together in the utility world, we haven’t accomplished our mission.”
How are the database giants reacting to the NoSQL challenge? Oracle has built its own NoSQL platform, but it’s tied to Oracle’s traditional RDBMS, making it less flexible than open-source or independently developed software. Even so, Oracle has a lot of market share in utility customer service and billing software, and is making big-data claims for its smart meter and distribution grid management platforms as well.
Likewise, IBM, a huge and central player in smart grid IT, has incorporated NoSQL features into its DB2 and Informix platforms, though it hasn’t specified smart grid applications for the technology yet. As the largest master integrator of smart grid projects in the world, IBM certainly has plenty of opportunity to find niches where object-oriented architectures fit in.
It’s important to note that object-oriented database architecture is not a magic cure for all the world’s database needs -- it has its flaws and quirks, just as relational database architectures do. But as a tool for managing millions of smart grid objects at the speed of electricity, its advantages are clear. We will be tracking the growth of NoSQL in the smart grid, and welcome your comments on how the technology is being applied in the industry. Stay tuned for more.