HELICS for Integrated Transmission, Distribution, Communication, & Control (TDC+C) Modeling SETO Workshop on Challenges for Distribution May 17, 2019 Washington, DC

Presented by: Bryan Palmintier

Henry Huang, Liang Min, Jason Fuller, Philip Top, Dheepak Krishnamurthy, Shri Abhyankar, Manish Mohanpurkar, Kalyan Perumalla, David Schoenwald

NREL/PR-5D00-73977

Grid modernization requires integrating multiple

infrastructures…

Transmission Distribution

Control & Communication

Transportation,Water,

Gas, etc.

And we have many, well trusted tools to model each…

PSLF PSS®E

Transmission

WindMil CYME

Synergi

Distribution

Control & Communication

SCEPTRE MATLAB

Transportation,Water,

Gas, etc.

SAInt

However they are largely used within their own silos of excellence.

Transmission

PSLF PSS®E

Control & Communication

SCEPTRE MATLAB

Distribution

Transportation,Water,

Gas, etc.

WindMil CYME

Synergi

SAInt

HELICS enables easily bringing together two or more existing tools, exchanging data as time advances, to form a tightly integrated co-simulation.

Transportation,Water,

Gas, etc.

SAInt

PSLF PSS®E

Transmission

WindMil CYME

Synergi

Distribution

Control & Communication

SCEPTRE MATLAB

MATPOWER Transmission/Bulk:

AC Powerflow, Volt/VAr

FESTIV Runtime plug-in

Bus Aggregator Bus Aggregator

Bus Aggregator

bus.py bus.py ...

ZMQ

MPI

IGMS-Interconnect

bus.py bus.py

Hom

e &

Appli

ance

Phy

sics

Hom

e &

Appli

ance

Phy

sics

Hom

e &

Appli

ance

Phy

sics

Alte

rnat

e Di

stribu

tion

Mod

elTim

eser

ies, e

tc. HT

TP

HTTP

HTTP

Scen

ario

Aut

omat

ion

Tran

smiss

ion

Distrib

ution

HELICS™: Hierarchical Engine for Large-scale Infrastructure Co-Simulation Project funding: GMLC 1.4.15

Capabilities: • Scalable: 2-100,000+ Federates • Cross-platform: HPC (Linux), Cloud,

Workstations, Laptops (Windows/OSX) • Modular: mix and match tools • Minimally invasive: easy to use

lab/commercial/open tools • Open Source: BSD-style. • Many Simulation Types:

• Discrete Event • QSTS • Dynamics

• Co-iteration enabled: “tight coupling” • APIs: C++,. C, Python, Java, Matlab,

Julia, FMI

Best of Existing Cosim

FESTIV: ISO Markets, UC & AGC

GRID

Lab-

DDi

stribu

tion

Powe

rflow

,

GRID

Lab-

DDi

stribu

tion

Powe

rflow

,

GRID

Lab-

DDi

stribu

tion

Powe

rflow

,

...

ISO

Build

ingAp

plian

ce

FNCS/ GridLAB-D

FSKit/ GridDyn

IGMS/ FESTIV

Use Case Requirements

New Platform Design

End-UseCont

rol

Markets

Communication

DistributionTransmission

“TDC Tool”

B. Palmintier, et al., “Design of the HELICS High-Performance Transmission-Distribution-Communication-Market Co-Simulation Framework,” Workshop on Modeling and Simulation of Cyber-Physical Energy Systems, Pittsburgh, PA, 2017.

v2.0.0 available now at https://www.github.com/

GMLC-TDC/HELICS-src Co-simulation Engine

Grid

DDER

Trans port ICT Water

Scalable, High-performance co-simulation to combine best-in-class tools for breakthrough grid modernization simulation and analysis

6

Computational Integration Workflows

Slide adapted from Dr. Wes Jones, NREL

Co-simulation “consists of the theory and techniques to enable global simulation of a coupled system via the composition of simulators. Each simulator is a black box mock-up of a constituent system, developed and provided by the team that is responsible for that system.”[1]

[1] Gomes, Cláudio et al. “Co-simulation: State of the art.” CoRR abs/1702.00686 (2017): n. pag. Figures: Gomes, Cláudio et al. “Co-Simulation: A Survey.” ACM Comput. Surv. 51 (2018): 49:1-49:33.

Co-modelling is “w[h]ere models are described in a unified language, and then simulated.”[1]

Computational Integration Workflows

[1] Gomes, Cláudio et al. “Co-simulation: State of the art.” CoRR abs/1702.00686 (2017): n. pag. Figures: Gomes, Cláudio et al. “Co-Simulation: A Survey.” ACM Comput. Surv. 51 (2018): 49:1-49:33.

Slide adapted from Dr. Wes Jones, NREL

“Cyber”-Physical Simulation e.g. Transmission-Distribution-Market

▶ Physical Data (Values) v Voltage, Frequency, Current

▶ Market Data (Messages) v Measured Load, LMPs

M Market Dispatch

Actual Generation

RT Prices (LMP)

Measured Load

T Transmission

DGridLAB-D

Load per phase

Voltage

DGridLAB-D

DGridLAB-D

DGridLAB-D

D Distribution

Large-scale DER-Market Interactions

NREL’s Integrated Grid Modeling System

GR

IDLa

b-D

Dis

tribu

tion

Pow

erflo

w,

Hom

e &

Appl

ianc

e Ph

ysic

s

Alte

rnat

eD

istri

butio

n M

odel

Tim

eser

ies,

etc

. HTT

P

(IGMS) provides a full-scale co-simulation with transmission-level markets, 1000s of distribution feeders, and 1Ms of DERs

FESTIV: ISO Markets, UC & AGC

MATPOWER Transmission/Bulk:

AC Powerflow, Volt/VAr

FESTIV Runtime plug-in

Bus AggregatorBus Aggregator

Bus Aggregator

bus.py bus.py ...

ZMQ

MPI

IGMS-Interconnect

bus.pybus.py

GR

IDLa

b-D

Dis

tribu

tion

Pow

erflo

w,

Hom

e &

Appl

ianc

e Ph

ysic

s

GR

IDLa

b-D

Dis

tribu

tion

Pow

erflo

w,

Hom

e &

Appl

ianc

e Ph

ysic

s

...

HTT

P

HTT

P

Scen

ario

Aut

omat

ion

ISO

Tran

smission

Distribu

tion

Building

App

lianc

eNew for IGMS v3: HELICS

co-simulation

9

Adding Controllers… e.g. Control Architecture Scaling & Performance

▶ Physical Data (Values) v Voltage, Frequency, Current

▶ Market Data (Messages) v Measured Load, LMPs

▶ Controller Data (Messages) v Sensor Readings, Control Signals

M Market Dispatch

Actual Generation

RT Prices (LMP)

Measured Load

T Transmission

DGridLAB-D

Load per phase

Voltage

DGridLAB-D

DGridLAB-D

DGridLAB-D

D Distribution

C Controller

Sensor Data

Sensor Data

Control Signals

Control Signals

Novel T&D Control Architecture Design: Predictive State Estimation & Machine Learning Control Grid Sim: Entire Island of Oahu, HI with >1M electric nodes.

10

Keeping the wires uncrossed

Inv 1&2 Substation

Legend Voltage Regulator

Capacitor Bank

PV Inverter RecloserController

Actual Deployment

Inv 1&2 Substation

Legend Voltage Regulator

Capacitor Bank

PV Inverter Inv 1&2

Substation Recloser

Legend Voltage Regulator

Capacitor Bank

PV Inverter RecloserController

Software Simulation

Keeping the wires uncrossed

Actual Deployment

Inv 1&2 Substation

Legend Voltage Regulator

Capacitor Bank

PV Inverter Recloser

Software Simulation

v Delays v Random drops v Other message effects (e.g.

packetization) v And more

▶ No changes to domain models

M Market Dispatch

Actual Generation

RT Prices (LMP)

Measured Load

T Transmission

DGridLAB-D

Load per phase

Voltage

DGridLAB-D

DGridLAB-D

DGridLAB-D

D Distribution

C Controller

Sensor Data

Sensor Data

Control Signals

Control Signals

CommFilter

CommFilter

Comm Filter

Comm Filter

… and Simple Communication e.g. Design-stage Cybersecurity Evaluation

▶ Built in “Filters” for

1313

Novel T&D Control Architecture Design: Predictive State Estimation & Machine Learning Control Grid Sim: Entire Island of Oahu, HI with >1M electric nodes.

1. Control signal spoofing

2. Control node compromise 3. Sensor data spoofing

4. Communication Denial of Service

X

5. Communication Latency Margin

Or Detailed Communication e.g. Protocol Comparison for Situational Awareness

▶ Full communication simulation: v Shared bandwidth

v Network Specific Vulnerabilities

v Potential Tools: ns-3, Opnet++, SCEPTRE, etc.

▶ No changes to domain models

M Market Dispatch

Actual Generation

RT Prices (LMP)

Measured Load

T Transmission

DGridLAB-D

Load per phase

Voltage

DGridLAB-D

DGridLAB-D

DGridLAB-D

D Distribution

1414

C Controller

Sensor Data

Sensors

C Communication

reroute

reroute

Controls

Projec

Protocol/Full-Stack Performance

t Ex: SuNLaMP Hybrid Comms

Ex: Power-Comm. Emulation

Some Other Use Cases

60.05 0% PV 10% PV 20% PV Large-scale DER-Market Sim 30% PV

60 • 35k feeders

frequ

ency

(Hz) 59.95 • WECC-240 trans.

59.9 • 25M homes 59.85 • Simplified CAISO-

59.8 0 1 2 3 4 5 6 7 8 9 10 style Market

Time (sec)

T&D frequency stability with high DER

Inv 1&2 Substation

Legend Voltage Regulator

Capacitor Bank

PV Inverter Recloser

Inv 1&2 Substation

Legend Voltage Regulator

Capacitor Bank

PV Inverter Recloser

Noon 3pm 6pm 9pmADMS Testbed and other PHIL Figure from Trevor Hardy, PNNL

15

HELICS Interfaces to Domain Simulators

Not exhaustive lists.

16

▶ Growing mix of tools

▶ Enable large-scale interdependency all-hazards studies: scale to 100,000+ domain simulators

▶ Diverse simulation types: v Continuous, discrete event,

time series

v Steady-state/dynamic/transient

v Any energy system

▶ Support standards: HLA, FMI, …

▶ APIs: C++,. C, Python, Java, Matlab, Julia, FMI

PSLF Windmil

Gas-Pipeline

Energy+ (buildings)

Transportation

T D

C More

GridDyn InterPSS

MATLAB (PST, MATPOWER)

FESTIV

GridLAB-D

MATLAB

NS3

HELICS built-in Comms Sim

OpenDSS

Existing Ongoing Waiting

Powerworld CYME

SCEPTRE

Thank You

Bryan Palmintier: bryan.palmintier@nrel.gov

HELICS v2.0.0 available at https://www.github.com/GMLC-

TDC/HELICS-src

NREL/PR-5D00-73977

Henry Huang, Liang Min, Jason Fuller, Philip Top, Dheepak Krishnamurthy, Shri Abhyankar, Manish Mohanpurkar, Kalyan Perumalla, David Schoenwald

This work was authored in part by Alliance for Sustainable Energy, LLC, the manager and operator of the National Renewable Energy Laboratory for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided by U.S. Department of Energy Offices of Electricity and of Energy Efficiency and Renewable Energy Solar Technologies Office through the Grid Modernization Initiative. The views expressed in the article do not necessarily represent the views of the DOE or the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, 17 worldwide license to publish or reproduce the published form of this work, or allow others to do so, for U.S. Government purposes.