Yokogawa & NextNine – Lessons Learned: Global Cybersecurity Management System at a Major Oil & Gas Company

Copyright © Yokogawa Electric CorporationCopyright © Yokogawa Corporation of AmericaCopyright © NextNine Inc. All rights reserved.

2015

Jeff Melrose CISSP-ISSEP YokogawaMichael Coden CISSP NextNine

Lessons Learned:First Year of Deployment and Operation of a Global Cybersecurity Management System at a Major Oil and Gas Company


- 2 -

• Overview of Global Cybersecurity System• 60 sites worldwide

• What went right• What went wrong• What processes needed to change• What technology/process changes needed to be

made• What new technology was developed

Agenda


- 3 -

Overview of Global Cybersecurity System


- 4 -Copyright © Yokogawa Electric CorporationCopyright © Yokogawa Corporation of AmericaCopyright © NextNine Inc. All rights reserved.

- 4 -

ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443ISA / IEC-62443

Automated and Direct Asset Discovery and Inventory

Automated Patch-AV DeliveryAnd Compliance/Enforcement Reports

Event and Incident Log Collection, Conditioning, and Transfer to Centralized SIEM for Analysis

Secure Remote Access and Secure Remote Device-to-Device Connection

Multi-Site File Transfer InfrastructureFor Multi-Site Backup / Restore

Centralizing, Connecting, & Automating Cybersecurity Processes Makes the “Cybersecurity Culture” Scalable



- 5 -

Yokogawa/NextNine Secured Remote Solution: System Overview

Central System

IC Systems IC SystemsIC Systems IC Systems

RemoteSite System

VSE

RemoteSite System

VSE

RemoteSite System

VSE

RemoteSite System

VSE

OS Patch Dist. ServerVSE VM BackupAnti-Virus

Dist. Server

Central SystemBusiness LAN (L4)

Business LAN (L4)

Remote Site System (L3.5)

PCN(L3)

Service Center

Anti-Virus Replication

Server

OS Patch Replication

Server

Auth. Server

Remote Operation

Solution Dashboard

(IP-VPN/Internet VPN/User Corporate WAN)

YokogawaIA System

OT SystemSupplier A

OT SystemSupplier B

Verifiedpatches

SecureTunnel


- 6 -

What Went Right


- 7 -

Customer Dashboard

• Security Center Dashboard was Created– Security Policy enforcement – Policy tracking green/red – Compliance Reporting– Patch status tracked



- 8 -

Auto-Asset Discovery & Inventory

DMZ

Central Security Center

Application Server

CommServer

Real-TimeDatabase

Server

Network& Security

Devices

Virtual Security Engine™

Devices, Systems, Applications

Remote Site/s

Internet

External UsersPartner / SI / OEM

Full Web UI

Full Web UI

Internal Users

WMISNMPOPCSSHHTTPTelnet (CLI)SFTPFTPProprietaryOthers

Solution supports all versions of: Windows (NT, XP, Vista, Win7, 2000,

2003, 2008, 2012) Unix (HP-UX, AIX, Solaris, ….) Linux (Red Hat, Ubuntu, ….) Any other product that can be accessed

via the protocols at the left.



- 9 -

Event & Sys-logs Converted to CEF & Transferred to Central SIEMVSE Continuously Scans Ports & Services vs. White/Black List

DevicesSystems

Applications

NetworkDevices

Virtual Security

Engineer™

LocalPeronnel

Network& Security

Devices

Virtual Security

Engineer™

LocalPersonnel


Internet


Field Service

Full Web UI

Cyber Security SIEM and Analysis Tools, e.g.: ArcSight, Q-Radar,

Nitro, ….

Detecting Rogue Devices, Ports, Services

Full Web UI

Internal Users

DMZSite


Application Server

CommServer

Real-TimeDatabase

Server

Nigeria

Qatar

VSE continuously collects logs, converts them to CEF (Common

Event Format) sends logs for analysis and detection of

malicious activities.

VSE continuously

scans Ports and Services –comparing

against Whitelist &

Blacklist.

Full Web UI

Cybersecurity Experts


- 10 -

Customer Dashboard• Data for Security Center Dashboard is Collected

Devices

Business Criticality Are Patches

up to date?Is Antivirus up to date?

Are Logs being sent

to the SIEM

Is Removable Media being

used?

Do Ports & Services

match the Black/

Whitelist?


- 11 -

Cybersecurity Management System - Governance

• Process can now be implemented for cybersecurity governance.– Every plant/facility can now be tracked on an

“as-like” basis– No more exceptions due to distance or region– One stop shop for a view of the organizations’

Cyber defensive profile



- 12 -

Assured encrypted access across IT networks world wide independent of media (satcom)

Virtual Security Engines:-All remote connectivity is through a single portoutbound only connection to specific IP address

-FIPS 140-2 Compliant & 1024-bit TLS Encrypted.

Remote Site A

Remote Site B

Remote Site C

Secure CenterCertificate

Something I have

CertificateSomething I have



Trusted Platform Module



– Data is compressed, encapsulated, encrypted– No possibility of VPN bleed or fake connections – A secure multipurpose tunnel to customer sites

Only 1 Firewall Rule to Manage for All Remote Connections



- 13 -

Bi-Directional File Transfer – to/from Anywhere: Off-site Backup/Restore, Production Optimization, Secure File Delivery

DevicesSystems

Applications

NetworkDevices

Virtual Security

Engineer™

LocalPeronnel

Network& Security

Devices

Virtual Security

Engineer™

LocalPersonnel


Internet


Field Service

Full Web UI

Backup Location# 2 With

Auto-Verify of Backups

Backup Location# 1 With

Auto-Verify of Backups

Full Web UI

Internal Users

DMZ

Houston


Application Server

CommServer

Real-TimeDatabase

Server

Nigeria

CaliforniaAmsterdam

Qatar


- 14 -

Audit Trail

• Audit Trail – Insider threat mitigation


- 15 -

Help desks established

• Established both Level 2 and Level 3 Helpdesks established in Europe– 24/7 coverage– Full visibility into plants supported

• Personnel• Lead Contacts• IT / OT local support • Escalation contacts• Vendor lead contacts for each plant


- 16 -

Industrial Controls – more like IT

• Fully Documented system in terms that IT and Cybersecurity personnel understand

• Plant’s connection to unified TCP/IP network went well.

• Initial Deployment process went well with IT related timelines met

• IT hardware delivered on time and in good condition (IT component procurement works!)

• Signoffs for Acceptance Testing occurred on time with minimal issues

• Initial Training on Cybersecurity Management was completed on time and budget



- 17 -

Secure ICS patch managementCentralized vetting of all patches with direct links to suppliers

WSUSePO

SEPM

WSUSePO

SEPM

DevicesSystems

Applications

NetworkDevices

Virtual Security

Engine™

Network& Security

Devices

Virtual Security

Engineer™


Remote Sites

Internet


Field Service

Full Web UI

Your Product PatchServer

Full Web UI

Internal Users

DMZ

CentralSecurity Center

Application Server

CommServer

Real-TimeDatabase

Server

Windows WSUS Server

McAfee ePO

Server

SymantecSEPMServer



- 18 -

Secure ICS patch management• Able to show delivery of patches to every

plant and track to completion of patching effort

Devices

Business Criticality Are Patches

up to date?Is Antivirus up to date?


- 19 -

What went wrong


- 20 -

IT integration is hard

• Review cycles on detailed site sign off were increased due to more people reviewing (IT and ICS)

• Delivery synchronization problems between: hardware, Virtual Hypervisor, OS’s, other software modules

• Installation and configuration of software longer than planned.


- 21 -

Regional Issues for Integration

• World wide integration is hard – Getting personnel – Legal to work personnel for that region– Site access (clearance issues)– Safety certification for personnel at plant– Extended encryption configuration for remote

sites


- 22 -

ICS personnel NOT familiar with IT integration cycles

• IT integrations cycles are quick reaction– ICS personnel can’t be called on like a

telephone repair man– Advanced planning needed to get person

familiar with install to return to plant– ICS Integration follows more of an Engineering

Process with Configuration Control.


- 23 -

IT components packaging

• IT components at HW level usually had all components needed

• IT SW however sometimes lacked complete deployment setup

• Training on the IT related components was lacking for certain configuration issues

• Handover to support could be more seamless without being a manual process (probably get better as more sites are set up)


- 24 -

Plant build out and provisioning

• Some Plant build out was delayed due to getting proper space to place components

• Provisioning at the network cloud to local plant was easy

• Last mile inside the plant provisioning was more complex (laying infrastructure inside an active plant is time consuming, and only local people can provide guidance on how long it may take)


- 25 -

What processes needed to change


- 26 -

IT vs OT/ICS

• Cats and Dogs need to declare peace ICS to IT joint meetings– More advanced warning of deployment plans

to plant personnel– More information to Plant personnel to smooth

integration– We plan on more briefings to Plants

– if possible


- 27 -

ICS/Operational Technology (OT) Controls Last Mile

• ICS and Plant Managers normally leads for Plant last mile

• It is important to have an engineering solution approach to IT within ICS domains

• Configuration Control, Review Process, Safety Checks, Pre-Briefs are all processes that need to be followed.


- 28 -

ICS / OT Runs on Maintenance Cycles

• Maintenance rules at the Plant• ICS / OT, IT and Cybersecurity personnel

must be understanding on these cycles• Times and locations convenient to IT, ICS and

Cyber may be completely bad for Plant operation

• ICS / OT, IT and Cyber personnel need to be the more flexible party

• Oh and when Plant maintenance says “you’re done” … you ARE Done for the day!


- 29 -

What technology/process changes needed to be made


- 30 -

Help Desk/Service Desk to Plant Communication and Integration

• Who to talk to when at what part of the Plant• Who tracked at coordination at Plant level• Better and more reliable IP based integrated

communication infrastructure to all Plants

– This integration drove• Larger bandwidth WAN to Plants• Class of Service management of the WAN• Partnering with international Telecom for WAN

infrastructure and provisioning


- 31 -

Yokogawa needed to invest in Security Training Coursefor Employee Engineers

As of April, 2014,about 700 certified Yokogawa engineers.

Yokogawa has supportedGICSP program since its first day.

- 31-


- 32 -

Yokogawa had to marshal and International Service Organization

Response Center

Service Office Network

A worldwide network of Yokogawa Response Centers, service offices,and service engineers provides a prompt response to all kinds of customer inquiries on an around-the-clock, 365-day-per-year basis.

Call center services byspecialistsRemote monitoring and patrol inspectionSupply of information on hardware/software revisions

Customers

On-site maintenance

Dispatch of engineers

Supply of spare parts

and components

Remote maintenance

Data collection and analysis

EscalationTechnical support from responsible

department

Korea

32

Singapore China BrazilIndiaBahrain USATheNetherlands TaiwanRussia


- 33 -

What new technology was developed


- 34 -

Remote Access Device Granularity

• Remote Access Users can be given restricted access by:– Site– Device(s)– Functions

• View• Edit• Delete


- 35 -

Password Vault

• Three opposing problems:– Many systems using default passwords or same

passwords– Ease of login required for safety operation– Third parties had passwords outside plant

• Solution = Password Vault in VSE– VSE contains credentials for systems with

different privileges– VSE uses correct password for each device

depending on Remote User’s privileges



- 36 -

Secure Remote Access – Third Parties Can Only Access Specific Devices at Specific Sites with Site Control

“Virtual Security Engineers:”– With Remote Access, Cyber Security and 3rd Party

experts can immediately connect to only specific devices at specific sites determined by your security policies

– Remote Site controls granting of access

– Remote Site can Supervise remote access

Remote Site A

Remote Site B

Remote Site C

Secure Center

End-customer approves remote access

VSE Interface



- 37 -

Secure Remote Access – Direct to Device“Virtual Security Engineers:”

– VSE Connects Experts Computer Directly to Target System– High Speed Real Time Desktop Sharing

-- Device to Device connection for any application

Remote Site A

Remote Site B

Remote Site C

Secure Center

– Sessions are video recorded at both Remote and Central Sites


- 38 -

Improved Asset Inventory Management• Device Properties Entry (require someone to input

info about devices, custodian, criticality etc)• Collected via NextNine VSE:

– Ipv4 Addresses– MAC Addresses– OS name and version– Application software name and

version– OS patches name and date– HW manufacturer and model– AV agent name and version– AV signatures file version and

date– AV service status– WSUS properties– Device Attributes

• Entered Into NextNine VSE:– Custodian– Criticality (C, E, N)– Type (Monitoring System,

Safety System, Workstation, Server, Firewall, Router, …)

– Vendor– Vendor Software– Function (Metering,

Engineering Station, DCS, PLC, …)

– Life-cycle (Active, Inactive …)– Deviation (free text)

Additional items may be added upon request.



- 39 -

Rapid Deployment of Exploit Scanners

Heartbleedscanner was

delivered in 48 hours!

DMZ


Application Server

CommServer

Real-TimeDatabase

Server

Network& Security

Devices

Virtual Security Engine™


Remote Site/s

Internet


Full Web UI

Full Web UI

Internal Users

• GUI based App Development Environment• Develop new Apps in a few hours• Distribute Apps to all VSE’s • No recompile or reboot of VSE is

required• App is used immediately

ShellShockscanner was

delivered in one week!


- 40 -

Ultimately the deployment yielded better central visibility into security policy enforcement across all plants

<Document Number>Copyright © Yokogawa Electric Corporation<date/time>


Thank You

May 2015

Jeff Melrose CISSP-ISSEP YokogawaMichael Coden NextNine