7/30/2019 06070155

1/6

Timing and Synchronization for Quasi-Real-TimeSystems Using IEEE 1588v2 Over Ethernet

Marc Cohn

Micrel, Incorporatedwww.micrel.com

LAN Solutions Business UnitSan Jose, California, USAmarc.cohn@micrel.com

Abstract Ethernet and IEEE 1588 are continuing to emerge in a

wide range of industries. Increasingly, they are replacing

industry-specific interconnects, delivering performance and cost

efficiencies. Initiatives including Industrial Ethernet, IEC 61850,

and LXI are all adopting commercial off- the-shelf technologies

to achieve broader communications at far lower costs. However,

significant investments that are warranted for large automation

systems become prohibitively expensive for a diverse set of less,ambitious Quasi-Real-Time (QRT) applications that also require

distributed communications and timing.

This paper proposes an approach to address the communications

and timing/synchronization requirements for QRT systems,

characterized by simple and low-cost devices interconnected and

synchronized over a real-time network. To achieve the aggressive

cost targets, a highly integrated attachment device is introduced

that integrates Ethernet communications, IEEE 1588v2

distributed synchronization, and precision I/O for local

synchronization in a single, energy efficient device. Systems issues

are also raised affecting the applicability for applications that can

exploit a QRT network.

Keywords: IEEE 1588v2, Real-Time Ethernet, Precision I/O,Synchronized I/O, Industrial Ethernet, Quasi-Real-Time

I. INTRODUCTIONEthernet, the predominant Local Area Network standard, is

emerging in a number of non-traditional applications,leveraging the tremendous momentum of an installed base inexcess of one billion ports. Economic realities have motivatedthe adoption of Ethernet in such diverse environments rangingfrom the factory floor, in-vehicle, to electrical powersubstations. While all very different, such applications sharean unmistakable and pervasive trend: point-to-pointinterconnects and industry-standard buses are being replaced

by robust, Ethernet-based networks.Increasing intelligence, counter-balanced by the ever-

present need for cost reduction, has necessitated integration ofcommunications and timing/synchronization, germane toIndustrial, Automotive, and Power Systems automation.Market and technology forces influenced the IEEE to upgrade arelatively obscure standard into prominence. Thus, IEEE1588-2008 [1] is gaining market momentum as the basis fornext-generation real-time automation systems and, in theprocess, is transforming the target industries.

Industrial Ethernet standards such as Ethernet/IP [2],Profinet [3], and PowerLink [4] seeking to capitalize upon thebenefits of Commercial Off-The-Shelf (COTS) technologies,are aggressively deploying Ethernet integrated with IEEE1588. Similarly, the power systems automation industry hasspecified an important standard to guide Power SubstationAutomation. IEC 61850 [5] breaks new ground in adoptingEthernet, IEEE 1588, and responsive fault tolerance schemesfor a comprehensive communications framework forSubstation Automation Systems (SAS).

Industrial and Power Systems Automation market leadersare aggressively promoting Industrial Ethernet standards, aswell as IEC 61850. However, significant up-front investmentand comprehensive software stacks render the standards tooexpensive for many applications. Ultimately, there are manylower-end applications that would benefit from reliablecommunications and tight synchronization, but without the richset of services and objects defined by the standard frameworks.

II. QUASI-REAL-TIME (QRT)NETWORKSTo distinguish from the large, hierarchical automation

networks, we introduce QRT systems and networks, whichcorrespond to a broad set of relatively mid- to low-endapplications built upon an integrated services network. Bymerging timing and communications into a unified, low-costnetwork, applications processing may be distributed,communications and synchronization jitter reduced, and greaterprecision may be achieved [6]. QRT systems are characterizedby:

Standardized, high-performance network architectureproviding communications and synchronization

Relatively low-end devices (~$100s - $1,000s):Sensors, Actuators, Motor Drives, etc.

Embedded CPUs, running the host application Relatively small number of nodes per sub-network

(typically less than 30)

Message latencies of less than 1 ms (short,

7/30/2019 06070155

2/6

As with most successful technologies, availability of low-cost, standardized building blocks spawn a diverse and broadset of applications. The convergence of Ethernet and IEEE1588 has paved the way for precision timing to catalyzedevelopment of QRT systems; by capitalizing upon the benefitsof COTS, proven communications and synchronizationtechnologies, and of course, reduction of life-cycle costs.

The key to enabling such applications is availability ofhighly integrated silicon that simplifies QRT node designs, andadapt to the widely varying interface requirements, including:

Comprehensive support for predominant standardsincluding IEEE 802.3 (Ethernet) and IEEE1588/PTPv2 distributed timing and synchronization

Multi-megabit communications (typically 100 Mbps) Standardized copper and fiber media options to

accommodate varying cable reaches

Flexible topologies:o

Centralized, star-wired topologies to exploitthe benefits of structured wiring

o Distributed, daisy-chained topologies (i.e.,rings and linear buses) to facilitatedynamically changing network configurations

o Hybrid topologies, exploiting the benefits ofboth Centralized and Distributed topologies

Optional, network fault tolerance mechanisms tomaintain communications availability

Support for distributed synchronization, which may beextended to locally connected devices

Hardware assist for communications andsynchronization to minimize the processing demandson the embedded processors

Rich set of I/O capabilities to accommodate a range ofdevices with varying operating characteristics

There are a number of complementary technologies that arealso needed to enable QRT systems/network design:

Real-Time, Systems Timing References, referred to inIEEE 1588 as Grand Master Clocks; highly accuratetiming standards for synchronizing distributed nodes

Robust Ring Recovery protocols capablereconfiguring around common network faults,

typically < 10 milliseconds

Streamlined Real-Time Operating Systems (RTOS),to ensure efficient communications, synchronization,and applications processing

Network management to enable the availability,performance, and configuration of the network to bemonitored for real-time and long-term analysis

Systems performance is driven by a complex combinationof these factors, making it difficult to apply generic rules ofthumb for QRT network configuration. Communicationslatency, synchronization jitter, ring recovery delays, etc. are allinter-related, c the need to meticulously analyze theapplications (with respect to the network), in order to achievethe desired systems behavior.

For instance, decreasing ring recovery times may not yield

appreciable benefits if the synchronization acquisition time isorders of magnitude higher. Increasing network bandwidth willnot necessarily address overcome the issue, nor will increasingthe processing speed necessarily improve network availability.

The challenge is architecting a QRT network architecturesufficiently broad to encompass a range of applications at avery low cost. Systems designers may then exploit standardizedbuilding blocks that attain the best of both worlds- the high-performance only achievable by silicon, at a cost-effectiveprice.

One prospective application for QRT networks is theemerging IEC 61850 standard for Power SubstationAutomation. IEC 61850 defines a hierarchical communications

architecture, based on Ethernet and IEEE 15888 technology.The lowest layer is referred to as the Process Bus, whichinterconnects Intelligent Electronic Devices (IED, as definedin the standard) that provides real-time communications.

Relays, switches, and other electrical power gear arecontrolled by the Process Bus, which is considered mission-critical by substation operators. The standard recognized theneed for a multi-service, scalable, robust, and high-performance network to achieve the explicit goal of reducingoverall operations costs- a key benefit of the QRT networks.

III. COST REALITIES-INTEGRATED ETHERNET/IEEE1588V2NETWORKS

Migrating from relatively simple serial interconnects andbuses to an integrated Ethernet/IEEE 1588v2 network in a cost-effective manner can be challenging. Considering the costdifference between an RS-485 interface and Ethernet port, itremains cost-prohibitive to directly connect low-end devices toEthernet networks.

Typically, the Ethernet MAC and PHY functions areimplemented in silicon in accordance with the IEEE 802.3standard [7]. Ethernet LAN are commercially available frommultiple vendors.

IEEE 1588v2 implementations are far less common, andare typically implemented in FPGAs, Physical Layer (PHY)

transceivers, or Micro-Controller Units (MCUs). IntegratingEthernet switching along with IEEE 1588v2 (and theassociated logic) is not a trivial exercise, and requiresadditional communications and synchronization processing (inthe host CPU).

Ethernet/IEEE 1588 attachments will multiplex multipledevices (through I/O pins) in order to achieve cost parity withpoint-to-point systems (see Figure 1). Migrating to a networkoffers greater flexibility and a compelling range of benefits.

7/30/2019 06070155

3/6

Figure 1. Per-Device Cost ComparisonSerial Interconnect vs. Serial Bus vs. Ethernet Network

Networks enable a rich set of communications servicessuch as multicast, Virtual LANs, and Quality of Service thatoffer unprecedented applications flexibility. Integrating IEEE1588 realizes sub-s synchronization of nodes distributedthroughout the network cost-effective and reliable. Far greaterbandwidths, often enumerated in orders of magnitude, arereadily achievable with the well-established 10/100 Ethernet

standards. In addition, cabling can be simplified as well, toaccommodate frequently changing operational needs.

The costs for an Ethernet/1588v2 attachment (assuming adistributed topology), include:

IEEE 1588v2 Facilitieso IEEE 1588v2 Precision Clocko 2 Time Stamp Units (TSUs) for PTP packetso PTP hardware support

3-port 10/100 Ethernet switcho 2 integrated Media Access Control (MAC)o 2 10/100BaseTX PHY Transceivers

Precision I/O (synchronized to the Precision Clock) PTP Software (typically running in the Host CPU)Cost challenges persist. Presently, an Ethernet/1588v2

attachment implemented in an FPGA, with a pair of externalEthernet PHY Transceivers, with an estimated cost measured in$20-30 USD or more. Assuming four to eight devices share asingle Ethernet/1588 attachment, the per-device costs are onthe order of $3 $5; multiples of the sub $1 USD cost for anRS-485 interface transceiver.

IV.

INTEGRATION OPPORTUNITIES In order to reduce the Ethernet/IEEE 1588 attachment costs

so as to be on par with serial interconnects, further integrationis necessary. This is primarily achievable through an ASIC. Asa result, Micrel Semiconductor recently introduced theKSZ84xx family of IEEE 1588v2-enabled Ethernet 3-port10/100 switches [8], for QRT applications. This Industrial-grade platform represents a single chip, highly integrated1588/Ethernet attachment, offering several tangible benefits:

Figure 2. KSZ84xx IEEE 3-port 10/100 / IEEE 1588v2 SwitchFunctional Block Diagram

Wire-speed, full-featured 3-port 10/100 Mbps switch Dual IEEE 1588v2 time stamp units, Precision Clock,

and distributed synchronization facility

o Synchronization performance is improved asTSUs reside between the MAC and PHY

o Grand Master, Master, Slave, and TransparentClock modes are supported

Dual, low-power 10BaseT / 100BaseTX PHYso Lowest power 100BaseTX PHY Transceiver

(< 150 mW per port)

o IEEE 802.3FX optical transceiver support Hardware support for communications and precision

clock synchronization, which reduces the overallprocessing load on the host CPU

Integrated I/O that can be synchronized to the overallsystem synchronization hierarchy

Advanced power management including IEEE 802.3azEnergy Efficient Ethernet (EEE) [9]

Compact size through a single-chip design(64-pin package, 10 mm x 10 mm)

Figure 2 depicts the high-level block diagram for the

KSZ84xx switch, illustrating the functional blocks. TheKSZ84xx is available with standard (MII or RMII) and generichost bus interfaces to support CPUs with and withoutembedded Ethernet MACs.

The Precision GPIO facility enables multiple devices toshare a single Ethernet/IEEE 1588 attachment. Precision GPIOis highly flexible and configurable to support a diverse set ofdevices with wide ranging operational and performancecharacteristics.

7/30/2019 06070155

4/6

Figure 3. IEEE 1588v2 Systems Timing Hierarchy

V. PRECISION I/O AND LOCAL SYNCHRONIZATION Devices connected to the QRT network through the

Precision GPIO facility must be synchronized to the node, andhence system reference timing. Figure 3 illustrates the systemtiming hierarchy implied by IEEE 1588v2, where:

A global, real-time reference (i.e., GPS) synchronizes aGrand Master Clock (GMC), the system reference

Multiple Master Clocks (MCs), each corresponding toa distinct synchronization domain, are synchronized tothe GMC

Each MC synchronizes the set of Slave Clocks (SCs)comprising a particular synchronization domain. ForIndustrial Ethernet, SCs are typically daisy-chainedtogether into a distributed topology.

Locally connected nodes can source or sink data and/orcontrol information through a Precision I/O interfacethat is synchronized with the local precision clock

The distinction is drawn between timing, where real-time isdistributed, and synchronization, which relates to a frequencyreference distribution across multiple, disparate equipment.

Synchronization may be provided through standardizedtiming interfaces (see Figure 4):

1 PPS reference for time distribution 10 MHz reference for synchronized clock distributionBecause timing and synchronization is application-specific,

versatility is required to tailor the timing interfaces to the needsfor specific designs. Additional reference signals (such as 100PPS timing output, 1 MHz and 5 MHz, etc.) may be necessary.Also, signal characteristics may also vary (e.g., pulse width,

voltage level, etc.), necessitating configurability, withoutcompromising synchronization in accordance with the systemstiming hierarchy.

Real-time distribution may be achieved through suchstandards as IRIG-B, published by the Inter-RangeInstrumentation Group. The IRIG standard [10], definesinternationally recognized Time Code Formats and sentencesthat reduce the need for real-time references. Real-timedistribution is being considered for future versions of theKSZ84xx platform.

Figure 4. QRT Node Timing Interfaces

A select set of applications may require synchronizationperformance far better than specified in the IEEE 1588-2008standard. Section 1 of the standard specfies 1 s Jitter(variability) value. However, the IEEE Precise NetworkedClock Synchronization Working Group recognized the need forsub-s performance for a range of specialized applications,

which is explicitly called out in the standard.In the Industrial Automation arena, defacto standards such

as EtherCat [11], and ProfinetIRT [12] have emerged to fill theneed for the most stringent real-time applications. At 100Mbps, these technology enable real-time system cycle timesbelow 500 s; at 1 Gbps < 500 s is feasible [13].Synchronization accuracies are measured in 10s of ns.

EtherCat and ProfinetIRT use proprietary approaches thatpreclude the use of standard Ethernet MACs (even though bothadopted the Ethenet 10/100/1000 Mbps PHY layer).

For higher-end applications, IEEE 1588v2 over Ethernetimplementations may be enhanced by proactively addressingthe error sources that deteriorate synchronization performance.Higher accuracy oscillators, enhanced power filtering, etc. arelikely to result in synchronization performance well-below100ns.

While the synchronization performance may be on par withthe real-time protocols, overall systems performance forEtherCat and ProfiNet IRT will be more suitable for precisionmotion, and other real-time systems which require the highestperformance.

VI. LOCAL TIMING &I/OIMPLEMENTATIONMicrels KSZ84xx Industrial Ethernet switch platform is an

ASIC specifically designed to enable QRT networks. Figure 5

depicts how multiple devices share a single Ethernet/IEEE1588 attachment through the KSZ84xx Precision I/O facility.

Precision I/O pins are synchronized with the local precisionclock, which in turn is synchronized to the system timinghierarchy. In-house testing yields synchronization jitters on theorder of 100 ns or less (see Figure 6), even under 99%+network loads. Such performance is attained throughpositioning the time stamp units, directly between the PHY andthe MAC (on-chip). As a result, measurement and time stamperrors are significantly reduced at the node-level.

7/30/2019 06070155

5/6

Figure 5. Multiple Devices Share a Common QRT Network Attachmentto Attain Cost Parity with Serial Interconnects

Most available 1588 implementations partition the MAC-to-PHY interface on two distinct devices, compromisingsynchronization performance. Multi-chip implementations alsoprove more costly, consume higher power, require additionalboard space, and are typically less reliable than an ASIC.

Another factor yielding improved synchronization

performance is the tightly coupled Precision Timing Protocol(PTP) stack, developed by OnTime Networks, Oslo, Norway.Hardware-assisted PTP operations (especially periodicTransparent Clock corrections) are not only more efficient, butmore importantly, conserve scarce Host CPU resources whichcan otherwise be dedicated to applications processing.

In order to support diverse interface needs anticipated forQRT systems, a broad set of I/O capabilities has been provided,as summarized in Table I. The I/O implementation is based ona set of logical I/O Event Units that may be assigned to any ofthe available I/O pins. By decoupling Event Units fromphysical pins, more sophisticated I/O operations are possible.

Input Event Units provide an efficient means of monitoring

external events and conditions on I/O input pins. For eachInput Event, a time stamp is generated whether for an Edge orPulse sensed on that pin. Input Event Units may be cascadedtogether to monitor compound events (i.e., sequences ofmultiple pulses, edges, and in essence bit patterns), offloadingthe Host CPU from low-level I/O manipulation.

Once detected, the time stamp corresponding to eachreceived event will be queued up for the host CPU to retrieve.Such an approach decouples host processing from the I/Oacquisition, further offloading the Host CPU. Improving hostefficiency is especially significant for QRT systems wheremultiple events may occur virtually simultaneously.

Upon retrieving each Input Events, the Host CPU must

analyze the time stamps to determine pulse widths, the timeintervals when the events occurred, etc.

For Output Events, a flexible range of I/O Output Units andassociated operations are supported. Each Output Event will beinitiated (i.e., triggered) when the local precision clock reachesa pre-configured value corresponding to the event. Thus, allOutput Events are timed in concert with the local clock, and inturn the overall systems timing hierarchy.

Figure 6. Sub-100 ns Synchronization Jitter (Master to Slave Clocks)

GPIO Output performance is governed by the Host CPUselection, host interface (MII, RMII, or Generic HostInterface), and complexity of the designated output operations.For each operation, the Host must configure the appropriateregisters (through the management interface), and set the

appropriate trigger time.

I/O

Capability

Applications Configurability Notes

Inputs

Edge

Monitor

Events

Alarms

Positive or

Negative

Polarity

May be cascaded for

more complicated bit

patterns

Pulse

Monitor

Interrupt

Control word

sensing

Timing (n x PPS)

Polarity

Pulse Width

May be used as a 1

PPS input for local

timing by an external

system

Outputs

Edge

Generator

Positive or

Negative

Polarity

May be cascaded for

more complicated bit

patterns

Pulse

Generator

Polarity

Pulse Width

May be used as a 1

PPS input for local

timing by an external

system

Waveform

Generator

t x PPS

Timing signal

Start time

Cycle time

# of Cycles

Pulse Width

Frequency

Generator

F MHz

Reference

Frequency,

Duty Cycle

1 Hz 12.5 MHz

Register

Mode

Control Word

outputs

Start time,

Repetitions

Shifts out contents of

a designated output

register

TABLE I. KSZ84xx Precision I/O Mechanisms

7/30/2019 06070155

6/6

Similar to the Input Event Units, Output Event Units maybe cascaded to enable more complex operations. For example,if a Logic Controller is required to send a 32-bit control wordto request a Sensor reading, 2 Register Mode Output Eventsmay be cascaded together. Once triggered (at the prescribedtime), both of the Register Mode Output Units will sequentiallyshift out the 16-bit contents of user-data (contained in aninternal register) onto a single I/O Output pin.

In addition to the relatively simple Edge, Pulse, andRegister modes, timing interfaces may be implemented usingthe highly configurable Output Trigger Unit, such as thecommon 1 PPS signal that distributes precision timing.

The 1 PPS Output may be configured to comply with theGPS Interface Control Document (ICD) for the Precision Timeand Time Interval (PTTI) Interface [14], or an alternativespecification.

The Waveform Generator Output Unit may be configuredto address a number of applications including:

N x PPS Output Interval (where N = 1 for 1 PPS) Start Time Pulse Width Number of pulses (if not continuous)Multiple timing signals may be initiated with different rate,

which provides a low cost, yet accurate timing reference.

High-frequency reference signals may also be generated,using the Frequency Generator Output Unit. FrequencyOutputs may be configured from 1 Hz to 12.5 MHz, on one ormore I/O output pins. While the frequency is synchronized,phase is not; the output may be used for syntonization.

VII. CONCLUSIONSQuasi-Real-Time systems, characterized by low-end

devices interconnected over a distributed Ethernet/IEEE1588v2 network in timing-based systems, are made viablethrough availability of highly integrated, silicon-basedimplementations.

Cost-effective, energy efficient network attachmentimplementations enable a range of distributed QRT monitoring,control, and automation applications that can exploitunprecedented communications and synchronizationperformance while reducing life cycle costs.

While QRT network performance remains coupled closelywith the applications, high-performance silicon such asMicrels KSZ84xx family renders IEEE 1588 over Ethernet asa viable option for a range of diverse applications that werepreviously cost-prohibitive. As costs are further reduced, QRTwill make it feasible to continue to migrate down-market,replacing point-to-point interconnects.

REFERENCES

[1] Standard for a Precision Clock Synchronization Protocol forNetworked Measurement and Control Systems, IEEE Standard 1588,2008.

[2] ODVA, Ethernet/IP Technology Overview, http://www.odva.org/[3] PI, Profinet- A Rich, Powerful, Ethernet Solution for Automation,

http://www.profibus.com/technology/profinet/

[4] EPSG, Ethernet PowerLink, http://www.ethernet-powerlink.org/[5] Communication networks and systems in substations - ALL PARTS,

IEC 61850, 2011.

[6] Ken Harris (January, 2009), An Application of IEEE 1588 to IndustrialAutomation, Rockwell-Automation Publication: 1756-WP005-EN-EAvailable:http://samplecode.rockwellautomation.com/idc/groups/literature/documents/wp/1756-wp005_-en-e.pdf

[7] IEEE Standard for Information technology - Telecommunications andinformation exchange between systems - Local and metropolitan areanetworks - Specific requirements Part 3: Carrier Sense Multiple Accesswith Collision Detection (CSMA/CD) Access Method., IEEE 802.3.,2008.

[8] Micrel Inc., KSZ84xx Data Sheet, 2011[9] IEEE 802.3az-2010 IEEE Standard for Local and Metropolitan Area

Networks - Specific requirements Part 3: Carrier Sense Multiple Accesswith Collision Detection (CSMA/CD) Access Method and PhysicalLayer Specifications Amendment 5: Media Access Control Parameters,Physical Layers, and Management Parameters for Energy-EfficientEthernet

[10] Inter-Range Instrumentation Group SERIAL TIME CODEFORMATS, IRIG Standard 200-98, Telecommunications and TimingGroup of the Range Commanders Council, 1998.

[11] EtherCat Technology Group, EtherCat- Ethernet for ControlAutomation Technology, http://www.ethercat.org

[12] PI, Profinet- A Rich, Powerful, Ethernet Solution for Automation,http://www.profibus.com/technology/profinet/

[13] Gunnar Prytz (2008), 13th IEEE International Conference on EmergingTechnologies and Factory Automation, pp. 408-415Available:http://www.ethercat.org/pdf/english/ETFA_2008_EtherCAT_vs_PROFI

NET_IRT.pdf

[14] GPS User Equipment III ICD for the Precision Time and Time Interval(PTTI) Interface, GPS ICD-GPS-060, GPS Navstar Joint ProgramOffice (US Govt.), 2002.