Senior DesignSenior DesignMay 07May 07--0606

High Performance Optical High Performance Optical InterconnectInterconnect

IntroductionIntroductionTeam MembersTeam Members

Adam FritzAdam FritzDavid SheetsDavid SheetsJay BeckerJay BeckerLayth AlLayth Al--JalilJalil

AdvisorsAdvisorsDr. Mani MinaDr. Mani MinaDr. Robert WeberDr. Robert WeberDr. Arun SomaniDr. Arun Somani

Client: Lockheed Martin

TerminologyTerminologyCOTS: COTS: CConsumer onsumer ooff ff tthe he sshelf, e.g. helf, e.g. ““COTS equipmentCOTS equipment””..NIC: NIC: NNetwork etwork iinterface nterface ccard, in this case, a card that accepts data ard, in this case, a card that accepts data over a PCIover a PCI--e bus and transfers it into a serial fiber optic signal.e bus and transfers it into a serial fiber optic signal.Node: A node is the combination of a network interface card and Node: A node is the combination of a network interface card and the host computer.the host computer.Primary Controller: This is the node that acts as the scheduler,Primary Controller: This is the node that acts as the scheduler, i.e. i.e. ““the masterthe master””..Secondary: This is a node that is controlled by the primary Secondary: This is a node that is controlled by the primary controller, i.e. controller, i.e. ““the slavethe slave””..LASER: LASER: LLight ight aamplification by mplification by sstimulated timulated eemission of mission of rradiation adiation RF: RF: RRadio adio ffrequency.requency.

TaskTaskDesign of a fiber optic network consisting of Design of a fiber optic network consisting of

at least three nodes intercommunicating at least three nodes intercommunicating at ten gigabits per second.at ten gigabits per second.

RelevanceRelevance

Avionics industry seeks greater throughput for every increasing system complexityMarket trend toward increased communication speeds among processorsFiber optic solutions are portable across Fiber optic solutions are portable across platforms.platforms.

System DesignSystem DesignTasked by client to develop a fiber optic network that Tasked by client to develop a fiber optic network that

provided 10Gbps throughput and COTS equipment.provided 10Gbps throughput and COTS equipment.The network shall be a ring topology.The network shall be a ring topology.Network shall employ a time division multiplexing scheme Network shall employ a time division multiplexing scheme to assign time intervals for each node to talk.to assign time intervals for each node to talk.

The figure at left shows nodes in a network that supports bidirectional data transport at 10Gbps.

System DesignSystem Design

Ring topology was chosen because of expandability Ring topology was chosen because of expandability issues.issues.

System DesignSystem DesignNetwork shall have fault tolerance and decentralized control.Network shall have fault tolerance and decentralized control.

Primary controller is assigned on system start up and assigns baPrimary controller is assigned on system start up and assigns backck--up up primary controllerprimary controllerIf primary controller fails, back up takes over scheduling and aIf primary controller fails, back up takes over scheduling and assigns a ssigns a new backup master.new backup master.This is a less troublesome approach to decentralized control. TrThis is a less troublesome approach to decentralized control. True ue decentralization would involve each node performing identical sydecentralization would involve each node performing identical system stem management tasks and necessitates that each node arrive at the smanagement tasks and necessitates that each node arrive at the same ame solution.solution.

TopologyTopologyThe architecture selected for this project will use a ring The architecture selected for this project will use a ring

topology: a fiber optic loop wherein every node has a topology: a fiber optic loop wherein every node has a coupler/splitter subcoupler/splitter sub--network that drives signals onto and network that drives signals onto and

couples signals off of a shared fiber optic line.couples signals off of a shared fiber optic line.

What is shown at left are nodes (i.e. the combination of an NIC and a host computer) interconnected via fiber optic cabling with fiber optic power couplers and fiber optic power splitters serving as data insertion and data extraction points.

TopologyTopologyRing topology supports bidirectional data transfer.Ring topology supports bidirectional data transfer.

This means that each node transmits a signal clockwise and This means that each node transmits a signal clockwise and counterclockwise about the topology, with the intention that shocounterclockwise about the topology, with the intention that should one uld one path fail that there is one spare path that will carry the signapath fail that there is one spare path that will carry the signal.l.The ring is comprised of passive elements.The ring is comprised of passive elements.Signal power level is directly tied to the number of nodes in thSignal power level is directly tied to the number of nodes in the network.e network.

Transmitting Node

(NODE A)

Receiving Node

(NODE B)

Signal f(t) f(t+t1) f(t+t2)

t1 is the delay from A to B using the shorter patht2 is thedelay from A to B using the longer path.

f(t+t1) will have more power then f(t+t2)

Fiber OpticsFiber OpticsFiber Optic Technology:Fiber Optic Technology:

What it is: an RF signal is input to a LASER transmitter What it is: an RF signal is input to a LASER transmitter which generates a signal centered on a specific frequency which generates a signal centered on a specific frequency with as much power at that specific frequency as possible. with as much power at that specific frequency as possible. This signal is then routed via waveguide conduits (i.e. This signal is then routed via waveguide conduits (i.e. fiber optic cable) to a transducer which converts optical fiber optic cable) to a transducer which converts optical energy into electrical energy. energy into electrical energy.

Fiber OpticsFiber Optics

Fiber Optic Networking:Fiber Optic Networking:Provides reliable high speed communication infrastructureProvides reliable high speed communication infrastructure•• Effectively immune from ambient EMIEffectively immune from ambient EMI•• Able to transmit higher serial data rates than conventional coppAble to transmit higher serial data rates than conventional copper er

conductorsconductors

COTS equipment is readily available to bring fiber optic COTS equipment is readily available to bring fiber optic signals off of a fiber optic network via signals off of a fiber optic network via NIC(sNIC(s))Because fiber optic technology is easily integrated into Because fiber optic technology is easily integrated into conventional network topologies and computer hardware, conventional network topologies and computer hardware, and because modern computing solutions demand lower and because modern computing solutions demand lower data latencies, implementation of an avionics fiber optic data latencies, implementation of an avionics fiber optic network is a highly realizable and cost effective industry network is a highly realizable and cost effective industry solution.solution.

Fiber OpticsFiber OpticsFiber Optic InterconnectFiber Optic Interconnect

Each node will have access to the other nodes in the network viaEach node will have access to the other nodes in the network via a fiber optic a fiber optic subsub--network. These subnetwork. These sub--networks are linked via a single fiber optic line.networks are linked via a single fiber optic line.Because only one node is talking at a given time, there is only Because only one node is talking at a given time, there is only one signal on one signal on the line at a given time.the line at a given time.

Fiber OpticFiber Optic

SubSub--network has four ports: network has four ports: NIC transmitter: Origin of laser NIC transmitter: Origin of laser signal; power emitted is equally signal; power emitted is equally split between I/O left and I/O split between I/O left and I/O right.right.NIC receiver: A portion of any NIC receiver: A portion of any signal incident to I/O left or I/O signal incident to I/O left or I/O right will be coupled into this.right will be coupled into this.I/O left and/or I/O right: As a I/O left and/or I/O right: As a signal propagates through the signal propagates through the loop, any signal incident to I/O left loop, any signal incident to I/O left will be split unevenly where most will be split unevenly where most of the signal power will propagate of the signal power will propagate out of I/O right and a small portion out of I/O right and a small portion into the receiver. This is true in the into the receiver. This is true in the reverse condition as well. reverse condition as well.

SubSub--Network:Network:

NICNIC

NIC Selection Criteria:NIC Selection Criteria:Support transmission speeds of 10 gigabits per secondSupport transmission speeds of 10 gigabits per secondSingle mode optical fiber compatibilitySingle mode optical fiber compatibilityBudget constraint issuesBudget constraint issuesAbility to support Ethernet and/or other prevalent data Ability to support Ethernet and/or other prevalent data encoding protocolsencoding protocols

NICsNICs should be configurable to either custom protocols or should be configurable to either custom protocols or should intrinsically support Ethernet.should intrinsically support Ethernet.

Must be PCIe or PCI-X 2.0 compatiblePCIe is preferable because with 2.5 Gbps per lane of theoretical

bandwidth, this format supports more data in fewer lines than PCI-X 2.0. PCIe is also a rising industry standard.

Time Division MultiplexingTime Division Multiplexing

The software used implements a polling system algorithm using a The software used implements a polling system algorithm using a TDM scheme. TDM scheme. Under this implementation, the primary controller polls each nodUnder this implementation, the primary controller polls each node as e as to how much data that node needs to transmit. Each node is then to how much data that node needs to transmit. Each node is then

assigned an amount time based on the formula:assigned an amount time based on the formula:

Data node has to transmitT(n)=Total data to be transmitted between all nodes

Rounded down to the nearest frame size, or rounded upto a frame size of one (if rounding down results in T(n) =

⎢ ⎥⎛ ⎞⎜ ⎟⎢ ⎥⎝ ⎠⎣ ⎦

0).Frame size is minimal sized data packet; e.g. 8 kB.

Basic TDMBasic TDMInitially, a simpler TDM scheme will be employed. Each Initially, a simpler TDM scheme will be employed. Each node will be given 1/nth of a predetermine time slice.node will be given 1/nth of a predetermine time slice.

Synchronization time is the time mainly accommodated Synchronization time is the time mainly accommodated for controller to initiate each time slot and round.for controller to initiate each time slot and round.

Guard time is the time accommodated mainly for each Guard time is the time accommodated mainly for each controlled cards to finish their transmission and for any controlled cards to finish their transmission and for any internal clock variation of each card.internal clock variation of each card.

Basic TDMBasic TDM

Advanced TDMAdvanced TDMAdvance TDM scheme will apply the same basic rules of Advance TDM scheme will apply the same basic rules of the basic TDM with the division of time into blocks that the basic TDM with the division of time into blocks that represent transmission windows.represent transmission windows.This scheme will apply more balanced data transmission This scheme will apply more balanced data transmission between cards in the network taking into consideration between cards in the network taking into consideration each nodeeach node’’s transmission needs.s transmission needs.The controller cardThe controller card’’s function of dividing the time slot s function of dividing the time slot windows (blocks) in each round is based on the windows (blocks) in each round is based on the feedback control packets, which arrive at the end of the feedback control packets, which arrive at the end of the previous round of transmissions. These feedback control previous round of transmissions. These feedback control packets indicate each nodepackets indicate each node’’s transmission needs for the s transmission needs for the upcoming round.upcoming round.

Advanced TDMAdvanced TDM

ImplementationImplementation

Company: Company: NetxenNetxenNIC Selection: NIC Selection: NXBNXB--10GXxR 10GXxR

A few relevant key features:A few relevant key features:Protocol offloadProtocol offloadMultiple protocol supportMultiple protocol supportProtocol implementation in firmwareProtocol implementation in firmwarePCIPCI--e 8x e 8x –– removes CPU bus removes CPU bus bottleneckbottleneckPotentially available through their Potentially available through their ““Beta Site ProgramBeta Site Program””

Hardware Specifics:

ImplementationImplementationSimulation:Simulation:

The first step in implementing our design is to The first step in implementing our design is to simulate the fiber optic network to ensure its simulate the fiber optic network to ensure its plausibilityplausibilityWhile simulation is not necessarily a required While simulation is not necessarily a required step before implementation, we feel that it will step before implementation, we feel that it will provide useful feedback about what potential provide useful feedback about what potential problems existproblems existFiber optic systems shall be tested using ADS, Fiber optic systems shall be tested using ADS, Cadence, and a network simulation tool called Cadence, and a network simulation tool called OpNetOpNet..

ImplementationImplementation

First StepsFirst StepsAfter simulation, we will begin working on building a After simulation, we will begin working on building a lower speed version of our design using lab equipmentlower speed version of our design using lab equipmentThe lab equipment available is only capable of The lab equipment available is only capable of approximately 2.5Gbps, but we need to make sure our approximately 2.5Gbps, but we need to make sure our fiber network layout works before becoming concerned fiber network layout works before becoming concerned with speedwith speedIn parallel with the low speed network we can begin In parallel with the low speed network we can begin testing the network cards in a point to point testing the network cards in a point to point arrangement to verify that they work and to determine arrangement to verify that they work and to determine their maximum throughputtheir maximum throughput

ImplementationImplementation

From walking to runningFrom walking to runningOnce we have confirmed both that the network cards Once we have confirmed both that the network cards work and that the fiber optic network architecture works, work and that the fiber optic network architecture works, it should be a relatively simple step to plug the 10Gbps it should be a relatively simple step to plug the 10Gbps cards into the networkcards into the networkThe difficult part in this step will be to get the software The difficult part in this step will be to get the software that maintains the TDM scheme working properlythat maintains the TDM scheme working properly

Closing CommentsClosing Comments

Summary:Summary:We believe our design for this fiber optic connection of We believe our design for this fiber optic connection of processor nodes will prove to be a cost effective and processor nodes will prove to be a cost effective and beneficial resource for Lockheed Martin and Iowa State beneficial resource for Lockheed Martin and Iowa State University in studying the capabilities and limitations of University in studying the capabilities and limitations of 10 gigabit communication technology. We look forward 10 gigabit communication technology. We look forward to producing a fully capable prototype next semester. to producing a fully capable prototype next semester.

Questions?Questions?