ODIN PRR

11

CERN

ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005

ODIN PRRODIN PRRTFC team:

Richard Jacobsson

Zbigniew Guzik

Associate

Arek Chlopik (10% 01-02)

Technical student:

Andrea Borga (03-04)

Summer students

Julie Advenard (05)

Pawel Bochinski (05)

Grzegorz Kasprowicz(03)

Ramy Abdel (03)

Pascal Koenig (02)

Agenda Functional description Hardware implementation Odin in 1MHz readout Firmware and simulation Testing Control software Production management Production testing Documentation and version control

Development history ODIN prototype 1 (1x) (2002) : Feasibility study with minimal version ODIN prototype 2 (2x) (2003) : Final prototype with full functionality ODIN Final 1 (2x) (2004) : Preproduction at Printca(Dk)/EFACEC(Pt) ODIN Final 2 (2x) (2005) : Preproduction at Printca(Dk)/ACAMAS(Fr) ODIN Final 2 (25x) (2005) : Production at Printca(Dk)/ACAMAS(Fr)

Today 6 ODINs (p2, v1, v2) installed and running (CERN, Zurich, Oxford)

22

CERN


TFC FunctionTFC Function

VELOL1 FE

TTCrx

VELOL0 FE

TTCrx

VELOL0 FE

TTCrx

ODINReadout Supervisor


Local trigger

L0 L1


THORTFC Switch

L1 MUNINL1 Throttle Switch

L0 MUNINL0 Throttle Switch

TTCtxOptical transmitter

TTC

sys

tem

TTCmiClock receiver/fanout

LHC clock

L1 (G

bE)

L0 (L

VDS)

Triggersplitter

VELOL0 FE

TTCrxVELOL0 FE

TTCrx

VELOTELL1

TTCrx

TTCocOptical splitter

VELOL1 FE

TTCrx

VELOL0 FE

TTCrx

VELOL0 FE

TTCrx

VELOL0 FE

TTCrxECALL0 FE

TTCrx

ECAL TELL1

TTCrx

HU

GIN

L0 /

L1 T

hrot

tle O

R

L1(G

bE)

FREJATTC monitoring

HU

GIN

L0 /

L1 T

hrot

tle O

R

BPIM








33

CERN


TFC installationTFC installation

10U

20U

30U

40U

50U

10U

20U

30U

40U

50U

TTCtxs

D3B08

TTCmi

OdinsTrigger splitter

Turbine

Heat exchanger

Deflector

Heat exchanger

Fan tray

Fan tray

ThorMunin_L0Munin_L1HuginL0DUTRM

Network patch panels

L0 patch panel

TTCmi

Turbine

Heat exchanger

Deflector

Heat exchanger

Fan tray

OdinsBPIMReserve

Network patch panels

Fan tray

D3B07

44

CERN


Context diagramContext diagram

ODIN

LHC accelerator Beam Phase and Intensity Monitor

Subdetectors

HLT farm

L1 trigger

‘Odin’ event bank

MultiDecisionPackets (MDP)

Bunch currentsBeam Synch,Timing info

(BST)

HW and run parameters

Run conditionsRun statistics

Detector status

55

CERN


Block diagramBlock diagram

LHC clockL0

Trigger generator

Trigger rate controller

Throttles

Cmd broadcastgenerator

Front-End

Event building

ECS interface

ECS L1

L1 broadcastgenerator

TTC

TTC Encoder

66

CERN


Overall dataflowOverall dataflow

BUNCH_INFO

TTCrsQ_MP

Q_L0

Q_L1

TTCrx

AFIFO

FE_BUFFER

DETECTOR_STATUS

GbE (EGRESS)Q_FE GbE

L1BROADCAST & COMMANDSTTC_DATA

L0_TRIGGER

L0_DATAL0_TRG

L0_DATA L0_DATA L1_DATA

L1_TRGL1_TRG

LHC_TTC TTC

GPSGbE (INGRESS)GbE

FPGA

FPGA

FPGA

FIFO

FIFO

MEZZANINE

FPGAMEZZANINE

CIRCUIT

CLOCK

ORBIT

77

CERN


Hardware implementationHardware implementation Design tools

Schematics and routing: Protel 99SE

Board dimensions 9U VME Same backplane as TELL1/UKL1 ~2.7mm PCB 1.6mm VME rails milling edges Holes for rigidity bar (only one backplane connector)

Main FPGAs

Top layerBottom layer

Front view

3mm

1.6mm

2.4÷2.7mm

Module FPGA type Speed grade

Pins / I/O pins* Gates used** Memory bits used**

Q_MP APEX 20K200E -1 240/123 52% 23% Q_L0 APEX 20K200E -1 240/123 67% 3% Q_L1 APEX 20K200E -1X(PLL) 240/123 ~35% 100% Q_FE APEX 20K100E -1X (PLL) 240/138 ~50% 50%

88

CERN


Board PowerBoard Power Use only +5V -5V, 3.3V, 2.5V and 1.8V made on board with regulators and DC/DC

• Distribution of local power with split power planes• > 2 x calculated worst case consumption per regulator• Total 6.4A@5V (“idle”) / <9A@5V (“full operation”)

Board can survive for extended periods without cooling• Add cooling towers in TTC encoder, no need on regulators

Back plane / PC power connector Power monitoring circuit

• Range 4.7 V – 5.3 V• LED and status bit in FPGA

Application Voltage Supply device

Main FPGA core 1.8V 2 x LT1963AEST-1.8 (1.5A)

JTAG to GbE mezzanine Special SPI-3 I/Os to GbE

2.5V 1 x LT1963AEST-2.5 (1.5A)

FPGA I/O LVPECL devices FIFOs I2C devices JTAG hub EPC4 configuration devices

3.3V 7 x LT1963AEST-3.3(1.5A)

LVDS receivers TTCrx LVPECL clock fan-out

3.3V 1 x LT1764AEQ-3.3 (3A)

GbE mezzanine 3.3V 1 x LT1084CT-3.3 (5A)

Encoder PLL PECL devices in TTC encoder Credit Card PC Glue light mezzanine

+5V -

TTC encoder PLL -3.3 1 x LM2991 (1A)

NECL devices in TTC encoder -5V Datel UWR-5/1600-D5(1.6A)

99

CERN


Function Chip Peripheral address

Odin board identifier 24LC024 0xA0

Hardware settings PCA9554 0x40

Clock adjustment PCA9555 0x42

TTCrx configuration TTCrx 0xEC

GbE mezzanine identifier GbE 0xAE

Board ControlBoard Control Control Interface: CCPC and Glue light

Ethernet I2C JTAG JTAG hub control Local Bus

I2C ~100 kHz Five devices:

JTAG ~100 kbits/s from CCPC FPGA programming and boundary scanning in-situ Fully configurable JTAG hub (programmed via header):

GPIO(7) GPIO(6) GPIO(5) GPIO(4) GPIO(3 .. 0)

RESET WRITE_ENB ADDR(0) ADDR(0) DATA(3 .. 0)

1010

CERN


Board controlBoard control JTAG bus architecture

All TCK lines filtered by 74HC244 and serial 100 ohm resistors TMS, TDI, TDO all have 33 ohm serial resistors

Q_MP FPGA

APEXEP20K200EQC240-1

Configurationdevice

Q_MP EPC4

GLUE lightCARD

JTAG bus

GPIO lines

LeveltranslatorMAX 3392

ExternalJTAG header

JTAG hub

MAXEPM7128AELC84-7

Hub control bus

JTAG master

3.3V

5VVoltage

JTAG headervoltage

Q_L0 FPGA

APEXEP20K200EQC240-1

Q_L1 FPGA

APEXEP20K200EQC240-1X

Q_FE FPGA

APEXEP20K100EQC240-1X

GigabitEthernet

mezzanine

Configurationdevice

Q_L0 EPC4

Configurationdevice

Q_L1 EPC4

Configurationdevice

Q_FE EPC4

JTAG 0

JTAG 1

JTAG 2

JTAG 3

JTAG 4

JTAG 5

JTAG 6

JTAG 7

JTAG 8

JTAG masterexternal

Altera Configuration buses

1111

CERN


Local bus Configuring, controlling and monitoring all TFC functionality in the FPGAs PLX 9030 Local Bus specification:

• 32-bit synchronous multiplexed address/data• Bus clock: system clock (BCLK) / 2 : ~20 MHz

Bus clock and control lines re-driven with 74ALVC16244 line driver

Resets and other hardware control lines:

System reset set after power up

Reset of CCPC and Glue card• VME reset from backplane ( Row 17,18,19 (A=B=C) nSysReset)• Front panel push button

Internal register Address Bit 7 Bit 6 Bit 5 Bit 4 Bit 3 Bit 2 Bit 1 Bit 0

Ouput port register (set out) 0x1

System reset

L1 Trigger LVDS

Encoder reset

Ext/Int System clock

Not used

Not used

Bunch info input

Detector status input

Port direction register 0x3 0 0 0 0 0 0 0 0

Board ControlBoard Control

Module LBUS base address

Q_MP 0x1000 Q_L0 0x2000 Q_L1 0x3000 Q_FE 0x4000

1212

CERN


Board ControlBoard Control Control of GbE MAC

Asynchronous u-processor bus Conversion between Local Bus and MAC control bus inside FPGA

• LBUS Address register for MAC internal register address • LBUS Data register for MAC configuration data

MAC reset vi control register in FPGA

Board id and code version Board identifier stored in I2C EEPROM (0xA0) Write protected (writing requires jumper)

FPGA code version• Local bus: (Main FPGAs: base address + 0xFC / Glue light : 0x3C)• YEAR(4) : MONTH(2) : DAY(2) : “HOUR”(2), that is for instance 2005071100

Board name Revision Number SystemID(TFC) BoardType Revision Serial number Hex

<31 .. 28> <27 .. 20> <19 .. 16> <15 .. 0>

OdinP2_00 P2 00 0010 00000000 0000 0000000000000000 0x20000000

OdinP2_01 P2 01 0010 00000000 0000 0000000000000001 0x20000001

OdinV1_00 V1 00 0010 00000000 0001 0000000000000000 0x20010000

OdinV1_01 V1 01 0010 00000000 0001 0000000000000001 0x20010001

OdinV2_00 V2 00 0010 00000000 0010 0000000000000000 0x20020000

OdinV2_01 V2 01 0010 00000000 0010 0000000000000001 0x20020001

OdinV2_02 V2 02 0010 00000000 0010 0000000000000010 0x20020002

OdinV2_nn V2 nn 0010 00000000 0010 nn 0x2002xxxx

1313

CERN


Board ControlBoard Control TFC Control and Status registers

Local Bus registers in the main FPGAs Configuration parameters

• H_name : Hardware parameters linked to the actual installation (Set once-only) • P_name: Parameters configuring running modes and operation (may not be changed during data taking)• R_name: Run related parameters which either enable/disable functions or operate the data taking

Status registers• S_name : Status bit for a particular function

– Distinction _INSTantaneous and _CONTinuous• C_name : Counter register

Reset registers• RST_name : Reset bits resetting individual functions• RST_CNT : Global reset of all counters and instantaneous status bits• RST_SEL_CNT : Selective reset of individual counters

Action registers• DMND_name : “Single-shot” bits to activate a function once on demand via ECS• UPDATE_CNT: Updates simultaneously all counter buffers for reading via ECS

1414

CERN


Clock and orbitClock and orbit Bunch clock

Two sources: External ECL AC coupled and internal ECL VXCO 80.158 MHz PLL clock driver for 40MHz, 80MHz, 160 MHz (MPC991 obsolete) Source selected by I2C Fine phase adjustment:

• LVPECL temperature compensated programmable delay chip 2.2ns – 12.2ns in 10 ps steps….(100EP195)

• Controlled via I2C• Compensate for phase shifts monitored by Beam Phase and Intensity Monitor• Initial alignment with TTCrx’s(250 x 104ps) and equal cable lengths

Clock distribution as differential LVPECL using clock fan-out 100LVEP111• Star to 10 destinations

Orbit signal Two sources: External ECL DC coupled and internal generated in FPGA from BCLK Source selected by the same I2C control line as for the bunch clock External is routed to FPGA (differential LVPECL) for edge detection and re-generation Re-generated bunch clock distributed to all FPGAs Orbit signal may be shifted internally In internal mode, orbit length is configurable…

1515

CERN


TTC signalTTC signal TTC Encoder on board

Similar to TTCvx Output is multiplexed A/B channel ECL AC-coupled TTC signal is fanned out by THOR (TFC Switch) Electrical to optical conversion by TTCtx

1616

CERN


L0 Trigger ProcessingL0 Trigger Processing

L0 Trigger Flow

L0 Trigger PriorityHandler

Front-End Buffer128K x 128 bits

L0 Accept FIFO128K x 18 bits

RandomTrigger

Generator

PeriodicTrigger

Generator

CalibrationTrigger

Generator

Resynch &L0 Pipeline

(fixed)

16

bit

TimingTrigger

Generator

L0 Pipeline(programmable)

Timing Trigger

Triggers

SynchronizationCheck Bunch ID

Counter12 bits

TriggerSequenceGenerator

L0

AC

CE

PT

WR

ITE

BX CrossingType

Sequencer

TTCEncoder

CH A

L0 Derand.Emulator

L1 BufferEmulator

GapGenerator

L1 TriggerProcessing

18

bits

Front-End

12

8 b

its1

28

bits

18

bits

External throttleECS throttleOther int. throttles

Safety throttle

L0 Event IDCounter

Au

xilia

ry T

rig

ge

r

Th

rott

les

Trigger type Encoded Priority

Trigger reject* 000 -

Physics trigger 001 1

Auxiliary trigger 010 2

Random trigger 011 3

Periodic trigger 100 4

Trigger for non-zero suppressed data** 101 -

Timing trigger 110 5

Calibration trigger 111 6

L0 throttle sources • External L0 throttle• ECS throttle• L0 derandomizer emulator • L1 Buffer Emulator• Gap generator• L0 Electronics Reset • L1 Electronics Reset• L0 Accept FIFO safety throttle• Front-End Buffer safety throttle• Automatic L0 trigger stop

1717

CERN



L0_STROBEL0_DECISL0_FORCEL0_TIM_TRGL0_BID[11:0]

Front PanelFront PanelFront PanelFront PanelFront Panel

AUX_TRIGL0_THR_EXT

ORBIT

BX_DATA[7:0]

Front PanelFront Panel

Q_MP

Front Panel

Q_L0 SUPERMODULE

L0_ERESETQ_MPL01_ERESETQ_MPECRQ_MP

CNT_RESETQ_MPCNT_UPDATDEQ_MPCAL_TRG_AQ_MPCAL_TRG_BQ_MPCAL_TRG_CQ_MP

L0_THR_L1EMUQ_L1

L0_THR_L0ERSTQ_MP

L0_ACCEPT_2

L0_ACC_LED

L0_SNC_ERRBX_TYPE[1:0]BX_DATA_OUT[7:0]

TTCrs

LED's

Q_MP+FE_BUFFE_BUFFE_BUF

L0_SNCERR_LEDL0_THR_LED

L0_QUAL[2:0]L0_FRC_OUT

LED'sLED's

AFIFO+FE_BUFAFIFO+FE_BUFAFIFO+FE_BUF

AFIFO_WE

AFIFO+FE_BUFBID[1:0]

FE_BUFBID[2:11]AFIFO+FE_BUFEID[11:0]

FE_BUFEID[23:12]

L0_RO_DONE Q_L1

nAFIFO_FULLAFIFOnFEB_FULLFE_BUF

L0_SNC_KEEP FE_BUF

On-board ConnectorPatch[12:0]

L0_OK Power Monitor

Glue LightLBUS[39:0]

BCLKTTCrs

LVCC_AVCCINT_1

SRESnI2C Register

FE_BUFL0_BID_OUT[11.:0]

JTAG HubJTAG_A[3:0]

(EP20K200EQC-240-1)

L0_ACCEPT_1 Q_MP

1818

CERN


L0 Trigger ProcessingL0 Trigger Processing L0 Trigger Input

Parallel 16-bit LVDS with twisted pair flat ribbon cable and 3M HE-10 connectors (distance ~1m) Implemented with MAX9179EUE Clock/data alignment:

• Latching on rising/falling edge• TTCrx in L0DU• L0 cable and TTC fibre length

Auxiliary L0 trigger input Single-ended ECL DC-coupled via LEMO

L0/L1 throttle inputs Dual twisted pair LVDS with RJ9 connectors

L0 Accept FIFO Two 128K x 9bits discrete synchronous FIFOs CY7C4292V

15 14 13 12 11 .. 0

Reserve L0_TIM_TRG L0_FORCE L0_DECIS L0_BID

17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

L0 Event ID (11 .. 0) FRC TYPE (2 .. 0) BID(1 .. 0)

1919

CERN


L0 Trigger ProcessingL0 Trigger Processing Front-End Buffer

Write 128 bits of L0 event data on every L0 trigger accept 16 128K x 9bits discrete synchronous FIFOs CY7C4292V

Detector status input Parallel 24-bit (2 bit/detector) LVDS with twisted pair flat ribbon cable and 3M HE-10 connectors Signal directly sampled in FIFO, must be correctly aligned

Bunch Information Input (Bunch intensity) General purpose 8-bit LVDS with twisted pair flat ribbon cable and 3M HE-10 connectors Signal directly sampled in FIFO, must be correctly aligned

FIFO pair

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

#0 L0 EVENT ID (9..0) FRC TRG TYPE(2..0) BID(1..0)

#1 BID(3..2) L0 EVENT ID(23..10)

#2 BX_DATA(7..0) BID(11..4)

#3 SNC KEEP

SNC ERR BXTYPE(1..0) BID(11..0) (L0DU)

#4 GPS_TIME(15..0)

#5 GPS_TIME(31..16)

#6 EXP_STATUS(7..0) GPS_TIME(39..32)

#7 EXP_STATUS(23..8)

2020

CERN


L0 Trigger ProcessingL0 Trigger Processing GPS time

Received from Beam Synchronous Timing system via onboard TTCrx as long broadcasts Extract 40 bits from 8 long broadcasts/turn with 8 bits (64 bits UTC) - ~ms resolution

• 4 bits year• 9 bits day of the year• 17 bits second of the day• 10 bits milliseconds

GPS time is processed in Q_MP• Time shift added

2121

CERN



L1 Trigger Flow

L1 TriggerDerandomizer

384 x 5 bits

L0 Accept FIFO128K x 18 bits

Gigabit Ethernetmezzanine

32 kB Ingress mem.>440 triggers

32

bit

MDPDisassembler

SynchronizationCheck

MinimumLatencyPipeline

AFIFOReader

L1 TriggerTransmission

L1 BufferEmulator

L0 TriggerProcessing 18 bits

TTCBroadcaster

External throttleECS throttleOther int. throttles

Th

rott

les

L1 Receiver(LVDS)

16

bits

Bro

ad

cast

AcknowledgeRe

qu

est

Front-End Processing

L1 Trigger InfoDerandomizer3072 x 32 bits

(8 words/trigger =>384 triggers)

Internal L1Trigger

Generator


L1 Event IDCounter

L1 Decision

L1 Info Transfer

Read

Read

L1 throttle sources • External L1 throttle• ECS throttle• L1 Electronics Reset• Trigger derandomizer safety throttle

2222

CERN



L1_THR_EXTFront Panel

nAFIFO_EMPTY

ORBIT

AFIFO_DATA[17:0]AFIFO

Q_MP

AFIFO

Q_L1 SUPERMODULE

L0_ERESETQ_MPL01_ERESETQ_MP

L1_SENT

Q_MP CNT_RESET

Q_MP

CNT_UPDATDE

L1_DATA_REQ_FE

Q_MP

nAFIFO_RE

L0_THR_L1EMU

REQ_L1_BRDCSTL1_BRDCST[4:0]

AFIFO

LED's

Q_L0

Q_MPQ_MP+FE_BUF

L1_SNCERR_LEDL1_THR_LED

LED's

Q_FEL1_DATA[15..0]

L1_ACCEPT_LED LED's

Back Panel L1_TRIGGER[15:0]

L1_OK Power Monitor

Glue MezzanineLBUS[39:0]

BCLKTTCrs

LVCC_AVCCINT_1

JTAG HubJTAG_B[3:0]

SRESnI2C Register

L1D_RFCLKL1D_RENB

L1D_RMOD[1:0]L1D_RPRTYL1D_RVALL1D_RSOPL1D_REOP

L1D_RXDAT[31:0]

L1D_RERRL1D_RSX

GbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_Ingress

On-Board Conn.Patch[9:0]

(EP20K200EQC-240-1X)

L1_LOCKED LED's

2323

CERN


L1 Trigger ProcessingL1 Trigger Processing L1 Trigger Input

Gigabit Ethernet mezzanine SPI-3 bus operated at 80 MHz, PLL in FPGA Voltage conversion for special control lines

Optional L1 Trigger input Parallel 16-bit LVDS with twisted pair flat ribbon cable and 3M HE-10 connectors

Decision 32-bit word 31 .. 24 23 .. 16 15 .. 8 7 .. 0

0

… Ethernet + IP header (see [4])

8

9 MDP Error block # of events

10 L0 Event ID

11 Error and status block L1 decision

12

… L1 trigger information

0

17

18 L0 Event ID

19 Error and status block L1 decision

20


1

25

N * 8 + 10 L0 Event ID

N * 8 + 11 Error and status block L1 decision

N * 8 + 12


N

N * 8 + 17

N * 8 + 18 CRC

15 14 13 12 11 .. 0

L1_STROBE L1_DECIS BID (1..0) L0_EID (11..0)

2424

CERN


Control CommandsControl Commands

TFC synchronous control commands generated by state machines Bunch Counter Reset L0 Event Counter Reset (L0 Event ID) L1 Event Counter Reset (L1 Event ID) Calibration pulse(s) L0 Electronics Reset L1 Electronics Reset Periodic User command

TFC asynchronous control commands L1 Triggers (accept interval 20 s, reject interval 400ns) L1 IP Destination (every n L0 trigger accepts) HLT IP Destination (every m L1 trigger accepts)

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 L1 IP destination 1 0 0 Flush R R Ethernet/IP address

HLT IP destination 1 0 1 Flush R R Ethernet/IP address

7 6 5 4 3 2 1 0

L1 trigger 1 Trigger type L0 EvID (ECR) (ECR)

Reset 0 1 R L1 EvID L1 FE L0 FE ECR BCR

Calibration 0 0 0 1 Pulse type (ECR) (BCR)

Command 0 0 1 Command type (ECR) (BCR)

2525

CERN


Control commandsControl commands TTC transmission based on priority scheme

Request/acknowledge mechanism Exact transmission time assured by forced pregap Synchronous commands get postponed until next turn Asynchronous commands get postponed until b-channel is free

TTC broadcast Encoded Priority

Bunch Counter Reset / Event Counter Reset

“00000001” / “0000011” 9

L0+L1 electronics reset “00001110” 8

L0 electronics reset “00000110” 7

Calibration command A “00010000” 6

Calibration command B “00010100” 5

Calibration command C “00011000” 4

User command “XXXXXXX” 3

IP destination

See Error! Reference source not

found. 2

L1 Trigger “1XXXXX00” 1

2626

CERN


Control CommandsControl Commands

ORBIT_EXTH_EXTRSVD_INERR_PWRL1_BRDCST[4:0]

Front PanelI2C RegisterFront PanelPower MonitorQ_L1

REQ_L1_BRDCST

L0_ACCEPT_1

CLKOCK40DES

Q_L1

Q_L0

TTCrx

Q_MP SUPERMODULE

TTC_SUBADDR[7:0]TTCrxTTC_DOUT[7:0]TTCrxTTC_BRCST[7:2]TTCrx

TTC_L1ACCEPTTTCrxTTC_EVCNTRESTTCrxTTC_DOUTSTRTTCrxTTC_BRCSTSTR1TTCrxTTC_READYTTCrx

TTC_BCNTRESTTCrx

ORBIT

ORBIT_EXT_LED

L0_ERESETL01_ERESETCNT_RESET

ALL + TTCrs

LED's

ALLALLALL

ECR

IP_ENA

FEB_RESETAFIFO_RESET

Q_L0

Q_FE

ALL

AFIFOFE_BUF

CNT_UPDATE

Q_L0L0_THR_L0ERST

Q_L0CAL_TRG_BQ_L0CAL_TRG_A

Q_L0CAL_TRG_C

CHB_BRDCST TTCrs


MP_OK Power Monitor


BCLKTTCrs

LVCC_AVCCINT_1

SRESnI2C Register

Q_L1+Q_FEL1_SENT

JTAG HubJTAG_D[3:0]

(EP20K200EQC-240-1)

FE_BUFGPS_TIME[39:0]

nTTCrx_RESETTTCrx

IP_DEST Q_FEIP_RSV Q_FE

2727

CERN


L1 Front-End ProcessingL1 Front-End Processing

L1 Front-End data flow

Gigabit EthernetTransmission Buffer

568 x 32 bits

Gigabit EthernetTransmitter


Gigabit Ethernetmezzanine

32 kB Ingress mem.>440 triggers

32 b

it3

2 b

it

MEP Handler

32 b

it

IP DestinationAssignment

IP destination word

Event FragmentBuffer

272 x 32 bits

32 b

it


32 b

it

Event FragmentBuilder16 bits

32 b

it


L1 Trigger InfoTransfer

2828

CERN


Front-End ProcessingFront-End Processing

nFEB_EMPTY

ORBIT

FEB_FATA[15:0]FE_BUF

Q_MP

FE_BUF

Q_FE SUPERMODULE

L0_ERESETQ_MPL01_ERESETQ_MP

L1_SENT

Q_MP CNT_RESET

Q_MP

CNT_UPDATDE

L1_DATA[15:0]Q_L1

Q_MP

L1_DATA_RE

nFEB_RE[8:1]

Q_L1

FE_BUF

GbEMBUS_DAT[15:0]

FE_OK Power Monitor


BCLKTTCrs

LVCC_AVCCINT_1

JTAG HubJTAG_C[3:0]

SRESnI2C Register(EP20K100EQC-240-1X)

DAQ_DTPA[1:0] GbE_EgressDAQ_STPADAQ_PTPADAQ_TADR

DAQ_TENBDAQ_TDAT[31:0]DAQ_TMOD[1:0]DAQ_TPRTYDAQ_TSOP

DAQ_TFCLK

DAQ_TEOPDAQ_TERR

GbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_Egress

DAQ_TSX GbE_Egress

L1_BRDCST[4:0]Q_L1


DAQ_PAUSE[1:0] GbE_IngressDAQ_PAUSED[1:0] GbE_Ingress

GbEMBUS_RDn

GbEMBUS_WRn

GbEMBUS_ALE

GbEMBUS_INTn

GbEMBUS_CONF[1:0]

GbEMBUS_RSTn

FE_LOCKED LED's

IP_ENAQ_MPIP_DESTQ_MPIP_RSVQ_MP

2929

CERN


Front-End ProcessingFront-End Processing Front-End Buffer read

FEB read in 8 clock cycles via 16 bit bus (pair of FIFO at a time)

ODIN Event Fragment 64 bytes Data transmission

Gigabit Ethernet mezzanine Egress SPI-3 bus operated at 80 MHz (PLL in Q_FE).

Level Data #bits

L0 Event ID 24

Bunch ID from Odin 12

Bunch ID from L0DU 12

GPS time 40

Trigger type 3

L0 Force bit 1

L0 synch error 1

L0 synch error (forced accept) 1

Detector status 24

BX type 2

L0 bunch current 8

L0

Subtotal 128

L1 Event ID 32

L0 Event ID in MDP 32

L1 Error/status block 32

L1 decision info (MDP) 6 x 32

L1

Subtotal 288

Total 52 bytes

3030

CERN


ODIN in 1MHz readoutODIN in 1MHz readout IP destination assignment, two schemes:

Static load balancing Dynamic load balancing

Additional long broadcast for MEP synchronization

ODIN Event Fragment transmission at 1 MHz Event fragment 68 byte Bandwidth sufficient even with packing factor of 1 at 1MHz

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 IP destination 1 0 0 Flush Ethernet/IP address

MEP synchronization 1 0 1 R L0 Event ID of first event in MEP

3131

CERN


ODIN in 1MHz readout schemeODIN in 1MHz readout scheme Static load balancing

L0 trigger path

Dest read ptr 0I

L0 AcceptFIFO

128K x 18 bits

L0 trigger path

IP destinationbuffer

2048 x 16 bits

CPU Category 0“100%”




12

bit

Dest read ptr 1

Dest read ptr 2

Dest read ptr 3

Static loadbalancer

1024 x 4 bits

CPU categories

TTC broadcaster

Request/Acknowledge

Re

q

Ack

Triggertype3 bits

L0 Event ID12 bits

IP Brdcst hdr

IP destination12 bits

Status4-bits

16

bit

Write ptr Cat. read ptr

0123

01212

......

CPU Cat.

Memory lock

Long broadcast derandomizer32 bit

Exclude CPUs onlinetemporarily or permanentby modifying CPU status

ECS L0 throttle

GbE RX:

- Receive CPU “link down” via SNMP

TELL1/UKL1 L0 throttle

16

bit

4 b

it

MEP synch. brdcst

16

bit

32

bit

L0 triggerprocessing

ECS:

- Throttle L0- Modify CPU status online

IP dest

Calibration CPUs Dest read ptr (cal)

IP state machine:

- Count L0 Accepts- Read CPU category for each MEP- Read destination(CPU cat.) for each MEP- Prepare IP brdcst header- Prepare MEP synchronization broadcast- Send broadcasts- Clear L0 Accept counter

- On special trigger (e.g. calibration):- Flush previous MEP- Read special destination- Flush special trigger- Clear L0 Accept counter

3232

CERN


ODIN in 1MHz readoutODIN in 1MHz readout Dynamic load balancing

L0 trigger path

Dest read ptr 0I

L0 AcceptFIFO

128K x 18 bits

L0 trigger path

IP destinationbuffer

2048 x 20 bits





12

bit

TTC broadcaster

Request/Acknowledge

Re

q

Ack

Triggertype3 bits

L0 Event ID12 bits

IP Brdcst hdr

IP destination12 bits

Credit4-bits

Write ptr

Memory lock

Long broadcast derandomizer32 bit

ECS L0 throttle

GbE RX:

- Receive MEP Requests withCredit from each farm node ~20kHz

- Increment Credit for destination

TELL1/UKL1 L0 throttle

16

bit

4 b

it

MEP synch. brdcst

16

bit

32

bit

L0 triggerprocessing

IP dest

Calibration CPUs Dest read ptr (cal)

IP state machine:

- Count L0 Accepts- Read destination for each MEP- Decrement Credit- Prepare IP brdcst header- Prepare MEP synchronization broadcast- Send broadcasts- Clear L0 Accept counter- Step to next destination with Credit /= 0

- On special trigger (e.g. calibration):- Flush previous MEP- Read special destination- Flush special trigger- Clear L0 Accept counter

3333

CERN


Front-PanelFront-Panel LEDs

Designator Function Normal Activity Abnormal Activity

D1-L Power Supply Status Power OK Green +5V Out of Range Blinking Red

D1-R System Reset OFF Green ON Red

D2-L All FPGA’s Configured Yes Green No Red

D2-R PLL Locked Locked Green Unlocked Red

D3-L External Orbit Present Yes Green No Red

D3-R External Clock Present Yes Green No Red

D4-L BCLK & Orbit Selection External Green Internal Red

D4-R L0 Sync Error No Blank Yes Red

D5-L L1 Accepts Present Yes Green No Blank

D5-R L1 Sync Error No Blank Yes Red

D6-L Reserve Yes Green No Blank

D6-R L0 Accepts Present Yes Green No Blank

D7-L L0 Throttle Present No Blank Yes Red

D7-R L1 Throttle Present No Blank Yes Red

3434

CERN


Front-panelFront-panel Connections

Designator Function Mode Signal Location Connector

P1-L Orbit In DC ECL Front Panel Dual LEMO

P1-R Orbit Out DC ECL Front Panel Dual LEMO

P2-L Bunch Clock In AC ECL Front Panel Dual LEMO

P2-R Bunch Clock Out AC ECL Front Panel Dual LEMO

P3-L TTC Channel A+B Out AC ECL Front Panel Dual LEMO

P3-R TTC Channel A+B Out AC ECL Front Panel Dual LEMO

P4-L TTC Channel A Out DC ECL Front Panel Dual LEMO

P4-R TTC Channel B Out DC ECL Front Panel Dual LEMO

P5-L Auxiliary Trigger In DC ECL Front Panel Dual LEMO

P5-R Reserve In TTL Front Panel Dual LEMO

P6 BST Info In Optical Front Panel ST/PC

L1 Trigger In GbEthernet GbE Mezzanine RJ45

P7-P10 DAQ Out GbEthernet GbE Mezzanine RJ45

P11 L0 Trigger In LVDS Front Panel 3M 34-pins

P12 Bunch Crossing Info In LVDS Front Panel 3M 14-pins

P13(0) L0 Throttle 1 In LVDS Front Panel RJ9




P15 Ethernet CCPC I/O Ethernet Front Panel RJ45

P16 L1 Trigger In LVDS Back Panel 3M 34-pins

P17 Experiment Status In LVDS Back Panel 3M 50-pins

VME-P1 Power Supply - - Back Panel DIN96_M

3535

CERN


Firmware and simulationFirmware and simulation

Tools No particular tool for VHDL coding Synthesis: Synplify Placement and routing: Quartus Simulation: Visual Elite

VHDL organization Counters, RAMs and FIFOs using Altera LPMs

Q_L0_v2:LBUS

ODIN_bid

ODIN_eid

ODIN_L0_pipeline

ODIN_L0_resynchronise

ODIN_L0_synch_chk

ODIN_handling.vhd

ODIN_l0d_emulator

ODIN_gap_generator

ODIN_per_trg

ODIN_rnd

ODIN_L0_counters

ODIN_bx_pipeline

ODIN_bx_type

Q_L1_v2:LBUS

ODIN_L1_trigger_GBE

ODIN_L1_trigger_LVDS

ODIN_L1_internal

ODIN_AFIFO_read

ODIN_L1_synch_chk

ODIN_L1_request

ODIN_l1b_emulator

ODIN_L1_counters

Q_MP_v2:LBUS

ODIN_orbit

ODIN_bcr_ecr

ODIN_ecr_internal

ODIN_cal_trg

ODIN_l0e_reset

ODIN_l01e_reset

ODIN_per_cmd

ODIN_IP_broadcast

ODIN_broadcaster

ODIN_MP_counters

ODIN_TTCrx

Q_FE_v2:LBUS

ODIN_egress_pll

Read_feb

ODIN_read_feb

ODIN_read_L1data

ODIN_data_format

ODIN_IP_handling

ODIN_GBE_ctrl

ODIN_DAQ_link

ODIN_FE_counters

Q_JTAGHUB_v2Q_JTAGHUB_v2.vhd

3636

CERN


SimulationSimulation Based on Visual Elite 3.1.7

Includes LHC bunch generator, L0DU, L1DU, TTCrx, THOR, ODIN, FREJA, Front-End, Glue Heavily used and maintained up to date

Functional and timing simulationC

CP

C

Q_L

0

AF

IFO

Q_L

1

Q_M

P

Gb

E

Q_F

E

FE

B

3737

CERN


TestingTesting Boundary scan

Local bus test

FIFO test

Gigabit Ethernet loop-back test GbE(Ingress->Q_L1->Q_FE->GbE(Egress)

Loop-back tests with onboard TTCrx

FREJA emulating L0FE, L1FE, L0DU,L1DU

Full system operation with complete control system

4 ODINs installed in sub-detector tests (CERN, Oxford, Zurich)

ODIN

FREJA

TT

C ca

ble

LHCMachine

Orbit

TTCdistribution

network

TT

C fib

er

Clock

3838

CERN


Control softwareControl software

Control operations via DIM ReadWriteRegister(struct[]{method, address, data, mask, r/w}), UpdateRegister(struct[]{address, data}), SubscribeCounters(struct[] {address, interval}), UpdateCounters(struct[] {address, data}), FPGADownload(id, STAPL data), FPGADownloaded(id, status).

UserInterface

CTRL

APIEventManager

DataManager

DatabeseServices

Commands

CCPC

DimServer

Electronics board

Boardelectronics

Control computers

DIM

On CCPC I2C read/write Local bus read/write FPGA/EPC programming GPIO control Boundary scan TFC server for PVSS control

Full TFC control system running centrally at CERN

3939

CERN


Production managementProduction management

Production archive

Company PCB: GPV Printca (Denmark) Mounting: ACAMAS (France)

Produce in total 25 boards 2 x 2 pre-production 6 production series PCBs made, in mounting 19 boards to be produced

4040

CERN


Production testing - checklistProduction testing - checklistStep Check

1 Examine PCB

2 Examine mounting

3 Fine-pitch components

4 Check power

5 Power monitoring

6 Clock distribution

7 CCPC + Ethernet

8 Glue card

9 I2C bus

10 I2C register

11 External clock

12 Clock phase alignment

13 Jitter

14 Backplane reset + reset button

15 Programming JTAG hub

16 Controlling JTAG HUB

17 Programming Q_MP

18 Programming Q_L0

19 Programming Q_L1

20 Programming Q_FE

21 Q_FE PLL

22 Q_L1 PLL

23 System reset

24 Test local bus

25 LEDs

26 TTC encoder

Step Check

27 TTC out

28 TTC channel A/B out with TTCvx

29 Orbit signal

30 Programming Q_MP EPC4

31 Programming Q_L0 EPC4

32 Programming Q_L1 EPC4

33 Programming Q_FE EPC4

34 JTAG BSCAN

35 FPGA interconnects

36 L0 Accept FIFO

37 L1 front-end buffer

38 L0 trigger input

39 L1 trigger input

40 Throttle inputs

41 Auxillary trigger input

42 Detector status inputs

43 GbE control bus

44 GbE sending

45 GbE receiving

46 TTXrx power and reset

47 TTCrx clock

48 TTCrx I2C bus

49 TTCrx L0 triggers

50 TTCrx short broadcasts

51 TTCrx long broadcasts

4141

CERN


Documentation and versionsDocumentation and versions

All to go in EDMS Production archive to go in EDMS Specifications Technical Reference …

Spare components

Documents

ODIN PRR