11
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
ODIN PRRODIN PRRTFC team:
Richard Jacobsson
Zbigniew Guzik
Associate
Arek Chlopik (10% 01-02)
Technical student:
Andrea Borga (03-04)
Summer students
Julie Advenard (05)
Pawel Bochinski (05)
Grzegorz Kasprowicz(03)
Ramy Abdel (03)
Pascal Koenig (02)
Agenda Functional description Hardware implementation Odin in 1MHz readout Firmware and simulation Testing Control software Production management Production testing Documentation and version control
Development history ODIN prototype 1 (1x) (2002) : Feasibility study with minimal version ODIN prototype 2 (2x) (2003) : Final prototype with full functionality ODIN Final 1 (2x) (2004) : Preproduction at Printca(Dk)/EFACEC(Pt) ODIN Final 2 (2x) (2005) : Preproduction at Printca(Dk)/ACAMAS(Fr) ODIN Final 2 (25x) (2005) : Production at Printca(Dk)/ACAMAS(Fr)
Today 6 ODINs (p2, v1, v2) installed and running (CERN, Zurich, Oxford)
22
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
TFC FunctionTFC Function
VELOL1 FE
TTCrx
VELOL0 FE
TTCrx
VELOL0 FE
TTCrx
ODINReadout Supervisor
ODINReadout Supervisor
Local trigger
L0 L1
ODINReadout Supervisor
THORTFC Switch
L1 MUNINL1 Throttle Switch
L0 MUNINL0 Throttle Switch
TTCtxOptical transmitter
TTC
sys
tem
TTCmiClock receiver/fanout
LHC clock
L1 (G
bE)
L0 (L
VDS)
Triggersplitter
VELOL0 FE
TTCrxVELOL0 FE
TTCrx
VELOTELL1
TTCrx
TTCocOptical splitter
VELOL1 FE
TTCrx
VELOL0 FE
TTCrx
VELOL0 FE
TTCrx
VELOL0 FE
TTCrxECALL0 FE
TTCrx
ECAL TELL1
TTCrx
HU
GIN
L0 /
L1 T
hrot
tle O
R
L1(G
bE)
FREJATTC monitoring
HU
GIN
L0 /
L1 T
hrot
tle O
R
BPIM
TTCtxOptical transmitter
TTCtxOptical transmitter
TTCtxOptical transmitter
TTCtxOptical transmitter
TTCocOptical splitter
TTCocOptical splitter
TTCocOptical splitter
33
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
TFC installationTFC installation
10U
20U
30U
40U
50U
10U
20U
30U
40U
50U
TTCtxs
D3B08
TTCmi
OdinsTrigger splitter
Turbine
Heat exchanger
Deflector
Heat exchanger
Fan tray
Fan tray
ThorMunin_L0Munin_L1HuginL0DUTRM
Network patch panels
L0 patch panel
TTCmi
Turbine
Heat exchanger
Deflector
Heat exchanger
Fan tray
OdinsBPIMReserve
Network patch panels
Fan tray
D3B07
44
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Context diagramContext diagram
ODIN
LHC accelerator Beam Phase and Intensity Monitor
Subdetectors
HLT farm
L1 trigger
‘Odin’ event bank
MultiDecisionPackets (MDP)
Bunch currentsBeam Synch,Timing info
(BST)
HW and run parameters
Run conditionsRun statistics
Detector status
55
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Block diagramBlock diagram
LHC clockL0
Trigger generator
Trigger rate controller
Throttles
Cmd broadcastgenerator
Front-End
Event building
ECS interface
ECS L1
L1 broadcastgenerator
TTC
TTC Encoder
66
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Overall dataflowOverall dataflow
BUNCH_INFO
TTCrsQ_MP
Q_L0
Q_L1
TTCrx
AFIFO
FE_BUFFER
DETECTOR_STATUS
GbE (EGRESS)Q_FE GbE
L1BROADCAST & COMMANDSTTC_DATA
L0_TRIGGER
L0_DATAL0_TRG
L0_DATA L0_DATA L1_DATA
L1_TRGL1_TRG
LHC_TTC TTC
GPSGbE (INGRESS)GbE
FPGA
FPGA
FPGA
FIFO
FIFO
MEZZANINE
FPGAMEZZANINE
CIRCUIT
CLOCK
ORBIT
77
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Hardware implementationHardware implementation Design tools
Schematics and routing: Protel 99SE
Board dimensions 9U VME Same backplane as TELL1/UKL1 ~2.7mm PCB 1.6mm VME rails milling edges Holes for rigidity bar (only one backplane connector)
Main FPGAs
Top layerBottom layer
Front view
3mm
1.6mm
2.4÷2.7mm
Module FPGA type Speed grade
Pins / I/O pins* Gates used** Memory bits used**
Q_MP APEX 20K200E -1 240/123 52% 23% Q_L0 APEX 20K200E -1 240/123 67% 3% Q_L1 APEX 20K200E -1X(PLL) 240/123 ~35% 100% Q_FE APEX 20K100E -1X (PLL) 240/138 ~50% 50%
88
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Board PowerBoard Power Use only +5V -5V, 3.3V, 2.5V and 1.8V made on board with regulators and DC/DC
• Distribution of local power with split power planes• > 2 x calculated worst case consumption per regulator• Total 6.4A@5V (“idle”) / <9A@5V (“full operation”)
Board can survive for extended periods without cooling• Add cooling towers in TTC encoder, no need on regulators
Back plane / PC power connector Power monitoring circuit
• Range 4.7 V – 5.3 V• LED and status bit in FPGA
Application Voltage Supply device
Main FPGA core 1.8V 2 x LT1963AEST-1.8 (1.5A)
JTAG to GbE mezzanine Special SPI-3 I/Os to GbE
2.5V 1 x LT1963AEST-2.5 (1.5A)
FPGA I/O LVPECL devices FIFOs I2C devices JTAG hub EPC4 configuration devices
3.3V 7 x LT1963AEST-3.3(1.5A)
LVDS receivers TTCrx LVPECL clock fan-out
3.3V 1 x LT1764AEQ-3.3 (3A)
GbE mezzanine 3.3V 1 x LT1084CT-3.3 (5A)
Encoder PLL PECL devices in TTC encoder Credit Card PC Glue light mezzanine
+5V -
TTC encoder PLL -3.3 1 x LM2991 (1A)
NECL devices in TTC encoder -5V Datel UWR-5/1600-D5(1.6A)
99
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Function Chip Peripheral address
Odin board identifier 24LC024 0xA0
Hardware settings PCA9554 0x40
Clock adjustment PCA9555 0x42
TTCrx configuration TTCrx 0xEC
GbE mezzanine identifier GbE 0xAE
Board ControlBoard Control Control Interface: CCPC and Glue light
Ethernet I2C JTAG JTAG hub control Local Bus
I2C ~100 kHz Five devices:
JTAG ~100 kbits/s from CCPC FPGA programming and boundary scanning in-situ Fully configurable JTAG hub (programmed via header):
GPIO(7) GPIO(6) GPIO(5) GPIO(4) GPIO(3 .. 0)
RESET WRITE_ENB ADDR(0) ADDR(0) DATA(3 .. 0)
1010
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Board controlBoard control JTAG bus architecture
All TCK lines filtered by 74HC244 and serial 100 ohm resistors TMS, TDI, TDO all have 33 ohm serial resistors
Q_MP FPGA
APEXEP20K200EQC240-1
Configurationdevice
Q_MP EPC4
GLUE lightCARD
JTAG bus
GPIO lines
LeveltranslatorMAX 3392
ExternalJTAG header
JTAG hub
MAXEPM7128AELC84-7
Hub control bus
JTAG master
3.3V
5VVoltage
JTAG headervoltage
Q_L0 FPGA
APEXEP20K200EQC240-1
Q_L1 FPGA
APEXEP20K200EQC240-1X
Q_FE FPGA
APEXEP20K100EQC240-1X
GigabitEthernet
mezzanine
Configurationdevice
Q_L0 EPC4
Configurationdevice
Q_L1 EPC4
Configurationdevice
Q_FE EPC4
JTAG 0
JTAG 1
JTAG 2
JTAG 3
JTAG 4
JTAG 5
JTAG 6
JTAG 7
JTAG 8
JTAG masterexternal
Altera Configuration buses
1111
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Local bus Configuring, controlling and monitoring all TFC functionality in the FPGAs PLX 9030 Local Bus specification:
• 32-bit synchronous multiplexed address/data• Bus clock: system clock (BCLK) / 2 : ~20 MHz
Bus clock and control lines re-driven with 74ALVC16244 line driver
Resets and other hardware control lines:
System reset set after power up
Reset of CCPC and Glue card• VME reset from backplane ( Row 17,18,19 (A=B=C) nSysReset)• Front panel push button
Internal register Address Bit 7 Bit 6 Bit 5 Bit 4 Bit 3 Bit 2 Bit 1 Bit 0
Ouput port register (set out) 0x1
System reset
L1 Trigger LVDS
Encoder reset
Ext/Int System clock
Not used
Not used
Bunch info input
Detector status input
Port direction register 0x3 0 0 0 0 0 0 0 0
Board ControlBoard Control
Module LBUS base address
Q_MP 0x1000 Q_L0 0x2000 Q_L1 0x3000 Q_FE 0x4000
1212
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Board ControlBoard Control Control of GbE MAC
Asynchronous u-processor bus Conversion between Local Bus and MAC control bus inside FPGA
• LBUS Address register for MAC internal register address • LBUS Data register for MAC configuration data
MAC reset vi control register in FPGA
Board id and code version Board identifier stored in I2C EEPROM (0xA0) Write protected (writing requires jumper)
FPGA code version• Local bus: (Main FPGAs: base address + 0xFC / Glue light : 0x3C)• YEAR(4) : MONTH(2) : DAY(2) : “HOUR”(2), that is for instance 2005071100
Board name Revision Number SystemID(TFC) BoardType Revision Serial number Hex
<31 .. 28> <27 .. 20> <19 .. 16> <15 .. 0>
OdinP2_00 P2 00 0010 00000000 0000 0000000000000000 0x20000000
OdinP2_01 P2 01 0010 00000000 0000 0000000000000001 0x20000001
OdinV1_00 V1 00 0010 00000000 0001 0000000000000000 0x20010000
OdinV1_01 V1 01 0010 00000000 0001 0000000000000001 0x20010001
OdinV2_00 V2 00 0010 00000000 0010 0000000000000000 0x20020000
OdinV2_01 V2 01 0010 00000000 0010 0000000000000001 0x20020001
OdinV2_02 V2 02 0010 00000000 0010 0000000000000010 0x20020002
OdinV2_nn V2 nn 0010 00000000 0010 nn 0x2002xxxx
1313
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Board ControlBoard Control TFC Control and Status registers
Local Bus registers in the main FPGAs Configuration parameters
• H_name : Hardware parameters linked to the actual installation (Set once-only) • P_name: Parameters configuring running modes and operation (may not be changed during data taking)• R_name: Run related parameters which either enable/disable functions or operate the data taking
Status registers• S_name : Status bit for a particular function
– Distinction _INSTantaneous and _CONTinuous• C_name : Counter register
Reset registers• RST_name : Reset bits resetting individual functions• RST_CNT : Global reset of all counters and instantaneous status bits• RST_SEL_CNT : Selective reset of individual counters
Action registers• DMND_name : “Single-shot” bits to activate a function once on demand via ECS• UPDATE_CNT: Updates simultaneously all counter buffers for reading via ECS
1414
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Clock and orbitClock and orbit Bunch clock
Two sources: External ECL AC coupled and internal ECL VXCO 80.158 MHz PLL clock driver for 40MHz, 80MHz, 160 MHz (MPC991 obsolete) Source selected by I2C Fine phase adjustment:
• LVPECL temperature compensated programmable delay chip 2.2ns – 12.2ns in 10 ps steps….(100EP195)
• Controlled via I2C• Compensate for phase shifts monitored by Beam Phase and Intensity Monitor• Initial alignment with TTCrx’s(250 x 104ps) and equal cable lengths
Clock distribution as differential LVPECL using clock fan-out 100LVEP111• Star to 10 destinations
Orbit signal Two sources: External ECL DC coupled and internal generated in FPGA from BCLK Source selected by the same I2C control line as for the bunch clock External is routed to FPGA (differential LVPECL) for edge detection and re-generation Re-generated bunch clock distributed to all FPGAs Orbit signal may be shifted internally In internal mode, orbit length is configurable…
1515
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
TTC signalTTC signal TTC Encoder on board
Similar to TTCvx Output is multiplexed A/B channel ECL AC-coupled TTC signal is fanned out by THOR (TFC Switch) Electrical to optical conversion by TTCtx
1616
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
L0 Trigger ProcessingL0 Trigger Processing
L0 Trigger Flow
L0 Trigger PriorityHandler
Front-End Buffer128K x 128 bits
L0 Accept FIFO128K x 18 bits
RandomTrigger
Generator
PeriodicTrigger
Generator
CalibrationTrigger
Generator
Resynch &L0 Pipeline
(fixed)
16
bit
TimingTrigger
Generator
L0 Pipeline(programmable)
Timing Trigger
Triggers
SynchronizationCheck Bunch ID
Counter12 bits
TriggerSequenceGenerator
L0
AC
CE
PT
WR
ITE
BX CrossingType
Sequencer
TTCEncoder
CH A
L0 Derand.Emulator
L1 BufferEmulator
GapGenerator
L1 TriggerProcessing
18
bits
Front-End
12
8 b
its1
28
bits
18
bits
External throttleECS throttleOther int. throttles
Safety throttle
L0 Event IDCounter
Au
xilia
ry T
rig
ge
r
Th
rott
les
Trigger type Encoded Priority
Trigger reject* 000 -
Physics trigger 001 1
Auxiliary trigger 010 2
Random trigger 011 3
Periodic trigger 100 4
Trigger for non-zero suppressed data** 101 -
Timing trigger 110 5
Calibration trigger 111 6
L0 throttle sources • External L0 throttle• ECS throttle• L0 derandomizer emulator • L1 Buffer Emulator• Gap generator• L0 Electronics Reset • L1 Electronics Reset• L0 Accept FIFO safety throttle• Front-End Buffer safety throttle• Automatic L0 trigger stop
1717
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
L0 Trigger ProcessingL0 Trigger Processing
L0_STROBEL0_DECISL0_FORCEL0_TIM_TRGL0_BID[11:0]
Front PanelFront PanelFront PanelFront PanelFront Panel
AUX_TRIGL0_THR_EXT
ORBIT
BX_DATA[7:0]
Front PanelFront Panel
Q_MP
Front Panel
Q_L0 SUPERMODULE
L0_ERESETQ_MPL01_ERESETQ_MPECRQ_MP
CNT_RESETQ_MPCNT_UPDATDEQ_MPCAL_TRG_AQ_MPCAL_TRG_BQ_MPCAL_TRG_CQ_MP
L0_THR_L1EMUQ_L1
L0_THR_L0ERSTQ_MP
L0_ACCEPT_2
L0_ACC_LED
L0_SNC_ERRBX_TYPE[1:0]BX_DATA_OUT[7:0]
TTCrs
LED's
Q_MP+FE_BUFFE_BUFFE_BUF
L0_SNCERR_LEDL0_THR_LED
L0_QUAL[2:0]L0_FRC_OUT
LED'sLED's
AFIFO+FE_BUFAFIFO+FE_BUFAFIFO+FE_BUF
AFIFO_WE
AFIFO+FE_BUFBID[1:0]
FE_BUFBID[2:11]AFIFO+FE_BUFEID[11:0]
FE_BUFEID[23:12]
L0_RO_DONE Q_L1
nAFIFO_FULLAFIFOnFEB_FULLFE_BUF
L0_SNC_KEEP FE_BUF
On-board ConnectorPatch[12:0]
L0_OK Power Monitor
Glue LightLBUS[39:0]
BCLKTTCrs
LVCC_AVCCINT_1
SRESnI2C Register
FE_BUFL0_BID_OUT[11.:0]
JTAG HubJTAG_A[3:0]
(EP20K200EQC-240-1)
L0_ACCEPT_1 Q_MP
1818
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
L0 Trigger ProcessingL0 Trigger Processing L0 Trigger Input
Parallel 16-bit LVDS with twisted pair flat ribbon cable and 3M HE-10 connectors (distance ~1m) Implemented with MAX9179EUE Clock/data alignment:
• Latching on rising/falling edge• TTCrx in L0DU• L0 cable and TTC fibre length
Auxiliary L0 trigger input Single-ended ECL DC-coupled via LEMO
L0/L1 throttle inputs Dual twisted pair LVDS with RJ9 connectors
L0 Accept FIFO Two 128K x 9bits discrete synchronous FIFOs CY7C4292V
15 14 13 12 11 .. 0
Reserve L0_TIM_TRG L0_FORCE L0_DECIS L0_BID
17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
L0 Event ID (11 .. 0) FRC TYPE (2 .. 0) BID(1 .. 0)
1919
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
L0 Trigger ProcessingL0 Trigger Processing Front-End Buffer
Write 128 bits of L0 event data on every L0 trigger accept 16 128K x 9bits discrete synchronous FIFOs CY7C4292V
Detector status input Parallel 24-bit (2 bit/detector) LVDS with twisted pair flat ribbon cable and 3M HE-10 connectors Signal directly sampled in FIFO, must be correctly aligned
Bunch Information Input (Bunch intensity) General purpose 8-bit LVDS with twisted pair flat ribbon cable and 3M HE-10 connectors Signal directly sampled in FIFO, must be correctly aligned
FIFO pair
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
#0 L0 EVENT ID (9..0) FRC TRG TYPE(2..0) BID(1..0)
#1 BID(3..2) L0 EVENT ID(23..10)
#2 BX_DATA(7..0) BID(11..4)
#3 SNC KEEP
SNC ERR BXTYPE(1..0) BID(11..0) (L0DU)
#4 GPS_TIME(15..0)
#5 GPS_TIME(31..16)
#6 EXP_STATUS(7..0) GPS_TIME(39..32)
#7 EXP_STATUS(23..8)
2020
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
L0 Trigger ProcessingL0 Trigger Processing GPS time
Received from Beam Synchronous Timing system via onboard TTCrx as long broadcasts Extract 40 bits from 8 long broadcasts/turn with 8 bits (64 bits UTC) - ~ms resolution
• 4 bits year• 9 bits day of the year• 17 bits second of the day• 10 bits milliseconds
GPS time is processed in Q_MP• Time shift added
2121
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
L1 Trigger ProcessingL1 Trigger Processing
L1 Trigger Flow
L1 TriggerDerandomizer
384 x 5 bits
L0 Accept FIFO128K x 18 bits
Gigabit Ethernetmezzanine
32 kB Ingress mem.>440 triggers
32
bit
MDPDisassembler
SynchronizationCheck
MinimumLatencyPipeline
AFIFOReader
L1 TriggerTransmission
L1 BufferEmulator
L0 TriggerProcessing 18 bits
TTCBroadcaster
External throttleECS throttleOther int. throttles
Th
rott
les
L1 Receiver(LVDS)
16
bits
Bro
ad
cast
AcknowledgeRe
qu
est
Front-End Processing
L1 Trigger InfoDerandomizer3072 x 32 bits
(8 words/trigger =>384 triggers)
Internal L1Trigger
Generator
Front-End Buffer128K x 128 bits
L1 Event IDCounter
L1 Decision
L1 Info Transfer
Read
Read
L1 throttle sources • External L1 throttle• ECS throttle• L1 Electronics Reset• Trigger derandomizer safety throttle
2222
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
L1 Trigger ProcessingL1 Trigger Processing
L1_THR_EXTFront Panel
nAFIFO_EMPTY
ORBIT
AFIFO_DATA[17:0]AFIFO
Q_MP
AFIFO
Q_L1 SUPERMODULE
L0_ERESETQ_MPL01_ERESETQ_MP
L1_SENT
Q_MP CNT_RESET
Q_MP
CNT_UPDATDE
L1_DATA_REQ_FE
Q_MP
nAFIFO_RE
L0_THR_L1EMU
REQ_L1_BRDCSTL1_BRDCST[4:0]
AFIFO
LED's
Q_L0
Q_MPQ_MP+FE_BUF
L1_SNCERR_LEDL1_THR_LED
LED's
Q_FEL1_DATA[15..0]
L1_ACCEPT_LED LED's
Back Panel L1_TRIGGER[15:0]
L1_OK Power Monitor
Glue MezzanineLBUS[39:0]
BCLKTTCrs
LVCC_AVCCINT_1
JTAG HubJTAG_B[3:0]
SRESnI2C Register
L1D_RFCLKL1D_RENB
L1D_RMOD[1:0]L1D_RPRTYL1D_RVALL1D_RSOPL1D_REOP
L1D_RXDAT[31:0]
L1D_RERRL1D_RSX
GbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_IngressGbE_Ingress
On-Board Conn.Patch[9:0]
(EP20K200EQC-240-1X)
L1_LOCKED LED's
2323
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
L1 Trigger ProcessingL1 Trigger Processing L1 Trigger Input
Gigabit Ethernet mezzanine SPI-3 bus operated at 80 MHz, PLL in FPGA Voltage conversion for special control lines
Optional L1 Trigger input Parallel 16-bit LVDS with twisted pair flat ribbon cable and 3M HE-10 connectors
Decision 32-bit word 31 .. 24 23 .. 16 15 .. 8 7 .. 0
0
… Ethernet + IP header (see [4])
8
9 MDP Error block # of events
10 L0 Event ID
11 Error and status block L1 decision
12
… L1 trigger information
0
17
18 L0 Event ID
19 Error and status block L1 decision
20
… L1 trigger information
1
25
N * 8 + 10 L0 Event ID
N * 8 + 11 Error and status block L1 decision
N * 8 + 12
… L1 trigger information
N
N * 8 + 17
N * 8 + 18 CRC
15 14 13 12 11 .. 0
L1_STROBE L1_DECIS BID (1..0) L0_EID (11..0)
2424
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Control CommandsControl Commands
TFC synchronous control commands generated by state machines Bunch Counter Reset L0 Event Counter Reset (L0 Event ID) L1 Event Counter Reset (L1 Event ID) Calibration pulse(s) L0 Electronics Reset L1 Electronics Reset Periodic User command
TFC asynchronous control commands L1 Triggers (accept interval 20 s, reject interval 400ns) L1 IP Destination (every n L0 trigger accepts) HLT IP Destination (every m L1 trigger accepts)
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 L1 IP destination 1 0 0 Flush R R Ethernet/IP address
HLT IP destination 1 0 1 Flush R R Ethernet/IP address
7 6 5 4 3 2 1 0
L1 trigger 1 Trigger type L0 EvID (ECR) (ECR)
Reset 0 1 R L1 EvID L1 FE L0 FE ECR BCR
Calibration 0 0 0 1 Pulse type (ECR) (BCR)
Command 0 0 1 Command type (ECR) (BCR)
2525
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Control commandsControl commands TTC transmission based on priority scheme
Request/acknowledge mechanism Exact transmission time assured by forced pregap Synchronous commands get postponed until next turn Asynchronous commands get postponed until b-channel is free
TTC broadcast Encoded Priority
Bunch Counter Reset / Event Counter Reset
“00000001” / “0000011” 9
L0+L1 electronics reset “00001110” 8
L0 electronics reset “00000110” 7
Calibration command A “00010000” 6
Calibration command B “00010100” 5
Calibration command C “00011000” 4
User command “XXXXXXX” 3
IP destination
See Error! Reference source not
found. 2
L1 Trigger “1XXXXX00” 1
2626
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Control CommandsControl Commands
ORBIT_EXTH_EXTRSVD_INERR_PWRL1_BRDCST[4:0]
Front PanelI2C RegisterFront PanelPower MonitorQ_L1
REQ_L1_BRDCST
L0_ACCEPT_1
CLKOCK40DES
Q_L1
Q_L0
TTCrx
Q_MP SUPERMODULE
TTC_SUBADDR[7:0]TTCrxTTC_DOUT[7:0]TTCrxTTC_BRCST[7:2]TTCrx
TTC_L1ACCEPTTTCrxTTC_EVCNTRESTTCrxTTC_DOUTSTRTTCrxTTC_BRCSTSTR1TTCrxTTC_READYTTCrx
TTC_BCNTRESTTCrx
ORBIT
ORBIT_EXT_LED
L0_ERESETL01_ERESETCNT_RESET
ALL + TTCrs
LED's
ALLALLALL
ECR
IP_ENA
FEB_RESETAFIFO_RESET
Q_L0
Q_FE
ALL
AFIFOFE_BUF
CNT_UPDATE
Q_L0L0_THR_L0ERST
Q_L0CAL_TRG_BQ_L0CAL_TRG_A
Q_L0CAL_TRG_C
CHB_BRDCST TTCrs
On-board ConnectorPatch[12:0]
MP_OK Power Monitor
Glue MezzanineLBUS[39:0]
BCLKTTCrs
LVCC_AVCCINT_1
SRESnI2C Register
Q_L1+Q_FEL1_SENT
JTAG HubJTAG_D[3:0]
(EP20K200EQC-240-1)
FE_BUFGPS_TIME[39:0]
nTTCrx_RESETTTCrx
IP_DEST Q_FEIP_RSV Q_FE
2727
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
L1 Front-End ProcessingL1 Front-End Processing
L1 Front-End data flow
Gigabit EthernetTransmission Buffer
568 x 32 bits
Gigabit EthernetTransmitter
L0 TriggerProcessing
Gigabit Ethernetmezzanine
32 kB Ingress mem.>440 triggers
32 b
it3
2 b
it
MEP Handler
32 b
it
IP DestinationAssignment
IP destination word
Event FragmentBuffer
272 x 32 bits
32 b
it
Front-End Buffer128K x 128 bits
32 b
it
Event FragmentBuilder16 bits
32 b
it
L1 TriggerProcessing
L1 Trigger InfoTransfer
2828
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Front-End ProcessingFront-End Processing
nFEB_EMPTY
ORBIT
FEB_FATA[15:0]FE_BUF
Q_MP
FE_BUF
Q_FE SUPERMODULE
L0_ERESETQ_MPL01_ERESETQ_MP
L1_SENT
Q_MP CNT_RESET
Q_MP
CNT_UPDATDE
L1_DATA[15:0]Q_L1
Q_MP
L1_DATA_RE
nFEB_RE[8:1]
Q_L1
FE_BUF
GbEMBUS_DAT[15:0]
FE_OK Power Monitor
Glue MezzanineLBUS[39:0]
BCLKTTCrs
LVCC_AVCCINT_1
JTAG HubJTAG_C[3:0]
SRESnI2C Register(EP20K100EQC-240-1X)
DAQ_DTPA[1:0] GbE_EgressDAQ_STPADAQ_PTPADAQ_TADR
DAQ_TENBDAQ_TDAT[31:0]DAQ_TMOD[1:0]DAQ_TPRTYDAQ_TSOP
DAQ_TFCLK
DAQ_TEOPDAQ_TERR
GbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_EgressGbE_Egress
DAQ_TSX GbE_Egress
L1_BRDCST[4:0]Q_L1
On-board ConnectorPatch[8:0]
DAQ_PAUSE[1:0] GbE_IngressDAQ_PAUSED[1:0] GbE_Ingress
GbEMBUS_RDn
GbEMBUS_WRn
GbEMBUS_ALE
GbEMBUS_INTn
GbEMBUS_CONF[1:0]
GbEMBUS_RSTn
FE_LOCKED LED's
IP_ENAQ_MPIP_DESTQ_MPIP_RSVQ_MP
2929
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Front-End ProcessingFront-End Processing Front-End Buffer read
FEB read in 8 clock cycles via 16 bit bus (pair of FIFO at a time)
ODIN Event Fragment 64 bytes Data transmission
Gigabit Ethernet mezzanine Egress SPI-3 bus operated at 80 MHz (PLL in Q_FE).
Level Data #bits
L0 Event ID 24
Bunch ID from Odin 12
Bunch ID from L0DU 12
GPS time 40
Trigger type 3
L0 Force bit 1
L0 synch error 1
L0 synch error (forced accept) 1
Detector status 24
BX type 2
L0 bunch current 8
L0
Subtotal 128
L1 Event ID 32
L0 Event ID in MDP 32
L1 Error/status block 32
L1 decision info (MDP) 6 x 32
L1
Subtotal 288
Total 52 bytes
3030
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
ODIN in 1MHz readoutODIN in 1MHz readout IP destination assignment, two schemes:
Static load balancing Dynamic load balancing
Additional long broadcast for MEP synchronization
ODIN Event Fragment transmission at 1 MHz Event fragment 68 byte Bandwidth sufficient even with packing factor of 1 at 1MHz
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 IP destination 1 0 0 Flush Ethernet/IP address
MEP synchronization 1 0 1 R L0 Event ID of first event in MEP
3131
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
ODIN in 1MHz readout schemeODIN in 1MHz readout scheme Static load balancing
L0 trigger path
Dest read ptr 0I
L0 AcceptFIFO
128K x 18 bits
L0 trigger path
IP destinationbuffer
2048 x 16 bits
CPU Category 0“100%”
CPU Category 1“95%”
CPU Category 2“80%”
CPU Category 3“50%”
12
bit
Dest read ptr 1
Dest read ptr 2
Dest read ptr 3
Static loadbalancer
1024 x 4 bits
CPU categories
TTC broadcaster
Request/Acknowledge
Re
q
Ack
Triggertype3 bits
L0 Event ID12 bits
IP Brdcst hdr
IP destination12 bits
Status4-bits
16
bit
Write ptr Cat. read ptr
0123
01212
......
CPU Cat.
Memory lock
Long broadcast derandomizer32 bit
Exclude CPUs onlinetemporarily or permanentby modifying CPU status
ECS L0 throttle
GbE RX:
- Receive CPU “link down” via SNMP
TELL1/UKL1 L0 throttle
16
bit
4 b
it
MEP synch. brdcst
16
bit
32
bit
L0 triggerprocessing
ECS:
- Throttle L0- Modify CPU status online
IP dest
Calibration CPUs Dest read ptr (cal)
IP state machine:
- Count L0 Accepts- Read CPU category for each MEP- Read destination(CPU cat.) for each MEP- Prepare IP brdcst header- Prepare MEP synchronization broadcast- Send broadcasts- Clear L0 Accept counter
- On special trigger (e.g. calibration):- Flush previous MEP- Read special destination- Flush special trigger- Clear L0 Accept counter
3232
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
ODIN in 1MHz readoutODIN in 1MHz readout Dynamic load balancing
L0 trigger path
Dest read ptr 0I
L0 AcceptFIFO
128K x 18 bits
L0 trigger path
IP destinationbuffer
2048 x 20 bits
CPU Category 0“100%”
CPU Category 1“95%”
CPU Category 2“80%”
CPU Category 3“50%”
12
bit
TTC broadcaster
Request/Acknowledge
Re
q
Ack
Triggertype3 bits
L0 Event ID12 bits
IP Brdcst hdr
IP destination12 bits
Credit4-bits
Write ptr
Memory lock
Long broadcast derandomizer32 bit
ECS L0 throttle
GbE RX:
- Receive MEP Requests withCredit from each farm node ~20kHz
- Increment Credit for destination
TELL1/UKL1 L0 throttle
16
bit
4 b
it
MEP synch. brdcst
16
bit
32
bit
L0 triggerprocessing
IP dest
Calibration CPUs Dest read ptr (cal)
IP state machine:
- Count L0 Accepts- Read destination for each MEP- Decrement Credit- Prepare IP brdcst header- Prepare MEP synchronization broadcast- Send broadcasts- Clear L0 Accept counter- Step to next destination with Credit /= 0
- On special trigger (e.g. calibration):- Flush previous MEP- Read special destination- Flush special trigger- Clear L0 Accept counter
3333
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Front-PanelFront-Panel LEDs
Designator Function Normal Activity Abnormal Activity
D1-L Power Supply Status Power OK Green +5V Out of Range Blinking Red
D1-R System Reset OFF Green ON Red
D2-L All FPGA’s Configured Yes Green No Red
D2-R PLL Locked Locked Green Unlocked Red
D3-L External Orbit Present Yes Green No Red
D3-R External Clock Present Yes Green No Red
D4-L BCLK & Orbit Selection External Green Internal Red
D4-R L0 Sync Error No Blank Yes Red
D5-L L1 Accepts Present Yes Green No Blank
D5-R L1 Sync Error No Blank Yes Red
D6-L Reserve Yes Green No Blank
D6-R L0 Accepts Present Yes Green No Blank
D7-L L0 Throttle Present No Blank Yes Red
D7-R L1 Throttle Present No Blank Yes Red
3434
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Front-panelFront-panel Connections
Designator Function Mode Signal Location Connector
P1-L Orbit In DC ECL Front Panel Dual LEMO
P1-R Orbit Out DC ECL Front Panel Dual LEMO
P2-L Bunch Clock In AC ECL Front Panel Dual LEMO
P2-R Bunch Clock Out AC ECL Front Panel Dual LEMO
P3-L TTC Channel A+B Out AC ECL Front Panel Dual LEMO
P3-R TTC Channel A+B Out AC ECL Front Panel Dual LEMO
P4-L TTC Channel A Out DC ECL Front Panel Dual LEMO
P4-R TTC Channel B Out DC ECL Front Panel Dual LEMO
P5-L Auxiliary Trigger In DC ECL Front Panel Dual LEMO
P5-R Reserve In TTL Front Panel Dual LEMO
P6 BST Info In Optical Front Panel ST/PC
L1 Trigger In GbEthernet GbE Mezzanine RJ45
P7-P10 DAQ Out GbEthernet GbE Mezzanine RJ45
P11 L0 Trigger In LVDS Front Panel 3M 34-pins
P12 Bunch Crossing Info In LVDS Front Panel 3M 14-pins
P13(0) L0 Throttle 1 In LVDS Front Panel RJ9
P13(1) L1 Throttle 1 In LVDS Front Panel RJ9
P14(0) L0 Throttle 2 In LVDS Front Panel RJ9
P14(1) L1 Throttle 2 In LVDS Front Panel RJ9
P15 Ethernet CCPC I/O Ethernet Front Panel RJ45
P16 L1 Trigger In LVDS Back Panel 3M 34-pins
P17 Experiment Status In LVDS Back Panel 3M 50-pins
VME-P1 Power Supply - - Back Panel DIN96_M
3535
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Firmware and simulationFirmware and simulation
Tools No particular tool for VHDL coding Synthesis: Synplify Placement and routing: Quartus Simulation: Visual Elite
VHDL organization Counters, RAMs and FIFOs using Altera LPMs
Q_L0_v2:LBUS
ODIN_bid
ODIN_eid
ODIN_L0_pipeline
ODIN_L0_resynchronise
ODIN_L0_synch_chk
ODIN_handling.vhd
ODIN_l0d_emulator
ODIN_gap_generator
ODIN_per_trg
ODIN_rnd
ODIN_L0_counters
ODIN_bx_pipeline
ODIN_bx_type
Q_L1_v2:LBUS
ODIN_L1_trigger_GBE
ODIN_L1_trigger_LVDS
ODIN_L1_internal
ODIN_AFIFO_read
ODIN_L1_synch_chk
ODIN_L1_request
ODIN_l1b_emulator
ODIN_L1_counters
Q_MP_v2:LBUS
ODIN_orbit
ODIN_bcr_ecr
ODIN_ecr_internal
ODIN_cal_trg
ODIN_l0e_reset
ODIN_l01e_reset
ODIN_per_cmd
ODIN_IP_broadcast
ODIN_broadcaster
ODIN_MP_counters
ODIN_TTCrx
Q_FE_v2:LBUS
ODIN_egress_pll
Read_feb
ODIN_read_feb
ODIN_read_L1data
ODIN_data_format
ODIN_IP_handling
ODIN_GBE_ctrl
ODIN_DAQ_link
ODIN_FE_counters
Q_JTAGHUB_v2Q_JTAGHUB_v2.vhd
3636
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
SimulationSimulation Based on Visual Elite 3.1.7
Includes LHC bunch generator, L0DU, L1DU, TTCrx, THOR, ODIN, FREJA, Front-End, Glue Heavily used and maintained up to date
Functional and timing simulationC
CP
C
Q_L
0
AF
IFO
Q_L
1
Q_M
P
Gb
E
Q_F
E
FE
B
3737
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
TestingTesting Boundary scan
Local bus test
FIFO test
Gigabit Ethernet loop-back test GbE(Ingress->Q_L1->Q_FE->GbE(Egress)
Loop-back tests with onboard TTCrx
FREJA emulating L0FE, L1FE, L0DU,L1DU
Full system operation with complete control system
4 ODINs installed in sub-detector tests (CERN, Oxford, Zurich)
ODIN
FREJA
TT
C ca
ble
LHCMachine
Orbit
TTCdistribution
network
TT
C fib
er
Clock
3838
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Control softwareControl software
Control operations via DIM ReadWriteRegister(struct[]{method, address, data, mask, r/w}), UpdateRegister(struct[]{address, data}), SubscribeCounters(struct[] {address, interval}), UpdateCounters(struct[] {address, data}), FPGADownload(id, STAPL data), FPGADownloaded(id, status).
UserInterface
CTRL
APIEventManager
DataManager
DatabeseServices
Commands
CCPC
DimServer
Electronics board
Boardelectronics
Control computers
DIM
On CCPC I2C read/write Local bus read/write FPGA/EPC programming GPIO control Boundary scan TFC server for PVSS control
Full TFC control system running centrally at CERN
3939
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Production managementProduction management
Production archive
Company PCB: GPV Printca (Denmark) Mounting: ACAMAS (France)
Produce in total 25 boards 2 x 2 pre-production 6 production series PCBs made, in mounting 19 boards to be produced
4040
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Production testing - checklistProduction testing - checklistStep Check
1 Examine PCB
2 Examine mounting
3 Fine-pitch components
4 Check power
5 Power monitoring
6 Clock distribution
7 CCPC + Ethernet
8 Glue card
9 I2C bus
10 I2C register
11 External clock
12 Clock phase alignment
13 Jitter
14 Backplane reset + reset button
15 Programming JTAG hub
16 Controlling JTAG HUB
17 Programming Q_MP
18 Programming Q_L0
19 Programming Q_L1
20 Programming Q_FE
21 Q_FE PLL
22 Q_L1 PLL
23 System reset
24 Test local bus
25 LEDs
26 TTC encoder
Step Check
27 TTC out
28 TTC channel A/B out with TTCvx
29 Orbit signal
30 Programming Q_MP EPC4
31 Programming Q_L0 EPC4
32 Programming Q_L1 EPC4
33 Programming Q_FE EPC4
34 JTAG BSCAN
35 FPGA interconnects
36 L0 Accept FIFO
37 L1 front-end buffer
38 L0 trigger input
39 L1 trigger input
40 Throttle inputs
41 Auxillary trigger input
42 Detector status inputs
43 GbE control bus
44 GbE sending
45 GbE receiving
46 TTXrx power and reset
47 TTCrx clock
48 TTCrx I2C bus
49 TTCrx L0 triggers
50 TTCrx short broadcasts
51 TTCrx long broadcasts
4141
CERN
ODIN Production Readiness Review, August 26, 2005ODIN Production Readiness Review, August 26, 2005
Documentation and versionsDocumentation and versions
All to go in EDMS Production archive to go in EDMS Specifications Technical Reference …
Spare components