ROD DAQ Workshop February 6-7, 2003

ROD DAQ WorkshopFebruary 6-7, 2003

DSP Software Update

• Isr and Data Handling

• MDSP initialization, Rev B & Rev C RODs

• Histogramming code

Douglas FergusonUW-Madison & LBL

ISR and Data Handling

B0F00000 EE1234EE 00000009 …………………… ………….. ………… …………………… 21008E91 9CF09EB0 ………..………… ………….. ………… …………………… ………….. ………… …………………… ………….. ………… …………………… ………….. ………… …………

………… ………….. ………… …………………… ………….. ………… …………………… ………….. E0F0000 …………………… ………….. ………… …………………… ………….. ………… …………………… ………….. ………… …………………… ………….. ………… 40xy0127

1 frame= 0x100 words

Word 27

Inserted by router

Event Based Data: Event based data allows the DMA ISR to completeits work quickly, since the event’s length is alwaysfound at the end of a frame.

Previously the router inserted a marker word (whichcannot appear in normal data) of 0x40ff llll, wherellll is the event length in words, at the end of theframe, if it encountered an E0F in that frame. If theevent ends on the frame boundary, the marker wordis inserted at the end of the next (empty) frame.

The ISR was responsible for checking the event forerrors by calculating the position of the error word inthe event trailer. The ISR also had to determine whichrouter trap had sent on the event by looking at a wordin the event header.

The new router code dispenses with those checks byusing a new marker word of 0x40xy llll. llll is as beforeand x = 0 or 1 indicates the router trap. If the event hasno errors, y = 0xF; if there are errors y = 0xE.

ISR and Data Handling (cont)

ISR code has been streamlined, resulting in modest speed improvements for normal events & much improved handling of cases where it was necessary to place data in slow memory:

• Previously for each incoming event, the ISR filled in a set of frame registers (in addition to the event data which goes onto the event queue). These permitted events to be processed by tasks & removed from the queue immediately (for multi-task situations).

Data word 1: 0x II LL XX ZZ, I= IDRAM starting frame, L= # of IDRAM frames, X= SDRAM starting frame, Z= # of SDRAM frames.

Data word 2: 0xEE TT LLLL , E= error bits: 31= overflow, 30= any error, 29= header err, 28= trailer err, & 27= data err, T= the trap function mask, from event trap setup. L= event length in words,

• With the availability of 4 slave DSPs, this is less likely, and in addition the queue of events is likely to be short under most circumstances, so the queue has been reduced to a simple FIFO. The space in IDATA (the section of fast memory reserved for significant program variables) which was reserved for the frame registers will now be filled with the event queue data (up to 256 events max), which has been compacted into two words per event:

ISR and Data Handling (cont)

• Previously if the fast memory became full, the ISR would dump all frames in fast memory into slow memory, & fix any pointers that need updating as a result. This was an attempt to minimize time spent re-organizing data when receiving large events (the only situation in which an overflow is likely), but resulted in accounting headaches. Additionally, in the unlikely case that an overflow occurred while a set of smaller events were still being processed, there was a performance penalty.

• The new scheme copies an incoming event frame by frame into slow memory once the fast memory fills. Once an event has begun dumping frames into slow memory, it will continue doing so even if fast memory frames become available meanwhile. (Succeeding events, however, can take advantage of the newly available memory).

MDSP Initialization & ROD boards

• The new Revision C boards are significantly better at event handling than the Rev. B boards. On Rev. B boards, the slave DSPs’ clocks cannot be phase-locked due to a fabrication error. This resulted in occasional errors while transferring the data from the router to the DSP. Slave DSP 1 is unusable, and DSP 3 is unreliable.

• On the new boards, slave DSP clocks are in a phase-lock loop with the router. This solves the data transfer problem, with one caveat:

The router FPGA has four global clocks; one is reserved for the router itself, andthree others are distributed to slave DSPs 0, 1 and 2.

DSP 3 shares its clock with DSP 0. The clock can either be in phase or 180o

out of phase. If it is out of phase, router errors are still possible.

MDSP Initialization & ROD boards (cont)

• To correct for this, code has been inserted inside the master DSP’s initialization. The MDSP checks registers in the router indicating the condition of the clock on slave DSP 3 (these are located at 0x0040240C, 4C, 8C and CC for DSP 0 – 3 respectively; if bit 12 is set, the clock is in phase).

If the clock is out of phase then the MDSP attempts to correct this by sending a signalto the rod controller FPGA that it should reset the slave’s clock. Each time this is donethere is a chance that the slave DSP clock will be put in phase; the MDSP will attemptup to 10 times to correct the clock.

If it is still out of phase by the 10th try (normally the clock is corrected by within 1-3attempts), the MDSP will give up and report an error. In this case the DAQ shouldcorrect the clock manually by sending the new resetDsp primitive to the MDSP.Additionally, if at any point slave DSP #3 is reset via VME (without a board reset),this primitive should be run (documentation forthcoming).

As a side benefit, the MDSP also configures the slave DSPs’ EMIF registers uponstartup, so that all areas of memory may be accessed freely via VME; it is no longernecessary to load them manually.

Histogram Control

• Is now designed around the concept of “module groups”, which are a group of modules that act in a similar fashion during a scan, and receive the same trigger. (A module’s group is determined by the data in the corresponding structure in the structure set). Four module groups are definable now (0-3); this will be increased to 8 in the next version.

Configuration Parameters:

slvBits: A bitfield indicating which slaves we are setting up. Bit 0 slave 0, etc.

configSctSet: Indicates which structure set is being used. INIT_MODULE_CONFIG, WORK_MODULE_CONFIG and SPARE_MODULE_CONFIG (primParams.h)

useRangeList: A debugging tool, will be removed soon. Set to FALSE for now

rangeList[2]: A debugging tool, will be removed soon. Ignored if useRangeList = FALSE.

dataSet: Indicates which type of data will be sent out during configuration (MODULE_BASIC, MODULE_TRIM and MODULE_ALL) Will be made automatic in the next version. (primParams.h)

Histogram Control (cont)

scanRegister[2]: The type of variable which will be scanned across during this histogramming run, eg. ST_QTHR. Two range lists are definable (discussed below), the two variables correspond to the different lists.

*dataPtr[2]: Pointers to the two range lists. These must either be pre-loaded into memory (and the pointer supplied here), OR the pointers set to the DEFAULT value (= 0xFFFFFFFF, from utilities.h), and the two lists included in the primitive list (inside the start task primitive, appended immediately after the TASK_STRUCTURES_IN union. The range lists give the floating point values of the scanRegister at each bin; a module group may use either of the two.

groupRangeMap[2]: Bitfields indicating which module groups use which range list.

cmdBuff: Indicates which command buffers (== serial ports) to send configuration data from. Can be either 0, 1 or both (= 2).

groupSPMap[2]: Bitfields indicating which module groups use which serial port.

groupDSPMap[4]: Bitfields indicating which module groups use which that SDSP.

Repetitions: The total number of events to send in a given bin.


nBins: The number of bins in this scan.

bin0: The bin to begin filling in upon start; for use in appending scans.

globalCtrl: Flag indicating whether this scan is under local (MDSP) or remote (SBC) control. Remote control will be implemented using control signals from the SBC to the MDSP which force it to step in sync with all other RODs in the crate.

syncLevel: Used with the global control flag; indicates the needed amount of ROD synchronization. To be defined.


triggerSequence[2]: Two 6 function command lists; list 0 is for serial port 0, list 1 is for port 1.

Trigger Parameters:

Command lists are structures defined in primParams.h: #define N_CMD_LIST_CMDS 6

struct CmdList {UINT16 cmd[N_CMD_LIST_CMDS];UINT32 data[N_CMD_LIST_CMDS];

};

Each cmd can be any of the commands listed in ABCDchip.h (L1 trigger, calibration pulse, delay, etc.) If a command takes data (eg. delay), it is putIn the corresponding data field.

calLineLoop: If set to TRUE, the master DSP will loop over the calibration lines before continuing on to the next bin.

distributionToggle: If set to MODULE_GROUP, the master DSP will set the command line and data link masks appropriately for each module group. Groups are looped over for each event. If set to ROUTER_DISTRIB, all groups receive their triggers simultaneously. This will be removed in the next version, and the distribution will be set using the groupDSPMap varaibles.


incCmd[2], incData[2]: If incData & incCmd are non-zero, the command given by incCmd inside the corresponding trigger sequence list is incremented by incData between bins. For cross-talk studies.

Histogramming Registers

Master DSP Slave DSPs

Histogram CMD/STAT 0 (really status)

current bin.

16 1 7 bits 8 bits

Histogram STAT 0/1: events received

DSP 3

16 16

DSP 2 Histogram STAT 0

event processing time (micro-sec)

errors in bin D P E R

R: ReadyE: Expecting

P: Processing

D: Done

Bit 8

Cal-lineEn

En= cal line enabled.

Error count

Histogram CMD/STAT 1 (really status):Event statistics (to be defined).

0:

DSP 1

16 16

DSP 01:

Histogram CMD/STAT 0:

current bin.

1 1 1 7 bits 8 bits

Cal-lineEn

C= new cal-line, N= New bin, En= cal line enabled.

NC

Histogram CMD/STAT 1: tot # of events

Histogram STAT 1: # of events received

Histogramming Task States

Histogram Control Task Histogram Task

INIT INIT

NEWBIN

• Task variable setup. • Task variable setup.

Slave Polling (all states)

• Copy slave HSTAT 0,1 to local variables. Update own HSTAT 0.

• Zero HSTAT 0,1. Place current bin & set in HCMD.

• Place current bin set, # rep in all slave HCMD.

• Zero out current repetitions

• Create & send configuration command stream. Prepare trigger command stream.

READY

• Wait for #repetitions in HCMD != 0. Then:

• Set bin, histogram set, # rep. this bin; copy bin & set to own HSTAT 0.

• Zero out events received variable.

• HSTAT 0 RDY HI• Check HRDY in slaves.

• HSTAT 0 DONE LO, EXP HI.

• Clear HSTAT 0 events received.

WAITEXP• Wait for all slave HSTAT EXP HI.

Histogram Control Task Histogram Task

PULSING EXPECTING

WAITING

• Wait for the # events received by all slaves == current rep per slave. (= curr. Rep./# slaves). Check for timeout. Then:

• Wait for events to arrive on queue. Then:

• Send out one trigger per slave, update current rep., HCMD. # rep

Histogramming Task States (cont)

• If Current repetitions < # rep. per bin, repeat.

• Wait for all slave HSTAT 0 DONE HI

• Histogram the event

• HSTAT 0 PROC LO.

• HSTAT 0 PROC HI.

• Increment events received, HSTAT 0 events received. HSTAT 0 PROC LO.

• If events received < #rep. this bin, repeat. Else:

• HSTAT 0 EXP LO, DONE HI

• Clear HCMD #repetitions.

• state READY

PREP

• If bins done == # points: state DONE. Else:

• Increment # bins done.

• Increment the data variables of the configuration & trigger lists • state NEWBIN

Event Trap Setup

• Event trap setup is an important primitive central to the event trapping; it is used to coordinate the actions of the slave DSPs and Router.

• Event trap setup should be sent from the host to the master DSP; the master will then set the Router’s registers, and pass a copy of itself onto the slave using the inter-DSP primitive list.

• The router has a set of registers for each slave DSP, and a global register set. Each DSP register set has two (mostly) independent traps. Trap 0 contains some Extra flags which are common to both traps:

Command Register 0 Command Register 1

Trap reset/load/flush Status Register

Trap Match 0Modulus/Remainder 0

Trap Match 1Modulus/Remainder 1

FIFO word count Internal Delay Count

• The Trap reset/load/flush & internal delay count are set by the DSP and generally should not be touched. The status & FIFO word count are read-only.

Global Command/Status Register

• Will be internally generated in next version of the histogram-control task.

Event Trap Setup (cont)• Input Parameters:

slvBits: A bitfield indicating which slaves we are setting up. Bit 0 slave 0, etc.

# of events: The total # of events to trap. Currently should be set to COLLECT_FOREVER (0).

timeout: Timeout for the reply, when the MDSP sends the primitive to a slave. 0x2000

extRouterSetup: Flag indicating that the router is set up independently; if set the master DSP will not write to the router’s registers. Use with care! FALSE

distribute: If set, the master DSP will distribute check that the input modulus (described later) is correct, and will distribute remainders among the input slaves. NO

permitBackPressure: Bit in the global CSR. Should only be applied in calibration mode (!!) Allows the router to apply back pressure to the EFB if large events are coming through and the DSP needs time to move data into SDRAM. (The slave DSP sets the interrupt mask from DSP inside the SR, if so). TRUE

format: Bit inside CR0 which controls the format in which events are sent to the DSP. This is common to trap 0 & 1. High= error format, Low= normal or “S-Link” format. NORMAL

Event Trap Setup (cont)• Input Parameters (cont):

dataMode: Bit inside CR0 which controls the whether the Router (for that DSP) outputs data in event mode (skipping to the end of a frame with a trailer, and writing 0x40ff +length, or in data mode, in which it waits until it has enough data to fill a frame. Common to both traps. FALSE

sLink: Bit in SR0; if set the trap match is overridden and only the modulus & remainder of trap 0 are used to distribute events among DSPs. Common to both traps. FALSE

• Trap Input Parameters:

config: The type of events to trap. If IDLE, trap is unused; otherwise ATLAS, ROD or TIM. If sLink is set, Trap 1 should be IDLE, and trap 0 set to SLINK. ROD

exclusionFlag: If set will trap all events *except* those in the match register. NOTE: for now this is disabled.

match: The type of event for the router to match. Up to 4 copies of ETS, one for each enabled slave DSP. Only difference: DSP 0: match = 0 … DSP 3: match = 3

• Trap Input Parameters (cont):

modulus/remainder: Determines which matched events are trapped. Events are trapped if mod(trap match counter, modulus)= remainder. modulus = 1, remainder = 0.

Event Trap Setup (cont)

• DSP Input Parameters:

releaseFrames: If set, the last task to operate on an event will release its frames as it processes them, freeing up space. FALSE

trapStray/iterLimit: Stray events are events which shouldn’t be there, but were trapped anyway, either through an electronics glitch or operator error. They gum up the works if not disposed of. If trapStray is set, they are put into a special “observation buffer” (after the 7th SDRAM frame buffer). iterLimit determines how long they should be kept around before being removed/sent for observation. Suggestion: trap strays, and set iterLimit to a low number, say 2 or so. Don’t care.

function: Determines what tasks will process the events coming from the trap. Functions are additive: HISTO= 1, TRAP= 2, OCCUPANCY= 4, etc. Used only by the slave DSPs.

Histogram Setup


Parameters:

*base: A pointer to the base of the histograms in memory. Normally set to the default given in memoryPartitions.h: HISTOGRAM_DEFAULT_BASE

rodType: The type of ROD this is: SCT_ROD (primParams.h)

nBins: The total number of bins.

padding[]: Adds delimiters to mark the histogram bins, if the arrangement is in blocks.

arrangement: HISTOGRAM_BLOCK or HISTOGRAM_SLICE. (primParams.h)

dataFormat: The format the data will arrive in. The formatters can be set to send the full hit pattern which arrives from the modules (3 hits), or send only the central hit. HISTOGRAM_CONDENSED or HISTOGRAM_FULL. FULL will place hit data into three separate data sets, located sequentially after one another in memory. (primParams.h)

calLine: Not used anymore; controlled solely by MDSP. Will be removed in next vers.

Histogram Setup (cont)

binSize: The size of the bins: HISTOGRAM_16BIT or HISTOGRAM_32BIT (primParams.h)

validModules[2]: Bitfield indicating which modules are contributing to this slave’s events. Word 0 is for modules 0-31, while the 16 LSB of word 1 are for modules 32-47. Histogram space will be reserved only for modules flagged as valid modules. All other modules’ data if any arrives (by an error) is put into a single module/bin “invalid data” section which is appended to the end of the histograms.

moduleRangeMap[2]: Bitfields indicating which modules use which range list.

dataType[2]: Indicates the SCT variables which correspond to the two range lists.

*xPtr[2]: Pointers to the two range lists. These must either be pre-loaded into memory (and the pointer supplied here), OR the pointers set to the DEFAULT value (= 0xFFFFFFFF, from utilities.h), and the two lists included in the primitive list (inside the histogram setup primitive, appended immediately after the HISTOGRAM_SETUP_IN structure.

Start Event Trapping


• Start event trapping is a simple primitive with no parameters. It is used to tell the slave DSP to trigger the router’s enable trapping flag. This register is connected to the SDSP’s serial port (both ports are configured for general purpose I/O). The state of this register is visible in the router’s status register for that slave DSP.

Start Histogram Task


• Start Task is used to set the histogramming task running on the slave DSPs; this must be done before the histogram control task is started on the MDSP. The histogram task has two parameters: the total # of events to analyze (set to COLLECT_FOREVER (0), as in Event Trap Setup), and a task control flag (set to MASTER_HREG (0) ).

• Event Trap Setup must be run beforehand for this slave DSP.

• Histogram Setup must be run beforehand for this slave DSP.

Documents

ROD DAQ Workshop February 6-7, 2003