Control and monitoring of on-line Control and monitoring of on-line trigger algorithms using a SCADA trigger algorithms using a SCADA

system system

Eric van HerwijnenEric van Herwijnen

Wednesday 15Wednesday 15thth February 2006 February 2006

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 2

ContentsContents Configuration of the LHCb DAQ Configuration of the LHCb DAQ The ProblemThe Problem A solutionA solution Gaucho architectureGaucho architecture PVSS backendPVSS backend PVSS performancePVSS performance RootRoot SummarySummary

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 3

Configuration of the LHCb DAQConfiguration of the LHCb DAQ

ECS

switch

CPUFarm

Readout Network

Front-end ElectronicsTFC

System

StorageSystem

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU50 Racks~1800 boxes~3600 CPU’s

FE FE FE FE FEFE

SwitchSwitchSwitch Switch

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 4

The problemThe problem

Control and monitor trigger (Gaudi) processes Control and monitor trigger (Gaudi) processes on Event Filter Farmon Event Filter Farm

Send monitoring data (counters, rates, Send monitoring data (counters, rates, histograms, status, error messages) to ECShistograms, status, error messages) to ECS

Sum monitoring informationSum monitoring information Tag summed information and make available Tag summed information and make available

for further analysisfor further analysis

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 5

A solutionA solution SCADA=Supervisory Control And Data AcquisitionSCADA=Supervisory Control And Data Acquisition LHCb’s Experiment Control System uses PVSSLHCb’s Experiment Control System uses PVSS PVSS for displaying counters, ratesPVSS for displaying counters, rates Root for displaying/manipulating histogramsRoot for displaying/manipulating histograms Histograms and counters accessed (e.g. via DIM)Histograms and counters accessed (e.g. via DIM) Automatic addition, display summed resultsAutomatic addition, display summed results

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 6

Gaucho (c++)Gaucho (c++) GAUdi Component for Helping OnlineGAUdi Component for Helping Online MonitorSvc to publish objectsMonitorSvc to publish objects OnlineMessageSvc publishes messagesOnlineMessageSvc publishes messages A status variable publishes the state of the job (ready, A status variable publishes the state of the job (ready,

running, stopped)running, stopped) Used by current LHCb trigger algorithm authorsUsed by current LHCb trigger algorithm authors Information only sent when subscribed to, no Information only sent when subscribed to, no

performance penalty performance penalty Current MonitorSvc/OnlineMessageSvc implemented Current MonitorSvc/OnlineMessageSvc implemented

using DIM, but this could be something elseusing DIM, but this could be something else

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 7

PVSS backendPVSS backend DIM clients subscribe to counters and histogramsDIM clients subscribe to counters and histograms Data viewed per job, summed per node, summed per Data viewed per job, summed per node, summed per

subfarmsubfarm Summed histograms are published by PVSS via DIM Summed histograms are published by PVSS via DIM

(PVSS acts as DIM server)(PVSS acts as DIM server)

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 8

PVSS backendPVSS backend Tree tier panel hierarchy: per subfarm, per node, per jobTree tier panel hierarchy: per subfarm, per node, per job

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 9

Size of monitoring dataSize of monitoring dataItemItem Test algorithmTest algorithm Typical Trigger algorithmTypical Trigger algorithm

# strings# strings 1 status, 5 comments1 status, 5 comments 1 status, 3 comments1 status, 3 comments

bytes for stringsbytes for strings 228228 188188

# counters# counters 4 ints, 1 long4 ints, 1 long 3 ints3 ints

bytes for countersbytes for counters 2020 1212

# histograms # histograms (nbins)(nbins)

4 1D (5,80,60,60), 1 4 1D (5,80,60,60), 1 2D (100)2D (100)

24 1D 24 1D (4x10,1X11,1X20,2x40,4x(4x10,1X11,1X20,2x40,4x150,12x80), 2 2D (200)150,12x80), 2 2D (200)

bytes for bytes for histogramshistograms

12201220 1004410044

Total #kbTotal #kb 1.51.5 10.510.5

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 10

Dataflow into PVSSDataflow into PVSS Total for 40 nodes, 4 Trigger jobs, 2 Test job/node: 1.6 Total for 40 nodes, 4 Trigger jobs, 2 Test job/node: 1.6

Mb every 20 secs (4 kbytes for counters, rest Mb every 20 secs (4 kbytes for counters, rest histograms)histograms)

Well within capacity of network (12 Mb/s=100 Mbit/s)Well within capacity of network (12 Mb/s=100 Mbit/s)

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 11

PVSS performancePVSS performance Tests over 40 nodes (240 jobs)Tests over 40 nodes (240 jobs) PVSS on Windows Xeon 3GHz CPU, 2 GB RAMPVSS on Windows Xeon 3GHz CPU, 2 GB RAM Counters: implemented via “datapoint functions”Counters: implemented via “datapoint functions” Updated and summed every 20 secsUpdated and summed every 20 secs

3-12 % CPU usage, 700 Mb memory

Histograms: updated and summed sequentially, once Histograms: updated and summed sequentially, once every 4 minutesevery 4 minutes

5-55 % CPU usage, 700 Mb memory

Memory use increases with time if update frequency Memory use increases with time if update frequency increasesincreases

PVSS can not display 2D histogramsPVSS can not display 2D histograms

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 12

RootRoot PVSS is good for displaying counters; need a more PVSS is good for displaying counters; need a more

scalable solution for manipulating histogramsscalable solution for manipulating histograms We plan to create stand alone “histogram adders”We plan to create stand alone “histogram adders”

finds out from PVSS which histograms to add (configuration) subscribes directly to Gaudi jobs publishes added histograms to clients (viewer, analysis programs)

A stand alone histogram viewer will allow selection of A stand alone histogram viewer will allow selection of histograms and displayhistograms and display

Possibly a mixture of PVSS and Root (Next release of PVSS uses QT)

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 13

Current root viewerCurrent root viewer A root program implements a Dim client to display 1D A root program implements a Dim client to display 1D

and 2D histogramsand 2D histograms Viewer needs to be closed when viewing different Viewer needs to be closed when viewing different

histogramshistograms Data can be saved on demand or at regular intervals in Data can be saved on demand or at regular intervals in

text files (counters) or root files (histograms)text files (counters) or root files (histograms)

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 14

Root ViewerRoot Viewer

Wed 15th February 2006 Control and monitoring of trigger algorithms using Gaucho 15

SummarySummary Can use PVSS to monitor countersCan use PVSS to monitor counters For a more scalable system, move manipulation of For a more scalable system, move manipulation of

histograms outside of PVSShistograms outside of PVSS Adders connect directly to Gaudi jobs, controlled by Adders connect directly to Gaudi jobs, controlled by

PVSS, send results for further analysis and feed the PVSS, send results for further analysis and feed the viewerviewer

A separate histogram viewer is planned in Root/PVSS A separate histogram viewer is planned in Root/PVSS which will allow easy viewing/saving of histogramswhich will allow easy viewing/saving of histograms

Build a histogram database to store details of Build a histogram database to store details of histograms for quality analysishistograms for quality analysis