The DZERO Level 3 Data Acquistion System
D. Chapin
M. Clements
D. Cutts
S. Mattingly
The DZERO experiment located at Fermilab has recently started RunII with an upgraded detector. The RunII physics program requires the Data Acquisition to readout the detector at a rate of 1 KHz. Events fragments, totaling 250 KB, are readout from approximately 60 front end crates and sent to a particular farm node for Level 3 Trigger processing. A scalable system, capable of complex event routing, has been designed and implemented based on commodity components: VMIC 7750 Single Board Computers for readout, a Cisco 6509 switch for data flow, and close to 100 Linux-based PCs for high-level event filtering. % To satisfy complex event routing requirements a Routing Master software component and control system has been implemented. % An overview of the system is presented, including descriptions of the hardware and the software, with particular attention paid to the event routing software and the control system. Results of almost a year of running will also be described, and a short discussion of scalability as well.
dzero_l3daq.tex
%% ****** Start of file slactemplate.tex ****** %
%%
%%
%%   This file is part of the APS files in the REVTeX 4 distribution.
%%   Version 4.0 of REVTeX, August 2001
%%
%%
%%   Copyright (c) 2001 The American Physical Society.
%%
%%   See the REVTeX 4 README file for restrictions and more information.
%%
%
% This is a template for producing manuscripts for use with REVTEX 4.0
% Copy this file to another name and then work on that file.
% That way, you always have this original template file to use.
%
\documentclass[twocolumn,twoside,slac]{revtex4}
\usepackage{graphicx}
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhead{} % clear all fields
\fancyhead[C]{\it {Computing in High Energy and Nuclear Physics,
La Jolla Ca, March 24-28, 2003}} \fancyhead[RO,LE]{\thepage}
\fancyfoot{} % clear all fields
\fancyfoot[LE,LO]{\bf MOGT002,TUGP010}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}
\renewcommand{\sfdefault}{phv}

\setlength{\textheight}{235mm}
\setlength{\textwidth}{170mm}
\setlength{\topmargin}{-20mm}


% You should use BibTeX and apsrev.bst for references

\bibliographystyle{apsrev}

\begin{document}

%Title of paper
\title{The DZERO Level 3 Data Acquistion System}


% Repeat the \author .. \affiliation  etc. as needed
%
% \affiliation command applies to all authors since the last
% \affiliation command. The \affiliation command should follow the
% other information

\author{D. Chapin, M. Clements, D. Cutts, S. Mattingly}
\affiliation{Department of Physics, Brown University, Providence, RI 02912 USA}
%
\author{B. Angstadt, G. Brooijmans, D. Charak, S. Fuess, A. Kulyavtsev, M. Mulders, D. Petravick, R. Rechenmacher, D. Zhang}
\affiliation{FNAL, Batavia, IL 60510, USA}
%
\author{R. Hauser}
\affiliation{Department of Physics, Michigan State University, East Lansing, MI 48824 USA}
%
\author{P. Padley}
\affiliation{TW Bonner Nuclear Lab, Rice University, Houston, TX 77251 USA}
%
\author{A. Haas, D. Leichtman, G. Watts}
\affiliation{Department of Physics, University of Washington, Seattle, WA 98195, USA}

\begin{abstract}
The DZERO experiment located at Fermilab has recently started RunII with an upgraded detector. The RunII physics program requires the Data Acquisition to readout the detector at a rate of 1 KHz. Events fragments, totaling 250 KB, are readout from approximately 60 front end crates and sent to a particular farm node for Level 3 Trigger processing. 
A scalable system, capable of complex event routing, has been designed 
and implemented based on commodity components: 
VMIC 7750 Single Board Computers for readout, a Cisco 6509 switch for 
data flow, and close to 100 Linux-based PCs for high-level event filtering.

% To satisfy complex event routing requirements a Routing Master software component and control system has been implemented.

% An overview of the system is presented, including descriptions of the hardware and the software, with particular attention paid to the event routing software and the control system. Results of almost a year of running will also be described, and a short discussion of scalability as well. 
\end{abstract}

%\maketitle must follow title, authors, abstract
\maketitle

\thispagestyle{fancy}

% body of paper here - Use proper section commands
% References should be done using the \cite, \ref, and \label commands
% Put \label in argument of \section for cross-referencing
%\section{\label{}}

\section{Introduction}
The Level 3 Data Acquistion System 
(L3DAQ)~\cite{L3DAQ_RT2003_PROCEEDINGS,L3DAQ_RT2003_JOURNAL} 
transports detector component data located in VME readout crates 
to the processing nodes of the Level 3 trigger filtering farm.
At a rate of 1kHz, sixty-three VME crates must be
read out for each event, each containing 1-20~kB of data distributed among VME
modules. The total event size is approximately 250 kilobytes. 

%The system's designed bandwidth is 250MB/s,
%corresponding to an average 250kB event size at a
%trigger accept rate of 1kHz.

As shown in figure \ref{fig:hardware}, the system is built around 
a single Cisco 6509\cite{CISCO} ethernet switch.
A schematic of the communication and data flow in the system is shown
in figure~\ref{fig:dataflow}. All nodes in the system are based on 
commodity computers and run the Linux operating system.
TCP/IP sockets implemented via the ACE~\cite{ACE} C++ network and
utility library
are used for all communication and data transfers.

\section{Operation}
%Sixty-three VME crates must be
%read out for each event, each containing 1-20~kB of data distributed among VME
%modules. 
The Supervisor process provides the interface between the 
main D0 run control (COOR) and the L3DAQ system. 
When a new 
run is configured, the Supervisor passes run and general
trigger information to the RM and passes the COOR-provided L3 filter 
configuration to the IO/EVB process on relevant farm nodes, 
where it is cached and passed on to the Level 3 filter processes.

A single-board computer (SBC) in each VME crate reads out the VME
modules and sends the data to one or more farm nodes
specified by routing instructions received from the Routing Master (RM)
process. An Event Builder (EVB) process on each farm node builds a complete
event from the received event fragments and makes it available to 
Level 3 trigger filter processes.

The SBCs are 
%VMIC 7750~\cite{VMIC} 
single-board computers with
dual 100~Mb/s Ethernet interfaces and a 
%Tundra UniverseII~\cite{Tundra} 
VME-to-PCI interface.
An expansion slot is occupied by a 
%BVM~\cite{BVM}
digital-I/O (DIO) module, 
used to coordinate the readout of VME modules over the VME user (J3) backplane.
A custom kernel driver on the SBC handles
interrupt requests from the DIO module which are triggered by
readout requests from the crate-specific electronics.
On each readout request the kernel module 
%instructs the UniverseII to 
performs the VME data transfers and stores the event
fragment in one 
of several buffers in kernel-memory.

A user-level process on the SBC receives  
route information from the Routing Master 
in the form of Route Tags that contain a
unique event identifier (L3 transfer number) and the indices of
the farm nodes to
which that event should be sent. %The process matches
If the Route Tag's L3 transfer number matches that of the transfer number 
embedded within the head event fragment in the kernel buffers, the event
fragment is sent to the specified farm nodes. 
%The user-level process alternates 
%sending between the two SBC ethernet interfaces, taking advantage
%of the full 25MB/s of available ethernet bandwidth.

%Several crates in the system, have an average event fragment 
%size that is large enough to saturate the
%12.5MB/s bandwidth limit of a single 100Mb ethernet interface at
%an event rate of 1kHz. In these crates, the user-level process on 
%the SBC alternates 
%sending between the two SBC ethernet interfaces, resulting in 
%an emprical bandwitdh limit of approximately 22MB/s.

%If there is a match, the user-level
%process sends the buffer's data to the specified farm nodes.
%In the case of an L3 transfer number mismatch, 
%either the event data or the Route Tag is discarded (based on which 
%L3 transfer number was behind), and a new match is attempted.

%A custom extender board is used to mechanically support the 6U commercial 
%SBCs in the 9U readout crates, propagate signals to the panel at the 
%front of the VME crate, and connect lines on the digital I/O card to 
%the J3 backplane. The SBCs run the Linux operating system, with 
%minor custom modifications to the kernel. 
%A relatively small (33MB) custom Linux distribution 
%resides on the flash memory or in a RAM disk if the SBC is net-booted. 

%Readout crate electronics (specific to the crate type) inform the 
%SBC that an event fragment is ready for reading by asserting a line (SLVDRY)
%on the J3 VME backplane. 
%triggered by changes on the SLVRDY line. 

%Upon completion of the VME data transfer, the kernel module informs the 
%crate electronics that it is ready to transfer another event fragment by
%asserting a second line (DONE) on the VME J3 connector.

An Event Builder process (EVB) on each farm node
collates the event fragments received from SBCs
into complete events, keyed by L3 transfer number. 
For each event the EVB receives an
expected-crate list from the RM in order to determine when
an event is complete. 
Complete events
are placed in shared memory event buffers 
%(approximately 20) 
for processing by the Level 3 filter processes.
%Incomplete events (those that are not 
%complete within one second) are discarded. 
The EVB routinely informs the RM of the number of
available event buffers that it has, so that the RM
will not route an event to a farmnode unless the
event can be processed immediately.

The Routing Master program executes on an SBC in a special VME crate
which contains a hardware interface to the D0 trigger framework (TFW).
The TFW provides trigger information and the L3 transfer number for
each event and allows the RM to asynchronously disable the firing
of triggers.
For each event the RM program chooses a node for processing based on 
the run configuration, 
the trigger information, and the number of available buffers 
in the set of nodes configured to process the type of event.  
A node is chosen in a round-robin fashion from amongst the set of nodes
with the most free buffers.
If the number of available free buffers is too low, the RM instructs the
TFW to disable triggers so that the farm nodes have time to catch up.

To avoid dropped packets in the main switch, data flow 
is limited by setting the TCP/IP receive window size 
on the farmnodes. The window size is set such that
the product of the number of connection sources and
the receive window size is smaller than the switch's 
per-port output buffer memory.


\section{Conclusion}
The DZERO Level 3 data acquistion system has been built from 
commercially available hardware: single board VME computers, ethernet
switches, and PCs. The software components rely upon
high level programming languages; the Linux operating system;
widely-used, open libraries; and standard networking protocols.
The system has performed reliably since commissioning in May 2002.
Additional details are availble from the 
references~\cite{L3DAQ_RT2003_JOURNAL}.



\begin{figure*}
%\begin{figure}[t]
\centering
\includegraphics[height=6.0in,angle=-90]{l3daq_network2.eps}
\caption{The physical network configuration of the L3DAQ system.}
\label{fig:hardware}
\end{figure*}

\begin{figure*}
\centering
\includegraphics[height=5.6in,angle=-90]{l3daq_flow.eps}
\caption{Schematic illustration of the information and 
dataflow through the L3DAQ system.}
\label{fig:dataflow}
\end{figure*}



%\begin{figure}
%\includegraphics[width=65mm]{JACpic_mc.eps}
%\caption{Layout of papers.}
%\label{l2ea4-f1}
%\end{figure}

% Create the reference section using BibTeX:
%\bibliography{basename of .bib file}
\begin{thebibliography}{9}   % Use for  1-9  references
%\begin{thebibliography}{99} % Use for 10-99 references

\bibitem{L3DAQ_RT2003_PROCEEDINGS} Proceedings of the IEEE 
NPSS Real Time Conference, Montreal, Canada, May 2003, RT-105.
\bibitem{L3DAQ_RT2003_JOURNAL} ``The DZERO Level 3 Data Acquistion System'', 
To be published in \emph{IEEE Transactions on Nuclear Science}.
\bibitem{CISCO} http://www.cisco.com/
\bibitem{ACE} http://www.cs.wustl.edu/\symbol{"7E}schmidt/ACE.html

\end{thebibliography}


\end{document}
%
% ****** End of file template.aps ******

physics0307070.extracted_bib
\begin{thebibliography}{9}   % Use for  1-9  references
%\begin{thebibliography}{99} % Use for 10-99 references

\bibitem{L3DAQ_RT2003_PROCEEDINGS} Proceedings of the IEEE 
NPSS Real Time Conference, Montreal, Canada, May 2003, RT-105.
\bibitem{L3DAQ_RT2003_JOURNAL} ``The DZERO Level 3 Data Acquistion System'', 
To be published in \emph{IEEE Transactions on Nuclear Science}.
\bibitem{CISCO} http://www.cisco.com/
\bibitem{ACE} http://www.cs.wustl.edu/\symbol{"7E}schmidt/ACE.html

\end{thebibliography}