Improving the Security and Performance of the BaBar Detector Controls System
K. D. Kotturi
It starts out innocently enough - users want to monitor Online data and so run their own copies of the detector control GUIs in their offices and at home. But over time, the number of processes making requests for values to display on GUIs, webpages and stripcharts can grow, and affect the performance of an Input/Output Controller (IOC) such that it is unable to respond to requests from requests critical to data-taking. At worst, an IOC can hang, its CPU having been allocated 100\% to responding to network requests. For the BaBar Online Detector Control System, we were able to eliminate this problem and make great gains in security by moving all of the IOCs to a non-routed, virtual LAN and by enlisting a workstation with two network interface cards to act as the interface between the virtual LAN and the public BaBar network. On the interface machine, we run the Experimental Physics Industrial Control System (EPICS) Channel Access (CA) gateway software (originating from Advanced Photon Source). This software accepts as inputs, all the channels which are loaded into the EPICS databases on all the IOCs. It polls them to update its copy of the values. It answers requests from applications by sending them the currently cached value. We adopted the requirement that data-taking would be independent of the gateway, so that, in the event of a gateway failure, data-taking would be uninterrupted. In this way, we avoided introducing any new risk elements to data-taking. Security rules already in use by the IOC were propagated to the gateway's own security rules and the security of the IOCs themselves was improved by removing them from the public BaBar network.
kdk.tex
%%
%%
%%   This file is part of the APS files in the REVTeX 4 distribution.
%%   Version 4.0 of REVTeX, August 2001
%%
%%
%%   Copyright (c) 2001 The American Physical Society.
%%
%%   See the REVTeX 4 README file for restrictions and more information.
%%
%
% This is a template for producing manuscripts for use with REVTEX 4.0
% Copy this file to another name and then work on that file.
% That way, you always have this original template file to use.
%
\documentclass[twocolumn,twoside,slac]{revtex4}
\usepackage{graphicx}
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhead{} % clear all fields
\fancyhead[C]{\it {Computing in High Energy and Nuclear Physics, La Jolla, California, March 24-28, 2003}} \fancyhead[RO,LE]{\thepage}
\fancyfoot{} % clear all fields
\fancyfoot[LE,LO]{\bf TUJP002}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}
\renewcommand{\sfdefault}{phv}

\setlength{\textheight}{235mm}
\setlength{\textwidth}{170mm}
\setlength{\topmargin}{-20mm}


% You should use BibTeX and apsrev.bst for references

\bibliographystyle{apsrev}

\begin{document}

\title{Improving the Security and Performance of the BaBar Detector Controls System}


% Repeat the \author .. \affiliation  etc. as needed
%
% \affiliation command applies to all authors since the last
% \affiliation command. The \affiliation command should follow the
% other information

\author{K. D. Kotturi}
\affiliation{For the BaBar Computing Group, SLAC, Stanford, CA 94025, USA}

\begin{abstract}
It starts out innocently enough - users want to monitor Online data and so run
their own copies of the detector control GUIs in their offices and at home. But
over time, the number of processes making requests for values to display on
GUIs, webpages and stripcharts can grow, and affect the performance of an
Input/Output Controller (IOC) such that it is unable to respond to requests
from requests critical to data-taking. At worst, an IOC can hang, its CPU
having been allocated 100\% to responding to network requests.

For the BaBar Online Detector Control System, we were able to eliminate this
problem and make great gains in security by moving all of the IOCs to a
non-routed, virtual LAN and by enlisting a workstation with two network
interface cards to act as the interface between the virtual LAN and the public
BaBar network. On the interface machine, we run the Experimental Physics
Industrial Control System (EPICS) Channel Access (CA) gateway software
(originating from Advanced Photon Source). This software accepts as inputs, all the channels which
are loaded into the EPICS databases on all the IOCs. It polls them to update
its copy of the values. It answers requests from applications by sending them
the currently cached value.

We adopted the requirement that data-taking would be independent of the
gateway, so that, in the event of a gateway failure, data-taking would be
uninterrupted. In this way, we avoided introducing any new risk elements to
data-taking.  Security rules already in use by the IOC were propagated to the
gateway's own security rules and the security of the IOCs themselves was
improved by removing them from the public BaBar network.

\end{abstract}

%\maketitle must follow title, authors, abstract
\maketitle

\thispagestyle{fancy}

% body of paper here - Use proper section commands
% References should be done using the \cite, \ref, and \label commands
% Put \label in argument of \section for cross-referencing
%\section{\label{}}
%\footnote{http://www.slac.stanford.edu/econf/}

\section{Introduction}

This paper describes the motivation behind and implementation of 
a CA Gateway \cite{gw-ref} in the BaBar Online Detector Controls System. 

\section{Motivation}

With the IOCs on the BaBar public network, there were no limits on the
number of clients (aside from the limit of 384 ssh sessions) that could connect and request values. 
These clients could run on any of the 63 workstations on the BaBar
public network. Remote access to the BaBar public network was possible
via ssh.

As a result, the number of
open file descriptors on an IOC could exceed the 150 limit (already increased
significantly from the default 50) and the CPU usage of an IOC could go 
as high as 100%. 

\section{Implementation}

The IOCs were moved to a private network and the four servers running EPICS
client software that were essential for data-taking were given a 
secondary network interface so that they could access the private network. 
CA gateway software was installed on one of these servers, "bbr-srv01". 
Statistics of the gateway are displayed instantaneously in Figure~\ref{stats} and over
time in Figure~\ref{graph}. 

\begin{figure}
\includegraphics[width=65mm]{gwStats.eps}
\caption{CA gateway statistics.}
\label{stats}
\end{figure}

\begin{figure}
\includegraphics[width=65mm]{gwStriptool.eps}
\caption{CA gateway statistics as a function of time.}
\label{graph}
\end{figure}

The gateway acts as a client to the IOCs, requesting channels and receiving replies at the rate of approximately 285 channels per sec (Figure~\ref{stats}). 

The gateway also acts as a server, providing a cached superset of all the IOCs' values to public EPICS clients. 
In Figure~\ref{stats}, we see that the number of channels active in the current cache is 15692 and the number of channels that will be held in the cache for up to two more hours is 17854. 

The server is receiving requests and responding to them at the rate of 1343.56 channels per second. The software was built without the flags to monitor the number of file descriptors in use, so this explains "FDs: 0".

Plotted against time (Figure~\ref{graph}), we see that the number of alive and active channels is near constant and that there are small fluctuations in the event and post rates.
 
The dual network interface card (NIC) servers required a special EPICS configuration to avoid seeing ambiguous replies to their request for data (since they could access both the IOCs directly and the CA gateway for values). 

The BaBar private network is not routed, so IOCs are hidden from the
public internet address space.
 
IOCs use security access files to define access security groups and rules
to specify which users can carry out which actions. For example, the
access security group ``dchexpert'' may contain a list of users who are
allowed to turn on the high voltage for the drift chamber.
A security access file for the gateway was constructed from the sum of the individual
IOC access security files. The userid running the gateway processes had 
to be added to the individual IOC's security rules since requests to do secure
transactions at the IOC level were being made by the gateway process
and not the expert user, if initiated from the public network.

\section{Results}

The number of file descriptors decreased by 25\% and the CPU usage
decreased by 20--40\%, on average for the 17 IOCs. There were fewer
IOC hangs/disconnects which helped to improve BaBar's data-taking
efficiency. 

Internal IOC security is maintained by propagating the IOC security access
definitions to the gateway security definitions. External IOC security
is improved since IOCs are no longer publicly visible. 

%\subsection{Acknowledgments} \label{Ack}
%Place acknowledgments including required mentions of the contract numbers in the Acknowledgements Section. See also
%Section~\ref{Fnotes}.

% Create the reference section using BibTeX:
%\bibliography{basename of .bib file}
\begin{thebibliography}{9}   % Use for  1-9  references

\bibitem{gw-ref}
http://www-csr.bessy.de/control/SoftDist/Gateway

\end{thebibliography}

\section{Appendix}
The poster displayed at CHEP 2003, in two halves (Figures~\ref{top} and
~\ref{bot}).
\begin{turnpage}
\begin{figure*}
%\centering
\includegraphics[width=240mm]{CHEP-top.eps}
\caption{CHEP03 poster.}
\label{top}
\end{figure*}
\end{turnpage}

\begin{turnpage}
\begin{figure*}
%\centering
\includegraphics[width=240mm]{CHEP-bottom.eps}
\caption{CHEP03 poster contd.}
\label{bot}
\end{figure*}
\end{turnpage}

\end{document}

cs0307026.extracted_bib
\begin{thebibliography}{9}   % Use for  1-9  references

\bibitem{gw-ref}
http://www-csr.bessy.de/control/SoftDist/Gateway

\end{thebibliography}