A Breakthrough in Astronomical Software Development
Raymond Plante
Doug Roberts
% R.\ M.\ Crutcher\altaffilmark{1}
% Leave intact % Place the text of your abstract here VIMOS main distinguishing characteristic is its very high multiplex capability: in MOS mode up to 800 spectra can be acquired simultaneously, while the Integral Field Unit produces 6400 spectra to obtain integral field spectroscopy of an area approximately 1$\times$1 arcmin in size. To successfully exploit the capabilities of such an instrument it is necessary to expedite as much as possible the analysis of the very large volume of data that it will produce, automating almost completely the basic data reduction and the related bookkeeping process. The VIMOS Data Reduction Software (DRS) has been designed specifically to satisfy these two requirements. A complete automation is achieved using a series of auxiliary tables that store all the input information needed by the data reduction procedures, and all the output information that they produce. We expect to achieve a satisfactory data reduction for more than 90\% of the input spectra, while some level of human intervention might be required for a small fraction of them to complete the data reduction. The DRS procedures can be used as a stand-alone package, but are also being incorporated within the VIMOS pipeline under development at the European Southern Observatory.
P1-25.tex
%ADASS_PROCEEDINGS_FORM%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% TEMPLATE.TEX -- ADASS VIII (98) ASP Conference Proceedings template
% Updated by N. Manset for ADASS IX (99) and F. Primini for ADASS 2000.
%
% Use this template to create your proceedings paper in LaTeX format
% by following the instructions given below.  Much of the input will
% be enclosed by braces (i.e., { }).  The percent sign, "%", denotes
% the start of a comment; text after it will be ignored by LaTeX.  
% You might also notice in some of the examples below the use of "\ "
% after a period; this prevents LaTeX from interpreting the period as
% the end of a sentence and putting extra space after it.  
% 
% You should check your paper by processing it with LaTeX.  For
% details about how to run LaTeX as well as how to print out the User
% Guide, consult the README file.  (The User Guide is also available
% on-line via http://hea-www.harvard.edu/ADASS/ in the Proceedings
% section) You should also consult the sample LaTeX papers,
% sample1.tex and sample2.tex, for examples of including figures, html
% links, special symbols, and other advanced features.
%
% If you do not have access to the LaTeX software or a laser printer
% at your site, you can still prepare your paper following the
% instructions in the User Guide.  In such cases, the editors will
% process the file and make any necessary editorial adjustments.
% 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% 
\documentstyle[11pt,adassconf]{article}  % Leave intact

\begin{document}   % Leave intact

%-----------------------------------------------------------------------
%			    Paper ID Code
%-----------------------------------------------------------------------
% Enter the proper paper identification code.  The ID code for your
% paper is the session number associated with your presentation as
% published in the official ADASS 2000 conference program.  You can
% find this number locating your abstract in the printed program
% that you received at the meeting or on-line via
% http://hea-www.harvard.edu/ADASS/; the ID code is the letter-number
% sequence proceeding the title of your presentation.
%
% This will not appear in your paper; however, it allows different
% papers in the proceedings to cross-reference each other.
%
% EXAMPLE: \paperID{O1-03}
% EXAMPLE: \paperID{P2-56}
%
% Note that you should only have one \paperID, and it should not
% include a trailing period.  

\paperID{P1-25}

%-----------------------------------------------------------------------
%		            Paper Title 
%-----------------------------------------------------------------------
% Enter the title of the paper.
%
% EXAMPLE: \title{A Breakthrough in Astronomical Software Development}
% 

\title{Automated spectral extraction for high multiplexing MOS and IFU
	observations}

%-----------------------------------------------------------------------
%		          Authors of Paper
%-----------------------------------------------------------------------
% Enter the authors followed by their affiliations.  The \author and
% \affil commands may appear multiple times as necessary (see example
% below).  List each author by giving the first name or initials first
% followed by the last name.  Authors with the same affiliations
% should grouped together. 
%
% EXAMPLE: \author{Raymond Plante, Doug Roberts, 
%                  R.\ M.\ Crutcher\altaffilmark{1}}
%          \affil{National Center for Supercomputing Applications, 
%                 University of Illinois Urbana-Champaign, Urbana, IL
%                 61801}
%          \author{Tom Troland}
%          \affil{University of Kentucky}
%
%          \altaffiltext{1}{Astronomy Department, UIUC}
%
% In this example, the first three authors, "Plante", "Roberts", and
% "Crutcher" are affiliated with "NCSA".  "Crutcher" has an alternate 
% affiliation with the "Astronomy Department".  The fourth author,
% "Troland", is affiliated with "University of Kentucky"

\author{Marco Scodeggio\altaffilmark{1}, Alessandra Zanichelli, Bianca Garilli}
\affil{IFC--CNR, Milano, Italy}
\author{Olivier Le F{\`e}vre}
\affil{LAM--CNRS, Marseille, France}
\author{Giampaolo Vettolani}
\affil{IRA--CNR, Bologna, Italy}
\altaffiltext{1}{On behalf of the VIRMOS Consortium}
%-----------------------------------------------------------------------
%			 Contact Information
%-----------------------------------------------------------------------
% This information will not appear in the paper but will be used by
% the editors in case you need to be contacted concerning your
% submission.  Enter your name as the contact along with your email
% address.
% 
% EXAMPLE:  \contact{Dennis Crabtree}
%           \email{crabtree@cfht.hawaii.edu}
%

\contact{Marco Scodeggio}
\email{marcos@ifctr.mi.cnr.it}

%-----------------------------------------------------------------------
%		      Author Index Specification
%-----------------------------------------------------------------------
% Specify how each author name should appear in the author index.  The 
% \paindex{ } should be used to indicate the primary author, and the
% \aindex for all other co-authors.  You MUST use the following
% syntax: 
%
% SYNTAX:  \aindex{LASTNAME, F. M.}
% 
% where F is the first initial and M is the second initial (if
% used).  This guarantees that authors that appear in multiple papers
% will appear only once in the author index.  
%
% EXAMPLE: \paindex{Crabtree, D.}
%          \aindex{Manset, N.}        
%          \aindex{Veillet, C.}        

\paindex{Scodeggio, M.}
\aindex{Zanichelli, A.}     % Remove this line if there is only one author
\aindex{Garilli, B.}
\aindex{Le F{\`e}vre, O.}
\aindex{Vettolani, G.}
%-----------------------------------------------------------------------
%			Subject Index keywords
%-----------------------------------------------------------------------
% Enter up to 6 keywords describing your paper.  These will NOT be
% printed as part of your paper; however, they will be used to
% generate the subject index for the proceedings.  There is no
% standard list; however, you can consult the indices for past ADASS
% proceedings (http://iraf.noao.edu/ADASS/adass.html). 
%
% EXAMPLE:  \keywords{visualization, astronomy: radio, parallel
%                     computing, AIPS++, Galactic Center}
%
% In this example, the author noticed that "radio astronomy" appeared
% in the ADASS VII Index as "astronomy" being the major keyword and
% "radio" as the minor keyword.

\keywords{data: reduction, spectroscopy}

%-----------------------------------------------------------------------
%			       Abstract
%-----------------------------------------------------------------------
% Type abstract in the space below.  Consult the User Guide and Latex
% Information file (http://hea-www.harvard.edu/ADASS/ in the
% Proceedings section) for a list of supported macros (e.g. for
% typesetting special symbols). 

\begin{abstract}          % Leave intact
% Place the text of your abstract here 
VIMOS main distinguishing characteristic is its very high multiplex
capability: in MOS mode up to 800 spectra can be acquired
simultaneously, while the Integral Field Unit produces 6400 spectra to
obtain integral field spectroscopy of an area approximately 1$\times$1
arcmin in size.  To successfully exploit the capabilities of such an
instrument it is necessary to expedite as much as possible the
analysis of the very large volume of data that it will produce,
automating almost completely the basic data reduction and
the related bookkeeping process.
The VIMOS Data Reduction Software (DRS) has been designed specifically
to satisfy these two requirements. A complete automation is achieved
using a series of auxiliary tables that store all the input
information needed by the data reduction procedures, and all the
output information that they produce.  We expect to achieve a
satisfactory data reduction for more than 90\% of the input spectra,
while some level of human intervention might be required for a small
fraction of them to complete the data reduction.
The DRS procedures can be used as a stand-alone package, but are also
being incorporated within the VIMOS pipeline under development at the
European Southern Observatory.  

\end{abstract}

%-----------------------------------------------------------------------
%			      Main Body
%-----------------------------------------------------------------------
% Place the text for the main body of the paper here.  You should use
% the \section command to label the various sections; use of
% \subsection is optional.  Significant words in section titles should
% be capitalized.  Sections and subsections will be numbered
% automatically. 
%
% EXAMPLE:  \section{Introduction}
%           ...
%           \subsection{Our View of the World}
%           ...
%           \section{A New Approach}
%
% It is recommended that you look at the sample papers, sample1.tex
% and sample2.tex, for examples for formatting references, footnotes,
% figures, equations, html links, lists, and other special features.  

\section{Introduction}

VIMOS is the first of a pair of imaging spectrographs that are being
built for the unit telescopes of the European Southern Observatory
Very Large Telescope. The instrument field of view is split into four
separate quadrants, each one covering approximately $7\times8$
arcmin. On one side of the instrument is anchored the head of the
Integral Field Unit (IFU), consisting of a lenslet array of 6400
lenslets organized in an $80\times80$ array that covers a field of
view of $54\times54$ arcsec. The light collected by the IFU head is
fed to the main spectrograph via optical fibers.

This instrument was designed specifically to carry out survey work,
and to have a very high multiplexing capability. Because of this, it
will also require a rather new approach to the process of data
reduction: it will be simply impossible to reduce by hand the very
large amount of data that VIMOS will produce, as the following example
demonstrates. Working ``by hand'' using IRAF (or a similar package),
even with a number of {\it ad hoc} scripts designed to carry out
repetitive tasks, an astronomer could reduce one spectrum in 3 to 5
minutes. As VIMOS in Multi Object Spectrograph (MOS) mode will produce
between 150 and 200 spectra per quadrant for each exposure, and it has
4 quadrants, it would then take between 30 and 70 hours of work to
reduce a single exposure. Therefore, to reduce just one night of
observations ``by hand'' it would require between 300 and 700 hours of
work, which is 40 to 150 working days (8 hours a day, doing nothing
else)!! With IFU observations, each one producing 6400 spectra, things
would obviously be even worse.

For this reason it was decided to implement a completely automatic
data reduction pipeline for the processing of VIMOS data. As part of
the agreement for the construction of the instrument, the VIRMOS
consortium is developing the core components of this pipeline, what we
call here the VIMOS Data Reduction Software (DRS). The European
Southern Observatory is in charge of putting this core component into
a completely automatic pipeline within the framework of their Data
Flow System. Although the DRS has been designed specifically for
VIMOS, its conceptual lay-out is general enough that it could be
easily adapted to any kind of instrument with capabilities comparable
to those of VIMOS.

\section{Instrument and individual mask calibrations}

It is very difficult to design completely automated tasks to work
blindly at the reduction of complex datasets like those produced by
VIMOS. It was thus decided that the DRS will {\it always and only}
work starting from a ``reasonable'' first guess about all calibration
parameters of the instrument. This will require the setup of an
instrument calibration database, and the periodical execution of
calibration procedures that will produce the necessary calibration
data. When reduced using DRS procedures, these data will produce:

\noindent\hangindent 15pt 1) a mapping between positions on the
slitlets mask and positions on the CCD frame where the slitlets images
are recorded;

\noindent\hangindent 15pt 2) a mapping of the distortions that affect
each individual spectrum;

\noindent\hangindent 15pt 3) a mapping of the wavelength dispersion
solution, that associates a wavelength to each pixel coordinate of a
spectrum image;

\noindent\hangindent 15pt 4) two mappings between celestial
coordinates and coordinates in the plane of the slitlets mask and in
the plane of the CCD frame.

\noindent
These mappings, generally in the form of second to fourth order
spatial polynomials, are stored in calibration matrices coefficients
inside FITS binary tables.

\begin{figure}
\epsscale{0.5}
\plotone{P1-25a.eps}
%\plotfiddle{P1-25a.eps}{9truecm}{0}{50}{50}{-120}{-100}
\caption{A block diagram scheme of the DRS spectral extraction process}
\end{figure}

%\section{Individual mask calibration}

Starting from the general instrument calibrations, specific
calibrations are obtained for each specific mask used for an
observation (see Figure 1):

\noindent\hangindent 15pt 1) the Mask Preparation Software used by the
astronomer to define the slitlets positions for MOS observations (see
Bottini, Garilli \& Tresse 2000) produces an Aperture Definition File
(ADF), containing the positions of all slitlets in the mask plane;

\noindent\hangindent 15pt 2) the ADF is inserted into the FITS file
header at the end of the observation (for IFU observations, where the
ADF is always the same, this is stored in a separate table);

\noindent\hangindent 15pt 3) using the slit positions derived from the
ADF, and the general calibration matrices, the approximate location of
each two-dimensional spectrum on the CCD frame is computed;

\noindent\hangindent 15pt 4) a flat field exposure is used to measure
the exact location of each two-dimensional spectrum on the CCD frame;

\noindent\hangindent 15pt 5) a calibration lamp exposure is used to
measure the position of a number of spectral lines and refine the
approximate wavelength solution provided by the general calibration matrix.


\section{Spectral Extraction}

The final result of the calibration phase is the complete definition
of a set of apertures to be used for the final spectral
extraction. The extraction itself is carried out following two
different procedures for MOS and IFU observations.

For observations carried out in MOS mode, it is assumed that slitlets
always contain a region of ``pure sky'', where the contribution of the
sky background to the composite spectrum of sky plus target
astronomical object can be estimated. Each two-dimensional spectrum is
collapsed along the wavelength dispersion axis, to produce a
mono-dimensional intensity profile along the entrance slit. Groups of
at least 3 adjacent pixels all with intensity above a certain
threshold (derived from the rms noise in the profile itself) are
considered as ``object'', while all other pixels are considered as
``sky''. One-dimensional spectra for each object so identified are
obtained using Horne (1986) optimal extraction method.

For observations carried out in IFU mode, first of all one-dimensional
spectra for each lenslet are obtained adding together the flux
collected by the corresponding optical fiber.  Each one of these
spectra can belong to one of two categories: either it is the
superposition of sky and astronomical object contributions, or it is a
pure sky spectrum. It is therefore necessary to identify those spectra
in this second category to build an estimate of the sky background
intensity, before subtracting it from all the spectra. This
identification is done on the basis of the distribution of total light
intensities registered in the various spectra, considering as pure sky
ones those that have an intensity lower than that corresponding to the
mode of the distribution.  For extremely crowded fields (or very deep
exposures), the number of pure sky spectra is expected to be a very
small fraction of the total number of spectra. In this case an
interactive tool will be provided, to allow the astronomer to specify
``by hand'' the spectra to be used to build the sky background
estimate.

After the pure sky spectra are identified, they are median-averaged
together, to build an estimate of the sky spectrum, which is
subtracted from all the 6400 one-dimensional spectra.  Since the
optical fibers redistribute the light collected by the lenslet array
over the four VIMOS quadrants following a rather complex pattern, the
one-dimensional sky-subtracted spectra are then re-assembled in a data
cube to provide a complete and spatially coherent reconstruction of
the given data-set. Also, all spectra can be collapsed along the
wavelength axis to provide a two-dimensional image of the area covered
by the IFU field of view.

\section{Final Remarks}

The VIMOS DRS is written entirely using ANSI C code. Highly specialized
tasks are carried out by external libraries, including the CFITSIO and
WCS libraries, and the SExtractor object detection software.
All input and output files are standard FITS files (image and
binary table format). Heavy usage is made of ESO hierarchical keywords
within headers (this means IRAF would have problems handling those
headers). 

The final DRS output will be: for MOS observations a 2-dimensional
FITS image, containing one extracted spectrum for each row of the
image, plus a table containing the identification parameters for each
one of those spectra; for IFU observations a 3-dimensional FITS image
containing the IFU data cube, plus a table containing the
identification parameters for each spectrum.


%-----------------------------------------------------------------------
%			      References
%-----------------------------------------------------------------------
% List your references below within the reference environment
% (i.e. between the \begin{references} and \end{references} tags).
% Each new reference should begin with a \reference command which sets
% up the proper indentation.  Observe the following order when listing
% bibliographical information for each reference:  author name(s),
% publication year, journal name, volume, and page number for
% articles.  Note that many journal names are available as macros; see
% the User Guide
% (http://hea-www.harvard.edu/ADASS/ in the Proceedings section)
% listing "macro-ized" journals.   
%
% EXAMPLE:  \reference Hagiwara, K., \& Zeppenfeld, D.\  1986, 
%                Nucl.Phys., 274, 1
%           \reference H\'enon, M.\  1961, Ann.d'Ap., 24, 369
%           \reference King, I.\ R.\  1966, \aj, 71, 276
%           \reference King, I.\ R.\  1975, in Dynamics of Stellar 
%                Systems, ed.\ A.\ Hayli (Dordrecht: Reidel), 99
%           \reference Tody, D.\  1998, \adassvii, 146
%           \reference Zacharias, N.\ \& Zacharias, M.\ 2000,
%                \adassix, \paperref{P7.6}
% 
% Note the following tricks used in the example above:
%
%   o  \& is used to format an ampersand symbol (&).
%   o  \'e puts an accent agu over the letter e.  See the User Guide
%      and the sample files for details on formatting special
%      characters.  
%   o  "\ " after a period prevents LaTeX from interpreting the period 
%      as an end of a sentence.
%   o  \aj is a macro that expands to "Astron. J."  See the User Guide
%      for a full list of journal macros
%   o  \adassvii is a macro that expands to the full title, editor,
%      and publishing information for the ADASS VII conference
%      proceedings.  Such macros are defined for ADASS conferences I
%      through X.
%   o  When referencing a paper in the current volume, use the
%      \adassix and \paperref macros.  The argument to \paperref is
%      the paper ID code for the paper you are referencing.  See the 
%      note in the "Paper ID Code" section above for details on how to 
%      determine the paper ID code for the paper you reference.  
%
\begin{references}

\reference Bottini, D., Garilli, B., \& Tresse, L.\ 2000, \adassx, 
	\paperref{P1-26}
\reference Horne, K. 1986, \pasp, 98, 609

\end{references}

% Do not place any material after the references section

\end{document}  % Leave intact
