Proceedings of the Third Conference of the EELA Project
R. Gavela, B. Marechal, R. Barbera et al. (Eds.)
CIEMAT 2007
© 2007 The authors. All rights reserved

 Analysis Facility infrastructure (TIER3) for ATLAS High Energy physics experiment

S. González de la Hoz1, L. March1, E. Ros1, J. Sánchez1, G. Amorós1, F. Fassi1, A. Fernández1, M. Kaci1, A. Lamas1, J. Salt1

1Instituto de Física Corpuscular de Valencia, Centro mixto Universitat de València - CSIC
{Santiago.Gonzalez,Luis.March, Eduardo.Ros, Javier.Sanchez, Gabriel.Amoros, Farida.Fassi, Alvaro.Fernandez, Mohammed.Kaci, Alejandro.Lamas, Jose.Salt}@ific.uv.es

Abstract

ATLAS project has been asked to define the scope and role of Tier-3 resources (facilities or “centres”) within the existing ATLAS computing model, activities and facilities. This document attempts to address these questions by describing Tier-3 resources generally, and their relationship to the ATLAS Software and Computing Project.
Originally the tiered computing model came out of MONARC (see http://monarc.web.cern.ch/MONARC/) work and was predicated upon the network being a scarce resource. In this model the tiered hierarchy ranged from the Tier-0 (CERN) down to the desktop or workstation (Tier 3). The focus on defining the roles of each tiered component has evolved with the initial emphasis on the Tier-0 (CERN) and Tier-1 (National centres) definition and roles. The various LHC projects, including ATLAS, then evolved the tiered hierarchy to include Tier-2’s (Regional centers) as part of their projects.
Tier-3’s, on the other hand, have (implicitly and sometime explicitly) been defined as whatever an institution could construct to support their Physics goals using institutional and otherwise leveraged resources and therefore have not been considered to be part of the official ATLAS Research Program computing resources nor under their control, meaning there is no formal MOU process to designate sites as Tier-3s and no formal control of the program over the Tier-3 resources. Tier-3’s are the responsibility of individual institutions to define, fund, deploy and support.
However, having noted this, we must also recognize that Tier-3’s must exist and will have implications for how our computing model should support ATLAS physicists. Tier-3 users will want to access data and simulations and will want to enable their Tier-3 resources to support their analysis and simulation work. Tiers 3’s are an important resource for physicists to analyze LHC (Large Hadron Collider) data.
This document will define how Tier-3’s should best interact with the ATLAS computing model, detail the conditions under which Tier-3s can expect some level of support and set reasonable expectations for the scope and support of ATLAS Tier-3 sites.

1. Introduction

The ATLAS computing model [1] describes a hierarchical distributed virtual computing facility within are defined Tier-1 and Tier-2 computing centres having certain specific MOU agreed roles and capacities to be used for the benefit and at the direction of ATLAS as a whole. ATLAS research program decides how these MOU pledged resources are used. In this model primary functions of the Tier-1 are to host and provide long term storage for, access to and re-reconstruction of a subset of the ATLAS RAW data, provide access to the Event Summary Data (ESD) [2], Analysis Object Data (AOD) [2] and TAG [3] data sets and support the analysis of these data sets. The primary functions of the Tier-2’s are simulation (they provide the bulk of simulation for ATLAS), calibration, chaotic analysis for subset of analysis groups and hosting of AOD, TAG and some physics group samples.
Tier-3 sites are institution-level non-ATLAS funded or controlled centres/clusters which wish to participate in ATLAS computing, presumably most frequently in support of the particular interests of local physicists (physicists at the local Tier-3 decide how these resources are used). These are clusters of computers which can vary widely in size.

2. Towards a user analysis local facility: the Tier3 prototype

According to the ATLAS Analysis Computing Model, analysis is divided into “group” and “on-demand” types. This analysis will be performed by physics groups, not by individual users, on Tier-2s. Therefore, the Tier-2 resources will be preferentially used by physics analysis groups. This means that users from universities and institutes need some extra computing resources, in addition to Tier-2 resources, to perform their own work and then contribute with their studies and algorithms to the group effort.
The ATLAS Computing Model has a hierarchical model (Event Filter, Tier-0/Tier-1/Tier-2) with specific roles and responsibilities: 
•	Event Filter Farm at CERN
o	Assembles data at CERN intro a stream to the Tier-0 Centre
•	Tier-0 Centre at CERN
o	Data archiving: raw data to mass storage at CERN and to Tier-1 centres
o	Reconstruction, calibration and alignment processing: Fast production of ESD, AOD
o	Distribution: ESD, AOD to Tier-1 centres and storage at CERN 
•	Tier-1 Centres distributed worldwide (10 centres)
o	Data steward: Re-reconstruction of raw data producing new ESD and AOD
o	Coordinated access to full ESD and AOD (all AOD, 20-100% of ESD depending on site)
•	Tier-2 Centres distributed worldwide (around 30 centres)
o	Monte Carlo simulation, producing ESD, AOD sent to Tier-1 centres
o	On demand user physics analysis of shared datasets
o	Scheduled working group activities
o	AOD processing, TAG extraction
Some local computer resources, beyond Tier-1 and Tier-2, for users at Universities and Institutes are required to do physics analysis, called Tier-3. A Tier-3 could be as small as a modern desktop computer on each physicist’s desk, or as large as a Linux farm, perhaps operated as part of a shared facility from an institution’s own resources.
The main goal of the Tier-3 is physics analysis on site with seamless access to all ATLAS Grid resources. For this purpose, analysis tools and access to data are very important, so these tools have to be installed in a Tier-3 infrastructure to facilitate the operations for users. Note that Tier-3 centres are outside the official infrastructure, but following the previous scheme, the specific role and responsibilities for a Tier-3 would be as follows:
•	Tier-3 Centres
o	Interactive analysis
•	Prototype: reconstruction, calibration and algorithm selection
•	Final selection: plots, studies, etc. 
o	Statistical analysis on AOD and Derived Physics Data (DPD) [2]
In a Tier-3 infrastructure physicists from an institute or a university could perform physics analysis on site and have access to the different ATLAS simulation and analysis facilities: tools, data, etc. Actually, according to the ATLAS Computing Model, users should send analysis jobs to sites where data are available and extract relevant output as ntuples or similar.
At Instituto de Física Corpuscular de Valencia (IFIC) the Tier-3 resources are being split into two parts:
•	Some resources are being coupled to IFIC Tier-2 resources in a Grid environment. These extra Tier-2 resources will be used preferently by Tier-3 users. While resources are idle, then they can be used by the ATLAS community.
•	A PC farm to perform interactive analysis, outside Grid.
Physicists need to produce small Monte Carlo productions in order to develop analysis algorithms, such as selection algorithms and background studies.
As a starting point, in order to perform a Tier-3 prototype at IFIC, the user requirements (IFIC users) to perform analysis are taking into account. One of the requirements is to produce small Monte Carlo simulations and store the output data for further analysis. In next subsection, the requirements for such a production using the ATLAS full chain simulation are described, base on our experience.

2.1. Requirements for a private Monte Carlo production

The Monte Carlo simulation production is going to be run at Tier-2 centres distributed worldwide according to the ATLAS Computing Model. In the following a real user example is given: We have produced a Monte Carlo simulation private production inside the ATLAS physics groups to perform the study of b-tagging at very high pT.
This private production consists of a set of three datasets. The physics processes are hadronic decays of ZH in the Little Higgs model [4], where the ZH decays into bbbar, uubar and ccbar. For each one of these hadronic decays a dataset has been made. Each dataset consists of 20000 events, so the total number of events is 60000. Each event has been processed using the ATLAS full chain simulation (FullSim events):
•	Generation
•	Simulation (Geant4) and Digitization
•	Reconstruction
Each steps of this private production is detailed as follows:

Generation step

Generator (Pythia) and generation option (one for each physics process) were validated by the Physics production managers; then they submitted these jobs to the ATLAS Tier-2 centres: 50000 events per dataset were generated into 10 event generation output files (5000 events per file). Each event generation file size was approximately 193 MB.

Simulation and Digitization step

In this case, two different steps from the ATLAS full chain simulation (Geant4 simulation and digitization) were run in the same job (Grid job). First, Geant4 simulation (simul) was run and, once the simulation output files was produced, then the digitization (digit) was run as well. So this job can be split into Geant4 simulation and the digitization step. The generation output files were the input data for Geant4 simulation. A usual simulation job is set to process only 50 events, since this is the most CPU-consuming step in the full chain. A simulation event usually takes 1400 kSI2k.sec = 700 seconds. For each job a simulation output file is produced, called simul.HITS. Each simulation output file has a size around 88 MB.
As commented before, simulation and digitization are handled by a common job. Once simulation is finished, the output is used as input for digitization that produced as output a digit.RDO file. Each digitization data file contains 50 events and its size is about 138 MB. A digitization event takes 21 kSI2k.sec = 10.5 seconds. The job submission scheme is displayed in Figure 1, where the user is logged into a User Interface (UI) and submits jobs using a Resource Broker (RB). Then the RB manages the jobs over all available Grid resources according to the user job requirements.

Figure 1. Job submission scheme of simulation private production

Reconstruction step

Reconstruction is the last step in the ATLAS full simulation chain. The output data files from digitization (digit.RDO files) are the input for reconstruction. As all official ATLAS production, a reconstruction job (recon) contains 250 events, which means that a reconstruction job has as input 5 digit data files. Each reconstruction job produced 3 output files: ESD, AOD and NTUP with 250 events per file. Each of the files had the following size; ESD around 760 MB, AOD around 134 MB and NTUP around 37 MB.
ESD are the first files produced using as input the digit data files, they required about 105 seconds per event on an Intel Pentium 4 CPU 3.20 GHz and cache size of 512 KB. Then AOD were produces using ESD input files. They took about 4.1 seconds per events on the same Pentium. Finally NTUP (DPD) files were produced using AOD as input. Note that all these data files from this private production were stored at IFIC storage element (SE) based on CASTOR. 
This private Monte Carlo simulation production needs a storage capacity of 0.5 TB (see Table 1). A total of 60 thousand full simulation events were produced. For future production, at the ATLAS start up, larger production and some local resources will be needed to perform these private productions.

Table 1. Number of events and size per output data file at each step of the Athena full chain simulation are shown. Number of files per dataset, total number of datasets and total size for each Monte Carlo production step are provided. Finally, total size of the production is shown as well.

	Events per file	Size per file (MB)	#files per dataset	# dataset	Total size (GB)	
Generation	5000	193	4	3	2.3	
Simulation	50	88	400	3	105.6	
Digitization	50	138	400	3	165.6	
Reconstruction	250	931	80	3	233.4	
Sum					469.9	

3. The Tier-3 prototype at IFIC- Valencia

ATLAS data taking is going to start on June 2008, less than one year left. For this reason, Tier-3 analysis facilities should be ready for physics analysis.
IFIC (as many other centres, institutes and universities) has a Tier-3 prototype in which different goals and steps should be done. As users at IFIC, our experience in Monte Carlo production and physics analysis is taken into account for several aspects. The first steps towards a Tier-3 infrastructure at IFIC are shown and described in this section.
Users at IFIC could get access to the ATLAS collaboration Tier-2 resources and Tier-3 resources (extra Tier-2 resources) from their desktops or laptops (see Figure 2). These extra Tier-2 resources would be used preferently by IFIC users.

Figure 2. User access from desktops or laptops to the ATLAS resources to perform physics analysis at IFIC are shown. The IFIC Tier-3 resources will be coupled to the IFIC Tier-2 resources in a Grid environment, considered as extra Tier-2 resources. In addition, a PC farm will be deployed to perform interactive analysis tests in a non-Grid environment. 

Otherwise, if these resources are free, they could be used, in addition to the IFIC Tier-2 resources, by ATLAS collaboration, ensuring and efficient use. On the other hand, other special requirements setup could be deployed for IFIC users as well, like a PC farm to perform interactive analysis.
An individual physicist can get access to the ATLAS software and analysis tools using its desktop or laptop. These software and analysis tools are already installed at IFIC as a first test. These tools are installed in a share file system area, called AFS [5], where IFIC users have access to them, in the same way as at CERN, in order to avoid installing these tools at each desktop or laptop.
These ATLAS software (Athena, Atlantis [6], etc) and distributed analysis tools (GANGA [7]) are installed under IFIC AFS area. Other useful tools, like ROOT or the Grid User Interface environment are also installed.
Users can use their desktop as a private exclusive-use User Interface and developer station. In this way, some first steps shown in Figure 2 has already started, like access to ATLAS Grid resources and tools as commented previously, but it needs more effort to install several ATLAS software tools (such as different atlas software releases and other useful tools).
IFIC Tier-3 Grid resources are going to be coupled to IFIC Tier-2 resources, but separating clearly them using different or shared queues with various priorities for the ATLAS Collaboration Tier-2 and IFIC Tier-3 users. IFIC Tier-2 resources are used by the whole ATLAS Collaboration, reaching the ATLAS resources requirements (disk storage and CPU). Extra IFIC Tier-2 resources (Tier-3) are going to be used by IFIC users to run local simulation and analysis productions and to store interesting data for analysis:
•	Production of Monte Carlo samples of special interest for the local institution
•	AOD private productions for further analysis
•	AOD analysis
Having access to the ATLAS software and Grid resources, users can perform local checks, running small Grid or non-Grid test jobs and develop their analysis code before submitting larger simulation or analysis productions to the Tier-2 or Tier-1 centres. The development of analysis code could motivate also a local copy of small number (perhaps a few thousands) of ESD, AOD or RAW data events.
In addition to the IFIC Tier-3 Grid resources, a PROOF (Parallel ROOT Facility) [8] farm for evaluation is going to be deployed for interactive analysis of ntuples. Direct access to ESD or AOD is not required only direct access where these ntuples are generated. At this point, we should note that this infrastructure, Tier-3 Grid and non-Grid resources, is going to use the same Storage Element, having access to the data in both cases. 
Interactive analysis is a very important issue for physics analysis. Usually, interactive analysis with DPD ntuples are analysed using ROOT. For this reason a PROOF farm with few powerful PCs (outside Grid environment) is going to be installed for evaluation and to do this interactive analysis. One important issue is that this farm must be well connected to the Storage Element in order to get a fast access to data.
Some data access performance tests have been performed using the IFIC local Storage Element, Lustre [9], and a b-tagging physics sample. ZH→bbbar. AODs and DPDs for the same events have been used to analyze them with the ATLAS framework (Athena) and ROOT, respectively. This data has been accessed in different ways: local disk, RFIO (CASTOR) and Lustre.
Figure 3 shows the results of the data access performance test using different ways of local storage, the Athena framework to analyze the data and 38 AOD’s as input files. Each AOD has 250 events (9500 in total) and the average size per AOD was around 160 MB. As Figure 3 shows, the slowest way to access this data is using local disk, analyzing around 6.43 events per second. RFIO took 6.96 events per second and the fastest way, Lustre, took 7.06 events per second. In general, the bandwidth used was around 4 MB/s.

Figure 3. Data access performance test using the IFIC local SE, the Athena framework and AOD’s as input data files.

Figure 4 shows the results of the data access performance test using different ways of local storage, the ROOT analysis tool to analyze the data and just one DPD with 9500 events as input data file. The size of this DPD is around 178 MB and contains all data from the previous AOD’s (9500 events), extracted from these AOD’s. As Figure 4 shows, the slowest way to access this data is RFIO, in this case, analyzing around 97.75 events per second. Lustre took around 101.24 events per second and the fastest way, local disk, took around 102.46 events per second The bandwidth used was around 2 MB7s, lower than in Figure 3. This is due to the data structure and data size, since the event size in a DPD is much smaller tan in a AOD.

Figure 4. Data access performance test using the IFIC local SE, ROOT analysis tool and one DPD as input data file.

As shown in Figure 3 and 4, the results obtained for data access performance test using AOD’s and DPD’a are different and show that DPD’s are analyzed faster than AOD’s. A fast data access for analysis is very important.
The aim of the work being done at the Tier-3 at IFIC is to provide a flexible computing resource; therefore the physicist could make an analysis work in an easy and convenient way. The Tier-3 will provide interactive and non-interactive CPU as well as enough storage capacity and bandwidth to the data sources, the closest Tier-2 (Spanish federated Tier 2) and other resources abroad.

3. Summary and conclusions

In this paper we have tried to develop a model for Tier-3 and Analysis Facility sites in ATLAS. Within the ATLAS model such sites will be used for interactive or batch analysis of, mostly, so called DPD data sets, which have been produced from AOD data using Grid distributed analysis tools. The definition of different possible DPD formats is been discussed in the ATLAS Analysis Model group and will be physics working group or even analysis specific. At IFIC in Valencia we are proposing a possible Tier-3 configuration and software setups that match the requirements according to the DPD analysis needs as formulated by the ATLAS Analysis Model Group. The results of this first evaluation are:
•	Some local resources, beyond Tier-1 and Tier-2, are required to do physics analysis in ATLAS.
•	These resources are termed Tier-3 and could be as small as a modern desktop computer on each physicist’s desk, or as large as Linux farm, perhaps operated as part of a shared facility from an institution’s own resources.
•	Support from the Tier-1 and Tier-2’s to such Tier-3 centres in terms of expertise (install, configure, tune, troubleshooting of ATLAS releases and the Grid stack) and services (data storage, data serving, etc.) follows from responsibility to support the user community.
We envision the following to be typical examples of uses of a Tier-3:
•	Interactive analysis of ntuples. It does require access to the data when these ntuples are generated.
•	Development of analysis code. This would motivate a local copy of a small number of ESD, AOD or RAW events.
•	Running small local test jobs before submitting larger jobs to the Tier-1 or Tier-2 via the Grid. This would motivate similar sized copies of the data as above. It also motivates having access to at least the appropriate subset of the TAGs at the Tier-3, because this it the same selection mechanism that will be used when the full scale job is run.
•	Running skimming jobs of the Tier-1 and Tier-2’s via the Grid, and copying the skimmed AOD (or rarely ESD) back to the Tier-3 for further analysis. The output of this skim must be a very small subset of the AOD – of order a few percent.
•	Analyzing the above skimmed data via ATLAS framework (Athena).
•	Production of Monte Carlo samples of special interest to the local institution.

References

[1]	D. Adams et al.“The ATLAS Computing Model”, ATL-SOFT-2004-007, CERN, 15 Dec 2004.

[2] ATLAS Event Data Model, https://twiki.cern.ch/twiki/bin/view/Atlas/EventDataModel

[3] Tag for Event Selection, https://twiki.cern.ch/twiki/bin/Atlas/TagForEventSelection

[4] G. Azuelos et al., “Exploring Little Higgs models with ATLAS at the LHC”, Eur. Phys.J.,C39S2:13, hep-ph/0402037, 2005.

[5] Andrew File System (AFS), http://en.wikipedia.org/wiki/Andrew_file_system

[6] Atlantis event display for ATLAS, http://www.hep.ucl.ac.uk/atlas/atlantis

[7] GANGA, http://ganga.web.cern.ch/ganga/

[8] PROOF: the Parallel ROOT Facility, http://root.cern.ch/twiki/bin/view/ROOT/PROOF

[9] Lustre Cluster File System: http://wiki.lustre.org