Sheet Music

Belle Computing System

We describe the present status of the computing system in the Belle experiment at the KEKB $e^+e^-$ asymmetric-energy collider. So far, we have logged more than 160 fb$^{-1}$ of data, corresponding to the world's largest data sample of 170M
of 6
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
    a  r   X   i  v  :  c  s   /   0   4   0   3   0   1   5  v   2   [  c  s .   D   C   ]   1   2   M  a  r   2   0   0   4 Belle Computing System Ichiro Adachi a ∗ , Taisuke Hibino a , Luc Hinz b , Ryosuke Itoh a , Nobu Katayama a , Shohei Nishida a ,Fr´ed´eric Ronga b † , Toshifumi Tsukamoto a and Masahiko Yokoyama aa IPNS, KEK, Oho 1-1, Tsukuba, Ibaraki 305-0801, Japan b LPHE, EPFL, Lausanne, Dsrcny CH-1015, Switzerland We describe the present status of the computing system in the Belle experiment at the KEKB  e + e − asymmetric-energy collider. So far, we have logged more than 160 fb − 1 of data, corresponding to the world’s largest datasample of 170M  B  ¯ B  pairs at the Υ(4 S  ) energy region. A large amount of event data has to be processed toproduce an analysis event sample in a timely fashion. In addition, Monte Carlo events have to be created tocontrol systematic errors accurately. This requires stable and efficient usage of computing resources. Here wereview our computing model and then describe how we efficiently proceed DST/MC productions in our system. 1. Introduction The Belle experiment[1] is the  B -factoryproject at KEK to study  CP   violation in  B  me-son system. The KEKB accelerator[2] is an asym-metric energy collider with 8 GeV electron to 3.5GeV positron, operating at Υ(4 S  ) energies. TheBelle group has been taking data since June 1999and has logged more than 160 fb − 1 of data un-til December 2003. This means that we have thelargest data sample of   B  meson pairs around theΥ(4 S  ) energy region in the world. The KEKBreached its design luminosity of 10 34 cm − 2 sec − 1 on May 2003. The KEKB performance is still be-ing improved and we can accumulate integratedluminosity of about 800 pb  − 1 per day recently.We have to promptly process all of beam datato provide them for user analyses. To do this, theDST production should be stable and the com-puting resources have to be used in an efficientway. The Monte Carlo( MC ) data should begenerated with statistics large enough to controlexperimental systematics. As a result, data sizewhich we should handle is extremely huge anda mass storage system has to be used to avoidnetwork traffic, and data management for entire ∗ corresponding author. tel: +81-29-864-5320. mail ad-dress: † present address: IPNS, KEK, Japan data sets should be carefully done without loosingflexibility, for instance, any modification of datadistributions.Provided a large amount of data sample, wehave published a variety of physics results re-lated to  B  meson decays, which is highlighted bythe first observation of the large  CP   violation in B  meson decays[3]. The quick and stable dataprocessing greatly contributed to this remarkableachievement.In this paper, we will describe a detail of ourcomputing system after a brief sketch of Bellesoftware utilities. In the next section, how weproceed DST as well as MC productions will bementioned, and then summary will be given. 2. Belle Software2.1. Core Utility In the Belle experiment, unique software frame-work called as B.A.S.F.( Belle AnalySis Frame-work ) for all phases in event processing from on-line data-taking to offline user analyses has beenemployed. This is “home-made” core software de-veloped by the Belle group. In this scheme, eachprogram, written in C++, is compiled as a sharedobject, and it is treated as a module. When onewants to run a program with this framework, themodules defined in the one’s script are dynami-1  2cally loaded.The data handling is done with the traditionalbank system, named as PANTHER, with a zlibcompression capability In this utility, data trans-fer between different modules is made and dataI/O is maniplated. PANTHER is only software tohandle our data in any stage of data processing.The typical event data size for rawdata is 35KB, and it is increased up to 60 KB for re-constructed DST data, which contains all of de-tector information after unpacking, calibrationand analysis. For user analyses, compact dataset (“mini-DST”) is produced, which is approxi-mately 12 KB for one hadronic event. 2.2. Reconstruction and Simulation Li-brary The reconstruction package is built when majorupdate of programs, which can affect final physicsanalysis, is made. Usually it is built once or twiceper year. Once new library is released, we need toprocess all of events to produce a consistent dataset for analysis.For MC data, the detector response is simu-lated based upon the GEANT3 library[4]. Herebackground events, calculated from beam data,are overlaied onto MC events. The same recon-struction codes are also applied to MC simulatedevents.The detector calibration constants are stored inthe database, for which PostgreSQL[5] is adoptedin the Belle experiment. Two database serversarerunning in our computing system, where one is forpublic usage and the other for DST/MC produc-tions. The contents of each server are periodi-cally mirrored to the other, and a backup tar filefor the contents of the srcinal database server isarchived once per week. For linux users, one canstart up own database server in her/his PC, ac-cording to a Belle instruction. User, if necessary,can download srcinal database contents from aKEK B computer for private purpose. 3. Belle Computing System3.1. Computing Model Overview Figure 1 indicates an overview of our comput-ing model for the Belle experiment. As can be Super-SINET to Univ’sDisk Storage(60TB)User PC farmsKEK B Computers10 login servers for User PC farms fast network  2001 Feb2003 Mar20022003 Jun2003 Jun Figure 1. Computing model overview.seen, it comprises of the four major elements.The first one is the KEK B Computers which hasbeen operated since February 2001[6]. This hasbeen a principal system, where data processingas well as analyses have been performed. In addi-tion to this, we have equipped a 60 TB disk stor-age area March 2003. In this space, more beamand MC data can be kept. As for a network in-side Japan, Super-SINET link has been availablesince 2002[7]. To enrich CPU’s for user analyses,PC farms dedicated to analysis jobs have beeninstalled in June 2003. These components are in-terlinked each other with a fast network of 1 ∼ 10Gbps. 3.2. KEK B Computers The KEK B computers are schematicallyshown in Figure 2. This system consists of thethree cardinal links. The first one is called “com-puting network”, which interconnects the 38 Sunservers with the PC farms, which will be de-scribed below. These Sun hosts are operated asa batch job system controlled by the LSF sched-uler. The computing network is also attached tothe DTF2 tape robotic device with 500 TB totalvolume which is used for rawdata and DST datastorage. The next network links the 9 work groupservers of the Sun hosts to the storage devices,which consists of the 8 TB file server and thehierarchy mass storage( HSM ) of 120 TB capac-  3Figure 2. Belle computing system.ity with the 4 TB staging disk. The work groupservers are connected via the “user network” to100 PC’s for users. With this network, user canlogin from his/her PC to one of the work groupservers, from which one can submit a batch job tothe 38 Sun compute servers. Table 1 summarizesthe CPU’s allocated in the KEK B computers. 3.3. PC farm More and more data have been accumulated,more and more computing needs for event pro-cessing have been seriously arising. To fulfill thisrequirements, a bunch of PC’s has been installedand connected into this existing network by con-sidering the best usage without making a bottle-neck. Table 2 tabulates the PC farm CPU’s in oursystem. As one can see, our PC farms are hetero-geneous from various vendors. The best choice atthe moment to get new PC’s usually made us topurchase from a variety of vendors, and it effec-tively reduces cost for CPU’s.In all PC’s, linux utilities as well as the Bellelibrary packages have been loaded and we can usethem as clusters of seamless PC systems to pro-cess event data.The primary purpose to add PC farms is thatwe have to increase CPU power for DST and MCproductions. In 2003, we expanded usage for ourPC’s by releasing new PC farms for user analyses,as a possible solution for ever increasing users’demand. The 10 login servers have been arrangedwith 6 TB local disk area, where user histogramfiles and analysis codes are located. From eachlogin server, job can be submitted to the user PCfarm consisting of 84 PC’s with dual Xeon 2.8GHz CPU’s. All PC’s are managed by the LSFqueuing utility. From user PC farms, beam and  4Table 1 A summary of CPU’s for the KEK B computers. host processor clock #CPU’s #nodescomputing server sparc 500MHz 4 38work group serve sparc 500MHz 4 9user PC P3 1GHz 1 100Table 2 A breakdown of the PC farm CPU’s. vendor processor clock #CPU’s #nodes total CPUDell P3 500MHz 4 16 32GHzDell P3 550MHz 4 20 44GHzCompaq P3 800MHz 2 20 32GHzCompaq P3 933MHz 2 20 37GHzCompaq Intel Xeon 700MHz 4 60 168GHzFujitsu P3 1.26GHz 2 127 320GHzCompaq P3 700MHz 1 40 28GHzAppro Athlon 1.67GHz 2 113 377GHzNEC Intel Xeon 2.8GHz 2 84 470GHzTotal 500 1508GHzMC data samples, which are stored in the 60 TBdisk mentioned previously, can be accessed. 3.4. Super-SINET at Belle A fast 1 Gbps network dedicated to academicactivities, Super-SINET[7], has been availablebetween KEK and major Japanese universities.This link enables us to copy a bulk of beam andMC data from KEK to other institutions andvice versa. Moreover, computing resources canbe shared as seamless system using Super-SINET.For example, one disk connected to PC at Nagoyauniversity can be mounted to the KEK computervia NFS as if it were located inside the KEK site.Then, output data can be written onto the diskdirectly from the KEK computer. It is possibleto make efficient and full use of total resourcescollaboration-wide. 3.5. Data Management A large volume of data is comprised of morethan 30 K data files including beam and MCevents. Basically each user has to go throughall of these files to obtain final physics results.So, it is very important to notify all of file loca-tions to users for their analyses. These data filesare distributed over a bunch of storage disks andsometimes data files have to be moved for variousreasons such as disk failure and so on. From ad-ministrative point of view, it is necessary to main-tain flexibility in data management despite of anychange of the file locations. To solve this, we haveregistered all of the attributes of data files suchas locations, data type and number of events intoour database, and they serves as “metadata” toaccess actual beam data. The central databasecontents for data file information is maintainedand updated whenever new data is available. Theweb-based interface between database and usersis prepared to extract necessary information. Inuser’s batch job, inquiry to the database is auto-matically issued and user can easily analyze eventdata without knowing actual file locations. 4. Data Processing4.1. Beam data production The scheme of the DST production is shown inFigure 3. In the first step of the DST production,one of Sun compute host is assigned as a tapeserver, and two DTF2 tapes, one is for raw dataand the other for DST data, are mounted. Then,raw data are read from the tape and are dis-tributed over PC nodes. In each PC node, event  5Figure 3. A schematic drawing of the DST production scheme.processing is performed in the B.A.S.F. frame-work. After the reconstruction is made, eventdata are sent back to the tape server, where dataare written onto the tape as DST.The next step, the event skimming, is carriedout in such a way that DST data are again readfrom the DST2 tape at the Sun compute server,and an appropriate selection criteria is imposedonto the data. In case that an event satisfies selec-tion conditions, it is saved onto disk as a skimmedevent. Each event is examined by a set of selec-tions such as  µ -pair event for a detector studyand hadronic event for a physics analysis.In our system, we can process about 1 fb − 1 of beam data per day with 40 PC nodes of quadIntel Xeon 700 MHz CPU’s each, and it can be in-creased up to 2 fb − 1 per day by adding PC nodes.Figure 4 shows a history of our beam data pro-cessing from 2001. In 2001, we completed the firstentire reprocessing with a single version of the re-construction package, providing 30 fb − 1 of datasample for 2000 summer conferences. In 2002,major update of the software was made in April2002 and the second reprocessing for 78 fb − 1 of data using this library was performed from Aprilto June. Last year, we added another 80 fb − 1 by reprocessing them, amounting to 159 fb − 1 of total beam data. 4.2. MC production We have three types of MC samples,  B 0  ¯ B 0 , B + B − and continuum events. They are producedon a run-by-run basis, where each MC sample filecorresponds to each beam data file. After dataproduction for one run beam file is done, run de-pendent information like beam interaction pro-file and background hit rate are calculated imme-diately and, based upon these information, MCsample data corresponding to the beam run file
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!