Business and Investment Development Agency

RUCIO

Rucio is a project that provides services and associated libraries for allowing scientific collaborations to manage large volumes of data spread across facilities at multiple institutions and organizations.

Rucio was originally developed to meet the requirements of the high-energy physics experiment ATLAS, and now is continuously extended to support the LHC experiments and other diverse scientific communities. Rucio is a highly scalable, and modular data management solution that can be used for the needs of different communities in the scientific domain like high-energy physics, astronomy, biology or anywhere else requiring large data storage and discovery. Rucio allows you to manage large data storage, discovery - any data federation.

Advantages

  • Moving data from public clouds to private clouds with a single API - Unified interfacing of heterogenous network & storage infrastructure
  • High-capacity storage of detector data, simulator data, and user data
  • Support for newer protocols in Storage & Network using plugins
  • Data Recovery
  • Adaptive Replication

Technical description

Rucio is based on a distributed system architecture & can be sectioned into four major layers:

Clients

The clients layer consists of components such as the command line clients (CLI), Python clients, and the Javascript-based web user interface and configuration.

Server

The server layer serves the purpose of authentication & provides a common API for interaction with clients & other external application, as also the Web UI.

Core

This layer consists of all the Rucio-level abstractions that are explained at length in the Concepts section.

Daemons

The daemons layer takes care of all the asynchronous & continuous workflows in the background.

In addition to the four main layers, we have the storage resources & transfer tools, as well as the underlying persistent layers. These are represented in the architecture diagram by the different queuing systems, transactional relational databases, & analytics storage on non-relational databases.

Storage & Transfer Tools layer

The Storage layer is responsible for the various interactions with different grid middleware tools & storage systems. The transfer tool interface is a feature of Rucio that enables the daemons to submit, query, and cancel transfers generically & independently from the actual transfer service being used. This layer is not a software that resides in a datacenter but is more representative of the various abstractions in a storage system. It can be configured dynamically & centrally to suite requirements.

Persistence Layer

This is the layer responsible for keeping all the data & application states for all daemons. Also known as the catalog, it requires a transactional database. The persistence layer supports many different types of transactions relational database management systems such as SQLite, MySQL, PostgreSQL, or Oracle 

We use cookies to ensure optimal functioning of our website. By continuing to browse the site, you are agreeing to our use of cookies. More information here
}