Data Management

The Abacus system is equipped with a fast storage device that is mounted via NFS on all machines under the path /storage. This storage serves as the primary location for storing and managing all data, projects, containers, and code. Within /storage, each unit has a dedicated directory named after the unit (e.g., /storage/DSH). These directories are assigned to the respective Linux group of the unit, allowing all members of that unit to read from and write to their designated folder. This disk is regularly backed up.

Users must avoid to put working data and model in their /home directory on the login. Use this for operational data, not for heavy files. For those you should use the storage dedicated to your unit and the NMVe storage on each machine.

When creating a folder within the unit’s directory, the folder will automatically be owned by the user who created it and assigned to their primary group. However, by default, also members out of the unit may still be able to read the contents. To restrict access so that only the folder owner can read and modify the contents, it is recommended to change the folder’s permissions using the command:

chmod 770 "path/to/folder"

<aside> 💡 Hint: You can change the default behavior by setting umask 007 in your ~/.bashrc file. While chmod <numbers> allows you to assign permissions, umask <numbers> removes them. A reasonable umask could be umask 022, which gives all permissions to the user but restricts write access for the group and others. For further info on Linux permissions, have a look here.

</aside>

<aside> 💡 For members of the DSH unit, the /storage/DSH directory is further divided into two subdirectories: projects and personal.

• The personal directory is intended for individual experiments and exploratory projects, such as testing new architectures.
• The projects directory is reserved for official lab projects (e.g., D34h).

When setting up a new project, please follow these steps:

1. Create a folder for the project under the projects directory, using the project’s name.
2. Request the admin to create a Linux group with the same name as the project.
3. Ask the admin to add all project members to that group.
4. Change the ownership of the project folder to the new group using the following command:

chgrp "group_name" "path_to_project"

</aside>

Additionally, every machine in the Abacus cluster is equipped with a local NVMe fast disk mounted at /mnt/md0/data. This storage is intended for temporary data and is optimized for high-speed I/O operations, such as reading data for training models or writing generated data. Please note that the contents of these disks are temporary and will be automatically deleted one week after their creation. Users should ensure that any important data is moved to permanent storage before this period elapses.

Disk Quota:

• 35GB for each user’s /home in the login node
• 1 TB in the storage folder for each unit

Typical Workflow for Handling Data on Abacus

You can perform this operations either on the login-node or by starting an interactive job with appropriate mounts (preferred).

1. Create a Project Folder: Start by creating a folder under your unit’s directory on the main storage (/storage/unit_name) for your specific project. This folder will be used to store data, container images, and codes relevant to the project.

   mkdir /storage/unit_name/project_name 
   

2. Prepare for Job Execution:

   • When submitting a job, you should mount both your project directory on the main storage as /storage/unit_name/project_name and the temporary NVMe storage (/mnt/md0/data) as /data within your container (please check here and here on how to do this).

3. Set Up Working Directory:

   • Inside the container, create a directory under /data for your project (e.g., /data/project_name).
   • Copy the necessary data from the main storage to this directory on the temporary NVMe storage (/mnt/md0/data).

4. Work with the Data: Perform your computations, data processing, or model training using the data stored in the /data/project_name directory within your container. save your outputs in the same /data/project_name directory.

5. Save and Backup Results:

   • Afterward, copy the results back to your project folder on the main storage (/storage/unit_name/project_name) for long-term storage and further analysis.