Riemann or cluster?

As of the current writing, the Abacus system offers two methods for accessing resources:

1. Using the SLURM Cluster
2. Using Riemann (you can directly ssh into the server)

Riemann Overview

Riemann is currently accessible exclusively to DSH users. However, please note that it might soon be integrated into the SLURM cluster. Users are encouraged to use Riemann for tasks such as moving data across different storage locations, developing containers, and coding. Its functionality is very similar to the Optimus system you are familiar with, except that in Riemann, the use of containers is mandatory.

Usage

• For DSH Users: You have access to the Riemann system and should utilize it for the aforementioned tasks. There are no strict usage limits, but it is important to use the system responsibly to ensure fair access for others.
• For Non-DSH Users: You do not have access to Riemann and must use the SLURM cluster for similar tasks.

Important Considerations

• Riemann should not be used for model training or other compute-intensive tasks. Specifically, avoid running any scripts or processes that exceed four hours in duration.
• For all compute-intensive jobs, you are required to use the SLURM cluster instead.

Please adhere to these guidelines to ensure optimal system performance and equitable resource allocation for all users.

Data Management

The Abacus system is equipped with a fast storage device that is mounted via NFS on all machines under the path /storage. This storage serves as the primary location for storing and managing all data, projects, containers, and code. Within /storage, each unit has a dedicated directory named after the unit (e.g., /storage/DSH). These directories are assigned to the respective Linux group of the unit, allowing all members of that unit to read from and write to their designated folder. This disk is regularly backed up.

When creating a folder within the unit’s directory, the folder will automatically be owned by the user who created it and assigned to their primary group. However, by default, also members out of the unit may still be able to read the contents. To restrict access so that only the folder owner can read and modify the contents, it is recommended to change the folder’s permissions using the command:

chmod 770 "path/to/folder"

<aside> 💡 Hint: You can change the default behavior by setting umask 007 in your ~/.bashrc file. While chmod <numbers> allows you to assign permissions, umask <numbers> removes them. A reasonable umask could be umask 022, which gives all permissions to the user but restricts write access for the group and others. For further info on Linux permissions, have a look here.

</aside>

<aside> 💡 For members of the DSH unit, the /storage/DSH directory is further divided into two subdirectories: projects and personal.

• The personal directory is intended for individual experiments and exploratory projects, such as testing new architectures.
• The projects directory is reserved for official lab projects (e.g., D34h).

When setting up a new project, please follow these steps:

1. Create a folder for the project under the projects directory, using the project’s name.
2. Request the admin to create a Linux group with the same name as the project.
3. Ask the admin to add all project members to that group.
4. Change the ownership of the project folder to the new group using the following command:

chgrp "group_name" "path_to_project"

</aside>