File Systems
File Systems Overview
Our HPC systems have a number of different file systems available for different purposes:
Environment Variable | Purpose | Quota | On Login Nodes | On Compute Nodes | Backup | Snapshots |
|---|---|---|---|---|---|---|
HOME | Home directory. Permanent small data. Per user account. | 50 GiB inspect with: | read-write | read-write | yes | yes |
PC2DATA | Permanent project data (e.g. program binaries, final results). Per project, full path | Requested at project application inspect with: | read-write | read-only | yes | yes |
PC2PFS | Parallel file system for computations. Temporary working data (but does not get erased periodically). Per project, full path | Requested at project application On Noctua inspect with: On Otus use | read-write | read-write | no | no |
PC2PFSN2 | PFS of Noctua 2 available at Otus | same as PC2PFS | read-write | read-only | no | no |
PC2DEPOT | Long-term backup of research data for members of Paderborn University. This filesystem is hosted and maintained by [ZIM]. | needs to be requested | read-write | not available | yes | no |
Some information about quotas
Most of the filesystems above have quotas enabled. Per default, every user gets 50GiB in his/her home directory. The quota on the group dirs and scratch filesystems varies according to your project application. You can display your quota usage on $HOME and $PC2DATA the following way:
$ df -h $HOME
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
ssd.fs.pc2.uni-paderborn.de:/ifs/pc2 50G 14G 36G 28% /pc2
$ df -h $PC2DATA/<project-acronym>
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
ssd.fs.pc2.uni-paderborn.de:/ifs/pc2 12T 1,3T 11T 11% /pc2On $HOME and $PC2DATA there are only hard limits set. This means, if you hit the limit, no more data could be written. So you should take care, that you don’t hit the limit. We will inform you at an usage of 90% of your limit. If you think, you need more quota, please contact us via pc2-support@uni-paderborn.de. Please state why you need more quota than the default settings.
Quota on Noctua PFS
On the parallel filesystem of the Noctua cluster, the method is a little bit different. Quotas are set for Unix group which corresponds to a project. You can display the current usage with lfs:
$ lfs quota -h -g <project-acronym> /scratch
Disk quotas for grp hpc-prf-*** (gid ***):
Filesystem used quota limit grace files quota limit grace
/scratch 245.1G 2T 2P - 105624 1000000 500000000 -On the lustre filesystem, there are two limits quota and limit. The quota is a soft limit. You can exceed this limit for a certain time (grace period, per default 14 days). After this time, no more data can be written. Beside this soft limit, there is a hard limit. If you hit this limit, writing of further data is prohibited immediately. The limits are set for the storage capacity and number of files. Within your application for a project, you have to request for these limits.
Quota on Otus
The quota system on Otus is different to the system used at Noctua clusters. First of all, there is a hard quota setting for each project. There is only one quota, so hitting the border will result in write failures. You will get a warning email at about 80% usage of your quota. You can inspect your current usage with
$ df -h /scratch/<project-acronym>
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
scratch 50T 55G 50T 1% /scratchTemporary Directories and Node-Local Storage
Users are strongly encouraged to utilize local temporary storage for IO-intensive jobs (frequent file updates and small temporary files), which do not need centralized storage (e.g. single node jobs), rather than relying entirely on the parallel file system (PC2PFS). Offloading these short-lived files significantly reduces the load on the shared network, prevents performance bottlenecks, and accelerates job runtimes.
There are two primary locations available for temporary storage:
/tmp (Local Disk or Parallel Disk Storage)
The /tmp directory is isolated between jobs and nodes to ensure security and prevent data conflicts:
Job Isolation: Jobs running on the same node cannot access each other's
/tmpdirectories.Node Isolation: A job running across multiple nodes will have an independent, individual
/tmpdirectory on each node.
Behind the Scenes & Node-Local Storage:
Nodes with Local Storage: On nodes equipped with local NVMe/SSD disks (see the Architecture Overview for a list of these nodes and the capacities), the job_container/tmpfs Slurm plugin is used. It transparently redirects anything a user writes to
/tmpinto a job-local directory created at/local/...user-idand automatically cleans up the directory after each job finishes.Nodes without Local Storage: If no local disk is present,
/tmpis mapped to an isolated directory hosted on the parallel file system.
/dev/shm (Shared Memory / RAM)
The /dev/shm directory resides entirely within the volatile main memory (RAM) of the node.
Memory Limits: Because it uses RAM, any data stored here counts directly toward the memory limit requested for your compute job. Exceeding this can cause your job to be terminated (OOM).
Isolation: Each job and node has its own isolated
/dev/shmdirectory.
Please refer to the Known Issues in case you experience issues with this configuration.
Accessing HOME and PC2DATA from Outside the Clusters
They are exported via NFS and CIFS and therefore accessible from Windows, MAC, Linux, ...
If you are not located at the University of Paderborn, you first have to establish a VPN to get access to our network.
A detailed description on how to connect the PC² file systems can be found here. Please change the used URL in the description to:
\\fs.pc2.uni-paderborn.de\home for HOME
\\fs.pc2.uni-paderborn.de\groups for PC2DATA Accessing the Parallel File System (PC2PFS) from Outside the Clusters
You can access the data on the parallel file systems of our cluster systems also from the outside with CIFS, NFS4 and rsync. Because the steps are cluster-specific, please have a look at Data Transfer / File Staging
Restoring a removed / earlier version of a file in HOME or PC2DATA
The filer of the directories HOME and PC2DATA makes daily snapshots. The snapshots of the last 90 days can be used to restore removed or older versions of a file.
Linux
In each directory, there is a hidden directory named .snapshot.
NOTE: You have to enter .snapshot to see its contents. An ls .snapshot will not show the contents.
For example, assume you have lost data in the directory MY_LIFE_DEPENDS_ON. To restore them do the following:
[xyz@n2login6 ~]$ cd MY_LIFE_DEPENDS_ON
[xyz@n2login6 MY_LIFE_DEPENDS_ON]$ cd .snapshot
[xyz@n2login6 .snapshot]$ ls -l
drwxrwx--- 23 xyz user 524 13. Jul 14:23 daily_pc2_snapshot_2022-07-10
drwxrwx--- 23 xyz user 524 13. Jul 14:23 daily_pc2_snapshot_2022-07-11
drwxrwx--- 23 xyz user 524 13. Jul 14:23 daily_pc2_snapshot_2022-07-12
drwxrwx--- 23 xyz user 524 13. Jul 14:23 SIQ-8d653f0d930f3780f885dc030729badc-latest
drwxrwx--- 23 xyz user 524 13. Jul 14:23 yesterday_pc2_backup
Search the file you want to restore and copy it to the old place or wherever you want.
Windows
Detailed information can be found here.
If you are not located at the University of Paderborn, you first have to establish a VPN to get access to our network.