File Systems Overview
Our HPC systems have a number of different file systems available for different purposes:
Environment Variable | Purpose | Quota | On Login Nodes | On Compute Nodes | Backup | Snapshots |
|---|---|---|---|---|---|---|
HOME | Home directory. Permanent small data. Per user account. | 20 GB inspect with: | read-write | read-write | yes | yes |
PC2DATA | Permanent project data (e.g. program binaries, final results). Per project, full path | Requested at project application inspect with: | read-write | read-only | yes | yes |
PC2PFS | Parallel file system for computations. Temporary working data (but does not get erased periodically). Per project, full path | Requested at project application inspect with: | read-write | read-write | no | no |
PC2PFSN1 | PFS of Noctua 1 available at Noctua 2 | same as PC2PFS | read-write | read-only | no | no |
PC2PFSN2 | PFS of Noctua 2 available at Noctua 1 | same as PC2PFS | read-write | read-only | no | no |
PC2DEPOT | Long term backup of research data for members of Paderborn University. This filesystem is hosted and maintained by [IMT]. | needs to be requested | read-write | not available | yes | no |
Some information about quotas
Most of the filesystems above have quotas enabled. Per default every user gets 20GB in his home directory. The quota on the group dirs and scratch filesystems varies according to your project application. You can display your quota usage on $HOME and $PC2DATA the following way:
$ df -h $HOME Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf ssd.fs.pc2.uni-paderborn.de:/ifs/pc2 20G 6,6G 14G 33% /pc2 $ df -h $PC2DATA/<project-acronym> Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf ssd.fs.pc2.uni-paderborn.de:/ifs/pc2 12T 1,3T 11T 11% /pc2
On the parallel filesystem, the method is a little bit different. Quotas are set for every project. You can display the current usage with:
$ lfs quota -h -g <project-acronym> /scratch
Disk quotas for grp hpc-prf-*** (gid ***):
Filesystem used quota limit grace files quota limit grace
/scratch 245.1G 2T 2P - 105624 1000000 500000000 -
File Systems in Calculations
There are NO node-local disks in nodes of Noctua 1 and Noctua 2 (except on the FPGA-nodes). Compute jobs as well as I/O-intensive jobs should use the Lustre file system, i.e., PC2PFS.
/tmp: The temporary directory/tmpon the nodes is mapped to an isolated directory on the parallel file system. The/tmpdirectory is isolated between jobs and compute nodes, i.e.jobs on the same node can’t access the other jobs /tmp directory
and a job running on multiple nodes has an individual /tmp directory for every node.
/dev/shm: The directory/dev/shmresides in the main memory of the node and the usage count towards the memory limit of your compute job. Each job and node has it's own/dev/shmdirectory.
Please refer to the Known Issues in case you experience issues with this configuration.
Accessing HOME and PC2DATA from Outside the Clusters
They are exported via NFS and CIFS and therefore accessible from Windows, MAC, Linux, ...
If you are not located at the University of Paderborn, you first have to establish a VPN to get access to our network.
A detailed description on how to connect the PC² file systems, can be found here. Please change the used URL in the description to:
\\fs.pc2.uni-paderborn.de\home for HOME \\fs.pc2.uni-paderborn.de\groups for PC2DATA
Accessing the Parallel File System (PC2PFS) from Outside the Clusters
You can access the data on the parallel file systems of our cluster systems also from the outside with CIFS, NFS4 and rsync. Because the steps are cluster-specific, please have a look at Data Transfer / File Staging
Restoring a removed / earlier version of a file in HOME or PC2DATA
The filer of the directories HOME and PC2DATA makes daily snapshots. The snapshots of the last 90 days can be used to restore removed or older versions of a file.
Linux
In each directory, there is a hidden directory named .snapshot.
NOTE:You have to enter .snapshot to see its contents. An ls .snapshot will not show the contents.
For example, assume you have lost data in the directory MY_LIFE_DEPENDS_ON. To restore them do the following:
[xyz@n2login6 ~]$ cd MY_LIFE_DEPENDS_ON [xyz@n2login6 MY_LIFE_DEPENDS_ON]$ cd .snapshot [xyz@n2login6 .snapshot]$ ls -l drwxrwx--- 23 xyz user 524 13. Jul 14:23 daily_pc2_snapshot_2022-07-10 drwxrwx--- 23 xyz user 524 13. Jul 14:23 daily_pc2_snapshot_2022-07-11 drwxrwx--- 23 xyz user 524 13. Jul 14:23 daily_pc2_snapshot_2022-07-12 drwxrwx--- 23 xyz user 524 13. Jul 14:23 SIQ-8d653f0d930f3780f885dc030729badc-latest drwxrwx--- 23 xyz user 524 13. Jul 14:23 yesterday_pc2_backup
Search the file you want to restore and copy it to the old place or wherever you want.
Windows
Detailed information can be found here.
If you are not located at the University of Paderborn, you first have to establish a VPN to get access to our network.