File Systems

1 File Systems Overview
2 Some information about quotas
3 File Systems in Calculations
4 Accessing HOME and PC2DATA from Outside the Clusters
5 Accessing the Parallel File System (PC2PFS) from Outside the Clusters
6 Restoring a removed / earlier version of a file in HOME or PC2DATA
- 6.1 Linux
- 6.2 Windows

File Systems Overview

Our HPC systems have a number of different file systems available for different purposes:

Environment Variable	Purpose	Quota	On Login Nodes	On Compute Nodes	Backup	Snapshots

Environment Variable	Purpose	Quota	On Login Nodes	On Compute Nodes	Backup	Snapshots
HOME	Home directory. Permanent small data. Per user account.	20 GB inspect with: `pc2status` or `df -h $HOME`	read-write	read-write	yes	yes
PC2DATA	Permanent project data (e.g. program binaries, final results). Per project, full path `$PC2DATA/<project-acronym>`	Requested at project application inspect with: `pc2status` or `df -h $PC2DATA/<project_acronym>`	read-write	read-only	yes	yes
PC2PFS	Parallel file system for computations. Temporary working data (but does not get erased periodically). Per project, full path `$PC2PFS/<project-acronym>`.	Requested at project application inspect with: `pc2status` or `lfs quota -hg <project_acronym> $PC2PFS`	read-write	read-write	no	no
PC2PFSN1	PFS of Noctua 1 available at Noctua 2	same as PC2PFS	read-write	read-only	no	no
PC2PFSN2	PFS of Noctua 2 available at Noctua 1	same as PC2PFS	read-write	read-only	no	no
PC2DEPOT	Long term backup of research data for members of Paderborn University. This filesystem is hosted and maintained by [IMT].	needs to be requested	read-write	not available	yes	no

Some information about quotas

Most of the filesystems above have quotas enabled. Per default every user gets 20GB in his home directory. The quota on the group dirs and scratch filesystems varies according to your project application. You can display your quota usage on $HOME and $PC2DATA the following way:

$ df -h $HOME
Dateisystem                          Größe Benutzt Verf. Verw% Eingehängt auf
ssd.fs.pc2.uni-paderborn.de:/ifs/pc2   20G    6,6G   14G   33% /pc2

$ df -h $PC2DATA/<project-acronym>
Dateisystem                          Größe Benutzt Verf. Verw% Eingehängt auf
ssd.fs.pc2.uni-paderborn.de:/ifs/pc2   12T    1,3T   11T   11% /pc2

On $HOME and $PC2DATA there are only hard limits set. This means, if you hit the limit, no more data could be written. So you should take care, that you don’t hit the limit. We will inform you at an usage of 90% of your limit. If you think, you need more quota, please contact us via pc2-support@uni-paderborn.de. Please state why you need more quota than the default settings.

On the parallel filesystem, the method is a little bit different. Quotas are set for unix group which corresponds to a project. You can display the current usage with lfs:

$ lfs quota -h -g <project-acronym> /scratch
Disk quotas for grp hpc-prf-*** (gid ***):
     Filesystem    used   quota   limit   grace   files   quota   limit   grace
       /scratch  245.1G      2T      2P       -  105624  1000000 500000000       -

On the lustre filesystem, there are two limits quota and limit. Quota is a soft limit. You can exceed this limit for a certain time (grace period, per default 14 days). After this time, no more data can be written. Beside this soft limit, there is a hard limit. If you hit this limit, writing of further data is prohibited immediately. The limits are set for the storage capacity and number of files. Within your application for a project, you have to request for these limits.

File Systems in Calculations

There are NO node-local disks in nodes of Noctua 1 and Noctua 2 (except on the FPGA-nodes). Compute jobs as well as I/O-intensive jobs should use the Lustre file system, i.e., PC2PFS.

/tmp: The temporary directory /tmp on the nodes is mapped to an isolated directory on the parallel file system. The /tmp directory is isolated between jobs and compute nodes, i.e.
- jobs on the same node can’t access the other jobs /tmp directory
- and a job running on multiple nodes has an individual /tmp directory for every node.
/dev/shm: The directory /dev/shm resides in the main memory of the node and the usage count towards the memory limit of your compute job. Each job and node has it's own /dev/shm directory.

Please refer to the Known Issues in case you experience issues with this configuration.

Accessing HOME and PC2DATA from Outside the Clusters

They are exported via NFS and CIFS and therefore accessible from Windows, MAC, Linux, ...

If you are not located at the University of Paderborn, you first have to establish a VPN to get access to our network.

A detailed description on how to connect the PC² file systems, can be found here. Please change the used URL in the description to:

\\fs.pc2.uni-paderborn.de\home       for HOME
\\fs.pc2.uni-paderborn.de\groups     for PC2DATA

Accessing the Parallel File System (PC2PFS) from Outside the Clusters

You can access the data on the parallel file systems of our cluster systems also from the outside with CIFS, NFS4 and rsync. Because the steps are cluster-specific, please have a look at Data Transfer / File Staging

Restoring a removed / earlier version of a file in HOME or PC2DATA

The filer of the directories HOME and PC2DATA makes daily snapshots. The snapshots of the last 90 days can be used to restore removed or older versions of a file.

Linux

In each directory, there is a hidden directory named .snapshot.

NOTE:You have to enter .snapshot to see its contents. An ls .snapshot will not show the contents.

For example, assume you have lost data in the directory MY_LIFE_DEPENDS_ON. To restore them do the following:

Search the file you want to restore and copy it to the old place or wherever you want.

Windows

Detailed information can be found here.

If you are not located at the University of Paderborn, you first have to establish a VPN to get access to our network.