The HLRN alliance jointly operates a distributed supercomputer system hosted at the sites Georg-August-Universität Göttingen und Zuse Institute Berlin (ZIB). The system delivery of the phases 1 and 2 of the HLRN-IV from the company Atos/Bull was completed in 2019. The total HLRN-IV system holds more than 230,000 cores with a total peak performance of about 16 PFlop/s.

The HLRN complex in Berlin at ZIB is named after Lise Meitner and it conatins 1270 compute nodes with 121,920 compute cores.

• 1270 compute nodes
  • 1236 nodes with 384 GB memory (standard node)
  • 32 nodes with 768 GB memory (large node)
  • 2 nodes with 1.5 TB memory (huge node)
  • per node 2 CPUs + 1 Intel Omni-Path host fabric adapter
  • per CPU 1 Intel Cascade Lake Platinum 9242 (CLX-AP), 48 cores
• Omni-Path interconnect configuration
  • 2 x 1162 port OPA100 director switches
  • 56 x 48 port edge switches
  • fat tree topology
  • 14 TB/s bisection bandwidth
  • 1.65 μs maximum latency
• 8 login nodes
  • per node 2 CPU with 384 GB memory
  • per CPU 1 Intel Cascade Lake Silver 4210 (CLX), 20 nodes

At the site Göttingen University the HLRN system phase 1 named Emmy (for Emmy Noether) is in operation since October 2018. In October 2020 the second phase was added.

• 974 nodes added in phase 2
  • 956 nodes with 384 GB memory (standard node)
  • 16 nodes with 768 GB memory (large node)
  • 2 nodes with 1,5 TB memory (huge node)
  • per nodes 2 CPUs and 1 Intel Omni-Path host fabric adapter
  • per CPU 1 Intel Cascade Lake Platinum 9242 (CLX-AP), 48 cores

• 448 compute nodes of phase 1
  • 432 nodes with 192 GB memory (medium node)
  • 16 nodes with 768 GB memory (large node)
  • per node 2 CPUs and 1 Intel Omni-Path host fabric adapter
  • per CPU 1 Intel Skylake Gold 6148, 20 cores
  • 1 x 480 GB SSD hard disk
• 3 GPU nodes
  • 2 x Intel Skylake Gold 6148 CPUs (40 cores per nodes)
  • 768 GB Memory
  • 1 x 480 GB SSD hard disk
  • 1 x Intel Omni-Path host fabric adapter
  • 4 x NVIDIA Tesla V100 32GB

There are three different file systems available for Göttingen and Berlin respectively.

The WORK file system

For large data at both complexes a WORK file system is available.

• 2 x 8.1 PB netto capacity
• 81 GB/s aggregated write and read performance

The WORK file system is based on a DDN EXAScaler and operates a DDN Lustre file system. Each site has two DDN ES18K embedded-storage systems. The work file system does not have automatic backups.

The HOME file system

The HOME file system has the following performance characteristics.

• 2 x 340 TB netto capacity
  • 2 x 280 TB for user data
  • 2 x 60 TB for snap shots and system data

The HOME file system is based on a DDN GRIDScaler (Spectrum Scale) with DDN SFA7700X block storage. The file system supports disk and inode quotas for groups and users, which are tested on all nodes for every file access. Additionally, all the usual tools e.g. hard- and soft-limits, grace periods are available on user, group and fileset level. The HOME file system is exported via NFS to the login and compute nodes. The HOME file system is constantly backed up and there are daily snapshots available.

The Tape archive

For long term data handling on both sites an archiv system based on tape storage is operated. At ZIB the archive system is a Hierarchical Storage Management (HSM) system with Sun StorEdge SAM-FS as a mangement software. In Göttingen a Tape-Library with StorNext file system software.