Page tree
Skip to end of metadata
Go to start of metadata


The NHR@ZIB Next-Generation Technology Pool of systems serves for the exploration and evaluation of new technologies for HPC and AI workloads. NHR@ZIB has a strong partnership with various vendors sharing the common goal to give experienced users insights and hands-on to future technologies.

Usage Policy

(info) Since no resource management system is currently installed, be gentle to other users, take care to not disturb experiments/measurements of others.

Systems

(1) Hosts with Intel Optane Memory

Hostnames: apass{1,2}

The two systems apass{1,2}  are equipped with Intel Optane Memory components (first generation Apache Pass): Storage Class Memory modules (SCM/NVRAM) and SSDs. The main difference between the two hosts is the memory capacity.

The Optane Memory, i.e. each of the systems, can be configured in two (three) modes:

  • Memory Mode: The Optane Memory is exposed as RAM to the OS. Although the hardware technology is different to DRAM, this mode allows transparent usage of the SCM memory. The main benefits are the gained memory capacity and easy usage (no modification of applications required). The DRAM memory acts as cache for Optane Memory. In this mode, the SCM is effectively not persistent.
  • AppDirect Mode: The Optane Memory is exposed as block device(s) to the OS (usually as /dev/pmemX, depending on actual configuration). On such block devices, a file system can be created which should support the direct access (DAX) option. This allows to map data on the persistent Optane Memory into an application's virtual address space while avoiding the OS' page cache. Thus, direct media access to the SCM is possible with load/store operations. Note that data in the Optane Memory is effectively cleared when the mode is changed. Therefore, data on /dev/pmemX should be considered as ephemeral.
  • (Mixed/Hybrid Mode: The System can also be configured to provide a portion of the Optane capacity for the Memory Mode and another portion for AppDirect mode.)

By default: apass1 is configured in Memory Mode while apass2 is configured in AppDirect Mode. If you need a different configuration contact S. Christgau Steffen Christgau . The mount points for the persistent memory are usually /mnt/pmemX. X often matches the NUMA domain of the socket/processor the memory is attached to. To be sure run lstopo from the hwloc environment module. Not every pmem device might be mounted or accessible if a system is in AppDirect mode because other software (DAOS, e.g.) may exclusively grab a device. Check the output of mount to find mount points of /dev/pmemX.

The login message (message of the day) displays the mode in which the system is currently running in. You can also check the CurrentVolatileMode property in the /var/run/optane/state file. As a further simple check for the given mode, you can run free -h. If the total memory capacity is around or larger than 3 TB the system is in memory mode. Further, if /dev/pmem[01] exists, the AppDirect (or Mixed/Hybrid) mode is in effect.

Hardware

CPU

2x Intel Xeon Platinum 8260L (24c, 2,4 GHz) Cascade Lake SP

SystemInspur NF5280M5
Memory

apass1:

  • 384 GB DDR4 (12 x 32 GB Micron 36ASF4G72PZ-2G9E2 PC4-2933 DIMMs, configured to 2666 MT/s)
  • 3 TB Optane/Apache Pass NVRAM (12 x 258496 MB Intel NMA1XXD512GPSU4, 2666 MT/s)

apass2:

  • 768 GB (12 x 64 GB Samsung M393A8G40MB2-CVF PC4-2933 DIMMs, configured to 2666 MT/s)
  • 6 TB Optane/Apache Pass NVRAM (12 x 514624 MB Intel NMA1XXD512GPSU4 DIMMs, 2666 MT/s)

All DIMM slots fully populated with Optane/DRAM pairs (2:2:2 configuration). The Optane DIMMs are interleaved and a single region spans over them (per socket)

Storage

apass1:

  • 240 GB Intel SSDSC2KB24 SATA, for OS/Home
  • 1x 8T Intel SSDPE2KX080T8 NVMe SSD, Scratch (ephemeral, might be wiped/unavailable at any time)

apass2:

  • 240 GB Intel SSDSC2KB24 SATA, for OS/Home
  • 2x 8T Intel SSDPE2KX080T8 NVMe SSD, Scratch (ephemeral, might be wiped/unavailable at any time )
NetworkSingle Port Omni-Path HFI Adapter 100 Series (back-to-back connected via Cu cable)

Software

  • OS: CentOS 7.9
  • more recent software (compilers, libraries, utilities) are available via environment modules (module avail). Module files are stored in /opt/local/modules.


click to enlarge

Pic 1: Server Board Layout

(2) NEC SX-Aurora TSUBASA A300-8

Hostname: aurora

NOTE: System software currently updated.

Hardware Configuration

CPU2x Intel Xeon Gold 6126 (12c, 2,6 GHz) Skylake
Memory

192 GB (DDR4-2666 ESS RDIMM)

Accelerators

8x NEC Vector Engines 1.0 (VE) Modell B
(4x per PCI root complex)

VE Configuration

per VE:

  • 8 cores, 1.4 GHz 
  • 48 GB HBM, 1600 MHz, 1.20 TB/s
  • peak pe: 2.15 TFLOPS
Network2x 100 Gb/s IB between the two PCI root complexes

Software

  • OS: CentOS 7.6
  • VE OS: tb
  • NEC Compiler Suite: tb




click to enlarge

Pic 2: Aurora Server with 8 VE's

(3) 3rd Gen Intel Xeon Cooper Lake

Hostname: t.b.a

Cooper Lake is Intel's codename for the third-generation of their Xeon scalable processors, developed as the successor to Cascade Lake.

Improvements:

  • New bfloat16 instruction
  • Support for up to 12 DIMMs of DDR4 memory per CPU socket


Hardware Configuration

CPU4x Intel Xeon Platinum 83xx CooperLake
Memory

384 GB

Network2x 10 Gb/s Ethernet

Software

  • OS: CentOS 8.4


(4) Intel Xeon Ice Lake

Hostname: icl

Hardware Configuration

CPU2x Intel Xeon Platinum 8360Y (36c, 2,4 GHz) IceLake
Memory

512 GB (DDR4-3200 RDIMM)

Network2x 10 Gb/s Ethernet

Software

  • OS: CentOS 8.4

Pic 3: Server Board Layout

Access

To gain access to the Next-Generation Technology Pool, contact support@hlrn.de. Please give a short description of your intention and the system you intend to use.

  • No labels