HLRN Accounting

-- Charge for System Usage --

HELP This document has been adapted to HLRN-III HELP


HLRN provides computational resources primarily to scientific institutions under public law that are located in the north-german states

  • Berlin
  • Brandenburg
  • Bremen
  • Hamburg
  • Mecklenburg-Vorpommern
  • Niedersachsen
  • Schleswig-Holstein

The details regarding the policies for resource allocation and payment are prescribed by the Fees And Regulations document Zulassungs- und Entgeltordnung (in German).


HLRN charges for system usage by batch jobs. The charge is calculated from

in units of so-called NPL (Norddeutsche Parallelrechner-Leistungseinheit, engl.: "north-german parallelcomputer work unit") .

The usage cost for the different HLRN-III node types are:

Node type / machine Cores per node NPL per node-hour NPL per core-hour
Cray XC30; MPP phase 1 24 2 (0.0833) [*]
Cray XC30; MPP phase 2 24 2.4 (0.1000) [*]
SMP nodes; phase 1 32 4 (0.125) [*]
SMP nodes; phase 2 40 6 (0.150) [*]
Data nodes 8 0.6667 0.0833
Pre/Post nodes; phase 1 32 6 0.1875
Pre/Post nodes; phase 2 40 9 0.2255

Numbers have been partially rounded to four decimal places.
[*] Important note: On the XC30 and SMP cluster nodes are always fully charged, independent of the number of cores actually used.

The details and more can be found in the following sections.

Charge Model

The charge for computational work is determined by the amount of reserved computational resources, the time these resources are reserved, and a charge rate constant:

Charge = Resources * Time * ChargeRate

Resources amount of computational resources reserved, measured in Processor Equivalents (PEQ)
Time usually the wallclock time duration
ChargeRate cost of 1 PEQ per time, measured in NPL (Norddeutsche Parallelrechner-Leistungseinheit)

This formula applies to batch jobs (including interactive batch jobs).

Interactive usage outside of batchjobs as well as usage of magnetic tapes or hard disk capacity is currently not accounted.

Definition of Resources

Resources are measured in Processor Equivalents (PEQ). A PEQ is an artificial resource, which combines several computational resources like CPUs or Memory into one single number. The definition is machine or node-type dependent:

Machine architecture
Node type
Cray XC30 / mpp1 1 PEQ = 1 Core [*]
Cray XC30 / mpp2 1 PEQ = 1 Core [*]
SMP nodes; phase 1 1 PEQ = 1 Core [*]
SMP nodes; phase 2 1 PEQ = 1 Core [*]
Data nodes 1 PEQ = 1 Core
Pre/post nodes; phase 1 1 PEQ = 1 Core
Pre/post nodes; phase 2 1 PEQ = 1 Core
historic resources:
Machine architecture
Node type
HLRN-II Intel E5472 (SGI ICE1, SGI XE) 1 PEQ = 1 Core
Intel X5570 (SGI ICE2) 1 PEQ = 1 Core
Intel X7560 (SGI UV) 1 PEQ = 1 Core
HLRN-I IBM Power4 1 PEQ = max(CPUs / 32, Memory / 64 GByte)

The number of PEQ reserved by a batch job depends on the batch job setup. Batch jobs, which request dedicated node usage, reserve all PEQs of the reserved nodes, regardless of the true number of cores or amount of memory the batch job really uses.

[*] Important note: On the XC30 and SMP cluster nodes are always fully charged, independent of the number of cores actually used.

Definition of Time

The time a batch job is charged for is the effective wallclock time duration of the job (not the requested wallclock time).

Definition of the Charge Rate

The charge for 1 PEQ per time depends on the architecture of the used machines.

Names of architectures are derived from the node types (like mpp1, smp1, etc.), which are used in the cluster types provided at HLRN (e.g. mpp1 is found in the Cray XC30).

See the Hardware Overview for current architectures at HLRN.

Machine architecture Charge Rate Example
Cray XC30; MPP phase 1 1 / (12 PEQs * 1 hour) 12 Cores (0.5 nodes) cost 1 NPL per hour [*]
Cray XC30; MPP phase 2 1 / (10 PEQs * 1 hour) 12 Cores (0.5 nodes) cost 1.2 NPL per hour [*]
SMP nodes; phase 1 1 / ( 8 PEQs * 1 hour)   8 Cores (0.25 nodes) cost 1 NPL per hour [*]
SMP nodes; phase 2 1 / ( 6.6667 PEQs * 1 hour) 20 Cores (0.5 nodes) cost 3 NPL per hour [*]
Data nodes; phase 1 1 / (12 PEQs * 1 hour) 12 Cores (1.5 nodes) cost 1 NPL per hour
Pre/Post nodes; phase 1 1 / ( 5.3333 PEQs * 1 hour) 16 Cores (0.5 nodes) cost 3 NPL per hour
Pre/Post nodes; phase 2 1 / ( 4.4444 PEQs * 1 hour) 40 Cores (1 node) cost 9 NPL per hour
historic definitions:
Machine architecture Charge Rate Example
HLRN-II Intel E5472 (SGI XE, SGI ICE1) 1 / (24 PEQs * 1 hour) 24 Cores (3 nodes) cost 1 NPL per hour
Intel X5570 (SGI ICE2) 1 / (12 PEQs * 1 hour) 12 Cores (1.5 nodes) cost 1 NPL per hour
Intel X7560 (SGI UV) 1 / (16 PEQs * 1 hour) 16 Cores (1 blade) cost 1 NPL per hour
HLRN-I IBM Power4 1 / (32 PEQs * 1 hour) 32 CPUs cost 1 NPL per hour

Numbers have been partially rounded to four decimal places.

[*] Important note: On the XC30 and SMP cluster nodes are always fully charged, independent of the number of cores actually used.

The design goals and properties of charge rates at HLRN are as follows:

  • A given computational work in terms of FLOP (number of Floating point operations) should be charged for roughly the same amount of NPL, regardless of the used machine architecture. This holds when averaging over the application mix at HLRN. However, when looking at a particular application, it will of course turn out that the use of a given architecture will be cheaper or more expensive than another one, but not by orders of magnitude.
  • The charge rates roughly reflect the relations of data transfer rates (memory and network bandwidths) of different architectures.
  • The charge rates represent a mechanism to smoothly enforce the intended usage models of different architectures by making architectures that are not suitable for a given work more expensive, when this work is done there.

Accounts, Projects

HLRN manages accounts in terms of a bank, which hold credits in units of NPL . These accounts are debited with the charge for batch jobs.

Credits are managed in quartely time frames.

Often the terms Account and Project are used as synonyms.

HLRN users have access to the HLRN account management system via the Service Portal.

For those users who have difficulties to understand bureaucratic english, here the translation of the above sentences into bureaucratic german:

HLRN pflegt Konten im Sinne einer Bank. Diese Konten enthalten Kontingente in NPL . Die Konten werden mit den Kosten für Batch-Jobs belastet.

Die Kontingente werden quartalsweise zugeteilt, Kosten werden quartalsweise abgerechnet.

Die Begriffe Konto und Projekt werden oft synonym verwendet.

HLRN-Nutzer haben Zugang zum HLRN Account-Management-System über das Service Portal.

Account Types

Personal Account

Every HLRN user owns a personal account, which is similar to a personal project. The account and project name is equal to the name of the UNIX account name of the user. The UNIX account must not be mixed up with the personal account (project). These are completely different things, only their names are equal.

The personal account is created, when the user gets granted access to the HLRN systems for the first time.

See How to Become a User for details.

The personal account holds 2500 NPL per quarter, as long as the user exists at HLRN (see also account lifetime). The default allocation may be increased on request (up to four times the default allocation, i.e. 10,000 NPL; justification required, approval by your HLRN consultant).

Project Account

HLRN users may apply for Projects. Project applications are reviewed by the HLRN Scientific Board. Essentially, a project application requests a certain amount of credits in units of NPL for some time period.

See How to Apply for Resources for details.

When a project application is submitted, a corresponding project account is created. When the application is approved, HLRN deposits the approved number of NPL into the project account.

Special Accounts

Special accounts are managed for HLRN administrators and consultants.

Connection between Batch Job and Account

Every batch job states the name of the account (project), which has to be charged for the resources requested by this job. This account is determined at submit time of the job.

The specification of the account is done with appropriate keywords of the used batch queueing system:

Batch system Job script Command line
Moab #PBS -A account msub -A account
historic systems:
LoadLeveler #@ account_no = account  
Torque #PBS -A account qsub -A account

When a batch job does not state an account (project), a default is taken from the account database. The default is the DefaultProject a user has chosen. It defaults to the personal project of the user.

A user may modify his DefaultProject by using the Service Portal.

The setting of the account for a given batch job is verified at submit time. The submit request will be rejected, if

  • the account does not exist
  • the job owner has no access to the account

Queued batch jobs will be checked periodically and at least at job start, whether their account setting is valid in the terms given above.

Queued batch jobs, that state an invalid account will be put into the SystemHold state, and will contain a descriptive message about the reason. Job owners may correct the reason to free these jobs. These actions may involve:

  • changing the account setting to something valid (use mjobctl for Moab jobs)
  • ask HLRN staff to move project account allocations between quarters
  • ask the HLRN Scientific Board to approve additional NPL
Please contact your responsible HLRN consultant in the latter cases.

Account Management

Every project account has at least one associated project administrator. A project administrator may

  • grant other users access to the project account or remove them from the project account (manage project members)
  • define additional project administrators

A project administrator uses the HLRN Service Portal for managing his project account. See also the page on options and policies for users and projects.

The HLRN user who submits the project application becomes the initial project administrator and the first member of a project.

All actions given above are on behalf of the project administrator. HLRN is not involved here.

Account Lifetime

Project Accounts

Project accounts live for an indefinite time, once they got the first allocation.

When a project account has got allocations for the current quarter, the account (project) is active.

When a project account has no allocation for the current and future quarters, the project is regarded as expired. If a member of this project stated this project as his DefaultProject, his DefaultProject is reset to his personal project.

An expired project account may get allocations at any time for any time in the future, which leads to reactivation of the project.

Personal Accounts

Personal accounts live as long as they hold allocations. The default lifetime is the quarter in which they have been applied for plus the two following quarters.

HLRN manages personal account allocations at every quarter start, in particular:

  • If the owner of the personal account has access to any active project account, then HLRN deposits 2500 NPL into the personal account for the current quarter.
  • If the owner has not been a member of any active project for two quarters, his personal account is not touched.
  • If the personal account holds no allocation at quarter start, it expires. The owner is still able to log into the HLRN machines, and may do interactive work. The owner is not able to submit batch jobs.
  • A personal account that expired in the preceeding quarter, becomes deleted. The owner is not able to log into the HLRN machines. All files on HLRN hard disks and magnetic tapes, which are owned by the associated UNIX account, are deleted.

Deleted personal accounts and their associated UNIX accounts cannot be reactivated.

Created by BerndKallies - 12 Dec 2008
Adapted for HLRN-III by WolfgangBaumann - 15 Jan 2014

Last modification: WolfgangBaumann - 12 Jun 2017 12:47 / 1 year, 1 week, 20 hours ago. (Version: 32)

Norddeutscher Verbund für Hoch- und Höchstleistungsrechnen
Back to top of page