Berlin: Data Migration from the SGI to the Cray System

The Berlin HLRN-II system will be switched off on August 26th 2013. Please take notice of the following informations about your data on the Berlin HLRN-II directories $HOME, $WORK, $WORK2 and $PERM (See also our previous migration announcements collected in our topic MigrationAnnouncements).

Summary and Important Notes

The home and the working file systems will be switched off together with the compute components of the HLRN-II. Data in your home and working directories $HOME, $WORK and $WORK2 will NOT be preserved. You are responsible for copying/moving your important data in these directories to the new HLRN-III file systems for later use. For this purpose, we offer an immediately starting

ALERT! data migration phase until August 26, 2013 ALERT!

(see below for additional known time restrictions)

Use our migration servers to copy all desired data from the old HLRN-II filesystems to the new Berlin HLRN-III directories. The important steps for data migration are:

On the HLRN-II SGI system in Berlin, examine your home and working directories:

  1. Delete all data, that is not needed anymore (e.g. because you have saved it to PERM or elsewhere before).
  2. Move important data, that is not needed within the next few months to the archive filesystem PERM
  3. For data, that is really needed immediately when the HLRN-III Cray system comes online, login to the migration servers and
    1. copy the remaining data from the home directory to the new home directory
    2. copy the remaining data from the working directories to the new working directory
We advise to perform the actions in parallel ("functional parallelism").

Schedule and Maintenance Restrictions

We strongly recommend to migrate your data as soon as possible. Due to possible insufficient network bandwidth, file transfers in the last days may not succeed.

Here is the schedule for the migration period in Berlin including maintenance windows for infrastruture work:

Jul 29 one day infrastruture maintenance; no data migration done
Jul 30 until
approx. Aug 09
open for data migration done
Aug 09 9:00-12:00: 3 hours migration system maintenance;
no data migration
done
Aug 09 from 12:00: reopen for data migration done
Aug 13, 8:00 until
Aug 14, 10:00
two days Cray system setup; no data migration done
Aug 14 until
Aug 26
reopen for data migration  
Aug 27 Final shutdown of the HLRN-II SGI system; end of data migration  

Home directory $HOME and working directories $WORK/$WORK2

On the migration servers you have

  • access to the HLRN-II directories $HOME, $WORK and $WORK2
  • access to your new HLRN-III directories $NEWHOME and $NEWWORK
Please note: the environment variables $NEWHOME and $NEWWORK are defined on the migration servers, only.

Please also note that you have only one work directory $NEWWORK on the new HLRN-III system. There is (and will be) no backup for this work directory.

To copy data from some HLRN-II directory olddir to an equivalent HLRN-III directory newdir, please follow these basic steps:

  1. Login to a migration server.
  2. Copy environment files from $HOME.
  3. Copy data from the old directory olddir to the new directory newdir.
  4. Verify the data correctness in newdir.

Login to the migration servers

The new migration nodes at Berlin HLRN are named bmig1, bmig2, bmig3 (update Aug 19: and bmig4. All migration servers have access to all old and new home and working directories. Access to the migration servers is possible only from the Berlin HLRN gateway nodes. Thus, please first login to a gateway node and from there to a migration server. Try to use one that has a lower load than others, as there is no automatic load balance active.

Example 1: Login with user account xyzname to bmig3:

$ ssh xyzname@bicegate.hlrn.de
********************************************************************************
*                                                                              *
*                 Welcome to HLRN-II Complex Berlin                            *
*                                                                              *
*                                                                              *
* Documentation:  https://www.hlrn.de/home/view/Service/WebHome                *
* Support:        mailto:support@hlrn.de                                       *
*                                                                              *
********************************************************************************
bicegate0:~ $ ssh bmig3
Last login: Thu Jul 25 10:43:45 2013 from bg1.hlrn.de
##############################################################################
#                                                                            #
#          Welcome to the HLRN Data Migration Servers in Berlin              #
#                                                                            #
#                       Data migration is possible                           #
#                    -->   until August 26, 2013  <--                        #
#                                                                            #
#  Documentation:  https://www.hlrn.de/home/view/Service/MigrationSgiToCray  #
#  Support:        mailto:support@hlrn.de                                    #
#                                                                            #
##############################################################################
bmig3:~ $ 

Copy environment files from $HOME

For a successful login to the future HLRN-III system, you need to have environment files (dot files) in your new HOME directory $NEWHOME, e.g. $HOME/.profile. With these files, you can control important environment variables via the command module.

Thus, we strongly recommend to copy important dot files and directories from $HOME to $NEWHOME (option -a preserves the original attributes of files and directories). Do this at least for your dot files and dot directories:

bmig3:~ $ cp -a $HOME/.??*  $NEWHOME

HLRN system administration will not transfer your dotfiles and such for you (for exceptions from this rule see below)

Copy data from olddir to newdir

By using copy commands you are able to copy data from any HLRN-II source to any HLRN-III target.

Example 2: Copy a single file expample.dat from some $HOME/olddir/ to some $NEWHOME/newdir/

bmig3:~ $ mkdir $NEWHOME/newdir
bmig3:~ $ cp -a $HOME/olddir/example.dat $NEWHOME/newdir/

Example 3: Recursively copy some subdirectory olddir from $HOME to $NEWHOME

bmig3:~ $ cp -a $HOME/olddir $NEWHOME

Example 4: Copy all data recursively from your old $WORK2 to a subdirectory of $NEWWORK

bmig3:~ $ mkdir $NEWWORK/oldwork2
bmig3:~ $ cp -a $WORK2/*  $NEWWORK/oldwork2/

Caution! Do not copy hundreds of GigaBytes or hundreds of thousands of files in one call. Depending on your network bandwidth share his may take lots of hours to finish. If transfer interrupts occur you will have to restart the copy. In such cases you should use the rsync example below to (re-)transfer only those files that did not make it to the new filesystem.

Example 5: Copy from $WORK to $NEWWORK using rsync

bmig3:~ $ rsync -a $WORK/*  $NEWWORK

Using rsync is also recommended if you change some data on the SGI after the initial transfer has completed. rsync transfers only those files again. See man rsync for additional options.

Verify for data correctness

It is a good practise to check both files, the original and the copy, to be equal. For that you calculate the checksums, e.g. with the programs md5sum or cksum, and compare for equalness.

Who copies your data from HLRN-II to HLRN-III?

You (the users) are responsible for copying/moving your important data to the new HLRN-III file systems. In cases where the migration cannot be accomplished by you within the given time frame until August 26 (e.g. you are on vacation or on leave with little or no access to HLRN), you may ask HLRN Support for assistance as soon as possible. In an email to Support please name all files and directories you wish to be migrated to HLRN-III. Be aware that for an orderly migration it may be necessary to lock your HLRN account by system administration until the end of the migration period. This means for you there will be no interactive access to HLRN and batch jobs will not be started.

Permanent directory $PERM

All user data in $PERM will continue to be available on the new HLRN-III system. For this data set, no data copy is necessary.


Created by -- MatthiasLaeuter - 18 Jul 2013
Last modification: WolfgangBaumann - 18 Aug 2013 21:03 / 5 years, 3 months, 3 weeks ago. (Version: 8)

 
Norddeutscher Verbund für Hoch- und Höchstleistungsrechnen
Back to top of page