Parallel Computation - running the "g-bcast" program interactively

Introduction

The programs for Gauss elimination are written in FORTRAN 77 using the MPI communication library. The g-bcast program will be run on the Data Star, by following the instructions given below.
If you are interested in more information regarding the use of the Data Star, it's recommended that you read the Data Star User Guide.

How to start

These programs can be run in the Linux environment. To start, you have two options:
- To use Linux machines. You have to start a terminal.
- To use the Windows NT machines. You have to use the "SSH" program. To start SSH, start from
Start-->Programs-->Network Applications-->SSH--->Secure Shell Client
Type "linux" as Host Name and click Connect. Don't be impatient if it doesn't immediately connect. Enter your CADLAB login and password. This will make a connection to one of the Linux machines in Engineering I. Now you are on your account via a Unix interface.

Login on the Data Star

To login on the Data Star you must use the secure shell command:

        % ssh -l UserID dspoe.sdsc.edu 

By this, you connect to the interactive nodes on Data Star. These are the nodes that have been set up for interactive use. There are 4 interactive nodes. Each has 8 POWER4 CPUs and 2 GB of memory. Interactive access to the nodes is shared -at times, there may be more than one user job running on a node, and for this, there may be a significant run-time variability.

Once you login the first time, you must change your password:

        % rlogin tfpasswd
        % passwd 

Transfer the files from the HPSS [High Performance Storage System] to the Data Star

To transfer the pge modules and auxiliary files to the Data Star, you have to use pftp. pftp is a ftp-like interface to HPSS. The following links and text below describe how to use the pftp utility:

        % pftp 
    pftp> cd /users/csb/u4078 
    pftp> get pge.tar 
    pftp> quit 
        %   

Compiling FORTRAN programs on the Data Star

In order to compile and run the pge module, you have to untar the file pge.tar first:

        %tar xvf pge.tar

In the directory pgauss, the following files are present:

        - Makefile
        - g-bcast.f

To compile the pge module, change to the pgauss directory (type "ls" to see what is inside the directory). To run make we have to have a file named Makefile in the directory:

        % cd pgauss
        % make 

These will create a new executable file bcast by using g-bcast.f.

Running programs on the Data Star

We can now run the bcast module using these input files. The command for this use is "poe".

The "poe" command invokes the Parallel Operating Environment for loading and executing programs on remote processor nodes. The operation of POE is influenced by a number of POE environment variables. Here are the explanations of the environmental variables that we are going to use, obtained by typing "man poe". Each flag (stating with -) is connected to the environmental variable starting with MP_ and followed by the name of the flag and sets them temporarily during the execution:

     
     MP_NODES
     Specifies the number of physical nodes on which to run the parallel
     tasks. It may be used alone or in conjunction with MP_TASKS_PER_NODE
     and/or MP_PROCS. The value of this environment variable can be 
     overridden using the -nodes flag.

     MP_TASKS_PER_NODE
     Specifies the number of tasks to be run on each of the physical nodes.
     It may be used in conjunction with MP_NODES and/or MP_PROCS,
     but may not be used alone. The value of this environment variable 
     can be overridden using the -tasks_per_node flag.    

     MP_RMPOOL
     Determines the name or number of the pool that should be used for
     non-specific node allocation. This environment variable/command-line
     flag only applies to LoadLeveler. Valid values are any identifying
     pool name or number. There is no default. The value of this
     environment variable can be overridden using the -rmpool flag.

     MP_EUILIB
     Determines the communication subsystem library implementation to use
     for communication - either the IP communication subsystem or the US
     communication subsystem. In order to use the US communication
     subsystem, you must have an SP system configured with its high
     performance switch feature. Valid, case-sensitive, values are ip (for
     the IP communication subsystem) or us (for the US communication
     subsystem). The value of this environment variable can be overridden
     using the -euilib flag.
     
     MP_EUIDEVICE
     Determines the adapter set to use for message passing. Valid values
     are en0 (for Ethernet), fi0 (for FDDI), tr0 (for token-ring), css0
     (for the SP system's high performance switch feature), and csss (for
     the SP switch 2 high performance adapter).

In our case, we use the following syntax:

        % poe bcast -nodes number_of_nodes -tasks_per_node number_of_tasks -rmpool 1 -euilib ip -euidevice en0

The number of processors used are equal to the product of number_of_nodes and tasks_per_node . The number of processors can be a maximum of 32. The following combinations are recommended:
4 processors -> -nodes 1 -tasks_per_node 4
8 processors -> -nodes 1 -tasks_per_node 8
16 processors -> -nodes 2 -tasks_per_node 8
32 processors -> -nodes 4 -tasks_per_node 8

After the execution, the bcast program will output the solution x.