Parallel Computation - running the "pfda" program and analyzing with VAMPIR

Introduction

The modules for the finite difference solution using an adaptive mesh for the dam seepage problem are written in FORTRAN 77 using the MPI communication library. The 'pfda' program will be run on the Data Star, by following the instructions given below.
If you are interested in more information regarding the use of the Data Star, it's recommended that you read the Data Star User Guide.

VAMPIR has been introduced earlier with "etch" module. Go over previous instructions, if you don't feel comfortable from "using the VAMPIR tool on the Data Star - Part I and II" for the "etch" module.

How to start

These programs can be run in Linux environment. To start, you have two options:
- To use Linux machines. You have to start a terminal.
- To use the Windows NT machines. You have to use the "SSH" program. To start SSH, start from
Start-->Programs-->Network Applications-->SSH--->Secure Shell Client
Type "linux" as Host Name and click Connect. Don't be impatient if it doesn't immediately connect. Enter your CADLAB login and password. This will make a connection to one of the Linux machines in Engineering I. Now you are on your account via a Unix interface.

Login on the Data Star

To login on the Data Star you must use the secure shell command, in LINUX environment:

        % ssh -lUserID dslogin.sdsc.edu

Transfer the files from the HPSS [High Performance Storage System] to the Data Star

To transfer the "fda" modules and auxiliary files to the Data Star, you have to use "pftp". "pftp" is an ftp-like interface to HPSS. The following links and text below describe how to use the "pftp" utility:

        % pftp 
    pftp> cd /users/csb/u4078 
    pftp> get fda.tar 
    pftp> quit 
        %   

Compiling FORTRAN programs on the Data Star

In order to compile and run the pfda module, you have to untar the file fda.tar:

        %tar xvf fda.tar

In the directory fda, the following files will be present:

        - Makefile
        - host.f
        - node.f
        - partial.f
        - whole.f
        - out

Change to the fda directory(type "ls" to see what is inside the directory). On file Makefile, we have to make a change to obtain the trace file of the parallel computation. For this, open Makefile with one of the easiest editors on Unix:Pico

        % cd fda
        % ls
        % pico Makefile

This will open the Makefile with Pico editor. Now you can add the link to the Vampir library, to obtain the trace file after running the program. For this add
"-L/usr/local/apps/vamp/lib -lVT -lm -lld" after the original third line:

        mpxlf  $(OBJS) -o pfda

to obtain new third line:

        mpxlf  $(OBJS) -o pfda -L/usr/local/apps/vamp/lib -lVT -lm -lld     

Now you can exit Pico editor with "^X" aka "Control+X", note that the file name should stay the same. Compile the fda module, by running make:

        %make

This will create a new executable file "pfda" which is linked with VAMPIR tracing library.

Running programs on the Data Star

To run the 'pfda' module you must be logged on into the so-called interactive nodes:

        % ssh -lUserID  dspoe.sdsc.edu

These are the nodes that have been set up for interactive use. There are 4 dspoe nodes. Each has 8 POWER4 CPUs and 2 GB of memory. Interactive access to the nodes is shared - at times, there may be more than one user job running on a node, for this, there may be a significant run-time variability.

To run the 'pfda' module with the VAMPIR tool, set the paths to the VAMPIR tool:

        % set PAL_ROOT=/usr/local/apps/vamp
        % set PAL_LICENSEFILE=/usr/local/apps/vamp/license.dat 
        % set path = ($path /usr/local/apps/vamp/bin)

and then invoke the parallel program using the poe command-line flags:

        % poe etch -nodes your_number_of_nodes -tasks_per_node your_tasks_per_node -rmpool 1 -euilib ip -euidevice en0

The program executes, generating a trace file called pfda.bpv.

Starting the VAMPIR session

After the program has finished executing, start a VAMPIR session:

         Set the DISPLAY variable to the name of the machine you are logged into
         [for example "ecipc004"]:
             % setenv DISPLAY ecipc004.engr.ucsb.edu:0.0
         Enter:   
             % vampir pfda.bpv

The VAMPIR Main window and the Global Timeline Display window open.

Open views to visualize the trace records

To do this, click on the Global Displays from the main window and open:

 
         * Summary Chart 
         * Activity Chart 
 

You can select any other views. To interpret the information these windows present, see VAMPIR User Guide.

View the trace file.

We'll start by viewing the time for the entire run. Zoom in on a section of the timeline:

    * Click and drag over a portion of the timeline with the left mouse. 
      This part will be magnified.
    * Continue zooming until most of the MPI function names are revealed.

View process statistics for the selected portion of the timeline.

From the Global Display menu, select Summary Chart view. A new view will open. Press the right mouse button within this window.

     * Select the Use Timeline Portion. 
     * Scroll the timeline, using the scroll bar at the bottom of the timeline
       window, and watch what happens in both displays.

View the "Activity Chart".

The "Activity Chart" display shows a statistic about the time spent in each activity individually for each process defined in the tracefile. With the default pie chart display you can recognize load imbalance at a glance in the trace program by comparing the different time consumptions of the activities over all processes.

End the VAMPIR session.

To do this, select:

        File --> Exit