Parallel Computation - running the "pfea" program and analyzing with VAMPIR

Introduction

The modules for the finite element solution using an adaptive mesh for the dam seepage problem are written in FORTRAN 77 using the MPI communication library. The 'pfea' program will be run on the Data Star, by following the instructions given below.
If you are interested in more information regarding the use of the Data Star, it's recommended that you read the Data Star User Guide.

VAMPIR has been introduced earlier with "etch" module. Go over previous instructions, if you don't feel comfortable from "using the VAMPIR tool on the Data Star - Part I and II" for the "etch" module.

How to start

These programs can be run in Linux environment. To start, you have two options:
- To use Linux machines. You have to start a terminal.
- To use the Windows NT machines. You have to use the "SSH" program. To start SSH, start from
Start-->Programs-->Network Applications-->SSH-->Secure Shell Client
Type "linux" as Host Name and click Connect. Don't be impatient if it doesn't immediately connect. Enter your CADLAB login and password. This will make a connection to one of the Linux machines in Engineering I. Now you are on your account via a Unix interface.

Login on the Data Star

To login on the Data Star you must use the secure shell command, in LINUX environment: 

        % ssh -lUserID dslogin.sdsc.edu

Transfer the files from the HPSS [High Performance Storage System] to the Data Star

To transfer the "fea" modules and auxiliary files to the Data Star, you have to use "pftp". "pftp" is a ftp-like interface to HPSS. The following links and text below describe how to use the "pftp" utility: 

        % pftp 
    pftp> cd /users/csb/u4078 
    pftp> get fea.tar 
    pftp> quit 
        %

Compiling FORTRAN programs on the Data Star

In order to compile and run the pfea module, you have to untar the file fea.tar: 

        %tar xvf fea.tar
In the directory fea, the following files will be present: 
        - Makefilepfea
        - Makefilegen
        - fea.f
        - node.f
        - minband.f
        - genmesh.f
        - decom.f
        - plotfea.f
Change to the fea directory (type "ls" to see what is inside the directory). On file Makefilepfea, we have to make a change to obtain the trace file of the parallel computation. For this, open Makefilepfea with one of the easiest editors on Unix:Pico 
        % cd fea
        % ls
        % pico Makefilepfea
This will open the Makefilepfea with Pico editor. Now you can add the link to the VAMPIR library, to obtain the trace file after running the program. For this: 
   
        add "-L/usr/local/apps/vamp/lib -lVT -lm -lld" after the original third line:

        mpxlf -o pfda $(OBJS)
        

        to obtain new third line:

        mpxlf -o pfea $(OBJS) -L/usr/local/apps/vamp/lib -lVT -lm -lld
Now you can exit Pico editor with "^X" aka "Control+X", note that the file name should stay the same.
The compilation of the pfea module is achieved by running make. To run make we have to have a file named Makefile in the directory. However, there are two types of Makefile in this directory, each for a different program. For this we have to run make for each in turn. So, we have to create a Makefile using one first, and run make, then create the other Makefile(overwriting the first) and run make again. 
        % cp Makefilepfea Makefile
        % make
        % cp Makefilegen Makefile  
        overwrite Makefile? y
        % make
These will create a new executable "pfea" file by using fea.f and node.f and a "gen" file by using minband.f, genmesh.f and decom.f.

Running programs on the Data Star

To run the 'pfea' module you must be logged on into the so-called interactive nodes: 

        % ssh -lUserID  dspoe.sdsc.edu
These are the nodes that have been set up for interactive use. There are 4 interactive nodes. Each has 8 POWER4 CPUs and 2 GB of memory. Interactive access to the nodes is shared - at times, there may be more than one user job running on a node, for this, there may be a significant run-time variability.

To run the adaptive mesh finite element analysis module with the VAMPIR tool, set the paths to the VAMPIR tool: 
        % set PAL_ROOT=/usr/local/apps/vamp
        % set PAL_LICENSEFILE=/usr/local/apps/vamp/license.dat 
        % set path = ($path /usr/local/apps/vamp/bin)
You can now obtain the trace files for the VAMPIR tool. Among our "pfea" and "gen" programs, only "pfea" runs parallel, so VAMPIR will be used for "pfea" only.

Run the "gen" and "pfea" programs as described before. The only thing that is different this time is that, after the "pfea" program executes, it generates a trace file called pfea.bpv.

Starting the VAMPIR session

After the program has finished executing, start a VAMPIR session: 

         Set the DISPLAY variable to the name of the machine you are logged into
         [for example "ecipc004"]:
             % setenv DISPLAY ecipc004.engr.ucsb.edu:0.0
         Enter:   
             % vampir pfea.bpv
The VAMPIR Main window and the Global Timeline Display window open.

Open views to visualize the trace records

To do this, click on the Global Displays from the main window and open: 

 
         * Summary Chart 
         * Activity Chart
You can select any other views. To interpret the information these windows present, see VAMPIR User Guide.

View the trace file.

We'll start by viewing the time for the entire run. Zoom in on a section of the timeline: 

    * Click and drag over a portion of the timeline with the left mouse. 
      This part will be magnified.
    * Continue zooming until most of the MPI function names are revealed.

View process statistics for the selected portion of the timeline.

From the Global Display menu, select Summary Chart view. A new view will open. Press the right mouse button within this window. 

     * Select the Use Timeline Portion. 
     * Scroll the timeline, using the scroll bar at the bottom of the timeline
       window, and watch what happens in both displays.

View the "Activity Chart".

The "Activity Chart" display shows a statistic about the time spent in each activity individually for each process defined in the tracefile. With the default pie chart display you can recognize load imbalance at a glance in the trace program by comparing the different time consumptions of the activities over all processes.

End the VAMPIR session.

To do this, select: 

        File --> Exit