The modules for the finite difference solution for the dam seepage problem are written in FORTRAN 77 using the MPI communication library. The "pfda" program will be run on the Data Star, by following the instructions given below.
If you are interested in more information regarding the use of the Data Star, it's recommended that you read the Data Star User Guide.
These programs can be run in Linux environment. To start, you have two options:
- To use Linux machines. You have to start a terminal.
- To use the Windows NT machines. You have to use the "SSH" program. To start SSH, start from
Start-->Programs-->Network Applications-->SSH-->Secure Shell Client
Type "linux" as Host Name and click Connect. Don't be impatient if it doesn't immediately connect. Enter your CADLAB login and password. This will make a connection to one of the Linux machines in Engineering I. Now you are on your account via a Unix interface.
To login on the Data Star you must use the secure shell command, in LINUX environment:
% ssh -lUserID dslogin.sdsc.eduOnce you login the first time, you must change your password:% rlogin tfpasswd % passwd
To transfer the "fda" modules and auxiliary files to the Data Star, you have to use "pftp". "pftp" is a ftp-like interface to HPSS. The text below describe how to use the "pftp" utility:
% pftp pftp> cd /users/csb/u4078 pftp> get fda.tar pftp> quit %
In order to compile and run the pfda module, you have to untar the file fda.tar:
%tar xvf fda.tarIn the directory fda, the following files will be present:- Makefile - host.f - node.f - plotfda.f - outTo compile the fda module, change to the fda directory (type "ls" to see what is inside the directory) and run "make". The purpose of the make utility is to determine automatically which pieces of a large program need to be recompiled, and issue the commands to compile them.(you can learn more about the commands by typing "man command_name"):% cd fda % ls % makeThis will create a new executable file "pfda" using host.f and node.f.
To run the 'pfda' module you must be logged on into the so-called interactive nodes:
% ssh -lUserID dspoe.sdsc.eduThese are the nodes that have been set up for interactive use. There are 4 dspoe nodes. Each has 8 POWER4 CPUs and 2 GB of memory. Interactive access to the nodes is shared - at times, there may be more than one user job running on a node, for this, there may be a significant run-time variability.
To run the 'pfda' module, we are going to use "poe" command. The "poe" command invokes the Parallel Operating Environment for loading and executing programs on remote processor nodes. The operation of POE is influenced by a number of POE environment variables. Here are the explanations of the environmental variables that we are going to use, obtained by typing "man poe". Each flag (stating with -) is connected to the environmental variable starting with MP_ and followed by the name of the flag and sets them temporarily during the execution:MP_NODES Specifies the number of physical nodes on which to run the parallel tasks. It may be used alone or in conjunction with MP_TASKS_PER_NODE and/or MP_PROCS. The value of this environment variable can be overridden using the -nodes flag. MP_TASKS_PER_NODE Specifies the number of tasks to be run on each of the physical nodes. It may be used in conjunction with MP_NODES and/or MP_PROCS, but may not be used alone. The value of this environment variable can be overridden using the -tasks_per_node flag. MP_RMPOOL Determines the name or number of the pool that should be used for non-specific node allocation. This environment variable/command-line flag only applies to LoadLeveler. Valid values are any identifying pool name or number. There is no default. The value of this environment variable can be overridden using the -rmpool flag. MP_EUILIB Determines the communication subsystem library implementation to use for communication - either the IP communication subsystem or the US communication subsystem. In order to use the US communication subsystem, you must have an SP system configured with its high performance switch feature. Valid, case-sensitive, values are ip (for the IP communication subsystem) or us (for the US communication subsystem). The value of this environment variable can be overridden using the -euilib flag. MP_EUIDEVICE Determines the adapter set to use for message passing. Valid values are en0 (for Ethernet), fi0 (for FDDI), tr0 (for token-ring), css0 (for the SP system's high performance switch feature), and csss (for the SP switch 2 high performance adapter).The number of processors used are equal to the product of number_of_nodes and tasks_per_node and can be maximum 64 for our program. In our case, we use the following syntax for the poe(To proceed, read the instructions carefully):% poe pfda -nodes number_of_nodes -tasks_per_node number_of_tasks -rmpool 1 -euilib ip -euidevice en0The program will request some inputs from the user before starting the computation. First the number of columns and then the number of rows of the subdomains are asked. The program will divide the rectangular domain of the dam into subdomains as specified by the given column and row numbers. Note here, that the number of subdomains should be equal to the number of processors.If the number of subdomains are greater than number of processors, you will get an error message like
ERROR: 0032-102 Invalid destination rank (4) in MPI_Send, task 0If the number of subdomains are greater than number of processors, the program will run forever.Once the subdomains are specified, the next input requested will be the "computation matrix size". This input specifies how many points are going to be computed on the square dam domain. The range of this input is [1,5].
The computation matrix size with respect to the integer input are as follows.
Input= 1 --> Computation Matrix Size= 21x21
Input= 2 --> Computation Matrix Size= 41x41
Input= 3 --> Computation Matrix Size= 61x61
Input= 4 --> Computation Matrix Size= 81x81
Input= 5 --> Computation Matrix Size= 101x101
The next input is the "relaxation factor". Since the program performs an SOR iteration, this is the relaxation factor for the SOR.
The execution summary is both printed on the screen and written in file out. This is an ASCII text file and can be viewed with "more" and "cat" commands or any of the editors, and will contain:
%more out ============================================================ column: row: rsize: number of nodes used: relaxation factor: maxtime used is node: ite #: time: ============================================================
The 'pfda' program outputs a set of files named as "da1","da2" etc. according to the number of the subdomains. Each output is from one processor.
There is a program available to interprete these data files and output new files which are in a plottable format. This program is plotfda.f. It can either plot the output for only one subdomain or plot the whole domain.
First, compile this program by typing% f77 -o plot plotfda.fSuccessful compilation will generate the executable plot file.
The "plot" program can create an output file in plottable format either for only one subdomain or the whole domain. It will require an input to choose between these once you run it:
% plot Do you want to print one subdomain(s) or the whole(w) domain?One subdomain option outputs a file plotting the values of the subdomain in 3-D (x-y-value). Whole domain option outputs two files, one is the 2-D(x-y) plot for the seepage boundary and the other is the 3-D (x-y-value) value plot of the whole domain.You can choose in which format you want the output to be between two different types. The program can give outputs in one of the two plot formats: Gnuplot or Matlab. The instructions for both type are given below.
Gnuplot is an interactive plotting software available on UNIX. If you choose to use Gnuplot, then follow the following steps to obtain the graphs.
First run the programs and obtain the plottable output files. Transfer the files to a UCSB machine(into your X:\ directory), using the SCP utility:To copy the file "pi.c" from a local machine to the Data Star: scp pi.c u4078@dslogin.sdsc.edu:pi.c To copy the file pi.c from Data Star to the Engineering domain machines: scp pi.c stefan@linux.engr.ucsb.edu:pi.c which will copy the "pi.c" file to the machines both in the "ENGR" domain. ["u4078" is the LoginID on Data Star and "stefan" is the LoginID on the Engineering domain machines]After, completing this you can exit the Data Star (b80 nodes) by typing "exit" or "logout". Now that you are on your engineering account, start the Gnuplot, by typing.% gnuplotFollowing are the commands on Gnuplot software to obtain the file in the Postscript format for printing.gnuplot> set term postscript gnuplot> set output 'your_output_name.ps'Note that you should set new output name for each file you want to plot, otherwise it will keep overwriting the same file. Here are a few commands to make your plots look neater(use these before plot command):gnuplot> set title 'title_of_your_plot' gnuplot> set xlabel 'x-axis' gnuplot> set ylabel 'y-axis'For the 3-D outputs of "plot" program:gnuplot> set contour surface gnuplot> set cntrparam levels incremental 0.01,200 gnuplot> splot 'your_file_name' w lThis will also show the seepage boundary. For the 2-D seepage boundary outputs of "plot" program:gnuplot> plot 'your_file_name'To quit Gnuplot:gnuplot> quitNow you can access your "*.ps" files through a File Manager, and use the available software for Postscript format (e.g. GhostView) to print your plots.
Matlab is a powerful mathematical software which also has a nice plotting interface. To plot your graphs in Matlab you should use the Matlab option of the plot program, and the type name of your files should be "*.m" .
Once you obtain your output files (in Matlab format), transfer the files to a UCSB machine(into your X:\ directory), using the SCP utility:To copy the file "pi.c" from a local machine to the Data Star: scp pi.c u4078@dslogin.sdsc.edu:pi.c To copy the file pi.c from Data Star to the Engineering domain machines: scp pi.c stefan@linux.engr.ucsb.edu:pi.c which will copy the "pi.c" file to the machines both in the "ENGR" domain. ["u4078" is the LoginID on Data Star and "stefan" is the LoginID on the Engineering domain machines]and exit Data Star(b80 nodes) by typing "exit" or "logout".
On Windows, you can start Matlab in conventional way.
On UNIX, you can use Matlab with Xwindow by% matlab -display your_computer_name:0.0 (your_computer_name e.g. ecipc004.engr.ucsb.edu)or by setting the display to your computer first and then running Matlab:% setenv DISPLAY your_computer_name:0.0 (your_computer_name e.g. ecipc004.engr.ucsb.edu) % matlab
Once you access Matlab, open your files and run them by either pressing F5 or from the menus
Debug --> (Save and) Run
Print your plots using Matlab figure window.