Unigird –

Data Grid Installation

Guide
(GSI-Enabled SRB)

Date: 2006/5/17

Version: 1.0
SRB Architecture
SRB is divided in three logic layers:
1. Metadata Catalog (MCAT): It stores metadata associated with data sets, users
and resources managed by the SRB. The MCAT server handles requests from the
SRB servers. These requests include information queries as well as instructions
for metadata creation and update.
2. SRB Server: It is responsible for carrying out tasks to satisfy the client requests.
These tasks include interacting with the MCAT service, and performing I/O on
behalf of the clients.
3. SRB Client Application: Client applications are provided with a set of API for
sending requests and receiving response to/from the SRB servers.
Figure 1 gives a simplified view of the SRB architecture. The model consists of three
components: the metadata catalog (MCAT) service, SRB servers and SRB clients,
connected to each other via network.
GSI in SRB
SRB supports two main methods of authentication, Grid Security Infrastructure
(GSI) and an SRB secure password system known as Encrypt1.
GSI in SRB makes use of the same certificates and Public Key Infrastructure
(PKI) as do Globus tools such as GridFTP. Users run grid-proxy-init which prompts
for the password and generates a secure proxy certificate which is good for a few
hours. During that period, Globus commands and Sinit and other SRB commands will
automatically authenticate using the proxy certificate. GSI uses SSL libraries to verify
signitures on certificates (using an RSA public/private key system) so that no
authenticating information is sent on the network in the clear, so a network
eavesdropper would still be unable to authenticate as the user. GSI authentication has
been an available SRB option since 1999.
GSI uses X.509 certificates to securely authenticate users across the network.
This same GSI authentication system that is used for Globus commands can also be
used for SRB commands. Starting September, 2004, SRB also supports the Grid
Toolkit 3 version of GSI.
1 MCAT Server Installation
1.1 Installation of Single MCAT Server

Before proceeding with the SRB installation, the following software is required:
1. SRB installation archive (SRB3.4.1.tar - http://www.sdsc.edu/srb/). The SRB
source code is available in encrypted form for educational and government use.
You should contact the SDSC SRB team to obtain a decryption key.
2. PostgreSQL installation archive (postgresql-7.4.3.tar)
3. Postgres ODBC APIs (psqlodbc-07.03.0200.tar)
4. SRB automated installation Perl script (install.pl)
5. Globus Toolkit 4.0.1

To build the SRB with GSI, you'll first need the Globus system, including the GSI
libraries, installed. You'll need to know the GLOBUS_LOCATION (this is a Globus
environment variable) under which the SRB can find the GSI include files, static
libraries, and dynamic link libraries. (Suppose we set the GLOBUS_LOCATION
=/opt/globus)

If the Globus has installed, we start the installation of SRB MCAT as follows.
Otherwise, the Globus Toolkit must be installed first.

[srb1@iisgrid06 srb_inst]$ ls

install.pl postgresql-7.4.3.tar psqlodbc-07.03.0200.tar SRB3.4.1rele_bf.tar

To decrypt the SRB tar, run the command (a decryption key is required):
[srb1@iisgrid06 srb_inst]$openssl enc -d -bf-cbc -in SRB3.4.1rele_bf.tar -out SRB3.4.1.tar

password:

After decrypting the SRB tar:

[srb1@iisgrid06 srb_inst]$ ls

install.pl psqlodbc-07.03.0200.tar SRB3.4.1.tar

postgresql-7.4.3.tar SRB3.4.1rele_bf.tar

Edit the automated installation script install.pl:

$SRB_FILE="SRB3.4.1.tar" # The SRB tar file

$SRB_FILE_ALREADY_DECRYPTED=1; # Set this to 1 if the SRB Tar file is already decrypted

$POSTGRES_FILE="postgresql-7.4.3.tar";

$ODBC_FILE="psqlodbc-07.03.0200.tar";

$DB_NAME="MCAT"; # Name of the Postgres database to create

$YOUR_ADMIN_NAME="yourAdminName"; # Change this to the name for the srb-admin

$YOUR_ADMIN_PW="yourAdminPass"; # Change this to a password you want to use

$YOUR_DOMAIN="yourDomainName"; # Change this to the SRB domain name you

# would like to use. For example, “iisunigrid”. $YOUR_ZONE="yourZoneName";

# Change this to the local zone name you

# would like to use. For example, “iis”

$RESOURCE_NAME="yourRescName"; # Name of the local SRB resource

$RESOURCE_DIR="resourcePath"; # The resource path that stores data objects.

# For example: “/home/srb1/srb_resc/601”

$IP_ADDRESS_LOCALHOST="yourHostIP";

$SRB_CONFIGURE_OPTIONS="--enable-installdir=$SRB_INSTALL_DIR --enable-psgmcat --enable-

psghome=$postgresInstallDir --enable-gsi-auth --enable-globus-location=/opt/globus --enable-globus-

flavor=gcc32dbg";

With the script and files in place, the installation can be performed with the command
shown as follows.
[srb1@iisgrid06 srb_inst]$ perl install.pl

This script is install.pl version 3.x, last updated February 21, 2006

This host is xxx.xxx.xxx.xxx

This host full network name is xxx.xxx.xxx.xxx

This host full network address is xxx.xxx.xxx.xxx

Your home directory is /home/srb1

Skipping gzip -d as the tar file is already uncompressed

Step A 1 completed.

running: tar xf postgresql-7.4.3.tar > /home/srb1/srb_inst/installA2.log

After the installation, a MCAT server and a SRB server are automatically started in
this machine. To check if the installation is successful or not, you can use the
following command:
[srb1@iisgrid06 srb_inst]$ perl install.pl ps
This script is install.pl version 3.x, last updated February 21, 2006

This host is xxx.xxx.xxx.xxx

This host full network name is xxx.xxx.xxx.xxx

This host full network address is xxx.xxx.xxx.xxx

Your home directory is /home/srb1

Note: SRB_DIR determined to be SRB3.4.1/

SRB3_4_1

Running srb server processes:

1 S 910 7214 1 0 76 0 - 3023 - ? 00:00:00 srbMaster-3.4.1

0 Z 910 7245 7214 0 76 0 - 0 exit ? 00:00:00 srbServer <defunct>

0 S 910 7249 7214 0 76 0 - 3503 pipe_w ? 00:00:00 srbServer

Running postgres server processes:

0 S 910 6794 1 0 76 0 - 4189 - pts/0 00:00:00 postmaster

1 S 910 6797 6794 0 75 0 - 1889 - pts/0 00:00:00 postmaster

1 S 910 6798 6797 0 75 0 - 1641 - pts/0 00:00:00 postmaster

1 S 910 7216 6794 0 76 0 - 4409 108168 pts/0 00:00:00 postmaster

1 S 910 7251 6794 0 76 0 - 4409 331208 pts/0 00:00:00 postmaster

Done listing processes at install.pl line 535.

To stop MCAT server, you can use the following command:

[srb1@iisgrid06 srb_inst]$ perl install.pl stop

This script is install.pl version 3.x, last updated February 21, 2006

This host is xxx.xxx.xxx.xxx

This host full network name is xxx.xxx.xxx.xxx

This host full network address is xxx.xxx.xxx.xxx

Your home directory is /home/srb1

Note: SRB_DIR determined to be SRB3.4.1/

SRB3_4_1

chdir'ing to: /home/srb1/srb_inst/SRB3.4.1/bin

You may see an error that a process is not killed,

this is normal as the killsrb script attempts to kill

each srb process twice.

running: ./killsrb < tty_input_yes

chdir'ing to: ../..

running: /home/srb1/srb_inst/pgsql/bin/pg_ctl stop

Done stopping SRB and Postgres servers

Done at install.pl line 452.

To start GSI-Enabled SRB MCAT, you need a pair of SRB host certificate and private
key. The SRB host certificate and private key can be the same with the certificate and
private key used for Globus host. Suppose we put the host certificate and private key
in “/home/srb1/srb_cert/hostcert.pem” and “/home/srb1/srb_cert/hostkey.pem”. Then
you have to modify the “runsrb” script in “/srb3.4.1/bin” to enable GSI authentication.

# X509_USER_KEY and X509_USER_CERT specify where the SRB server's GSI

# key and certificate are located.

X509_USER_KEY=/home/srb1/srb_cert/hostkey.pem

X509_USER_CERT=/home/srb1/srb_cert/hostcert.pem

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/globus/lib

To start MCAT server, you can execute the following command:

[srb1@iisgrid06 srb_inst]$ perl install.pl start

This script is install.pl version 3.x, last updated February 21, 2006

This host is xxx.xxx.xxx.xxx

This host full network name is xxx.xxx.xxx.xxx

This host full network address is xxx.xxx.xxx.xxx

Your home directory is /home/srb1

Note: SRB_DIR determined to be SRB3.4.1/

SRB3_4_1

running: /home/srb1/srb_inst/pgsql/bin/pg_ctl start -o '-i' -l /home/srb1/srb_inst/pgsql/data/pgsql.log

chdir'ing to: /home/srb1/srb_inst/SRB3.4.1/bin

sleeping a second

running: ./runsrb

mv: cannot stat `./../data/srbLog': No such file or directory

rm: cannot remove `./../data/lockDir/.[a-z]*': No such file or directory

rm: cannot remove `./../data/lockDir/CVS': Is a directory

findServerExec: found "/home/srb1/srb_inst/SRB3.4.1/bin/./srbServer" using argv[0]

logFile: ../data/log/srbLog.4.21.6 opened successfully.

running: ps -el | grep srb | grep -v grep

If the srb server is running OK, you should see srbMaster and srbServer here:

1 S 910 24645 1 1 76 0 - 3321 - ? 00:00:00 srbMaster-3.4.1

0 S 910 24648 24645 2 76 0 - 3606 pipe_w ? 00:00:00 srbServer

Once installation is complete, the easiest way to test your environment is by using the
S-commands, which provide a set of command-line client programs similar to UNIX
commands. Before testing the S-commands, the environment should be properly set
up with a shell script
SRB_ROOT=/home/srb1/srb_inst/

export SRB_INSTALL=$SRB_ROOT/SRB3_4_1

export PATH=$SRB_INSTALL/utilities/bin:$PATH

With the environment in place, the list of Scommands can be used for testing.
[srb1@iisgrid06 srb_inst]$ Sinit
[srb1@iisgrid06 srb_inst]$ Sls
/Z06/home/admin06.iisgrid06:
[srb1@iisgrid06 srb_inst]$ Sexit

Use “mcatAdmin.jar” to manage your MCAT server, such as adding users or adding
resources. You can get the file at http://www.sdsc.edu/srb/tarfiles/mcatAdmin.jar and
run the tool at a machine with a java runtime environment:
java –jar mcatAdmin.jar

For more detailed information about “mcatAdmin”, please see

http://www.sdsc.edu/srb/index.php/Admin_Tool
 1.2 Federation of MCAT servers

The Federated MCAT implementation allows users to access resources and data
across zones. Suppose that there are two zones (MCAT servers) installed, for example
Z06 and Z08.

For each MCAT server, MCAT administrator should execute “Stoken Zone”
command to generate a zone profile in a specified file, e.g. “Z06.profile”.
[srb1@iisgrid06 srb_inst]$ Stoken Zone > Z06.profile

[srb1@iisgrid06 srb_inst]$ cat Z06.profile

--------------------------- RESULTS ------------------------------

zone_id: Z06

local_zone_flag: 1

netprefix: xxx.xxx.xxx.xxx:NULL.NULL

port_number: 0

auth_scheme: ENCRYPT1

distin_name: NULL

zone_status: 1

zone_create_date: 2003-08-01-00.00.00

zone_modify_date: 2006-04-24-19.36.32

zone_comments: Created Mon Apr 24 19:36:32 CST 2006

zone_contact:

user_name: xxx

domain_desc: xxx

locn_desc: xxx.xxx.xxx.xxx

-----------------------------------------------------------------

Exchange zone profiles between two zones, for example Z06 sends Z06.profile to
Zone08 and Z08 sends Z08.profile to Zone06. Then save the profile in the MCAT dir
of SRB source, for example, we save the profile in
“/home/srb1/srb_inst/SRB3_4_1/MCAT”, and perform the follow command to ingest
remote zone.
[srb1@iisgrid06 MCAT]$ zoneingest.pl Z08.profile

zone_id=Z08

netprefix=xxx.xxx.xxx.xxx:NULL.NULL
port_number=0

zone_status=1

user_name=xxx

domain_desc=xxx

locn_desc=xxx.xxx.xxx.xxx

Look OK? Enter yes (or y) to proceed:y

chdir'ing to bin

running: ./ingestToken Domain xxx home

You can run “Stoken Zone” command to see if the remote has been ingested or not.

To Synchronize zones of a zone federation, each zone administrator should perform

the following command in the “SRB3_4_1/utilities/admin_tool” folder, for example
“/home/srb1/srb_inst/SRB3_4_1/utilities/admin-bin” to synchronize metadata of
remote zones.
[srb1@iisgrid06 admin-bin]$ Szonesync.pl -r -d

chdir'ing to: ./SzonesyncTmpDir

There are 2 zones.

The local zone is: Z06

Remote zones are: Z08

Remote ports are: 0

Remote hosts are: xxx.xxx.xxx.xxx

Doing zone Z08 which is hosted at host xxx.xxx.xxx.xxx port 0

After zone synchronization, you can utilize “mcatAdmin.jar” to see the metadata
synchronized from remote zones. For example,
2 SRB Server Installation
Before proceeding with the SRB installation, the following software is required:
1. SRB installation archive (SRB3.4.1.tar ).
2. Globus Toolkit 4.0.1

Extract all files from SRB3.4.1.tar (suppose that we extract the archive file in
“/SRB3_4_1” folder)
tar -xf SRB3.4.1.tar

Run the configure script in “/SRB3_4_1” (suppose we set the path of srb server in
“../SRBServer” )
[srb2@iisgrid07 SRB3_4_1]$ ./configure --enable-installdir=../SRBServer --enable-gsi-auth --enable-globus-
location=/opt/globus --enable-globus-flavor=gcc32dbg

[srb2@iisgrid07 SRB3_4_1]$ gmake

[srb2@iisgrid07 SRB3_4_1]$ gmake install

After the installation completes, you must tell the SRB server which MCAT host it is
supposed to connect to. This is done by editing the file: SRBServer/data/mcatHost
[srb2@iisgrid07 data]$ cat mcatHost

# The MCAT enabled host file.

# A SRB federation (zone) can be configured to run with a single Master MCAT

# plus zero or more Slave MCAT. Lines 1, 2 and 3 configures the Master MCAT

# which is required. Line 4 is optional which is used to configure a Slave

# MCAT

# Line 1 is the host address of the MCAT enabled host.

xxx.xxx.xxx.xxx

# Line 2 is the AUTH_SCHEME used to connect the MCAT enabled server.

# valid schemes are : PASSWD_AUTH, SEA_AUTH, SEA_ENCRYPT,

# GSI_AUTH, GSI_SECURE_COMM

ENCRYPT1

# Line 3 is the Dn of the MCAT enabled server.

/C=US/O=NPACI/O=DICE/UID=srb/CN=Storage Resource Broker/Email=srb@sdsc.edu

# Line 4 is option which uses the keyword slaveMcatHost to configure a

# Slave MCAT. For example:

# slaveMcatHost = srbbrick9.sdsc.edu

Before starting SRB server, you must set the user environment used for starting the
SRB server and connecting to MCAT server. The user must have a role with system
administrator permission specified in the zone. The user environment includes two
file, .MdasAuth and .MdasEnv, and must be put in ~/.srb folder. The templates of
.MdasAuth and .MdasEnv can be found in /SRB3_4_1/utilities/envFiles

.MdasEnv
mdasCollectionHome '/ZoneName/home/UserID'

mdasDomainHome 'domainName'

srbUser 'userID'

srbHost 'thisHostAddress'

# mcatZone defines the user's home zone

mcatZone 'zoneName'

defaultResource 'rescName'

# For 1.1.7 and up AUTH_SCHEME defines the auth scheme to use.

# valid schemes are : 'ENCRYPT1', 'GSI_AUTH', 'GSI_SECURE_COMM' and 'GSI_DELEGATE'

AUTH_SCHEME 'ENCRYPT1'

# SERVER_DN specify the Gsi Dn of the Server. Valid only if AUTH_SCHEME is set

# to 'GSI_AUTH' or 'GSI_SECURE_COMM'.

SERVER_DN '/C=US/O=NPACI/OU=SDSC/USERID=srb/CN=Storage Resource Broker/Email=srb@sdsc.edu'

.MdasAuth
# This is the Mdas password
yourAdminPass

As the same with the setting of GSI-enabled SRB MCAT mentioned in previous
section, to start GSI-Enable, you need a pair of SRB host certificate and key. The SRB
host certificate and key can be the same with the certificate and key used for Globus
host. Suppose we put the host certificate and key in
“/home/srb2/srb_cert/hostcert.pem” and “/home/srb2/srb_cert/hostkey.pem”. Then
modify the “runsrb” script in “/ SRBServer /bin” to enable GSI authentication.
# X509_USER_KEY and X509_USER_CERT specify where the SRB server's GSI

# key and certificate are located.

X509_USER_KEY=/home/srb2/srb_cert/hostkey.pem

X509_USER_CERT=/home/srb2/srb_cert/hostcert.pem

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/globus/lib

To start SRB server, run “runsrb” in /SRBServer/bin

[srb2@iisgrid07 bin]$ runsrb

mv: cannot stat `./../data/srbLog': No such file or directory

rm: cannot remove `./../data/lockDir/CVS': Is a directory

findServerExec: found "/home/srb2/srb_inst/SRBServer/bin/./srbServer" using argv[0]

logFile: ../data/log/srbLog.4.21.6 opened successfully.

srb2 18654 1 0 17:16 ? 00:00:00 ./srbMaster-3.4.1 -d 1 -S

srb2 18663 18650 0 17:16 pts/0 00:00:00 grep srbMaster

To stop SRB server, run “killsrb” in /SRBServer/bin

[srb2@iisgrid07 bin]$ killsrb

1 S srb2 18654 1 0 75 0 - 2722 - 17:16 ? 00:00:00 ./srbMaster-3.4.1 -d 1 -S

0 S srb2 18656 18654 0 76 0 - 3333 pipe_w 17:16 ? 00:00:00

/home/srb2/srb_inst/SRBServer/bin/./srbServer -Q -p8 -P9

Will kill srbServers with pid = 18656

Will kill srbMaster with pid = 18654

Enter y to do so:

To test if the SRB server is started with GSI authentication, you can use Scommand
with GSI to connect to the SRB server.

To examine GSI-enabled Scommand, you need Globus and Scommand installed. You
also have to apply a user certificate from CA (NTHU is the CA of UniGrid), and then
we can obtain a certificate and a private key (usercert.pem and userkey.pen). The
applied certificate and private key have to be stored in the directory “~/.globus”
[srb2@iisgrid07 .globus]$ ls
usercert.pem userkey.pem

Edit .MdasEnv in directory ~/.srb

mdasCollectionHome '/ZoneName/home/UserID'

mdasDomainHome 'domainName'

srbUser 'userID'

srbHost 'thisHostAddress'

# mcatZone defines the user's home zone

mcatZone 'zoneName'

defaultResource 'rescName'

# For 1.1.7 and up AUTH_SCHEME defines the auth scheme to use.

# valid schemes are : 'ENCRYPT1', 'GSI_AUTH', 'GSI_SECURE_COMM' and 'GSI_DELEGATE'

AUTH_SCHEME ' GSI_AUTH '

# SERVER_DN specify the Gsi Dn of the Server. Valid only if AUTH_SCHEME is set

# to 'GSI_AUTH' or 'GSI_SECURE_COMM'.

SERVER_DN 'connectSRBHostDN'

Run “grid-proxy-init” and enter pass phrase for private key

[srb2@iisgrid07 ~]$ grid-proxy-init

Your identity: /O=NTHU/OU=Unigrid/OU=simpleCA-

uniblade01.cs.nthu.edu.tw/OU=cs.nthu.edu.tw/CN=unigridtest

Enter GRID pass phrase for this identity:

Creating proxy ..................................................................................... Done

Your proxy is valid until: Sat May 20 08:33:48 2006

Finally, execute Scommand to test if the setting is correct.

[srb2@iisgrid07 ~]$ Sinit

[srb2@iisgrid07 ~]$ Sls

/Z06/home/test61.iisgrid06:

aaa.txt

C-/Z06/home/test61.iisgrid06/tmp
3 Client Application Installation
There are a set of SRB clients implemented for various purposes that can access SRB
system.
1. inQ: It is a browser/query tool for SRB which offers the traditional file and
directory functionality of the old SRB Browser as well as support for metadata
and nested queries on both collections and datasets. Metadata, access rights, and
queries can be viewed along-side collections and datasets in inQ's 'Windows
Explorer-like' interface. inQ is multithreaded and has drag and drop support. inQ
is written using the SRB client library, the SRB C++ wrapper, and the MFC
library for MS Windows platforms.(see http://www.sdsc.edu/srb/index.php/InQ)
2. Jargon: It is a pure java API for developing programs with a data grid interface.
The API currently handles file I/O for local and SRB file systems, as well as
querying and modifies SRB metadata. The API is also easily extensible to other
file systems. File handling with JARGON closely matches file handling in Sun's
java.io API, hopefully a familiar API to most java programmers. (see
http://www.sdsc.edu/srb/index.php/Jargon)
3. Kepler: It recently provides support for the Storage Resource Broker storage
management system as an interface to provide efficient storage functionality.
Using SRB in Kepler provides an efficient storage management; data access and
transfer functionality, server side processing of data using proxy commands and
efficient search functionality using metadata. Moreover, as Kepler is a cross-
project collaboration extending to various scientific disciplines, SRB provides
data access to diverse repositories using a single namespace. Other benefits of
using SRB in Kepler include third party transfer, shipping and handling of data,
and persistent archiving. (see http://www.sdsc.edu/srb/index.php/Kepler)
4. Matrix: It has been designed to work on data grid workflows and data grid
administration of SRB based data grid systems. In conjunction with SRB, Matrix
performs data grid and compute-intensive tasks along with other grid middleware.
(see http://www.sdsc.edu/srb/matrix/index.html)
5. MySRB: It is a web-based browse and search interface to the Storage Resource
Brok er (SRB) developed at the San Diego Supercomputer Center (SDSC). The
SRB facilitates information sharing by allowing users (1) to access files stored on
heterogeneoues resources including disks, tapes and databases on different
machines through logically organized catalogs; and (2) to manage and share data
collections in a secure manner. (see http://www.sdsc.edu/srb/index.php/MySRB)
6. Scommands: They are command line utilities that run in the Unix, Windows or
Mac OS command shells. Most Scommand names are preceded by an "S". These
are Unix-like commands for accessing SRB data and metadata. (see
http://www.sdsc.edu/srb/index.php/Scommands)