Académique Documents
Professionnel Documents
Culture Documents
Coders' Corner
ABSTRACT
A SAS Risk Dimensions based system was implemented for use by an investment group of a bank. The system requires up to date market data as input. An interface to Bloomberg was implemented which consists of a set of macros which request and download required market data automatically from Bloomberg server every business day. One of the requirements for the process was that the transfer of data files to and from Bloomberg server should be completely automatic and as fail-proof as possible. Another requirement was that the Bloomberg login credentials as well as reply files from Bloomberg need to be encrypted, for security reasons. The password encryption requirement was handled by PROC PWENCODE in conjunction with filename FTP engine which accepts encrypted password. The more challenging task of making the process near fail-proof was handled by a combination of SAS functions which will be discussed in this paper. The requirement was challenging in this case because the files are generated dynamically on the fly on Bloomberg server, and in some situations the files might not be available for several reasons. The process was implemented in such a way that it would gracefully recover in abnormal situations.
INTRODUCTION
SAS/BASE provides a convenient tool to transfer files to and from a remote server through filename FTP engine. The typical situation presents little challenge in terms of execution, and in case something goes wrong, for example if transfer fails, another attempt could easily be made. In this paper we present a situation where the FTP needs to be done in an automated way with little user interaction, while being able to gracefully recover in abnormal situations. First, the context in which a secure, automated FTP process is needed is presented. Then conventional FTP statement is presented, along with its limitations. Finally, a FTP process is presented to overcome these limitations. SAS/Risk Dimensions is an Enterprise Risk Management solution to help financial institutions manage their risk. To implement a market risk solution, SAS/RD requires two major inputs, portfolio data and market data. The acquisition and management of portfolio and market data presents a significant challenge in a typical SAS/RD implementation. While portfolio data acquisition can be managed more easily, the market data management presents some unique challenges. Market data can be sourced from several vendors, including Bloomberg. While Bloomberg provides different ways to obtain market data, for automated SAS/RD solution, the appropriate method is to use the batch FTP process. This process needs to be secure, automated and near fail-proof; Secure, to keep the data confidential; Automated, to help run the SAS/RD solution with no user interaction; A brief explanation about the process to obtain market data from Bloomberg follows, to help better understand the challenges involved. The first step is to create a request file, which contains a description of required market data, in a format defined by Bloomberg. These files end with an extension .req, and are uploaded to Bloomberg server by FTP. Once a .req file is placed on Bloomberg server, Bloomberg examines the file, and if the file is in correct format, generates a .copied file. And finally Bloomberg retrieves market data and places the reply in a .reply file. Once a .reply file appears on Bloomberg server, the file can be downloaded for further processing. It needs to be noted that, a reply file is not always guaranteed to be generated. There might be several reasons for this, for example, incorrect format of request file. The transfer of files, both request and reply files can be done using a conventional FTP statement; A brief overview of FTP filename engine is presented next.
The above statement defines a handle ftpdir. reqfile is a macro variable which resolves to request file to be uploaded, &username and &password are login credentials to remote server. 1
NESUG 2009
Coders' Corner
The following statements make use of the handle and execute the FTP statement: filename locfile "&outdir.\&reqfile"; data _null_; infile locfile end=eof; input; file ftpdir("&reqfile"); put _infile_; run; The result of the above statements is to upload a file to the remote server. While this approach works well usually, very rarely it fails and issues the message ERROR: Physical file does not exist". There are several reasons for this message to appear, including the possibility that the file in question does not exist, or access to that file has been denied for some reason. However, the message might also appear even if the file exists and is available for download. It is possible that this could be a bug, and in fact SAS Tech support does mention this bug and offers two remedies. Re-run the FTP code. Apply a fix on remote server as suggested in :
http://support.sas.com/kb/5/883.html Changing the setting is ruled out in this case as the remote server in question is Bloomberg. In our experience, re-running the code always works, however the goal is to create a fail-proof process which can be ported to a production environment. It was important to avoid a failure in the first place, because that would mean significant delays for the client. The bug was traced to INFILE statement in the above code snippet, and the solution was to avoid the INFILE statement altogether. We provide an alternative approach in the next section.
FOPEN(fileref<,open-mode<,record-length<,record-format>>>) FCLOSE(file-id) The fopen function opens an external file and returns a file identifier. In Figure 1.0 fopen is shown using cHandle, and file_id is the identifier returned. If the file open fails the returned identifier is 0, and a non-zero return indicates a successful open. The argument I, indicates that the file is opened in read-only mode. The fclose function closes the file referenced by file_id. Once the remote file is open, fread and fget functions can be used to read the contents and write them out to a local file. Their syntax is: FREAD(file-id) FGET(file-id,variable<,length>)
The fread function reads a record from an external file into the File Data Buffer (FDB) while the fget function copies data from the File Data Buffer (FDB) into a variable. The use of fread and fget functions is demonstrated in Figure 2. The advantage of this approach is that it offers tight error control, i.e. a series of checks can be done and if any step fails, some kind of action can be taken. That kind of flexibility is not available with FTP filename engine. 2
NESUG 2009
Coders' Corner
CONCLUSIONS
An approach is presented for transferring files to and from a remote server. The approach offers better error control compares to conventional FTP approach.
DISCLAIMER:
The contents of this paper are the work of the author and do not represent the opinions, recommendations, or practices of SAS Institute.
REFERENCES
Paul D. Sherman and Arthur L. Carpenter. "Secret Sequel: Keeping Your Password From the Log." Proceedings of the SAS Global Forum 2009 Conference.
ACKNOWLEDGMENTS
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are registered trademarks or trademarks of their respective companies.
CONTACT INFORMATION
Please contact the author if you have any questions or comments: Shankar Yaddanapudi Shankar.stat@gmail.com
NESUG 2009
Coders' Corner
NESUG 2009
Coders' Corner
NESUG 2009
Coders' Corner