Install and use Starfish ETL Suite

Starfish ETL Suite
Installation & Usage Technical Guide

Technology Advisors, Inc
PLEASE NOTE: It is recommended to use our Wiki, instead of this

document. The Wiki will always be the most up to date – so check here
first for the most recent help. http://wiki.starfishetl.com
Starfish ETL Version 3.7
11/11/2015
Starfish Technical Guide Technology Advisors, Inc.
Table of Contents
Installation ............................................................................................................ 4
Overview ............................................................................................................ 4
System Architecture .............................................................................................. 4
Starfish Engine ................................................................................................... 5
Engine Web.Config Settings ................................................................................... 7
Starfish Admin .................................................................................................... 8
Starfish Scheduler ............................................................................................ 10
Scheduler Config Settings .................................................................................... 12
Admin Usage ........................................................................................................ 13
Server-wide Settings ........................................................................................ 13
Email Settings .................................................................................................... 13
License Key ....................................................................................................... 14
Job-specific Settings ......................................................................................... 15
Notification Settings ............................................................................................ 15
Job Chaining ...................................................................................................... 15
Jobs .................................................................................................................. 16
Creating a New Job ............................................................................................. 16
Copying a Job .................................................................................................... 16
Configuring the Origin ...................................................................................... 17
Configuring the Destination .............................................................................. 18
Using Transactions .............................................................................................. 18
Mapping ............................................................................................................ 20
Creating a Stage ................................................................................................ 20
Options ............................................................................................................. 21
Mapping Screen Overview .................................................................................... 22
Straight Field-to-Field Mapping ............................................................................. 24
Function Field Mapping ........................................................................................ 25
SQL Lookup..................................................................................................... 27
Scripted Value ................................................................................................. 28
Exec Before/After Operations ............................................................................... 29
Action Types ................................................................................................... 29
Exec When ...................................................................................................... 30
VBScript Starfish Class ........................................................................................ 31
Cross-reference (Xref) ......................................................................................... 36
Xref List Initialization ....................................................................................... 36
Xref Writes ...................................................................................................... 37
Xref Reads ...................................................................................................... 38
Inserting Variables .............................................................................................. 39
Scheduling Jobs ................................................................................................ 41
Previewing Jobs ................................................................................................ 42
Running Jobs .................................................................................................... 43
Run Options ....................................................................................................... 43
Using Multi-threading .......................................................................................... 44
Saving/Deploying Jobs ..................................................................................... 46
Starfish Updater ................................................................................................... 48
System Backup Wizard ......................................................................................... 50
Connector-specific Guidelines .............................................................................. 56
SData ............................................................................................................... 56
Salesforce .......................................................................................................... 56
Confidential Page 2 11/11/2015

Microsoft CRM .................................................................................................... 56

SugarCRM ......................................................................................................... 56
NetSuite ............................................................................................................ 57
Email ................................................................................................................ 57
SageCRM ........................................................................................................... 57
Oracle ............................................................................................................... 57
Troubleshooting ................................................................................................... 58
Appendix A: Sample Import Exercise ................................................................... 60

Installation
Overview
The Starfish ETL suite is, in a nutshell, an import tool with a lot of flexibility and power.
There are 3 essential components to that make up the ETL suite: The Engine, the Admin
tool, and the Scheduler. The Starfish Engine runs as a web service, and requires IIS to run.
The Scheduler runs as a Windows Service. The Admin tool runs as a Windows GUI
application, which allows the end user to configure jobs, mappings & schedules.
System Architecture
Admin Tool
Scheduler 3rd Party
- Windows Application
- Configure Jobs - Windows Service - SOAP Calls
- Set Schedule - Event Logging - Pass Params
- Set Notifications - Notifications
- Run Jobs in Real Time
Engine
- Web Service
Origin - Multiple Jobs Destination
- Multiple Stages per Job
SQL - XML formatted SQL
OLEDB OLEDB
ODBC Methods ODBC
SOAP - GetConfig () SOAP
XML - DeployConfig (XML) XML
CSV - StartJob (JobID) CSV
- GetJobStatus ()

Starfish Engine
The server where the Engine will be installed must have IIS and ASP.NET installed.
Microsoft .NET Framework 4.0 is required. This may be a server on your network; it does
not have to be a computer where the end users have physical access to it. The only other
requirement is that this server will need direct access to the databases and files that the
jobs will be operating on.
Installation Steps
1. Run the StarfishEngineSetup.exe file in the StarfishEngine folder on the desired
server.
2. No configuration is needed. The installation will automatically place the files in a new
Virtual Directory (StarfishEngine) your IIS Inetpub directory (typically
C:\Inetpub\wwwroot). It will also create a new Application Pool for use with the
Virtual Directory.

3. Click Next and complete the installation. After the install, you can test to make sure
it worked correctly by opening a browser to
http://localhost/StarfishEngine/StarfishService.asmx
(http://<server_name>/<virtual_directory>), and you should get a window that
looks like the one below.

Engine Web.Config Settings
Setting Description
LogPath Path to save log files in when LogToFile is enabled.
LogToFile Boolean, Default False. If enabled, a log file (at the “High”
level) with be created within the specified LogPath location.
CommandTimeout Length in seconds for SQL commands to execute
ProxyEnabled Boolean, Default False. If enabled, Starfish will use the
specified Proxy server when validating it’s license.
ProxyURL URL for proxy server
ProxyPort Port proxy server is running on
ProxyUseDefaultCredentials Boolean, Default False. If enabled, default network
credentials will be used.
ProxyUseSuppliedCredentials Boolean, Default False. If enabled, the credentials specified
below will be used.
ProxyCredentialsUsername Proxy username
ProxyCredentialsPassword Proxy password
ProxyCredentialsDomain Proxy user’s domain
RowThreadCount Default: 1. The default number of threads to use when
executing a Starfish job.
UseParamQueries Boolean, Default False. If enabled (default), Starfish will
build parameterized SQL queries. If disabled, queries will
instead be sent as normal text. Useful for performance
tweaking.
UseNoLock Boolean, Default False. If enabled, Starfish will append the
SQL “WITH(NOLOCK)” command to generated lookup
statements. Useful for performance tweaking.
CountStatement Default “COUNT(*)”. When doing lookups, Starfish will
perform a statement such as “SELECT COUNT(*) FROM
TABLE WHERE …” to determine if a record exists or not.
Changing this value to something like “COUNT(1)” may help
performance for some database systems.

Starfish Admin
You will install the Admin tool on the workstation where you plan to do your job
development. The installation requires the .NET Framework 4.0, and will need the ability to
access the web service set up in the previous step. An easy way to test this is open the web
service in a browser, substituting “localhost” for the actual server name.
Installation Steps
1. Run the Setup.exe file in the StarfishAdmin folder on the desired Workstation.
2. Select the installation folder, and the default access.

3. Click next and complete installation.

4. The first time you start Starfish Admin, it is recommended you set the Starfish
Engine Web Service URL. Click the round Starfish button, then “Set Web Service
URL” from the menu.
5. In the window, set the location (default if on the same machine is

http://localhost/StarfishEngine/StarfishService.asmx)

Starfish Scheduler
The Scheduler is a run as Windows Service. It may installed on any server which has access
to the web service, but for the sake of simplicity it is recommended to be installed the same
server as the Starfish Engine. The Scheduler reads in the jobs’ configured schedules and
launches them at the appropriate times.
Installation Steps
1. Run the Setup.exe file in the StarfishScheduler folder on the desired server.
2. Select the installation folder, and the default access.

3. Click next and complete installation.

4. It is recommended to update the default web service location. To do this, open the
.config file in the install path (“C:\Program Files\Technology Advisors\Starfish
Scheduler\StarfishScheduler.exe.config”) with a text editor.
Near the end of the file, change http://servername/StarfishEngine.asmx to the

location of your new web service
(http://<server_name>/StarfishEngine/StarfishEngine.asmx).

5. A number of other options may be changed through this config file.
Scheduler Config Settings
Setting Description
SchedulePollMS Interval in milliseconds on how often the Service should cycle to
check if it needs to execute any Jobs based on the schedules.
Default is 5 seconds.
Commit True/False, whether Jobs executed through the Scheduler should
be committed to the destination database. Default is True.
ChainJobs True/False, whether Jobs executed through the Scheduler should
automatically call the next Job configured in Job Changing.
Default is True.
LoggingLevel Level of logging to return through notification.
0=None, 1=Low, 2=Medium, 3=High
FromAddress The email address notifications should come from.
FromDisplay The display name of the notification email address.
SMTPPort Port of your SMTP server. Default is port 25.

Admin Usage
For new installations, when you first start Admin you will have to create a New “Bindings
File” from the File->New menu. This will allow you to continue and create new jobs. Only
one Bindings File may be active on the Starfish Engine server at a time, but each file may
contain any number of Jobs and Stages.
Server-wide Settings
Email Settings
When a job is run through the Scheduler, logs can be automatically emailed to an
administrator. Use these fields to set the SMTP email server used transporting these
messages.
Field Caption Description

SMTP Server Address SMTP server host address
SMTP Port Server Port where SMTP service is running
SMTP Auth. Username If your SMTP server requires authentication, supply your
username here.
SMTP Auth. Password If your SMTP server requires authentication, supply your password
here.
From Address The Email Address where messages from Starfish should appear
to come from.
From Display The Display Name of the email address where messages from
Starfish should appear to come from.

License Key
To run a job using Starfish, you must supply your license key. If you do not have a license
key, please contact Technology Advisors to obtain one. To apply your key, from the General
tab, click the “Update License” button.
Then paste your key in the box, click OK.

Job-specific Settings
Notification Settings
When a job is run through the Scheduler, logs can be automatically emailed to an
administrator. Use these fields to set the email address(es) for the logs to be mailed to, and
the level of logging that you want reported back. These settings can be changed for each
job.

Email Address Email address to deliver notifications to. Multiple addresses can be
separated by a comma:
(address1@domain.com,address2@domain.com)
Notify On - Information Will return the # of rows affected for each stage.
Notify On – Warning Will return any warning messages.
Notify On – Error Will return any error messages.
Job Chaining
You can configure a Job to automatically call another Job in order when its parent is started.
Any number of jobs can be “daisy chained” together in this manner.

Next Job Select the next Job in order after the currently selected Job. Leave
blank (Select) to disable this feature. You cannot select the
current Job as the Next Job. (the Admin will not allow this as it
may cause an endless loop)

Jobs
Jobs are a series of tasks that can be performed against a singe data source. Each Jobs may
contain different Stages, which allow you to move the data to different destinations. You
may have an unlimited number of Stages per Job, and an unlimited number of Jobs per
Bindings File.
Creating a New Job

On the General tab, click the New Job button. Enter a meaningful name for the new Job.
This will create a new, blank definition for the database connections and mappings.
Copying a Job
On the General tab select the Job you want to copy, then click the Copy Job button. Enter a
meaningful name for the new Job. This will copy all database connection settings, mappings,
and schedules. The only setting that is (intentionally) not copied over is the Next Job for Job
Chaining.

Configuring the Origin

The Origin is the data SOURCE that will be used in a Job. For MS SQL database connections,
use the OLE DB connection.
ODBC Note: Use the Count Rows option if your driver/database platform does not support
the TOP function.
Example for OLE DB (MS SQL Server)

1. On the Origin tab, select the OLE DB tab.
2. To build the connection string, click the ellipses (…) next to the box.
3. Use the Data Link screen to create the connection. On the provider tab, select
“Microsoft OLE DB Provider for SQL Server” or “SQL Native Client”, depending on
your version.
4. For Data Source, enter your SQL Server host/instance name. Configure your
authentication settings, and select your source Database from the Initial Catalog
dropdown, then click OK. The Connection String box should now be populated.
5. Enter a SELECT SQL statement to configure the entire recordset source of your
connection. This will be all of the data that will be transformed or moved into your
destination.
6. Click the Test Connection button. Results should read “Successful” and display the
number of rows returned by your SQL SELECT statement.

Configuring the Destination

The Destination is the data TARGET that will be used in a Job. For MS SQL database
connections, use the OLE DB connection.
Example for OLE DB (SalesLogix)

1. On the Origin tab, select the OLE DB tab.
2. To build the connection string, click the ellipses (…) next to the box.
3. Use the Data Link screen to create the connection. On the provider tab, select
“SalesLogix OLE DB Provider”.
4. Select your SalesLogix Server and database name. Enter your username & password,
being sure to check Allow Saving Password.
5. Click OK, the Connection String should be populated.
6. Click the Test Connection button. Results should read “Successful”.
Using Transactions
If you wish to use commit/rollback transactions, select the Check Box “Use Transactions” on
the Destination tab for OLE DB or ODBC connections. When enabled, all database operations
to the destination will be performed, but not committed until the end of the Job. If an error
occurs while processing the Job, all actions are rolled back.


Mapping
The Mapping tab is where Stage, Exec Operations and Data field mapping is configured.
Creating a Stage
1. On the Mapping tab, click the Stages… button
2. In the Stages dialog window, click the button

3. Enter a meaningful Stage Name, select Stage Type, and select the destination Table
(where data will be Inserted or Updated), then click OK. A description of these
options are listed below.

4. The New Stage should now be displayed in the list on the left.
5. Repeat steps 2-4 for each additional stage.
Options
Option Name Description
Stage Type - Update Will perform an update-only operation on matched records for each
Origin row. Unmatched records will be skipped.
Stage Type – Insert Will perform an insert-only operation for each Origin row. Use this
operation when performing a “dump” of the data.
Stage Type – Delete Will perform a delete operation on the Destination database for each
Origin row, based on the match.
Stage Type – Will first perform a lookup, based on matches. If a match is found,
Update/Insert and update operation will be performed. If no match is found, an
insert operation will be performed.
Skip Blank Writes If checked, the insert/operation will ignore NULL/blank fields while
building the SQL statement – resulting in NULL values on the
destination.
Only Update on For update operations, this setting will first check the matched
Difference record for values that are different and will only actually update the
record if a mismatch of data values occurs and it’s actually needed.

Mapping Screen Overview

From the Stage drop-down, select the stage you want to modify.
Screen Layout Definition

Origin Fields Screen creates a drop-down field for each destination field. Drop-
down list contains each a list of all the fields from the Origin data
source, plus a special option called “Function Field”.
Destination Fields List of each Destination field.
Match For Update and Delete operations, check this box to use the result
as part of the destination row-matching criteria. Think of the
Match fields as a “WHERE”-clause of a SQL statement.
Insert Only This option can be checked to only process a field if an
Update/Insert stage type results in an Insert operation. Useful for
fields that generate values such as a unique ID or record creation
date.
Update Only This option can be checked when the stage’s option “Only Update
on Difference” is set. It causes the engine to ignore this mapping
during an update comparison operation, but will allow it to update
the field if an update operation is necessary based on other fields.
For example, this is useful for mapping a “Modify Date” field to

the current time. You don’t want the field to be compared,

because the engine will always think an update is necessary.
Operation An overview of what kind of function has been selected for
Function Fields.

Straight Field-to-Field Mapping

To relate an Origin field directly to a Destination field, select from the Origin Fields drop-
down. The fields on both sides will turn BOLD to indicate the map has been created.

Function Field Mapping

To begin creating a Function Field, you can do this 3 ways.
1. Select “Function Field” from the Origin Fields drop-down.
2. OR click the ellipsis inside the Operation column (or double-click this field).
3. OR click the + button
Select from the following types of Function Fields, more information is provided below.
Hardcoded Value Allows the user to specify a static value, although you can also insert
a Variable here as well. It is not necessary to include quotes for
text/string values.
SQL Lookup Performs a database lookup on the Destination database. Supply a
SQL SELECT statement, will return the first column from the first
row. More on Server-Side Lookups
Smart Lookup Performs a lookup on the destination, returning a single field’s value.
Allows caching of results so the multiple runs of the same lookup
don’t have to make round-trips to the destination. This cache is
stored in memory, so it’s a good idea to only use it when there is a

high chance of the same queries, otherwise an out of memory

exception could occur when caching large amounts of data.
Scripted Value Allows the user to write a VBScript function to perform custom
operations for each field. The VBScript function provides a skeleton
function called “ScriptedField”. The value that will be used is
whatever is returned out of this function. More on Scripted Values
Format Conversion Convert source data to another format.
Options are:
Rich Text (RTF) -to- Plain Text
HTML -to- Plain Text
Base64 -to- Binary
Binary -to- Base64
File -to- Base64
Current Date/Time Inserts the current Date/Time at the server. Formats include
Date/Time, Date Only, Time Only.
SalesLogix ID For OLE DB destinations using the SalesLogix OLE DB adapter, this
function will be available. Automatically generated a new Saleslogix
ID for this field, based on the Stage’s destination table.

SQL Lookup
Server-Side Lookup Function Fields can be used to obtain data from other tables within the
Destination database. Below is an example of a lookup returning an AccountID based on an
Account-name lookup. The Account name is provided by the Origin field “Account”. Note
how variables can be used within the SQL statement itself. The Variable code will be
replaced with the actual value at run-time.
Server-Side lookups are also useful for generating ID’s. The example below calls a
SalesLogix stored procedure to generate an ID for the Account table.

Scripted Value
Scripted Value fields allow for extreme flexibility when assigned values to a Destination
field. Basically, any function that can be performed with VB Script can be performed within a
Scripted Value field. Variables can also be referenced from within the script.
Below are a few examples of how Scripted Value fields can be used.
Samples
Return the Current Date/Time Return the left-most 5 characters from
“ZIPCODE” Origin field
Function ScriptedField Function ScriptedField
Dim var
ScriptedField = Now() var = "@@ORG:ZIPCODE@@"
End Function ScriptedField = Left(var,5)
End Function
Trim an Origin field Return different values, based on a Variable

value
Function ScriptedField Function ScriptedField
Dim var If "@@VAR:vMyVariable@@" = "" Then
var = "@@ORG:TYPE@@" ScriptedField = "Value 1"
ScriptedField = Trim(var) Else
End Function ScriptedField = "Value 2"
End If
End Function
Press the button to test the results of your script. Keep in mind; this will not
replace variable values.

Exec Before/After Operations

Any number of functional operations can be performed before or after a Job is processed. To
set up the operations, click the Add/Edit link for Exec Before (on the top of the screen) or
Exec After (on the bottom of the screen)
Enter a meaningful name, select the Action Type, assign a variable name where necessary,
set the Exec When option, and configure the Action.
Action Types
The Action Type defines what type of operation will be performed. Variable-type actions
create a User-defined variable for use through the Job.
Action Type Description

Variable/SQL Set a variable through the execution of a SQL SELECT statement run
against the Destination database. Set the variable value equal to the
first column of the first row returned by the statement.
Variable/VBScript Set a variable through the execution of a custom VBScript function.
A shell function is provided called “ScriptedVariable”, the value of
the variable will be what is returned out of the function.
Execute SQL Execute any kind of statement against the Destination database.
Useful for calling DELETE or UPDATE statements before or after a job
for cleanup purposes.
VBScript Procedure Execute any statements desired in a VBScript Sub. A shell sub called
“VBScriptProcedure” is provided. Any kind of VBScript processing
can be called here. Useful for before-job file manipulation, or any
kind of pre- or post-processing.

Exec When
An Exec operation can be performed at different times throughout a job run. The table
below defines what each of these settings mean.
Exec Before Operations

Repeat Each Row Operation will be repeated once before processing each data origin
row.
Once Before Conn Operation will be performed only once per job, before the
connections have been opened. Useful for setting up files to be used
during the data load.
Once Before Query Operation will be performed only once per job, just after the
connections have been opened, but before the Origin Query is
opened.
Once After Conn Operation will be performed only once per job, just after the
connections have been opened and after the Origin Query is opened.
Repeat Each Stage Operation will be repeated before each stage (multiple times per
each data origin row), depending on the number of stages defined in
the Job.
Exec After Operations

Repeat Each Row Operation will be repeated once after processing each data origin
row.
Once Before Conn Operation will be performed only once per job, just before the
connections are closed at the end of the job.
Once After Conn Operation will be performed only once per job, just after the
connections are closed at the end of the job. Useful for backing up
files after a load.
Repeat Each Stage Operation will be repeated after each stage (multiple times per each
data origin row), depending on the number of stages defined in the
Job.

VBScript Starfish Class

From within VBScript operations (ScriptedVariable, VBScriptProcedure, ScriptedField) in
additional to normal VBScript functions, the developer has access to a custom “Starfish”
class. The tables below define its usage.
Class Properties
Property Name Description
CurrentStageName Read-only. Returns the name of the current stage. Useful during
“Repeat Each Stage” Exec Operations to determine path of
execution/next steps.
TotalRows Returns the total number of Origin Rows in the job.
CurrentRow Returns the current row number.
ReturnValue Sets the return value to the calling application. Useful for passing an
output back when the Job was called with an argument.
Origin ( String ) Returns the current Origin row’s value for the field specified.
Class Methods
Method Name Description
GotoStage ( String ) Directs Starfish Engine to go directly to a Stage
(skipping any others before it). String Parameter is
the Name of the stage to go to. Once GotoStage has
been called within a Job, it is considered in a “manual
operation mode”. No more stages will be called in
order and it will be up to the user to implement a
After Each Stage Exec Operation to control stage flow.
GotoNextRow () Skips all Stages and moves to the next Origin row.
EndJob ( String ) Cancels the Job execution immediately, ending with
an error message as passed into the String param.
LogMessage ( String, MsgBoxStyle ) Appends a custom message to the Log which will
always be displayed (even if Logging Level is set to
None). Message is the passed in String parameter.
Second parameter is a valid MsgBoxStyle
(vbInformation, vbQuestion, vbExclamation,
vbCritical)
GetFields ( String ) Executes a SQL SELECT statement returns all field
values from the first row, comma-delimited. VBScript
Split() function may be used to convert to an array
within your function. Can improve performance when
needing multiple fields from a single table.
ExecScalar ( String ) Executes a SQL SELECT statement on the Destination
database. Returns the first column of the first row
returned by the query. Compatible with OLE DB
destinations.
ExecScalarCache ( String ) Same as ExecScalar, except that results are cached
and before a query is executed, a search is performed
on a locally-stored set of results to prevent execution
of identical queries, thus boosting performance.
Should only be used for queries where multiple
identical queries may be run (such as looking up user
id’s or picklist values) – otherwise could consume

resources needlessly.
ExecSQL Executes a non-query statement, such as UPDATE or
DELETE against the destination database. Returns
number of rows affected. Compatible with OLE DB
destinations.
SendEmail ( String, String, String ) Sends an email directly from within the VBScript
function/procedure.
Parameters:
To_Addr – Email Recipient Address
Subject – Subject of the Email
Body – Body of the Email
ParseName ( String, String ) Parses as string containing a person’s name, and
returns the part of the name requested.
Parameters:
String – Input String containing entire name
Part – Part of the name to return. Valid parts are
“Title”,”First”,”Last”,”Middle”,”Pedigree”,”Degree”.
GetStageValue ( Integer, String) Returns the destination value used on a previous
stage.
Integer – Index number of the stage, 0-based.
String – Field name of the value from this stage to
return.
For some of the CRM Native API’s (SData, Microsoft),

generated ID’s will be automatically stored in a field
named “Key”.
SmartLookup ( String, String, Performs a lookup against the destination database,
String, [Boolean] ) and returns one field value based on supplied filter.
String – Table to perform lookup in
String – Name of field to return
String – Filter code (such as Where clause, not
including keyword “Where”)
Boolean – Default False. Enable cache for this lookup.
SmartQuery ( String, [String], Performs a lookup query against the destination
[String]) database, returns all columns and all rows from query
as a multi-dimensional array.
String – Table to perform the query in
String – Filter code
String – Default All Columns. Comma-separated list of
columns to return
If both last 2 parameters are left blank, SmartQuery
performs a full operation as specified in the first
parameter.
XrefInitWrite ( String ) Initializes a cross-reference file for writing. If file
exists, new data will be appended to it.
String – cross-reference name
XrefInitRead ( String ) Initializes an existing cross-reference for reading. This
reads all ids into memory so they will be available
within your job.
XrefWrite ( String, String, String ) Writes a new ID reference entry to an initialized file.

String – ID 1 (the ID you’ll look up by in future jobs)

String – ID 2 (the ID you’ll want returned by future
lookups)
XrefRead ( String, String ) Reads an ID entry from an initialized file. (Returns ID
2 from above)
String – ID 1
BlackbirdMessage ( String, String, Send a Blackbird Alert message (server is specified in
String, String, [String], [String] ) Web.config)
String – Message Title
String – Message Text/Body
String – To (user or group to receive message)
String – URL where message link will forware to
String – Icon, Default “Info”. Choose from: Info,
Warning, Error, Question, Star, Check, Search, Globe,
Phone, Users, Lock, Task, Bell, Cellphone, Envelope,
Calendar, Flag, Charts, Calculator, Fax
String – System, Default <blank>. Name of External
System for use with To-ID matching.
IsEmailValid ( String ) Returns True if supplied email address is in a valid
email format; otherwise false.
FormatDate ( String, String ) Returns date in supplied format.
String – Date to format
String – Format String
Example:
newdt = FormatDate(olddt, “yyyy-MM-dd hh:mm:ss”)
MoveFile ( String, String ) Moves a file on the server.
String – From filename path
String – To filename path
CopyFile ( String, String ) Copies a file on the server.
String – From filename path
String – To filename path
DeleteFile ( String ) Deletes a file on the server.
String – Filename path
ReadFile ( String ) Reads a file in text mode and returns the contents as
a string.
WriteFile ( String, String ) Writes the supplied contents to a file in text mode.
Entire file is overwritten.
String – contents to write to the file
MS Dynamics CRM-Specific
GetMCRMPicklistValue( String, Returns the integer value for a supplied picklist string.
String, String ) String – Entity for the picklist (“account”)
String – Fieldname of the picklist (“accounttype”)
String – Value for the picklist (“Prospect”)
MCRMSetRelationship ( String, Sets a relationship between two entites.
String, String, String, String ) String – Relationship Name
String – Entity 1 Name
String – Entity 1 ID
String – Entity 2 Name
String – Entity 2 ID

SugarCRM-Specific
SugarSetRelationship (String, Sets a relationship between two modules.
String, String, String ) String – Module 1 Name
String – Module 1 ID
String – Module 2 Name
SugarSetRelationshipv4 (String, Sets a relationship between two entities, compatible
String, String, String ) also with custom modules.
String – Relationship Name
SugarGetRelationships ( String, Gets comma-separated a list of ID’s related to an
String, String ) entity
Email (IMAP)-Specific
MoveIMAPMessage ( String ) Moves the current email message to the specified
folder on the server.
String – Name of IMAP folder to move message to
Samples
Jump to a Specific Origin Row
(first create a Global Include and dim a variable)
Dim Counter
(then create a repeat each row procedure – start at row 250)

Sub VBScriptProcedure
Counter = Counter + 1
If Counter < 250 Then
Starfish.GotoNextRow
End If
End Sub
Skip an Origin Row if a field is Blank
If "@@ORG:PERSONNAME@@" = "" Then
Starfish.LogMessage("Person Name Blank, Skipping Record")
End If
End Sub
Manual Stage Flow-Control
Select Case Starfish.CurrentStagename
Case "Stage 1"
Starfish.GotoStage("Stage 2")
Case "Stage 2"
Starfish.GotoStage("Stage 3")
Case Else
End Select
End Sub
Execute a delete or skip row, based on a SQL Query

Dim sql
sql = "SELECT Count(*) FROM Table WHERE UniqueID=@@ORG:UNIQUEID@@"
If Starfish.ExecScalar(sql) > 0 Then
Starfish.ExecSQL("DELETE FROM Table WHERE UniqueID=@@ORG:UNIQUEID@@")
Else
End If
End Sub
Send an email notification
Starfish.SendEmail("person@domain.com","Email Subject","Email Body")
End Sub
Perform a SmartQuery Lookup, loop through results
Dim mdarray
mdarray = SmartQuery("Table", "Column='sample'", "Column1,Column2,Column3")
for i = 0 to UBound(mdarray) 'Loop through all rows
for j = 0 to UBound(mdarray, 2) 'Loop through all columns in the row
LogMessage "(" & i & "," & j & "): " & mdarray(i, j)
next
next

Cross-reference (Xref)
Starfish ETL has built-in methods for maintaining relationships between different systems.
Put simply, you will be storing an ID from your Origin, and the new related ID in your
destination for the record.
For an example, if you are migrating Company records from one system to another, and
later in the process you’ll also be importing Contacts that will be related to these companies
you’ll need a method to select which company the contact belongs to. One method would
simply be to store the Old (Origin) primary company ID somewhere inside of the new
company table on your destination. Then you could use a SQL/SmartLookup and pull down
the ID by a filter clause. This is perfectly acceptable; however you’ll take a performance hit
because for each row you’ll have to perform another lookup operation. Using Xref functions,
these ID relationships are stored in memory, thus making the lookup operation
instantaneous – speeding up your process. The idea will be to write out the ID relationships
for accounts while that job is running. Then, when the Contacts job runs it can initialize and
read from this list to quickly & easily relate to the correct Company.
Internally, Xref Lists are stored in your StarfishEngine\Xref folder as simple text files with
an .ids file extension. You may modify these files by hand if necessary.
Xref List Initialization

To begin, you must initialize an Xref List before you can read or write to it. This is done on a
per-job basis through the Xref Lists button on the Mapping tab.
Write to a new list to store a relationship between an Origin Primary Key and the
Destination Primary Key. The checkbox indicated whether the list should be appended to.
When it is written to, by default, the entire list will be blanked and written to as new when
the job is run. Check to box to instead append to the end of the existing list. (for instance if
you need to run multiple jobs which write to the same list, or need to run the job multiple
times for some reason).

Reading from an existing list will allow you to retrieve the “New ID” for a system, given an
“Old ID”. The drop-down on the Read side gets a list of Xref Lists that are available on the
server.
Rather than using the built-in screens for Xref List Initializations, you may also use the
XrefInitRead() and XrefInitWrite() VBScript functions.
Xref Writes
To write to an initialized Xref list, you may use the Exec Operation “Xref Write”. Select the
list from the Xref List drop-down. Then supply a value or variable to use as the Origin/Old
ID, and one for the Destination/New ID.
Alternatively, you can use the XrefWrite() VBScript function to write to an Xref list.

Xref Reads
To read from an initialized Xref list, you may use the “Xref Read” function field in your
mapping.
Alternatively, you can use the XrefRead() VBScript function to read from an Xref List.

Inserting Variables
Variables that have been created during an Exec Before/After Operation can be inserted as
the end-result value in any kind of Function Field. Place your cursor inside the text box, and
click the “Insert Variable” link.
From this list, you can choose a User-Defined variable, any Origin field, any Destination field
(that comes before it in the sort order, and that has been assigned a value), or a Special
Character. At run-time, the Starfish Engine will “replace” the variable code with the actual
value for the current Origin row.


Scheduling Jobs
Each Job can be set up to have its own Schedule. When a schedule has been configured and
enabled for a Job, it will be automatically executed by the Starfish Scheduler Windows
Service.
There are four different types of schedules that can be configured. (As of version 1.0, only
Time Interval and Daily are supported)
Schedule Type Description

Time Interval Job will be launched on every specified interval of minutes or hours.
Daily Job will be launched at the same time every day. If an error occurs,
you can define the Scheduler should react.
Weekly Similar to Daily, but can specify which days the Job is launched as
opposed to every day of the week.
File Not yet supported.
Be sure the “Starfish Scheduler” service is running on the desired server. Check the Event
Viewer for error and informational logs from the Scheduler service. To make sure everything
is being read correctly by the service, after the service starts, check the Event Viewer. An
Informational event should be created for each Job, showing the next time the Job is
scheduled to run.

Previewing Jobs
Before running a job for the first time, you can check to make sure all your formulas and
translations are working correctly by stepping through each row on the Preview tab. No data
will be modified in your destination system, and you may click Log to review the high-level
output logging from the Starfish Engine.

Running Jobs
If you wish to run a Job directly from the Starfish Admin, follow the steps below.
1. Deploy the current configuration to the Web Service.
2. On the General tab, select the Job that you want to run from the drop-down.
3. Click the Run Job tab.
4. Configure the options for the run.
5. Click the Start button. The status box should show “Running” and will display in real-
time, the # of Origin Rows processed.
The Message box will display any error messages encountered during the run. The Log box
will display the entire log for the Job run (when it’s finished), depending on the Logging
Level specified at the beginning of the run.
Run Options
Commit Valid for OLE DB and ODBC destination-type jobs, where “Use
Transaction” has been enabled. If checked, all transactions
against the destination database will be committed at the end of
the Job. If unchecked, no actual changes will be made to the
database (all transactions will be Rolled Back). This is very

helpful during testing/debugging.

Chain Jobs If the current Job has Job Chaining enabled, this checkbox
specified whether it should stop after this Job, or continue on
following the daisy-chained jobs.
Auto Deploy Ensure the project document gets deployed automatically so you
don’t have to click the Deploy button each time a change is made
and rerun.
Logging Level - None Only a total record count will be returned at the end of the Job.
Logging Level – Low Returns an overview of each row/stage processed.
Logging Level – Medium Returns Low Level messages, and displays all SQL statements
that are executed.
Logging Level - High Returns Low and Medium Level messages, and breaks down each
field during evaluation. This shows the actual VBScript executed
and return values from Function Fields.
Argument Allows user to pass in a run-time argument, which is available
during the Job through the variable “@@VAR:JobArgument@@”.
Job can pass a Return value out of the Job, through the VBScript
Starfish Class property “ReturnValue”.
Thread Count Number of threads to spawn for processing data into the
destination. Leave blank to use default. (set in StarfishEngine
Web.config)
Begin at Row Begin at Row/Leave blank to start at the first row
End at Row End at Row/Leave blank to process through to the end of all
rows.
Salesforce Download Only available at the end of a SFDC Bulk job. Downloads the raw
Results CSV results of each batch from SFDC servers and opens in Excel.
Using Multi-threading
A new feature in Starfish 2.1 is the ability to use multithreading. This ability has been built-
in natively to the Starfish Engine, so there is no need set up any additional message queue
systems. The default number of threads is set to 1 in the Engine Web.config file. To override
this, set a value in the Thread Count box in the Starfish Admin Run tab, or modify the value
in the Config file.
How many threads to use depends on a number of factors and finding the “sweet spot” may
involve some trial and error. Multi-threading is most beneficial in scenarios where there may
be network latency, such as integrations between backend systems and cloud-based or
hosted solutions. In cases where data is being moved between two systems on the same
database server or even within the network, using more than one thread may not produce
better results (and may degrade performance). However, in ground-to-cloud migrations
very significant performance gains will likely be seen.
For these cases, we recommend starting with a value of 8 threads. This number can be
increased if performance allows it. Starfish does not block an upper-limit of the number of
threads that can be used. You’re only limited by the memory and speed of your server and,
realistically, the limitations of your network bandwidth and the server you’re communicating
with. For this reason, we recommend not exceeding 24 threads.
NOTE: Multi-threading hands off the next available data row to the next available thread as
it because available. This means, rows being processed can be in any state at any given

time. Rows will not complete in the sequential order they are fed into the queue. So, for
example if you’re using 8 threads, Row #6 may finish before Row #1, then Row #3 may be
next, and so on. The results will basically be random. For this reason, you cannot use
multiple threads if you require data to be handled in a sequential order. For instance, if your
job performs an Insert/Update operation and relies on this lookup to work properly based on
data from previous rows.

Saving/Deploying Jobs
The Project Document, which is an encrypted XML file, contains the definition for the Jobs.
This file can be saved to the local Admin workstation as a “.SPD” file.
However, before a Job can be executed, it must be “deployed” to the Starfish Engine. This is
done through the Set Web Service URL menu option.

Change the Web Service URL to the server/address location of your Starfish Engine server.
After you change it and deploy the first time, Admin will remember the location. You should
receive the message below if the file deployed correctly.

Starfish Updater
Starfish ETL has a built-in updater. To start, click the Starfish button and select “Check for
Updates” or launch StarfishUpdater.exe directly from within the Starfish Admin installation
path.
The first time the Updater is run, you’ll have to supply the paths to the Starfish ETL
components. If your Starfish Engine is installed on another server, UNC paths are
supported. If you don’t have a certain component installed (such as the Scheduler), you
may leave the path empty.

Now you may click the “Look for Updates” button. If the updater finds out of date or missing
files, the “Update Now” button will become available. (Files are compared by MD5 hash)
Before proceeding, it is recommended that you close out Starfish Admin and shut down the
Starfish Scheduler service if running. Now you may press the “Update Now” button. This will
download and replace the necessary files from Starfish ETL servers.

System Backup Wizard

The Starfish ETL System Backup Wizard will walk you through a series of steps to create a
complete backup framework for any system. The System Backup Wizard does not actually
back up any data during this process, it only creates the tables and jobs to do so. When the
wizard has completed, then you must run the jobs to backup the actual data.
You will first have to verify your Origin and Destination connections for compatibility. The
connections for the currently selected Job will be used. The wizard works with any Origin
connection type. The Destination must be of a type which support SQL statements (typically
any current RDBMS).
When you complete the wizard, it will automatically:

 Create Tables inside of your backup database to match the schema or your Origin
tables.
 Create Data maps for each table to back them up.
It may be necessary for you to manually alter the generated table definitions and maps to
fit your needs.
Follow the steps below:
1. Create a new job. Set up the Origin & Destination. Your origin query/table selected
does not matter in the wizard, but will be necessary to populate to get a valid
connection.
2. Start the wizard by clicking the “System Backup Wizard” button on the General tab.

3. Welcome to the System Backup Wizard

Read the introduction & click Next.
4. Connector Compatibility Validation

Next your connections will be validated. If your connections test ok and your
destination is of a compatible type, you will be allowed to proceed.
5. Select Origin Tables to include in Backup Scripts

You will be presented with a check list box with all of the tables in your Origin. Select

any table(s) that you will want to include in your backup.
6. Table Creation Settings

Fill in the information for creating the new destination backup tables. This is where
the data will be backed up to.
Option Description
Generate matching Uncheck this option to skip this step during the wizard (if you
schema already have created the necessary tables)
Table Prefix Will prefix each table name with the value supplied (ie: prefix “bak_”
for table “account” will become “bak_account”.
Default Type If a data type is not given by the connector or cannot be resolved
through the data_type.map translation file, this value will be used.

Default Length If a type is specified, but not a length this value will be used.
Maximum Length If a value is supplied, and the length specified by the Origin is
greater than this number, the length will be changed to this value.
Change if Maximum If the length specified by the Origin is greater than the Maximum
Length is exceeded Length, the field data type is changed to this.
Drop Existing Tables Before a table is created by the Wizard a Drop Table command will
be issued.
7. Map Generation Settings

Fill in the settings for how you would like your jobs created.
Option Description
Generate Jobs Uncheck this option to skip this step during the wizard.
Job Name This value will be used when creating each job’s name. The
“%tablename” variable will be replaced with the actual table name
when the wizard runs.
Stage Type One stage will be created for each job, with each field mapped
directly across. This specifies that Stage Type: Insert, Update,
Delete, Update/Insert.
Guess Job’s Primary Valid only for Update, Delete, Update/Insert Stage Types. Attempts
Key for Match. to place the Match checkbox on the table’s Primary Key.
Here is the logic used for each connector:
Microsoft CRM: The Primary Key field is retrieved through the

Metadata API for each table.
NetSuite: The "internalID" field will be used for each table.
Sage CRM: The Table ID will be using matching format (tableprefix
+ "_" + tablename + "ID") ie: comp_CompanyID
Salesforce: The "Id" field will be used for each table.
SData: The "Key" field will be used for each table.
SQL Server: First the table is evaluated for a Primary Key. If one

does not exist then Unique Indexes are used. If there are no Unique
Indexes, then regular indexed fields are used.
SugarCRM: The "id" field will be used for each table.
Chain all Jobs All jobs will be chained together to run after another.
together
8. Last Run Settings

This option will create the following:
a. an additional table in your Destination database to track individual job last
run dates
b. a default filter on your Origin query to only select data from the last run date
c. logic to look up this last run date, and save/overwrite it when the job finishes
Then press Start to begin.
Option Description
Only pull data Check this option to process this step during the wizard.
changed from last run
Last Run Table Name of the table inside your Destination where the Last Run Dates
for each job will be stored.
Default Filter For SQL-based connectors, will be appended as a WHERE clause to
Origin query; otherwise will be populated as the Origin criteria box.
Default Format Date format to use in in filter/criteria.
Initial Date Date to initially populate in the Last Run table for each job. The first
time the job is run, it will pull data starting from this date based on
the filter clause.
9. Backup Wizard is now Generating

Please wait while each table & job is created

10. System Backup Wizard Complete

You may click the link to view the log file generated, or open it directly from the
Starfish Admin path, file named systembackup.log.

Connector-specific Guidelines
SData
1. Filters (where clauses) must use SData format. Such as ‘eq’ instead of ‘=’. Details for
the query language are provided at:
http://interop.sage.com/daisy/sdata/AnatomyOfAnSDataURL/QueryLanguage.html
2. To insert data into related tables you must select it as a Resource when setting up
your stage [All Fields].
3. To supply an ID to related entity you must select the table as a Resource when
setting up your stage [Key Only].
Salesforce
1. Passwords must contain the security token generated by Salesforce.
2. Destination: Updates and Deletes MUST match on the ID field only.
3. Filters containing quotes or single quotes must be escaped. IE: “John’s Pizza” must
be escaped to “John\’s Pizza”. If referencing an origin field such as
“@@ORG:AccountName@@”, change ORG to SFO to allow Starfish to automatically
escape these characters, example “@@SFO:AccountName@@”. This is not necessary
for standard field mappings, but only when part of a filter clause.
Microsoft CRM
1. Filters (where clauses) should use fully qualified field name (tablename dot field
name). ie: account.name, contact.firstname, etc
2. Filter statements may use the following operands: =, <>, >, <, >=, <=.
3. Strings in filter must be surrounded by single or double quotes. ie: name = ‘John
Smith’
4. Filter statements may use multiple logical operands (AND/OR), but of only one type
per statement. You may not use both an AND and OR in the same statement but
may use multiple ANDs or ORs in a statement.
5. Parenthesized filters are not possible due to limitations in the CRM API.
6. It is not necessary to generate an ID first for an insert, the API generates one for
you. If you need to get the newly generated ID within the same job on a future
stage, use the GetStageValue() function. If you need to get the ID on a different job,
then use a Smart Lookup.
7. For Lookup-type fields, provide data in following format:
MSCRMLOOKUP,table_name,guid
8. For Owner (Team/User) lookup type fields provide data in following format:
USER: MSCRMOWNER,USER,user_guid TEAM: MSCRMOWNER,TEAM,team_guid
9. To force a NULL/blank in an update operation into a particular field, write out
“MSCRMNULL”.
10. You may also use MSCRMNULL in filter statements (ie: name <> MSCRMNULL).
11. You may set the createdby/modifiedby field to a valid systemuserid guid.
SugarCRM
1. For many types of relationships, you cannot just supply the ID – you’ll need to make
use of the SugarGetRelationship and SugarSetRelationship functions.

NetSuite
2. Related fields (such as Addresses) are loaded for the Destination only and allow 2 of
each type.
3. For RecordRef type fields, supply the Internal ID of the entity being linked to.
Email
1. Protocol uses IMAP, not POP.
2. Default server port is 143, Secure IMAP is 585.
3. When connection to a MS Exchange Server IMAP account, use port 993 with SSL
checked.
SageCRM
1. In SageCRM, the following options should be configured as specified below.
(Administration > System > Web Services)
a. Enable Web Services: Yes
b. Make WSDL Available to all: Yes
c. Dropdown fields as strings in WSDL: Yes
d. Force web service log on: Yes
2. Ensure the user you are using has “Allow Web Service Access” enabled in their user
profile.
3. Sub-entities are only enabled for Company and Person parent entities. This is a
limitation of the SageCRM web service API.
Oracle
1. In order for the Oracle connector to work properly, you must first install the 32-bit
version of the Oracle Data Provider for .NET (ODP.NET). Download the latest version
from here:
http://www.oracle.com/technetwork/database/windows/downloads/index-
101290.html

Troubleshooting
Problem: Starfish Engine installation fails with message “Installation was
interrupted”.
Solution: On some versions of Windows, in addition to IIS needing to be installed – “IIS 6
Management Compatibility” will also need to be installed. This is added through the “Turn
Windows features on or off” function in Control Panel and is located in the “Internet
Information Services\Web Management Tools” Program group. Be sure to check all items
under the “IIS 6 Management Compatibility” group.
Problem: Scriptcontrol error when deploying bindings file.

Solution: Some newer versions of Windows Server does not have the ms script controls
registered on the server by default. From the command line, run “regsvr32
msscript.ocx”. This file may be located in System32 or SysWow64, depending on your
version of Windows. If you continue to get similar errors, you may also need to register
“vbscript.dll” and “jscript.dll”.
Problem: Error message on deploy: “Could not load file or assembly

‘Microsoft.Crm.Sdk’ or one of its dependencies. An attempt was made to load a
program with an incorrect format.”
Solution: On some 64-bit systems, IIS disables 32-bit mode which Starfish requires.
For IIS 7: In IIS, select the Application Pool where Starfish is running, and go to the
Advanced Settings and set “Enable 32-Bit Applications” to True.
For IIS 6: Follow the instructions outlined here:
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/405f5bb5-
87a3-43d2-8138-54b75db73aa1.mspx?mfr=true
Problem: ODBC DSN entries do not show up in the drop-down list, or DSN entry not found
error when running.
Solution: On 64-bit systems, the DSN must be configured using the 32-bit version of the
ODBC manager, it is typically located in: C:\Windows\SysWow64\odbcad32.exe.
Problem: When deploying, you get an error message “The HTTP request is
unauthorized with the client scheme ‘Anonymous’. The authentication header
received from the server was ‘Negotiate,NTLM’”.
Solution: In IIS Manager, expand Sites, Default Web Site, and select StarfishEngine. Under
IIS, select the “Authentication” item, and make sure “Anonymous Authentication” is set to
“Enabled”.
Problem: Permission denied error to starfish.spd file when deploying.

Solution: Grant modify permission on the C:\Inetpub\wwwroot\StarfishEngine folder to the
server’s IIS_WPG (IIS6) or IIS_IUSRS (IIS7) user.
Problem: When testing the Starfish Engine website, you get the following (or similar) error
message: Could not load file or assembly 'Microsoft.Crm.Sdk' or one of its
dependencies. An attempt was made to load a program with an incorrect format.
Solution: On 64-bit systems, this can be caused if 32-bit applications have been disabled
for IIS.
For IIS6, go to a command prompt. Change to the “C:\Inetpub\AdminScripts” folder and
execute the following command:
cscript.exe adsutil.vbs set W3SVC/AppPools/Enable32BitAppOnWin64 true

For IIS7+, go to IIS Manager, select the Application Pool that Starifh is running under, select Advanced
Settings, and set "Enable 32-Bit Applications" to "Enabled".
Problem: Errors on Insert & Update stages when using SData Destination with Saleslogix
7.5.3, such as “Object not set to an instance of an object”.
Solution: This is caused by a bug in Saleslogix 7.5 Service Pack 3. If you are running 7.5.3,
be sure to apply 7.5.3 Hot Fix 1, which will fix this issue. SP, Hot Fix 3 may also be
necessary to get more detailed messages from SData (such as ValidationException
reasons).
Problem: Permission error when deploying on path

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files.
Solution: Some server environments require write permissions be given to the NETWORK
SERVICE user (or the user defined in Application Pool Identity tab).
Problem: Permission error when deploying “Access to the path

‘C:\Inetpub\wwwroot\StarfishEngine\starfish.spd’ is denied”.
Solution: Some server environments will require write permissions to be given to the
IIS_WPG user. Access should be given to the Inetpub\wwwroot\StarfishEngine folder.

Appendix A: Sample Import Exercise

Goal: To get you acquainted with Starfish by doing a simple import of Accounts into
SalesLogix that will use many different Starfish features.
Criteria: After installing the Engine, Admin tool, and Scheduler; use a provided
bindings file - training.oct to import a provided .csv file of new customers who
need to be added into SalesLogix as Accounts. We will be importing the data
into the SalesLogix Eval database.
1. Make sure the SalesLogix Eval database is set up.
NOTE: We will not address setting up the Eval database in this exercise.
You may wish to do the import on a development environment of your own.
This is perfectly fine.
2. Give Full Control to the IIS_WPG user in C:\Windows\Temp
3. Give Read access to the IIS_WPG user in the directory where you extracted the
Starfish_ETL_v1.1.zip. This is where the Customers.csv file we will be importing
is located.
4. Create a Text DSN to CSV file. This will allow you to set your .csv as an ODBC
data source Origin.

- Go to the Start menu>Administrative Tools>Data Sources (ODBC)
- Under the System DSN tab, hit the Add button
- Select Microsoft Text Driver (*.txt; *.csv) option from the list and click
Finish.

- .Give the Data Source a Name and Description similar to what you see below,
and click OK.
5. Log into the Starfish Admin and choose the File>Load Bindings File option.
Navigate to the folder where you unzipped the Starfish_ETL_v1.1.zip file and
choose to open the training.oct file.

NOTE: We will be using an existing Bindings file that has a Job set up, as well
as the Mappings for an import to Accounts in SalesLogix. We will need to
tweak the Origin and Destination, however.
6. The General Tab should appear. Within the Jobs dropdown menu, select
Import Customers. The rest of the tabs will then appear.
NOTE: An error will appear after selecting. This is because the Origin that was
originally mapped has not been configured yet. Click OK to get through the
error. A few seconds will go by and another related error will appear. Click OK
on that, as well. We will configure the Origin in a few minutes.

7. Click on the Origin tab.
- Click on the ellipses next to the DSN. Choose the option for .csv Files. This is
available only after you have set up a DSN for .csv files, as we did earlier.
- Click on the ellipses next to Available Tables. Choose your Customers.csv file
that should appear here.
- Click the Test Connection button. You should see a Test Result of
‘Successful.’
8. Click on the Destination tab.
- We will be importing into SalesLogix, which uses an OLE DB Connection. Click

on the OLE DB tab and then click on the ellipses that sits to the right of the
Connection String. This will bring up the Data Link Properties window.

- The existing connection information is left behind on the form. Use your
SalesLogix information to populate this form – enter your server on top, then
choose the database you wish to import the data into. Be sure to type in
your username and password in the User Settings area, and hit the Test
Connection button to be sure you will be properly connected.

- Click the Test Connection button on the right to be sure your Destination
connection succeeds.
9. Click on the Mapping tab.
- Click the dropdown next to Stages, and you will see three stages already
mapped: Create Account, Create Account Summary, and Create Address.
NOTE: The Account, Account Summary, and Address tables need to be

populated any time we do a new Account import in SalesLogix. The same thing
goes for the Contact, Contact Summary, and Address tables when importing
Contacts.
- Before we look at a few of the actual mappings in the stages, let’s first look at
the Exec Before section, where there are already two Exec Operations set up.
These particular Execs allow a SalesLogix stored procedure to be called that
creates IDs for new Accounts and Addresses, respectively. You will need to
map these IDs several times in the import, and the great thing about this
function is that it will allow you to set the Account and Address IDs as
variables that you can call each time you need to map the field.

o Get Account ID
o Get Address ID
- In the Stages dropdown again, select the Create Account Stage. This shows
all of the Bindings between our .csv and SalesLogix fields (as mentioned,
these have all been set up already).
As noted in the manual, some fields are mapped straight to each other.

Other fields, like the AccountID and AddressID fields use a Function Field.
This allows you to Edit the Function, and enter in a Hardcoded Value.
Within the Hardcoded value box, you may Insert a Variable.
If you choose from the User-Defined Variable Class, you can select the ID’s
that were created by the Exec Before statements we just looked at.

- Feel free to look at the other fields in the various stages that have been
mapped out here. They may help to give you some ideas for your own
imports.
10. Now that we have our Origin and Destination set up, as well as our Mappings, we
are ready to run our import (we are not going to configure the Scheduler for this
import, as it is a one-time import). Before we run the import, we need to
Deploy to Web Service. As noted in the manual, this is how we get all of the
information we have just configured to the Starfish Engine.
- Select the option at File>Deploy to Web Service:
- Depending on how and where you installed the Starfish Engine, use a path
similar to the following:
- Click OK, and you should get the following notice:

11. Click on the Run Job tab.

- Because we selected the option to ‘Use Transactions’ on the Destination tab,
we have the option to Commit, or not Commit the import to the Destination.
If unchecked, you can consider it to be a ‘test’ import, as all transactions will
be rolled back. Let’s start with it unchecked to make sure the import goes
through ok. Make sure the Commit checkbox is unchecked, and hit the Start
button.
- After a minute, you should get word that the Import Customers job is
Complete. The amount of information you see in the log correlates with the
Logging Level you specified on the right hand side.
- If you were to check the SalesLogix Eval database right now, you would not
see your data. This is because you need to Commit the data. Now that we
know the import worked from our test run, check the Commit box, and hit the
Start button again. You should get the same result.

- From here, you may login to the SalesLogix Eval database and make sure the
data is there.
NOTE: The data in the Customers.csv file is originally from the SalesLogix Eval
database, so you should see two Abbott Ltd. Accounts, amongst others if your
import was successful.

Install and use Starfish ETL Suite

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Install and use Starfish ETL Suite

Transféré par

Droits d'auteur :

Formats disponibles

Starfish ETL Suite

Installation & Usage Technical Guide

PLEASE NOTE: It is recommended to use our Wiki, instead of this

Starfish ETL Version 3.7

Confidential Page 2 11/11/2015

Microsoft CRM .................................................................................................... 56

Confidential Page 3 11/11/2015

Confidential Page 4 11/11/2015

Confidential Page 5 11/11/2015

Confidential Page 6 11/11/2015

Engine Web.Config Settings

Confidential Page 7 11/11/2015

2. Select the installation folder, and the default access.

Confidential Page 8 11/11/2015

3. Click next and complete installation.

5. In the window, set the location (default if on the same machine is

Confidential Page 9 11/11/2015

2. Select the installation folder, and the default access.

Confidential Page 10 11/11/2015

3. Click next and complete installation.

Near the end of the file, change http://servername/StarfishEngine.asmx to the

Confidential Page 11 11/11/2015

5. A number of other options may be changed through this config file.

Scheduler Config Settings

Confidential Page 12 11/11/2015

Field Caption Description

Confidential Page 13 11/11/2015

Then paste your key in the box, click OK.

Confidential Page 14 11/11/2015

Field Caption Description

Field Caption Description

Confidential Page 15 11/11/2015

Creating a New Job

Confidential Page 16 11/11/2015

Configuring the Origin

Example for OLE DB (MS SQL Server)

Confidential Page 17 11/11/2015

Configuring the Destination

Example for OLE DB (SalesLogix)

Confidential Page 18 11/11/2015

Confidential Page 19 11/11/2015

2. In the Stages dialog window, click the button

Confidential Page 20 11/11/2015

5. Repeat steps 2-4 for each additional stage.

Confidential Page 21 11/11/2015

Mapping Screen Overview

Screen Layout Definition

For example, this is useful for mapping a “Modify Date” field to

Confidential Page 22 11/11/2015

the current time. You don’t want the field to be compared,

Confidential Page 23 11/11/2015

Straight Field-to-Field Mapping

Confidential Page 24 11/11/2015

Function Field Mapping

3. OR click the + button

Confidential Page 25 11/11/2015

high chance of the same queries, otherwise an out of memory

Confidential Page 26 11/11/2015

Confidential Page 27 11/11/2015

Trim an Origin field Return different values, based on a Variable

Confidential Page 28 11/11/2015

Exec Before/After Operations