Académique Documents
Professionnel Documents
Culture Documents
CONTROL FLOW: The execution flow of the package always start from the
control flow level then after it will move to the DATA
FLOW.
DATA FLOW: Data flow is purely used for data loading purpose, whenever
we want to load the data from source to destination we will
use these data flows.
EVENT HANDLER: Whenever an event got occurred in the package level or task
level then automatically event handler will be raised and
the specified operation will be performed which we
configured in it.
PACKAGE EXPLORE: This package explorer will give you the package information
like TASKS, CONTAINERS, and VARIABLESetc.
HOW TO CONNECT SSIS IN 2008
SELECT AND EXPAND SQL SERVER 2008 AND SELECT SQL SERVER BUSSINESS
INTELLIGENT DEVELOPMENT STUDIO
CLICK ON NEW
CLICK ON PROJECT
CLICK ON OK
DATE FLOW SOURCES: we have different kind of sources to extract the data
from corresponding file or database to load the data into different kind of
destinations.
We have following sources available in the SSIS level.
1) FLAT FILE SOURCE
2) EXCEL SOURCE
3) RAW FILE SOURCE
4) XML SOURCE
5) ADO.NET SOURCE
6) OLEDB SOURCE
1) FLAT FILE SOURCE: Flat file source is used to load the data from TEXT,
CSV files.
NAVIGATION:
Go to Data flow
Drag flat file source
Right click edit
Click on new to create new connection manager for the created file.
Provide the connection manager name which you required.
Browse the file from the location in file name.
Enable the check box column name in the first data row.
Go to column observe the row delimiter and column limiter
Click on ok
Now go to column and observe the output columns properly coming or not.
Click on ok
Open the sources file copy the delimiter which are present in source
Go to SSIS level
Go to column delimiter
Paste the copied item
Click on refresh it
Then automatically it will split the column based on the delimiter
Go to connection manager
Go to header rows to skip option as you required.
2) EXCEL SOURCE: Excel source component is used to load the data from
excel file into destinations.
Navigation:
Go to data flow
Drag excel source
Right click edit
Click on new
To create new connection for excel
Browse the excel file from the location
Click on ok
Select the sheet name from drop down list
Go to columns tab in the left side
Click on ok
Drag OLEDB destination and configure it
3) RAW FILE SOURCE: We can load the RAW DATA by using RAW source to load
the destination.
Prepare one flat file and configure with flat file source
Drag RAW file destination
And configure it.
Navigation:
Drag RAW file source
Right click and edit
Browse the file from the location Where it got located
Go to columns
Click on it
Drag OLEDB destination and configure it
4) ADO.NET SOURCE: ADO.NET source is used to load the data from data
bases this component will run based on .NET providers
Navigation:
5) OLEDB SOURCE:
6) XML SOURCE: XML source will extract the data from the XML file
into database to load this XML file we need.
.XML
.XSD
.XML: XML is extendable markup language which can contain the source
data inside of it.
.XSD: XSD is XML schema definition file which will store the meta data
of XML file.
<ROOT>
<GOO>
<ID>201</ID>
<NAME>SAMBA</NAME>
<SAL>45000</SAL>
</GOO>
<GOO>
<ID>301</ID>
<NAME>SRIKANTH</NAME>
<SAL>55000</SAL>
</GOO>
<GOO>
<ID>401</ID>
<NAME>NARESH</NAME>
<SAL>53000</SAL>
</GOO>
</ROOT>
CLICK SAVE AS
CLICK ON SAVE
THEN AUTOMATICALLY THE ENTIRE DATA WILL BE STORED IN THE FORM OF XML IN THE
PROVIDED FORMAT.
NAVIGATION
CSV FILE: CSV file is COMMA SEPERATED VALUES, which will have the column
delimiter as , always.
To load the CSV file we have to use flat file source and configure it same as
.txt
1) ADO.NET DESTINATION
2) OLEDB DESTINATION
3) EXCEL DESTINATION
4) RAW FILE DESTINATION
5) FLAT FILE DESTINATION
6) SQL SERVER DESTINATION
OLEDB DESTINATION: OLEDB DESTINATION will load the data into different
kind of databases from the sources generally in real time we will use 90%
destination as OLEDB.
NAVIGATION:
FLAT FILE DESTINATIO: We can extract the data from different sources and
load the data in to flat file destination.
RAW FILE DESTINATION: By using this destination we can create the raw file
with the help of source data, there is no execution for this raw file and we
cant create this files manual way.
NAVIGATION
Create always
Create once
Append
Truncate and append
CREATE ALWAYS: This option will create always a new file and delete the old
one which is available in the same path.
CREATE ONCE: This option will create only once in the destination path, if
we are trying to create same file one more time then automatically it will
fail.
APPEND: This option will always add the rows in to the existing file with
same name as new records.
TRUNCATE AND APPEND: It will clean the old data which is already present in
the existing file and re-write the new data in to the same file.
EXCEL DESTINATION If we want to load the data in excel then we will use
excel destination.
NAVIGATION
ADO.NET DESTINATION
Configure source
Drag ado.net destination
Connect with source
Right click edit
Provide the database connection and table connection
Click on mapping
Click on ok.
SQL DESTINATION; Sql server destination as same as OLEDB and it will
be used for purely bulk data load
NAVIGATION
ADO.NET
OLEDB
XML FILE
RAW FILE SQL SSIS
SQL SSIS VARCHAR STRING
VARCHAR STRING INT FOUR BYTE INT
ADO.NET
OLEDB
SQL SSIS
SQL SSIS
VARCHAR VARCHAR
VARCHAR VARCHAR
INT INT
INT INT
TRANSFORMATIONS Transformations are used to transform the data
in b/w source and destinations. In SSIS level we have different kind of
transformations as per the requirement.
NAVIGATION
Example: DIAGRAM
DATA
CONVERSION
TRANSFORMATION
OLEDB
DESTINATION
EXCEL FILE SOURCE
EXCEL
LEVEL ID NAME SAL
DOUBLE DOUBLE
DATATYPES PRESICION UNICODE STRING PRESICION
DATA CONVERSION
SSIS LEVEL TRANSFORMATION
D_ID D_NAME D_SAL
DATA
TYPES FOUR BYTE INT STRING FOUR BYTE INT
SQL
SERVER OLEDB DESTINATION
LEVEL
D_ID D_NAME D_SAL
INT VARCHAR INT
COPY COLUMN: We can copy data of the existing column which will act as new
columns the map into the destination level to populate the same data.
NAVIGATION
Example diagram
COPY COLUMN
CUSTNO
TRANSFORMATION
Example:
SQL TABLE
NAVIGATION
Example
SQL TABLE
CHARACTER MAPPING
TRANSFORMATION
1 A 1000
2 B 2000
SQL TABLE
2 B 2000 PANDILLA\SANDEEP
PACKAGE2
3 C 3000 PANDILLA\SANDEEP PACKAGE2
OLEDB DESTINATION
NAVIGATION:
Prepare one flat file source as we mentioned above.
Drag flat file source and configure it.
Drag audit transformation connect with source right click edit.
Select the required column which you are going to add in destination
level like username, package etc. in audit type
Click on ok
Do the data conversion for the input column to map with destinations
Drag OLEDB source and configure it.
NAVIGATION:
DERIVED COLUMN: Derived will help us to derive a new column in SSIS level.
HOW TO ADD NEW COLUMNS IN DERIVED COLUMN BASED ON THE EXISTING COLUMN
NAVIGATION:
Click on ok
NAVIGATION
CONDITIONAL SPLIT: we can split the data based on the given condition
in the conditional split transformation send it to the multiple destinations.
NAVIGATION
UNION ALL: union all will combine different types of data sources and load
it in to single destination. We can combine any type of data sources by using
this transformation.
NAVIGATION
Prepare 3 sources flat file, excel and OLEDB source with same column
names and data types.
Drag flat file, excel and OLEDB sources and configure it.
Do the required data conversion for flat file source and excel source
Drag UNION ALL transformation
Connect with source in order OLEDB, EXCEL, FLAT FILE.
Note: the first connected source will store the data type for the
output column and remaining sources have to be can data types flat file
column
Right click and edit
And map the appropriate columns.
Click on ok
Drag OLEDB destination and configure it.
MUTICAST: we can send the same copy of the source to different kind of
destinations at a time
NAVIGATION
NAVIGATION:
NAVIGATION:
ROW COUNT: Row count will help us to compare the no of rows following between
source and destination.
We will capture this count into a variable in SSIS level and we can use
anywhere in the package.
Go to control flow
Right click on empty space and select variables then automatically
variable window will open.
Click on add variable button
Provide the variable name and appropriate data type
NAVIGATION
Prepare one flat file source and excel same columns and configure it.
Drag sort transformation and do the sorting for excel and flat file
sources.
Drag merge transformation connect it for flat file and excel sources as
input1 and input2.
Right click and edit.
Select the merge input1 and merge input2 column as following.
Drag OLEDB destination and configure it.
Here we can perform all types of joins like INNER LEFT RIGHT AND FULL JOIN
Prepare two source files flat file and excel with some matching records
to do the join operations.
Drag flat file and excel sources and configure it.
Drag data conversion and do the required conversion for flat file and
excel.
Drag the sort transformation and do the sorting both flat file and
excel.
Drag merge join transformation connect left and right inputs.
Right click and edit and provide the required join type from drop down
and select the required column for the left and right inputs.
Click on ok.
Note: if we want to do the right outer join we can select left outer
join from drop down list and click on SWAP INPUT button. Then
automatically left table will be right table and right table act as
left table.
Drag OLEDB destination and map the required columns for destination
table.
Here we will check the data in the lookup table by using the base table
based on the key column.
Example: if the existing customer is coming from the source then we have to
redirect to matching output.
If the new customer is coming then we have to redirect as new customer
NAVIGATION:
Prepare one flat file with some sample records, having following
headers.
Drag flat file source and configure it.
Drag lookup transformation right click and edit.
Go to general tab
Select the connection type as OLEDB connection manager
Provide specify how to handle rows with no match entries as
IGNORE FAILURE: if any new record is coming which is not matched with
the destination then automatically package will ignore those new
records.
REDIRECT ROWS TO ERROR OUTPUT: all the records which are not matched
with the destination will redirect to the error output.
REDIRECT ROWS TO NO MATCH OUTPUT: all the records which are not matched
with the destination will come as no match output as a separate output.
FAIL COMPONENT: if any new customer is coming which are not there in
the destination level automatically lookup component will be fail.
Go to connection tab.
Provide OLEDB connection manager for the lookup table
Select lookup table from the drop down list.
Go to columns tab
Add the join column as following.
Click on ok
Drag OLEDB command and OLEDB destination.
Connect with match and no match outputs
click on ok
Go to columns mapping
Provide the mapping as following
Click on ok.
SLOWLY CHANGING DIMENSION: Slowly changing dimension will update the changes
for the selected records which will is coming from the source based on the
provided change type.
1) SCD type 1
2) SCD type 2
SCD TYPE 1: it will maintain only updated records and it will not maintain
any historical data in table. This as similar as lookup concept.
SCD TYPE 2: it will always maintain historical data for the records based on
the start date and end date in the table.
Prepare one flat file source with some data and destination with
following structure.
SOURCE STRUCTURE:
DESTINATION STRUCTURE:
Click on ok.
Select the dimension columns and required change type for type 2 as
following
Click on next.
Use start date and end date to identify current and expired records.
Select start date and end date from drop down list.
Provide variable to set date values option as
Click on next
Click on next
Click on finish.
SCD TYPE 1 NAVIGATION:
SOURCE STRUCTURE:
DESTINATION STRUCTURE:
Click on next.
Provide the change type for remaining dimension columns as following.
Click on next.
Select the check box for changing attributes
Click on next.
Click on next.
Click on finish.
EXAMPLE TABLE:
EXPECTED OUTPUT:
NAVIGATION:
Prepare the source as following
Drag flat file source and configure it.
Drag Unpivot transformation and connect with source.
Right click and edit.
Select the years using left side check box as following.
Provide destination columns and pivot key values as following.
Provide the pivot key value name as YEAR. To get the years list in the
year column and same data in the sale column.
Click on ok.
Drag OLEDB destination and configure it.
PIVOT: pivot is used to transform the rows into multiple columns here no of
columns will be increased and rows will be decreased.
INPUT TABLE:
EXPECTED OUTPUT;
NAVIGATION:
Prepare one flat file source having above data and destination.
Drag pivot transformation and connect with flat file source
Right click and edit.
Go to input column tab select all the column.
Go to input and output properties tab
Expand the input columns.
Set the pivot usage as following for the input column.
Default input columns. Pivot usage
Pid -0
Pname -1
Year -2
Sale -3
Expand pivot default output.
Select on output column.
Click on add column 5 times then automatically it will add 5 columns in
the output column.
Rename those column as following.
Default output column.
Pid,pname,2001,2002,2003.
Output column pivot key value source column
Pid pid linage id of pid(i/p)
Pname pname linage id of pname(i/p)
2001 2001 linage of sale(i/p)
2002 2002 linage of sale(i/p)
2003 2003 linage of sale(i/p)
Drag OLEDB destination and configure it.
IMPORT COLUMN: By using this import column we can upload images and different
types of documents into SQL table.
NAVIGATION:
EXPORT COLUMN: Export column will help us to extract the required files which
are uploaded to the SQL server.
We have a SQL server table with the columns file path, file name,file
type,file data.
Note 2: we can change the format as well for the supported images by updating
table.
FUZZY GROUPING: Fuzzy group will group the data which is coming from the
source based on the threshold value which is provided by the user.
NAVIGATION:
FUZZY LOOKUP: Fuzzy lookup will look the words into lookup table which is
available in the source based on the threshold value and provide the
similarity and confidence values for each and every row.
Data:
Rose is RED
Sun is hot
Apple is red
Bangalore is cool
board is white
Ice is cool
Lookup values
RED
COOL
NAVIGATION:
FUZZY GROUPING: fuzzy grouping will group the UN cleaned data which is coming
from the source based on the threshold value.
FUZZY LOOKUP: Fuzzy lookup will look the key words in the lookup table based
on the source and provide the list of rows.
In fuzzy lookup the no of rows will be depend on key words available in the
lookup table.
Id,name,sal
101,A,10000
102,B,25000
103,C,30000
Rowno,id,name,sal
1,101,A,10000
2,102,B,25000
3,103,C,30000
TERMLOOKUP: Term lookup is the one of the powerful transformation in the data
flow level, this transformation is basically help to analyze the set of text
records which are coming from the source.
This will look for the noun or noun phrase which are
available in the source based on the provided keyword in the lookup table.
Sno, keywords
1, touch screen
2, screen
3, application
4, phone
NAVIGATION:
Prepare the some sentences in the flat file source which are having
nouns and noun phrase.
Drag flat file source and configure it.
Drag data conversion and convert the column to Unicode string.
Drag term extraction right click and edit.
Select the column which you have nouns and noun phrase by using check
boxes.
Go to advanced tab
Select the term type using radio button whether it is noun or noun
phrase or noun and noun phrase
Click on ok.
Drag OLEDB destination and configure it.
Once this got executed it will provide the selected term types and
score for that.
Following are the important containers and tasks available in the control
flow level.
CONTAINERS
TASKS
INIT EXPRESSION: Init expression will initialize the value for the loop, so
the loop always starts from initial value.
Example @A=1
Here A is a variable
EVAL EXPRESSION: Eval expression is evaluate expression which will check the
condition and execute for loop until the condition false.
Example @A<=@B
ASSIGN EXPRESSION: Assign expression will increase or decrease the value for
every execution and gives increment value to the condition
Example @A = @A+1
NAVIGATION:
Configure one data flow with one flat file having some sample records
Drag for loop container
Create 2 variable with the scope of package as following
If we want to change the properties of the tasks for all in the container no
need to change each and every task, just we can change container property
then it will applicable for all the tasks.
BULK INSERT TASK: Bulk insert task will load the data very fast using the
fast load options from flat file source to the SQL SERVER destination.
Our source should be flat file and destination should be SQL SERVER.
NAVIGATION:
DATA PROFILING TASK: Data profiling task will validate the data which is
coming from the source based on the provided profile request in the table.
We will have different kind of profile request like column length, null
ratio, statistics etc. to validate source data.
NAVIGATION:
Prepare one table with some null values in a table in specific column.
Drag profiling task.
Right click edit.
In general tab options provide the destination types as file connection
choose destination path and format by browsing in destination.
Provide the override destination option as true.
Go to profile request tab.
Select the profile type as we required.
For example column null ratio profile request.
Click on request id, then it automatically generates.
Select it in go to the request properties.
Connection manger: database file source connection
Table name : source table name
Column: request column name for null ratio.
Click on ok.
SEND MAIL TASK: send mail task will help us to send the email notification
for starting of the package and success of the package and failure as well.
NAVIGATION:
SMTP CONNECTION: SMTP connection is the gateways email to the one domain
to other domain to configure this. Select the new SMTP connection drop
down list and provide the SMTP server name default port no is 25.
FROM TO CC BCC
Provide the corresponding email address for the above tabs.
MESSAGE SOURCE TYPE: we have 3 type of message source as following.
DIRECT INPUT: directly we can provide the message source in the column.
FILE CONNECTION: create one file and provide the subject which you want
inside of it file and select the file from location.
VARIABLE: select the priority as we required like LOW, HIGH, NORMAL
ATTACHMENT: if we have any attachments browse it for mail.
SUJECT: we can hard code the subject message in the subject tab.
If we want to change the subject from server to server then we need to
handle this in dynamic way.
Go to express tab
Click on right side corner symbol
Select the properties as subject.
Click on expression and write the following expression for this.
the package got started on + @[system::Machine Name] +
at the time of + (DT_STR,50,1252)substring((DT_STR,50,1252)get
date(),1,19)
Message source: expression shown below.
The package got completed successfully server + @ [system::Machine Name]
+ \n\n Regards, \n support team.
EXECUTE PROCESS TASK: Execute process task is used to call the applications
in the SSIS level by using .EXE, .BAT file etc.
To call the applications first we need to provide the executable file where
it was installed in the load machine in the executable tab.
Generally we will use this task for extracting the ZIP or RAR archive in the
file using WinZip or WinRAR software.
NAVIGATION:
For UNRAR
EXECUTE PACKAGE TASK: Execute package task will help us to call the SSIS
package as with in a package.
NAVIGATION:
NAVIGATION:
FILE SYSTEM TASK: file system task is used to do the operations like, copy
file, delete file, move file, and folder operations like delete directory,
create directory.. etc.
NAVIGATION:
Drag file system task create two folders named as source and destination
Create two variables in SSIS level as source path and destination path.
Right click the task and click on edit.
Provide the operation which you required.
Set the property as source path and destination path is true. And provide
variables as source variable and destination variable for list.
Click on ok.
Execute for task
We have following operations for file system task.
Copy file
Move file
Delete file
Delete directory
Create directory
Delete directory contents
Move directory
SCRIPT TASK: script task is same as script component here also we can write
the code C#, VB as per our requirement.
We can do any kind operations by using script task in the control flow level.
SCRIPT language: script language is which are going to write in the script
task either VB or C#.
READ ONLY VARIABLE: The variable which are select under read only variable
will provide the variable as input to do the operations in the script. We
cant override the output value in these variables after dong the operation.
READ WRITE VARIABLES: The variable which are select under read only variable
will provide the variable as input to do the operations in the script. We can
override the output value in these variables after dong the operation.
NAVIGATION:
Msgbox(fileloc + + datetime.now)
Dts.taskresult =scriptresult.sucess
EXECUTE SQL TASK: execute SQL task will help us to do the SQL level operation
like calling procedures, executing SQL statements, storing results of SQL
statements in to variables etc.
This is only the component available in the control flow to do the SQL level
operations.
NAVIGATION:
Provide all the connections and provide and paste the query which you
required in the task on following.
Create one procedure with required logic and assume name as usp_emp
Exec usp_emp
Parameter mapping: parameter mapping concept is the values which are present
in the variable will be passed to sql statements.
As
Begin
End
Go to SSIS
Exec sp_update ?
Click on ok
Conclusion: while executing the task the values which we provide in the
variable will be passed to procedure as input.
RESULT SET: the concept of the result set is storing the sql statement
results in to a user variable created in the SSIS level.
Single row:
To see this variable value at the time of execution time set the break
point pre and post execution.
FULL RESULT SET: it will store the complete result the which is coming
from the table in the form of rows and columns.
Provide the sql query as following
Select * from empdata
Provide it in the task
Provide the result set as full result set
Create one variable as following
Click on ok
XML: we can store the xml string into one user variable in SSIS level
Prepare the query as following
Select * from empdata for xml auto
Drag one execute sql task
Set result set as xml
Paste the above query
Create the variable with string datatype
Go to result set tab and add one variable and results name as 0 and
variable as user::xmaldata
Click on ok
Add break points to see the output at the execution time.
FOREACH LOOP CONTAINER: For each loop container is used to iterate the loops
based on the type of enumerator.
NAVIGATION:
Create one folder named as source folder and prepare sample files as
src1.txt, src2.txt, src3.txt.
All the files should be in the same format with same extension
Create 3 variables as following.
Click on ok.
Go to variable mappings map as following.
Click on ok.
We have 3 options in the collections tab to store the filename in the
variable.
1) Fully qualified: it will store entire path and filename in the
variable.
2) Name and extension: it will store only the file name and extension in
the variable.
3) Name only: it will store only the file name in variable.
Drag data flow task configure the flat file connection with any one of the
file located in the location.
Drag OLEDB destination and configure it.
In the connection manager select the flat file connection click on F4.
Go to expression in the property and provide the connection string
property as following.
Click on ok.
Drag for each loop container inside of for each loop container.
Right click and edit.
Go to collection tab.
Provide enumerator as for each ADO.NET schema rowset enumerator.
Select the new connection in the connection
Click on new.
Select provider type as Microsoft office 12.0 access database engine OLEDB
provider.
Click on ok.
Provide any one file name.
Server or filename: E:\source\src1.xlsx
Go to all.
Go to advanced options.
Provide the extended property as following
Click on ok.
Drag one dataflow task inside of it.
Drag one excel source.
Provide the connection through the any one of the file.
Select the data access mode table name or view name variable.
Provide the variable name as following.
Click on ok.
Drag OLEDB destination
Configure the destination table.
Click on ok.
Click on ok.
Click on columns
Click on ok.
Drag flat file destination and configure it.
Create on variable destination and provide the destination path in it.
Go to connection string option in the flat file connection.
Provide the connection string expression as following.
SPECIAL PROPERTIES
Logging
Check points
Precedence constraints
Configurations
Deployment
Scheduling
Select the provider type as SSIS log provider for sql server
Click on add
Provider configuration for database connection
Default log name as
Sysssislog ----------in 2008,2012
DTS90 ---------------in 2005
Once we executed the package log table will be created under systables
with the following names.
CHECK POINTS: Check points are used to restart the package at the point of
failure
If we will identify the failure point of the package and create one check
point file in the specified location by using this failure point if identify
and restart from that.
HOW TO CONFIGURE:
Constraint
Expression
Expression and constraint
Expression or constraint
In the above example cnt is the variable which we declared in the SSIS
level and 11 is the value which is stored in the variable.
If this value is not equal to 11 in the variable then below task will not
be executed.
XML configuration file: The required package properties will be stored in the
XML configuration which will help us to handle the package properties
dynamically through the configuration file.
NAVIGATION:
If we change the values inside of the table then those will be passed to the
package dynamically through the table.
HOW TO CONFIGURE:
PARENT PACKAGE VARIABLE: We want to pass a value from one package to the
other package which are available in the variable then we will use parent
package variable.
HOW TO CONFIGURE:
DEPLOYMENT: Deployment is used to move the package from one area to other
area to using the deployment.
We have two types of deployment in SSIS level.
FILE SYSTEM DEPLOYMENT: File system deployment is nothing but moving the
package one drive to another drive it is just copy paste option.
NAVIGATION:
Go to project location
Go to the bin area
Open the deployment folder
Double click on the deployment manifest file
Click on next
Then it will ask for the type of deployment
Select file system deployment by using radio button.
Click on next
Browse the location where you want to move package in the drives.
Click on next
Click on finish.
SQL SERVER DEPLOYMENT: SQL server deployment is moved in the packages from
drive to MSDB folder which is available under stored packages.
NAVIGATION:
Connect to the integration services then it will show stored packages and
running package folders
Expand stored packages
Right click on MSDB folder
And create one folder by using new folder options
Running packages
Stored packages
File system
MSDB
Data collector
Maintenance plans
My packages(user folders)
Package1
Package2
Go to job properties.
Go to schedule tab.
Click on new to create new schedule.
Provide schedule name as we required.
Select the frequency occurs either weekly or monthly.
Specify the time in the daily frequency at what we have to run.
EVENT HANDLERS: Event handlers will rise an event based on the provided event
type in the package level or task level.