How to automate your

SSIS and SQL Agent Job

Deployment
Yong Ji
About me
A guy who likes DB and ping pong
Blog : http://speaksql.wordpress.com/
SSIS
Requirement
Should support 2012 +
Should deploy to a destination where you
can easily backup and maintain
Should deploy right configuration/variable
value when deploying SSIS package
Configuration/variable value should be
managed in a single location
Should support delta deployment
Deployment Model
Package deployment
The project was simply a Visual Studio concept
The unit of deployment is a package
Project deployment
A project is now a real SSIS object, and it contains all
the SSIS packages defined in the project, project
parameters and the project connection managers.
The unit of deployment is a project
Project Deployment
Parameters
Project/Package
Resides in SSDT project
Environments
variables
Resides in SSISDB
SSISDB Catalog
SSIS DB
SPs and Views

Link to SSISDB ERD

Deployment Destinations
File system MsDtsS

Save the package on a physical location on hard drive or any shared folder <?xml v
<DtsSe
SQL Server xmlns:x
MSDB <StopE
SSIS Package Store down>
<TopL
File system <Fol
Default Root Folder : %ProgramFiles%\Microsoft SQL Server\110\DTS\Packages <Na
<Se
RootFolder is controlled by MsDtsSrvr.ini.xml(..\110\DTS\Binn)
</Fo
SSIS Catalog <Fol
<Na
SSISDB <St
Default Folder : [YourFolderNmae]\Projects </Fo
</Top
</DtsSe
Multi-environment Configuration
Configuration file
Single configuration file on the network share
Configuration table
Config tables + System environment variable/registry key/SQL Server
Alias on every server and developer computer
Single configuration table on a centralized server
SSIS multi-environment configuration in a single SQL Server
table by Scott Coleman
SSISDB Environment
SSISDB per environment
Can not reference remote servers SSISDB environment
One ETL server for all environment
My Choices
Deployment model - Project
Deploy destination - SSIS Catalog
Configuration management
Store in DB table(CMDB)
SSIS configuration will be
deployed to SSIS catalogs
environment variable
CMDB will be a common
configuration repository
 Deployment Process
-Source Control Archive Deployment
Deployment Test
Test
Build Deploy Test
Artifact

SCM tool Build tool Artifact Deployment tool Testing tool

- TFS - MSBuild Repository - OctopusDe - NBi
- Git - VS - Artifactory ploy - NUnit
- SVN executable - Nexus - Scripting - etc
- etc - etc - etc - MSBuild

CI tool
- Jenkins
- TeamCity
- TFS Release Manager
- etc
 Build
-Source Control Archive Deployment
Deployment Test
Test
Build Deploy Test
Artifact

MSBuild doesnt support SSIS project file (..dtproj extension) by default

Visual Studio executable(devenv.exe)
Make MSBuild working
Need dll that knows what to do - Microsoft.SqlServer.IntegrationServices.Build.dll
Microsoft SQL Server Community Samples: Integration Services) by MattCarroll
Building SSIS 2012 using MSBuild from https://speaksql.wordpress.com
MSBuildSsis2012 by RichardPhoenix
Call the task defined in dll
Building SSIS 2012 using MSBuild from https://speaksql.wordpress.com
Build output - ispac file
 Deploy
-Source Control Archive Deployment
Deployment Test
Test
Build Deploy Test
Artifact

SSIS Catalog
ISDeploymentWizard.exe
catalog.deploy_project in SSISDB
MOM(Management Object Model) - Powershell, .net
SSIS Deployments with SQL Server 2012 by Koen Verbeeck
MSBuild with Microsoft.SqlServer.IntegrationServices.Build.dll
MSDB
dtutil.exe
Scripting SSIS Package Deployments by CHAD MILLER
File system
copy
 Test
-Source Control Archive Deployment
Deployment Test
Test
Build Deploy Test
Artifact

Checklist for various test type

Data Warehouse Testing by Wayne Yaddow
Testing
NBI
ssisUnit
BI.Quality
PASS 2014 presentations
Unit and Integration Testing for SSIS Packages [BIA-313] by John Welch
A NEW APPROACH TO UNIT TESTING AND INTEGRATION TESTING IN SSIS [BIA-312]
AGILE BI: UNIT TESTING AND CONTINUOS INTEGRATION [BIA-309] by Davide Mauri
Deployment flow
Implementation example 1

SSIS 2012 Continuous Integration using Jenkins and Octopus from https://speaksql.wordpress.com
Implementation example 2
Demo Environment
VS2012 - SSDT,SSDT-BI Jenkins plugin
TFS 2013 Role-based Authorization Strategy
CloudBees Folders Plugin
Jenkins Environment Injector Plugin
build-name-setter
Team Foundation Server Plug-in
MSBuild Plugin
Hudson PowerShell plugin
Scriptler
Jenkins Dynamic Parameter Plug-in
Dynamic Parameter Plug-in
Extensible Choice Parameter plugin
Demo

SSIS deployment
References
Deployment model
http://www.sqlservercentral.com/articles/Integration+Services+(SSIS)/101240/
http://msdn.microsoft.com/en-us/library/hh213290.aspx
http://sqlmag.com/sql-server-integration-services/ssis-deployment-sql-server-2012
http://nexxtjump.com/2013/01/14/ssis-2012-configuration-guide-part-1-introduction/
Deployment destination
http://stackoverflow.com/questions/11240761/understanding-stored-packages-on-integration-services
http://help.pragmaticworks.com/dtsxchange/scr/FAQ%20-%20What%20are%20the%20advantages_disadvantages%20of%
20storing%20SSIS%20packages%20to%20MSDB%20vs%20File%20System.htm
https://msdn.microsoft.com/en-us/library/ms137789(v=sql.110).aspx
http://www.mssqltips.com/sqlservertip/2061/how-to-manage-ssis-packages-stored-in-multiple-sql-server-database-instances/
Deployment process
http://www.infoq.com/articles/orch-pipelines-jenkins#_Toc2
http://msdn.microsoft.com/en-us/library/ff953020(v=sql.105).aspx
https://speaksql.wordpress.com/2013/06/07/a-journey-to-db-deployment-automaton-ssis-build-using-msbuild/
http://www.sqlservercentral.com/articles/Integration+Services+(SSIS)/101240/
http://sev17.com/2012/11/06/scripting-ssis-package-deployments/
http://ibmdatamag.com/2013/07/data-warehouse-testing-part-1/
http://ibmdatamag.com/2013/07/data-warehouse-testing-part-2/
https://nbi.codeplex.com/
http://ssisunit.codeplex.com/
http://biquality.codeplex.com/
Delta deployment
Change list
Diff
If you deploy to file system, you can diff using tools
like rsync, robocopy, etc.
Changeset list from SCM
Get changeset number of last successful deployment
Jenkins - you can get this using Groovy script
Generate change list after that changeset number
TFS - Team Foundation Server Power Tools
Microsoft Team Foundation PowerShell Snap-In
Demo

Delta deployment
Reference
Changeset number
Jenkins
http://stackoverflow.com/questions/21236268/access-to-build-environment-
variables-from-a-groovy-script-in-a-jenkins-build-st
TeamCity
http://stackoverflow.com/questions/20332397/teamcity-how-to-get-names-of-the-
files-edited
Change list
TFS
http://programmaticponderings.wordpress.com/2012/08/10/using-powershell-to-
generate-a-tfs-change-manifest-for-artifact-delivery/
Job
Requirements
Deploying jobs using CI tool(Jenkins)
Job script should be managed by source
control manager(TFS)
Being able to use variable in the job script so
that value changes as environment changes.
Job execution history should be kept after
the deployment
Challenges
No automated tool
SQL Server Database Project(SSDT) doesnt
support SQL Agent Job
Redgate SQL Source control, Apex SQL Source
control and other third party tools dont support
either
This means
You need to script out and check in to TFS
manually
Theres no automated way to generate a change
script(ex; publish.sql)
Some approaches
Post deployment script of SSDT database project
SSDT SQL Agent Job Deployment Template by Kent Chenery
Script out final version and check into source control
Database Deployment: The Bits - Agent Jobs and Other Server Objects
by Phil Factor
Some people tried to generate change script using tables
in MSDB but didnt work well
 Yet another approach
Create Job Script out using Replace actual value Check in Deploy using
on Local DB Powershell with a SQLCMD variable script to TFS Jenkins

Check SQLCMD variable exists

PowerShell will modify
in SSIS CM
the following things You need to add manually to
Declare SQLCMD SSIS CM
variable
Check existence using
name instead of id
Pick up enable/disable
status
Get right SSIS package
object id
Call custom SP to
update job instead of
drop and create
Limitations
Doesnt support Multi-server jobs
http://msdn.microsoft.com/en-us/library/ms191305.aspx
http://sqlmag.com/database-administration/multiserver-task-administration

Doesnt support Shared schedule

http://www.mssqltips.com/sqlservertip/2223/how-to-setup-and-manage-sql-server-
agent-shared-job-schedules/

SSIS should be deployed SSIS catalog

Operator, Proxy and Alert are not included
in the deployment
FAQ
1. What's going to happen if you deploy when a job is
running?
a. The running job will be terminated immediately.
2. Will the existing execution history be cleared if you
deploy?
a. No, history will be kept.
3. Will the job be enabled?
a. If job exists, it will keep the current status. If that job was disabled, it will stay disabled.
b. If it is new job, it is controlled by SQLAgentJobDefaultStatus SSIS CM variable
i. Production - enabled
ii. Other environment - disabled
Implementation example 1
Demo

Job deployment
Reference
http://blog.sqlconcepts.co.nz/2012/09/ssdt-sql-agent-job-deployment-template.html
https://www.simple-talk.com/sql/t-sql-programming/database-deployment-the-bits---agent-
jobs-and-other-server-objects/
http://blog.sqlconcepts.co.nz/2012/09/ssdt-sql-agent-job-deployment-template.html
http://msdn.microsoft.com/en-us/library/ms191305.aspx
http://sqlmag.com/database-administration/multiserver-task-administration
http://www.mssqltips.com/sqlservertip/2223/how-to-setup-and-manage-sql-server-agent-
shared-job-schedules/
Q&A
Blog : http://speaksql.wordpress.com/
Email : kulmam92@gmail.com