Extending the Data Collector to Monitor

SQL Server effortlessly

Gianluca Sartori
@spaghettidba
sqlconsulting.it

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Sponsors

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Organizers

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Gianluca Sartori
 Independent SQL Server consultant

 SQL Server MVP, MCTS, MCITP, MCT

 Works with SQL Server since version 7

 DBA @ Scuderia Ferrari

 Blog: spaghettidba.com
 Twitter: @spaghettidba

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Agenda
 Why Data Collection?
 Data Collector Overview
 Data Collector Architecture
 What it does out of the box
 Extending the Data Collector
 Introducing Extended T-SQL Collector
 Reading collected data

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Why Data Collection?
 Goals:
 Troubleshooting
What happened tonight at 03:40?
 Tune performance
Which queries consume most resources?
 Capacity Planning
How fast are my databases growing?
 Alerting
Hey, look: something’s wrong here!
 Monitoring
Is the system behaving normally?

#sqlsatPordenone
February 28, 2015 #sqlsat367
 How should I monitor?
 Buy a commercial suite
 SQLSentry, RedGate, SolarWinds…
 $$$
 Use open source/free tools
 Often unreliable/incomplete
 Code your own
 Are you sure?
 Use SQL Server built-in tools
 SQL Server Data Collector

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Challenges
Want to code your own?

 What to collect?
 How to collect?
 How often?
 Where to persist?
 Which retention policy?
 How to analyze?

#sqlsatPordenone
February 28, 2015 #sqlsat367
 What to collect?

 Performance indicators
 Dynamic Management Objects
 DMV
 DMF

 Performance Counters
 Query Stats
 Wait Stats

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Dynamic Management Objects

 sys.dm_os_ OS information
 sys.dm_io_ Disk performance
 sys.dm_exec_ Sessions, requests, connections
 sys.dm_db_ Database info
 sys.dm_tran_ Transactions, locks
 sys.dm_... Audits, broker, CDC, FileStream,
FTS, HADR, Repl, RG, XE…

 Which DMOs?
 SQL Server 2014 contains 200 DMVs and DMFs!

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Performance Counters

 CPU
 Memory
 Disk
 SQL - specific
 …

 Which counters?
 SQL Server 2014 exposes more than 400 counters!

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Data Collector: challenge accepted!

 What to collect  System Collection Sets

 How to collect  Data Collector Infrastructure
 How often  Default Schedules
 Where to persist  Management Datawarehouse
 Which retention policy  Cleanup jobs
 How to analyze  Canned reports

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Data Collector Overview
Management Studio Components

Data Collector Collection Set

Configuration Reports

Background Components

Data Storage

Instance «A» Data Collector

DataCollector

Background Components Management Data

Warehouse

Instance «B» Data Collector

DataCollector

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Concepts
 Data Provider Data Provider

 Source of information
 Collector Type Performance
Counter
T-SQL Query Trace

 Reads data from data providers

 Collection Item
Collector Type

 Instance of a collector type

SSIS

 Collection Set
 Group of collection items
Collection Item
Collection Item
Collection Set Collection Item

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Architecture

SQL Server Agent

msdb

 configuration
 schedules

Data Collector Runtime

(dcexec.exe)

Collect Cache Upload

MDW

Target Instance

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Architecture
Collection Job
(SSIS runtime)

Collected Data

Data Collector Cache

Upload Job
(SSIS runtime)

MDW database

#sqlsatPordenone
February 28, 2015 #sqlsat367
 What it does out of the box
 System Collection Sets
 Server Activity
 DMV snapshot
 Sessions
 Wait Stats
 Latches
 …
 Performance counters
 Query Statistics
 Query Activity
 Disk Usage
 Data files
 Log Files

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Built-in Reports
 Based on system collection sets
 Server Activity
 Performance dashboard
 Many drill down reports
 Query Statistics
 Most expensive queries
 Drill down
 Disk Usage
 Monitor growth trends

#sqlsatPordenone
February 28, 2015 #sqlsat367
 DEMO

Data Collector out-of-the-box

#sqlsatPordenone
February 28, 2015 #sqlsat367
 What it does NOT
 Limited blocking information
 No deadlock monitoring
 No alerting
 No GUI
 No reporting for all the metrics collected

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Extending the Data Collector
 Custom collection sets
 Based on generic T-SQL collector
 Some limitations apply
 No varchar(max)
 No XML
 … no LOBs at all!
 No GUI

#sqlsatPordenone
February 28, 2015 #sqlsat367
 DEMO

Creating a custom collection set

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Extending the Data Collector
 Custom collector types
 Complicated
 No documentation
 Requires SSIS ninja skills
 Some undocumented proprietary tasks

#sqlsatPordenone
February 28, 2015 #sqlsat367
 DEMO

System Collector Types

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Introducing Extended T-SQL Collector
 Provides a GUI for the Data Collector
 Incorporates 2 new collector types:
 Extended TSQL Collector Type
 Adds support for LOB columns
 Extended XE Reader Collector Type
 Reads data from Extended Events sessions
 Incorporates alerting
 Free and Open Source
http://extendedtsqlcollector.codeplex.com

#sqlsatPordenone
February 28, 2015 #sqlsat367
 DEMO

Working with Extended T-SQL Collector

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Extending the Data Collector
 Custom reports
 Integrate in SSMS
 Distribute to all DBAs?
 Deploy to Reporting Services
 Bill Ramos’ MDW reports
 Analyze with PowerBI tools
 PowerQuery
 PowerView

#sqlsatPordenone
February 28, 2015 #sqlsat367
 DEMO

Using Bill Ramos’ MDW reports

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Bottom line
 Data collection provides out-of-the-box tools
to collect and analyze performance data
 Free - Ships with SQL Server
 Some limitations apply
 Free tools to the rescue!
 Viable alternative to commercial monitoring
tools when money is a constraint

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Resources

 Extended T-SQL Collector

http://extendedtsqlcollector.codeplex.com

 Bill Ramos – MDW Reports

http://blogs.msdn.com/b/billramo/archive/tags/mdw+reports/

 Bart Duncan – Query Hash Statistics

http://code.msdn.microsoft.com/QueryHashStatistics

#sqlsatPordenone
February 28, 2015 #sqlsat367
 Q&A

Questions?

#sqlsatPordenone
February 28, 2015 #sqlsat367
 #sqlsatPordenone
#sqlsat367

Feedback form: http://speakerscore.com/D3D4

THANKS!

#sqlsatPordenone
February 28, 2015 #sqlsat367