Activating The Data Warehouse

CRITICAL SUCCESS FACTORS for Activating the Data Warehouse
Roger E. Mann
Critical Success Factors For Activating the Data Warehouse
What is Active Enterprise Intelligence?
Active

Better, faster decisions that drive actions Responsive and agile
Enterprise

Consistency Single view of the business Scope Across appropriate business functions Reach New operational users, processes, and applications
Intelligence

Strategic Intelligence, aligned to drive Operational Intelligence
2
Insights into Action 2007 Teradata PARTNERS
Accelerating Decisions
Business event
Data captured Value
Intelligence delivered Action taken

Engaged with customer Truck at loading dock Customer has left the store Truck has left the dock Time
Opportunity
Insights into Action
Missed Opportunity
3
2007 Teradata PARTNERS
Activating the Warehouse

ACTIVATING MAKE it happen!
Query complexity grows Workload mixture grows Data volume grows Schema complexity grows Depth of history grows Number of users grows Expectations grow Business Value
PREDICTING WHAT WILL happen? ANALYZING WHY did it happen? REPORTING WHAT happened?
OPERATIONALIZING WHAT IS happening?
Event-Based Triggering Takes Hold
Continuous Update & Time-Sensitive Queries Become Important Batch Analytical Modeling Grows Ad Hoc Analytics Continuous Update/Short Queries Event-Based Triggering
Increase in Ad Hoc Analysis Primarily Batch & Some Ad Hoc Reports
Data and Process Sophistication

4
What are the Implementation Challenges at Each Stage?
REPORTING WHAT happened?
ANALYZING WHY did it happen?
PREDICTING WHAT WILL happen?
Business Requirement:
Standard batch reports Known Questions Limited Ad-hoc Increased Ad-hoc analysis Answers to complex questions that involve cross functional queries Deep analytics used to understand cause/ effect & correlation Formulation of business models based on analytics & observations Timely access to data Integration of strategic and operational business functions Event notification services Automation of Customized offers
IT Challenges:
Data Quality Model Integration Platform scalability Responding to changing data requirements Data Model Integration Data Transformation Analytical tools Ability to handle increased demands for more information Ad-hoc query performance Query performance & user concurrency Timely response to additional data request Tactical query tuning Mixed workload management Changes to the data integrated data model EAI tools Data loading SLAs Integration with CRM System availability & DR Usage of triggers & SP
5
Teradata Functionality at each stage
REPORTING WHAT happened?
ANALYZING WHY did it happen?
PREDICTING WHAT WILL happen?
Primarily Batch & Some Ad Hoc Reports
Increase in Ad Hoc Analysis
Analytical Modeling Grows
Continuous Update & Time-Sensitive Queries Become Important
Event-Based Triggering Takes Hold
Self-Management Model Integration Full Parallelism Fault Tolerant File System
Optimizer OLAP in DBMS 64 way joins Ref. Integrity
Triggers Data Mining Scalability Single source
T-Pump Mixed Workload TASM TDQM PPI, Join Indexes
Dual Active Reference Arch. External Triggers Hot Stand-by System Mgmt
Views and Indexes Applications
6
Active Enterprise Intelligence

Active Data Warehouse from Teradata
Active Events
Active Access
Active Enterprise Integration
Teradata Warehouse
Active Workload Management Active Availability
Active Load
7
Strategic Intelligence
Back Office Corporate Users

Tens to hundreds of users Knowledge workers in Strategic Planning, Marketing, Finance, Manufacturing, Quality Assurance, Supply Chain
Intelligence for Strategic Analysis
Sales Reporting, Forecasting, Inventory Analysis, Product Profitability Analysis, Financial Management, Customer Segmentation, Customer Profitability, Compliance, etc.
8
Operational Intelligence
Front Line users and customers
Thousands to tens of thousands of users and customers Gate Agents, Cashiers, Dock Workers, Bank Tellers, Sales, Customer Service Agents, Customers, and Suppliers Self-Service Systems: POS, ATMs, Web
Intelligence for Operational Execution
customer/product look up, individualized customer offers, transaction exceptions, supply chain visibility, event detection and notifications
9
Different People, Different Needs

Strategic Intelligence Users Population Response Time Needed Data Latency Access Operational Intelligence
Call centers, Sales agents, Gate Planners, analysts, managers, Agents, Suppliers, Logistics, Finance, Marketing Consumers 10s-100s Seconds to hours Daily updates
BI Workbench tools, Excel, Applications Business Critical Low tolerance for down time
10s-10,000s Web Speed: 1 5 seconds; Up to minutes Within minutes or hours

Portals, Dashboards, Applications, Alerts Mission Critical No tolerance for down time; + Disaster Recovery plan
Availability
10
Organizational and Cultural Considerations
A tale of two data centers
Predictable, regimented world of OLTP meets adhoc, ask any question world of data warehousing
Best In Class OLTP Practices

SW Development Methodologies Built in QA and Testing Processes Rigid Change Control Practices Strict Problem Resolution and Escalation Procedures
11

Understand the need for cultural change in the IT organization Balance between DSS and Operational users needs The need for clear expectations setting has never been greater SLAs exist whether explicit or not Education and training requirements are continuous
12
Before You Begin
Understand the Business Requirements
What are your business goals?

Clearly understand requirements Business must own it, IT must manage it Management support
What is the value to the business?

ROI should be stated and understood Must be calculated, measured and published
Understanding and Agreement on SLAs

Response times Load refresh cycles Load times Availability requirements

13
Active Data Warehouse Success Common Themes

Well understood DSS environment Mature management and control Infrastructure to manage workflow to the warehouse Core Teradata team with deep expertise Leveraged outside resources when appropriate
Teradata and Teradata knowledgeable consultants and services Start small and grow POC
Incremental approach

Ongoing education
14
Understanding Your DSS Environment

- Single Integrated Copy
Single View of the Business Single Version of the Truth

ORDER ITEM BACKORDERED QUANTITY CUSTOMER CUSTOMER NUMBER CUSTOMER NAME CUSTOMER CITY CUSTOMER POST CUSTOMER ST CUSTOMER ADDR CUSTOMER PHONE CUSTOMER FAX ITEM ITEM NUMBER QUANTITY DESCRIPTION
ORDER ORDER NUMBER ORDER DATE STATUS
ORDER ITEM SHIPPED QUANTITY SHIP DATE
Integrate Once, Use Many

Event-driven/ Closed Loop
Strategic Users
Operational Reporting Users OLAP Users
Data Miners
Data and Channel Consistency

15

Do you know where your data is? Is it consistent?

Across time? Across functions? Across platforms?
Is it secure and complete?

Access audit cradle to grave visibility
Are you prepared for everything to change?

16
Architect an enterprise solution not a point solution.

Enterprise Logical Data Model (ELDM) deployment. Promote data sharing across analytic applications (strategic and tactical). Create a roadmap that goes beyond a single Business Improvement Opportunity (BIO). Educate the business on the Network Effect in data content and the attractive TCO implications of an enterprise approach.
17
In the past, data warehousing was considered a back-office IT service.

Regional sales reports Inventory analysis Cash flow analysis Customer profiles & Trends etc.
But the Active Data Warehouse is an Operational System, just like the transactional systems. So
Q: How does one integrate the ADW with other front line business services? A: Use the Enterprise Reference Architecture to design, implement, & deploy ADW services that interoperate throughout the enterprise.
18
Teradatas Real-Time Enterprise Reference Architecture

Legacy Environment
C/S EDI Consumers
Enterprise Users (Browsers and/or Portal)

Suppliers Internal Partners
Legacy Environment
EDI C/S
WAN / VAN
Internet / Intranet
WAN / VAN
Transactional Services
Analytic & Decision Making Services
NW TX1 APPL DA-MW
MSG-MW
MSG-MW TX3 APPL DA-MW
MSG-MW TX4 APPL DA-MW ASP / JSP
MSG-MW Strategic APPL DA-MW
MSG-MW Tactical APPL DA-MW
MSG-MW BI APPL DA-MW
NW BI APPL DA-MW
TX2
APPL DA-MW
Service Brokers Enterprise Message Bus
QD MSG-MW Event Notification DA-MW MSG-MW Business Rules DA-MW MSG-MW Event Detection DA-MW
QD
EDW A
EDW B
RS
OLTP1
OLTP2
OLTP3
OLTP4
Business Process Automation
RDBMS Based Event Processing
Streaming Batch
Transactional Repositories
Data Acquisition & Integration
Analytic & Decision Making Repositories 2007 Teradata PARTNERS
19
Stages 1 thru 3: Traditional Data Warehouse

Legacy Environment
C/S EDI

Consumers Suppliers Internal Partners
Legacy Environment
EDI C/S
WAN / VAN
Internet / Intranet
WAN / VAN
Analytic and Decision Making Services
NW TX1 APPL DA-MW
NW BI APPL DA-MW
MSG-MW Event Notification DA-MW
MSG-MW Business Rules DA-MW
MSG-MW Event Detection DA-MW RS
EDW A
EDW B
OLTP1
OLTP2
OLTP3
OLTP4
Streaming Batch
Data Acquisition and Integration
Analytic and Decision Making Repositories
20
Stage 4: Operationalized Decision Making

Legacy Environment
C/S EDI

Legacy Environment
EDI C/S
WAN / VAN
Internet / Intranet
WAN / VAN
NW TX1 APPL DA-MW
MSG-MW
NW BI APPL DA-MW
TX2
APPL DA-MW
QD
EDW A
EDW B
RS
OLTP1
OLTP2
OLTP3
OLTP4
Streaming Batch
21
Stage 5 Automated Decision Making

Legacy Environment
C/S EDI

Legacy Environment
EDI C/S
WAN / VAN
Internet / Intranet
WAN / VAN
NW TX1 APPL DA-MW
MSG-MW
NW BI APPL DA-MW
TX2
APPL DA-MW
QD
EDW A
EDW B
RS
OLTP1
OLTP2
OLTP3
OLTP4
RDBMS Based Event Processing
Streaming Batch
22
Questions to Answer

So, what is your corporate objective for Enterprise Intelligence? What is your current state?

Data integration Performance management Process versus platform independence
What are your current major challenges?

Master Data Management Usage and measurement of value Executive support and governance
Do you have a roadmap for data or applications? What current part of your warehouse could you leverage in an active mode?
23
Understand the Differences

Strategic Intelligence
Expected Users Expected Response Data Latency Integration Query Tuning 10s, 100s or a few thousand Seconds to hours Monthly, Weekly and Daily updates ETL, BI Tools
Mostly all-AMP operations Index options for repetitive queries
Operational Intelligence
100s to 10,000+ Web Speed: Sub second to 5 seconds; Up to minutes Within minutes or hours EAI, SOA, ETL, transaction or web servers. Triggers, SP and UDFs
Single AMP, planned, tested, tuned PPI, Global, Sparse, JI, AJI Indexing Target tables,
Performance
Minimal priority settings
TASM, DQM, Priority Scheduler. Segregation of strategic, tactical, loading and reporting functions Mission Critical No tolerance for down time; Dual Active Systems
24
Availability
Business Critical Low tolerance for down time
Active Data Warehouse Success

Success in ADW depends upon achieving an acceptable balance between the best-in-breed characteristics of ODS and DSS Systems
3 Areas of Concentration for ADW Success

Systems Management Best Practices Development Best Practices Organization and Cultural Considerations
25
System Management Best Practices - Summary

Application Differences Active Data Warehouse requires best in class Performance Management Develop and use a solid baseline benchmark Proactively monitor and measure data growth Capacity Planning and Management Escalation Processes Disaster Recovery and Fault Resilience Clearly define and quantify SLAs
26
System Management Best Practices - Performance Management

Proactive Performance Management Understand workload and job mix

Understand current resource usage levels at system and granular level Collect data and information Identify heavy hitters Analyze, trend, and predict Canary Queries
Analyze, trend, and predict
Query and performance measurements

Alert System
What actions to take
Customer Comment:
Manage your workload and pay attention to performance when the system is running well, not just when you have an emergency; By then its too late to start
27
Workload Management:
TDQM & Priority Scheduler
Teradata Dynamic Query Manager (TDQM)

Users can pre-schedule queries TDQM can intervene to delay or postpone queries Sensitive to resource-usage levels inside Teradata
Teradata Priority Scheduler

BYNET Interconnects
SMP Node1 SMP Node2 SMP Node3 SMP Node4
Tactical Queries Analytical Queries Loading Reporting
PE
PE
AMP
PE
PE
AMP
PE
PE
AMP
PE
PE
AMP
AMP
AMP
AMP
AMP
AMP
AMP
AMP
AMP
AMP
AMP
AMP
AMP
Enforces sharing of resources for jobs inside of Teradata Each category of work has its own partition that can be weighted Within each partition users are given priority assignments Comprehensive Priority Scheme Dynamic Priority Changes via Query Milestones Expedited Reserved Query Slots for Extreme Cases
28
System Management Best Practices - Performance Assurance
Develop a solid baseline benchmark

Start with application SLA requirements Design the benchmark to reflect the characteristics of production job mix Design to scale to predictable size of system Assess impact and trade-offs of design choices Run and save EXPLAINS Use TSET to recreate exact EXPLAINS for production system Feed the results into capacity planning efforts Visual Explain Teradata System Emulation and Test Index Wizard Statistics Wizard TASM
29
Teradata Analyst Pack

Understanding Tactical Decision Support

Performance: Critical, measured in seconds Flexibility: Important, but not ad-hoc Availability: Critical, OLTP-like requirements Update Frequency: Constant, near real-time Query Type: Narrow and deep data retrieval Index Usage: Primary key frequently used Result Size: Up to tens of thousands of rows Data Source: Operational and DSS systems Referential Integrity: Handled in operational systems and ETL
30
System Management Best Practices - Data Growth
Proactively Monitor and Manage Data Growth

Record and trend data growth by database and table Understand thresholds and the red zone Use compression for reduced space and improved performance Review rules for collect statistics frequency
Critical for Primary Index and join columns
Periodically check for skewed data Use spool limits to govern queries
31
System Management Best Practices - Capacity Planning
Track and Predict Capacity Consumption
Size for Performance

Existing data capacity growth Existing workload growth New workloads & data impacts Peak period strategy User Data + Indexes and Summaries + Fallback + Working Space
Size for Data Capacity
Consider Impact to Configuration Options

Priority Scheduler Compression Multi-Temperature Data Other Teradata Features
Customer information:
The more detailed the system capacity/performance data, the more precise the config options can be evaluated
32
System Management Best Practices - Escalation Processes
You have to change your mindset on handling EDW issues and how they are escalated

DSS query problems often handled at lower priorities Tactical query problems typically create more user dissatisfaction The Active Data Warehouse needs to be handled similarly to an operational or ODS environment
Customer Example:
DSS support processes are integrated with enterprise operational standards
33
System Management Best Practices - DR/Fault Tolerance
Problem Detection
Proactive monitoring for system problems

Platform vs database Alert System with defined actions
Availability
Maximize single system availability attributes
Fallback, Clique size, capacity threshold
Review BAR relative to application SLA Disaster Recovery

Service or On-site systems
Dual Active Systems
34
Teradata Dual Active Solution
A Dual Active Solution is comprised of several components:

Multiple Teradata Systems

NCR Hardware Nodes Teradata Database Software
Data Synchronization

Dual Load Batch and near real-time Cross-Feed Synchronization Dual SQL Apply Replication V2R6
Teradata System A Users/ Applications
Systems Management

Monitoring Administration Operational Control

Data Synchronization
Monitoring Administration Operational Control
Workload Management

Teradata Query Teradata Query Director Director
Users/ Applications
Teradata Query Director Teradata Active Systems Management
Services

Professional Services Customer Services
Teradata System B
Users/ Applications
35
Development Best Practices - Summary
Understand and manage the applications impact to the entire data warehouse platform - data model, capacity, performance management settings, etc. Performance starts with Application Planning and Design Develop Obsessive Compulsive Disorder about your queries Employ a Third Normal Form (3NF) Data Model Optimize the data model for a balance of ODS and DSS Change control process rigor
36
Development Best Practices - Application Planning and Design
Customer Comment:
Pay Attention to Database Design and especially the queries. You cant just throw anything at the system and expect it to work forever. DSS experience drives practices which favor flexibility over predictability and tolerate resource inefficiency Tactical query requirements run counter to this Design and review process often needs improvement to focus on appropriate tactical query design
37
Development Best Practices - Application Planning and Design
Customer Example - Ping:

Entire enterprise transaction systems running on Teradata Zero latency data freshness 90% Operational Queries / 10% DSS Queries
Because its primary role [of Teradata as perceived by the industry] is as a data warehouse, you must engineer the application for the Teradata environment. It is not conventional thinking. When people build applications, they do not think about doing it all in one environment. We have proven that it works.
38
Development Best Practices - Query OCD
Develop Obsessive Compulsive Disorder over your Queries
Queries are the single most significant database performance factor Strive for Single or Few-AMP operations
When necessary utilize most appropriate index type Understand the trade-offs
Develop internal processes and/or organizations that are responsible for reviewing and optimizing queries Develop and optimize queries with the understanding of the entire data warehouse application environment Develop Coach or Mentor roles
39
Development Best Practices - Query OCD
Customer Situation:
Original OLTP Application was developed with little or no thought to SQL design (SQL designed to get the required information) For predictable query performance in an ad hoc environment use tactical queries techniques:

Single or few AMP operations Retrieved through a primary index or unique secondary index, Selecting a small number of rows Dont forget to control DDL operations
40
Development Best Practices - Data Model
Use a 3NF Model
3NF Model provides the most flexibility for extension and integration 3NF more accurately models the business without the bias of specific application needs Understand the impact of denormalizations to all applications and processes
Customer Observation:
[Using a 3NF Model] is an advantage to integrate and extend our data.
41
Development Best Practices - Data Model
Each data model must be examined to find the proper balance between DSS and ODS
ODS will favor

Small Data Blocks Flat, De-normalized structures Light use of indexes and aggregations Single AMP operations on most frequent access paths
DSS will usually favor the opposite
Must balance operational performance with ask any question, of any data, at any time approach
42
Active Data Loading (Data Freshness)
Goal: Provide data freshness service levels to meet business requirements
Not all applications or users need the same levels

Tactical users require very up-to-date information for operational usage Strategic users require use of historical data for deep analytics Some analytical users typically do not want data changing with analytics underway Other users need near real-time data (as real-time as it needs to be)
Provide point-in-time views to meet user needs

Success Factors

Drive loading SLA from the business requirements, not the technology Without duplication of data
43
Development Best Practices - Load Processes
Many Load tools and options exist

Daily batch with Fastload and Multiload Frequent or Rapid batch to staging table
Then insert/select to destination
TPump mini batch TPump continuous

Each have pros and cons

Trade offs change with database design changes Volume and frequency of load will influence choice Access control will influence choice
Re-examine previous decisions based upon evolving application needs
44

Building an Information Enabled Business doesnt happen overnight. Culture, process and procedures must change to fully exploit information throughout the organization - Carrie Ballinger / Stephen Brobst
Teradata
45
Both sides must get ready to adapt
OLTP people must learn to live in a world that is less structured and more unpredictable DW people must learn to be more process oriented and formalized in development and operational practices
46

Training and Education Utilize Everything

Teradata Education Network - bite size and right time Orange Books - an expanding resource Lunch and learn sessions - awareness and education Leverage User Groups - regional and Partners Class room - onsite or remote Mentoring - internal and external resources Inventory, Gap Analysis, & Action Plan
Plan and Track
47
Active Enterprise Intelligence

Active Data Warehouse from Teradata
Active Events
Active Access
Active Enterprise Integration
Teradata Warehouse
Active Workload Management Active Availability
Active Load
48
How To Get Started
Your Teradata Data Warehouse = the foundation
You are most of the way there
Good News!
Start (relatively small) & simple pick a pilot project

Conduct AEI Workshop Clearly define business value Keep project focused along the journey Map out the business enhancement, process, & users Find an executive sponsor Deploy pilot Ensure process & users actually use operational intelligence Communicate your success Roger.Mann@teradata.com
49

Activating The Data Warehouse

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Activating The Data Warehouse

Transféré par

Droits d'auteur :

Formats disponibles

CRITICAL SUCCESS FACTORS for Activating the Data Warehouse

What is Active Enterprise Intelligence?

Better, faster decisions that drive actions Responsive and agile

Strategic Intelligence, aligned to drive Operational Intelligence

Data captured Value

Intelligence delivered Action taken

Activating the Warehouse

OPERATIONALIZING WHAT IS happening?

Event-Based Triggering Takes Hold

Increase in Ad Hoc Analysis Primarily Batch & Some Ad Hoc Reports

Data and Process Sophistication

What are the Implementation Challenges at Each Stage?

REPORTING WHAT happened?

ANALYZING WHY did it happen?

PREDICTING WHAT WILL happen?

OPERATIONALIZING WHAT IS happening?

ACTIVATING MAKE it happen!

Insights into Action

Teradata Functionality at each stage

REPORTING WHAT happened?

ANALYZING WHY did it happen?

PREDICTING WHAT WILL happen?

OPERATIONALIZING WHAT IS happening?

ACTIVATING MAKE it happen!

Primarily Batch & Some Ad Hoc Reports

Increase in Ad Hoc Analysis

Analytical Modeling Grows

Continuous Update & Time-Sensitive Queries Become Important

Event-Based Triggering Takes Hold

Self-Management Model Integration Full Parallelism Fault Tolerant File System

Optimizer OLAP in DBMS 64 way joins Ref. Integrity

Triggers Data Mining Scalability Single source

T-Pump Mixed Workload TASM TDQM PPI, Join Indexes

Views and Indexes Applications

Active Enterprise Intelligence

Active Enterprise Integration

Active Workload Management Active Availability

Back Office Corporate Users

Intelligence for Strategic Analysis

Front Line users and customers

Intelligence for Operational Execution

Different People, Different Needs

10s-10,000s Web Speed: 1 5 seconds; Up to minutes Within minutes or hours

Organizational and Cultural Considerations

A tale of two data centers

Best In Class OLTP Practices

Organizational and Cultural Considerations

Before You Begin

Understand the Business Requirements

What are your business goals?

What is the value to the business?

Understanding and Agreement on SLAs

Response times Load refresh cycles Load times Availability requirements

Insights into Action

Active Data Warehouse Success Common Themes

Insights into Action

Understanding Your DSS Environment

Single View of the Business Single Version of the Truth

ORDER ORDER NUMBER ORDER DATE STATUS

ORDER ITEM SHIPPED QUANTITY SHIP DATE

Integrate Once, Use Many

Operational Reporting Users OLAP Users