Metadata: The Critical Foundation for Data Governance

May 17, 2013

Ravi Chivukula Director, Business Solutions

Copyright 2013 Allen Systems Group Inc. This document is unpublished and the foregoing notice is affixed to protect ASG in the event of inadvertent publication. All rights reserved. No part of this document may be reproduced in any form, including photocopying or transmission electronically to any computer, without prior written consent of ASG. The information contained in this document is confidential and proprietary to ASG and may not be used or disclosed except as expressly authorized in writing by ASG

www.asg.com

www.asg.com

Copyright2010 Allen Systems Group, Inc.

Agenda
What Metadata is

Challenges with Data Governance

Features of Metadata that help address Governance challenges

Types of Metadata and where it can be found

How to use Metadata to build an effective Governance program

1
www.asg.com
Copyright2010 Allen Systems Group, Inc.

What is Metadata?
Business Metadata Information Business Policies Rules Services Service Directory Service Users Interfaces

Business Policies

I Dont Know What Metadata Is

Information Usage Data Governance Information Data Quality Quality SLAs Entity Models Information Architecture Relational Master Object Tables Directory Data Elements Data Element Definitions

Access Control

But I Can Sure Tell When I See It !!

Data Formats Aliases & Synonyms

Critical Data Elements

Conceptual Domains

Value Domains

Reference Metadata Reference Tables Business Definitions Definitions

Mappings

Concepts

Business Terms

Semantics

2
Source: Master Data Management by David Loshin
www.asg.com
Copyright2010 Allen Systems Group, Inc.

Challenges with Data Governance

www.asg.com
Copyright2010 Allen Systems Group, Inc.

Highly Complex Data Architectures

Cloud

Visualization

Acct Mktg HR Sales CRM

Reports Dashboards

Analytics

E T L

Data Warehouse
Corporate

Operational Data store

Cross-Channel Analysis

Social Media

MDM

Multi-platform Usage

4
www.asg.com

Copyright2010 Allen Systems Group, Inc.

Metadata: Roadmap for Data Governance

Source physical entities Analyze physical entities Develop Logical and Conceptual entities Map physical to Logical and then to Conceptual entities

Source Conceptual mappings from LOB Develop Enterprise entities Consolidate and map to Enterprise entities

Identify Enterprise entities for Governance Identify Attributes of governable entities Identify governanable attributes

LOB Repository

Enterprise Repository

Enterprise Attributes

5
www.asg.com
Copyright2010 Allen Systems Group, Inc.

Devil maybe in the data, but sinful to ignore Rules!

Most Metadata is benign but Business rules are not! Business Rules manipulate your data when you are not looking!
Legal Corporate IT Finance Analysts

Rule Rule Rule

Rule Rule Rule

Rule Rule Rule

Rule Rule Rule

Rule Rule Rule

Business Rules represent the collective thinking of the enterprise

Governance reaches full potential when it manages Business Rules

Business rules is the metadata that glues your enterprise data
6
www.asg.com
Copyright2010 Allen Systems Group, Inc.

Analyze and Troubleshoot Issues

Metadata Stack
Business Glossary /Conceptual layer
Enterprise Logical layer LOB/Application Logical layer

Upstream/Downstream Data traceability Enables robust impact analysis at the Enterprise level

Physical layer

Database schema
7
www.asg.com
Copyright2010 Allen Systems Group, Inc.

Regulatory Compliance

Business Vocabulary

Root-cause Anaysis.

Data Transparency

Data Lineage

Enterprise Scale
Metadata

History / Versioning

8
www.asg.com
Copyright2010 Allen Systems Group, Inc.

Deep Dive Into Metadata

www.asg.com
Copyright2010 Allen Systems Group, Inc.

Versions and Configurations

Versioning allows storing different versions of an item, each displaying only those properties and details about the item that are relevant for a particular context.

A Configuration is a set of logically associated items (e.g., the items pertaining to a software application at different development stages) and thus define particular views of the items of a database.

10
www.asg.com
Copyright2010 Allen Systems Group, Inc.

Simple Lineage
Traditional Lineage identifies that an item on a report is derived from information that originated in one or more Operational Systems Data Stores

Operational Systems Data Stores

ETL

Data Warehouse

Query

Analytics

Real Data Lineage is more complex, with multiple transformations and governance demands to know the transformation details
11
2013 Allen Systems Group, Inc.

www.asg.com

Extended Data Lineage

A good lineage solution will allow this flow to be diagrammed, with user recognizable names in the boxes

Operational Transform Staging Transform Data System Database Warehouse Operational System Operational System

Data Mart

Data Mart

Data Mart

Report

Report

Report
12
2013 Allen Systems Group, Inc.

www.asg.com

Deeper Data Lineage

Data Lineage can be very complicated, and governance requires the ability to dive deep into lineage

Business Process Implemented By Process / Data Flow Application Code

Process / Data Flow

The flow may not just be data it could be messages in a queuing system or even business processes

Implements Transform

The flow may be across many systems and from the highest level of abstraction to the most granular detail

13
2013 Allen Systems Group, Inc.

www.asg.com

Metadata Sources for Lineage

Business Terms Business Process Models

Operational Stores

ETL Technology

Data Warehouse / Data marts

Reporting Technologies

Data Models

Applications

Traditional Deep Lineage Lineage

14
2013 Allen Systems Group, Inc.

www.asg.com

Predominant Metadata Sources

Metadata Repository ASG-Rochade / Business Glossary

DBMS's
DB2/UDB Greenplum MS SQL Server Oracle Sybase Teradata

Transformation Modeling Tools Language

ARIS Casewise Erwin PowerDesigner Rational Tools System Architect C/C++ COBOL JAVA SQL / Stored Procedures

ETL Tools Data Stage Informatica SSIS

BI Tools Business Objects Cognos SSAS/SSRS

15
2013 Allen Systems Group, Inc.

www.asg.com

The Components Of A Comprehensive Lineage Approach

Data Model

Process Model

Display Model

Lineage

Solution

16
2013 Allen Systems Group, Inc.

www.asg.com

A View Of A Core Data Model

Contains Concepts for broad and deep lineage

Is technology agnostic

Connects naturally to other Repository Information Model constructs

17
2013 Allen Systems Group, Inc.

www.asg.com

Basic Process Flow

Build Knowledge Base

Discover Capture Rationalize

Analyze Lineage (Where) Impact (How)

Metadata Capture

18
2013 Allen Systems Group, Inc.

www.asg.com

Display Model
Visualization can be selected to Display only selected attributes

19
2013 Allen Systems Group, Inc.

www.asg.com

A Look at Business Metadata

Entity

Process (App/ETL)

Table
Attribute

Database
Column

Data Model

Relationship

Business Term
Concepts Definitions

Attributes

Contexts

Conceptual Model

Relationship

Metrics/Standards

20
www.asg.com
Copyright2010 Allen Systems Group, Inc.

How Does a Glossary help Governance

BUSINESS TERMS
Identifier Customer Authentication

Unique number assigned by the company

Password created by the Customer/user

Identifier Social Security Number Authentication

Unique number assigned by the Govt.

Password created by the Customer/user

Harmonization Through Collaboration The collaborative platform exposes variances that impact semantic consistency Governance Through Collaboration A collaborative platform allows interested individuals to track those business terms, data elements, concepts, and reference metadata that is relevant to your business 21
www.asg.com
Copyright2010 Allen Systems Group, Inc.

Business Terms Help Lineage

Customer

Identifier

Unique number assigned by the company

Data Element Type First Name VARCHAR Last Name VARCHAR CustID VARCHAR Start Date DATE

Multiple definitions for same term can be captured and Shared through a central repository. Discrepancies can be resolved through collaboration

Data Element First Name Middle Last Name Customer_Id Telephone

Type VARCHAR CHAR VARCHAR VARCHAR VARCHAR

22
www.asg.com
Copyright2010 Allen Systems Group, Inc.

An Integrated Wide Ranging Management Environment

Business Terminology / Concepts ASG-metaGlossary

Higher level / Architectural metadata

ASG-Rochade

Detailed/ Application metadata

ASG-becubic

140 + metadata sources (databases, languages, models, DW/BI tools)

Operational and Decision Support (BI/DW) environments

Mainframe and distributed environments
23
2013 Allen Systems Group, Inc.

www.asg.com

Questions

Thank You For Attending!!

Ravi Chivukula Director, Business Solutions Email: ravic@asg.com Tel: 817-652-6323

Carl Roberts Senior Account Executive Email: carl.roberts@asg.com Tel: 303-249-3607

24
www.asg.com
Copyright2010 Allen Systems Group, Inc.