2014 Fall2014 Fall 11

Chapter 11:Chapter 11: Data Warehousing Data Warehousing

楊立偉教授台灣大學工管系

註 註 :: 於於 1111 版為版為 Chapter 10Chapter 10

Chapter 11 22

DefinitionDefinition Data WarehouseData Warehouse: :

A subject-oriented, integrated, time-variant, non-A subject-oriented, integrated, time-variant, non-updatable collection of data used in support of updatable collection of data used in support of management decision-making processesmanagement decision-making processes

Subject-oriented:Subject-oriented: e.g. customers, patients, e.g. customers, patients, students, productsstudents, products

Integrated: Integrated: Consistent naming conventions, formats, Consistent naming conventions, formats, encoding structures; from multiple data sourcesencoding structures; from multiple data sources

Time-variant: Time-variant: Can study trends and changesCan study trends and changes Non-updatable: Non-updatable: Read-only, periodically refreshedRead-only, periodically refreshed

Data MartData Mart:: A data warehouse that is limited in scope A data warehouse that is limited in scope Ex. Corporate (Data Warehouse) v.s. Department (Data Mart)Ex. Corporate (Data Warehouse) v.s. Department (Data Mart)

Chapter 11 33

History Leading to Data History Leading to Data WarehousingWarehousing

Improvement in database technologies, Improvement in database technologies, especially relational DBMSs especially relational DBMSs 資料庫技術的進步資料庫技術的進步

Advances in computer hardware, including Advances in computer hardware, including mass storage and parallel architecturesmass storage and parallel architectures配合大量儲存與平行運算架構的進步配合大量儲存與平行運算架構的進步

Emergence of end-user computing with Emergence of end-user computing with powerful interfaces and tools powerful interfaces and tools 使用者自行操作使用者自行操作

Advances in middleware, enabling Advances in middleware, enabling heterogeneous database connectivity heterogeneous database connectivity 相互整合相互整合

Recognition of difference between operational Recognition of difference between operational and informational systems and informational systems

體認到日常作業與資訊分析是不同的體認到日常作業與資訊分析是不同的

Chapter 11 44

Need for Data WarehousingNeed for Data Warehousing Integrated, company-wide view of high-quality Integrated, company-wide view of high-quality

information (from disparate databases) information (from disparate databases) 整合各種資訊整合各種資訊 Separation of Separation of operationaloperational and and informationalinformational systems and systems and

data (for improved performance) data (for improved performance) 獨立於日常作業外獨立於日常作業外

Table 11-1 – Comparison of Operational and Informational Systems

Chapter 11 55

Issues with Company-Wide Issues with Company-Wide ViewView

Inconsistent key structures Inconsistent key structures 不一致的不一致的PKPK

Synonyms Synonyms 同義字問題同義字問題 Free-form vs. structured fieldsFree-form vs. structured fields Inconsistent data values Inconsistent data values 資料值不一致資料值不一致 Missing data Missing data 缺值問題缺值問題

See figure 11-1 for exampleSee figure 11-1 for example

Chapter 11 66

Figure 11-1 Examples of heterogeneous data

Chapter 11 77

Organizational Trends Organizational Trends Motivating Data Motivating Data

WarehousesWarehouses No single system of records No single system of records Multiple systems not synchronizedMultiple systems not synchronized Organizational need to analyze activities in a Organizational need to analyze activities in a

balanced waybalanced way

從組織角度從組織角度 , , 看資料需要整體來看看資料需要整體來看 for Customer relationship management for Customer relationship management

(CRM) and Supplier relationship (CRM) and Supplier relationship

management (SRM)management (SRM)

Chapter 11 88

Data Warehouse Data Warehouse ArchitecturesArchitectures

Independent Data MartIndependent Data Mart Dependent Data Mart and Dependent Data Mart and

Operational Data StoreOperational Data Store Logical Data Mart and Real-Logical Data Mart and Real-

Time Data WarehouseTime Data Warehouse Three-Layer architectureThree-Layer architecture

All involve some form of extraction, transformation and loading (ETLETL)

Chapter 11 99

Figure 11-2 Independent data mart data warehousing architecture

Data marts:Data marts:Mini-warehouses, limited in scope

E

T

L

Separate ETL for each independent data mart

Data access complexity due to multiple data marts

Chapter 11 1010

Source: adapted from Strange (1997).

Table 11-2 – Data Warehouse Versus Data Mart

Chapter 11 1111

Data CharacteristicsData CharacteristicsStatus vs. Event DataStatus vs. Event Data

Status

Status

Event = a database action (create/update/delete) that results from a transaction

Figure 11-6 Example of DBMS

log entry

Chapter 11 1212

Other Data Warehouse Other Data Warehouse ChangesChanges

加新欄位加新欄位 New descriptive attributesNew descriptive attributes New business activity attributesNew business activity attributes New classes of descriptive attributesNew classes of descriptive attributes Descriptive attributes become more Descriptive attributes become more

refinedrefined 加描述資料加描述資料 Descriptive data are related Descriptive data are related

to one anotherto one another 加新資料源加新資料源 New source of dataNew source of data

Chapter 11 1313

Derived Data Derived Data 加入衍生性資加入衍生性資料料

ObjectivesObjectives Ease of use for decision support applicationsEase of use for decision support applications Fast response to predefined user queriesFast response to predefined user queries Customized data for particular target audiencesCustomized data for particular target audiences Ad-hoc query supportAd-hoc query support Data mining capabilitiesData mining capabilities

CharacteristicsCharacteristics Detailed (mostly periodic) dataDetailed (mostly periodic) data Aggregate (for summary)Aggregate (for summary) Distributed (to departmental servers)Distributed (to departmental servers)

Most common data model = star schemastar schema(also called “dimensional model”)

與一般資料庫設計準則略有不同

Chapter 11 1414

Figure 11-9 Components of a star schemastar schemaFact tables contain factual or quantitative data

Dimension tables contain descriptions about the subjects of the business

1:N relationship between dimension tables and fact tables

Excellent for ad-hoc queries, but bad for online transaction processing

Dimension tables are denormalized to maximize performance

→日常作業（有寫入或修改）還是要用正規化後的表格

Chapter 11 1515

Figure 11-10 Star schema example

Fact table provides statistics for sales broken down by product, period and store dimensions

Chapter 11 1616

Figure 11-11 Star schema with sample data

Chapter 11 1717

Issues Regarding Star Schema Issues Regarding Star Schema (1)(1)

Dimension table keys must be Dimension table keys must be surrogatesurrogate (non-(non-intelligent and non-business related),intelligent and non-business related), because: because: Keys may change over timeKeys may change over time Length/format consistencyLength/format consistency

資料的精細度資料的精細度 Granularity of Fact Table–what Granularity of Fact Table–what level of detail do you want? level of detail do you want? Transactional grain–finest levelTransactional grain–finest level Aggregated grain–more summarizedAggregated grain–more summarized Finer grains Finer grains better better market basket analysismarket basket analysis capability capability Finer grain Finer grain more dimension tables, more rows in fact table more dimension tables, more rows in fact table

Chapter 11 1818

Issues Regarding Star Schema Issues Regarding Star Schema (2)(2)

資料期間 資料期間 Duration of the database–how much Duration of the database–how much history should be kept?history should be kept? Natural duration–13 months or 5 quartersNatural duration–13 months or 5 quarters Financial institutions may need longer durationFinancial institutions may need longer duration Older data is more difficult to source and cleanseOlder data is more difficult to source and cleanse

Chapter 11

Size of Fact TableSize of Fact Table Depends on the number of dimensions and the grain of the fact tableDepends on the number of dimensions and the grain of the fact table Number of rows = product of number of possible values for each dimension Number of rows = product of number of possible values for each dimension

associated with the fact tableassociated with the fact table Example: Assume the following for the next Figure:Example: Assume the following for the next Figure:

Total rows calculated as follows (assuming only half the products record Total rows calculated as follows (assuming only half the products record

sales for a given month):sales for a given month):

1919

Chapter 112020

(new) Figure 9-12 Modeling dates

Fact tables contain time-period data Date dimensions are important

Chapter 11

Slowly Changing Dimensions Slowly Changing Dimensions (SCD)(SCD)

How to maintain knowledge of the past How to maintain knowledge of the past Kimble’s approaches:Kimble’s approaches:

Type 1: just replace old data with new (lose Type 1: just replace old data with new (lose historical data)historical data)

Type 2: for each changing attribute, create a Type 2: for each changing attribute, create a current value field and several old-valued fields current value field and several old-valued fields (multivalued)(multivalued)

Type 3: create a new dimension table row each Type 3: create a new dimension table row each time the dimension object changes, with all time the dimension object changes, with all dimension characteristics at the time of change. dimension characteristics at the time of change. Most common approachMost common approach

2121

Chapter 112222

(new) Figure 9-18 Example of Type 2 SCD Customer dimension table

The dimension table contains several records for the same customer. The specific customer record to use depends on the key and the date of the fact, which should be between start and end dates of the SCD customer record.

Chapter 112323

(new) Figure 9-19 Dimension segmentation

For rapidly changing attributes (hot attributes), Type 2 SCD approach creates too many rows and too much redundant data. Use segmentation instead.

Chapter 11

10 Essential Rules for Dimensional 10 Essential Rules for Dimensional ModelingModeling

Use atomic factsUse atomic facts Create single-process Create single-process

fact tablesfact tables Include a date Include a date

dimension for each dimension for each fact tablefact table

Enforce consistent Enforce consistent graingrain

Disallow null keys in Disallow null keys in fact tablesfact tables

Honor hierarchiesHonor hierarchies Decode dimension Decode dimension

tablestables Use surrogate keysUse surrogate keys Conform dimensionsConform dimensions Balance requirements Balance requirements

with actual datawith actual data

2424

Chapter 11

Columnar databasesColumnar databases for Big Data (huge volume, often unstructured)for Big Data (huge volume, often unstructured) Columnar databases optimize storage for Columnar databases optimize storage for

summary data of few columns (different need summary data of few columns (different need than OLTP)than OLTP)

Data compression Data compression NoSQLNoSQL

““Not only SQL”Not only SQL” Deals with unstructured dataDeals with unstructured data Hadoop HBase, Cassandra, MongoDBHadoop HBase, Cassandra, MongoDB

2525

Other Data Warehouse Other Data Warehouse AdvancesAdvances

Chapter 11 2626

Online Analytical Processing (OLAP) Online Analytical Processing (OLAP) ToolsTools

The use of a set of graphical tools that provides The use of a set of graphical tools that provides users with multidimensional views of their data users with multidimensional views of their data and allows them to analyze the data using and allows them to analyze the data using simple windowing techniquessimple windowing techniques

Relational OLAP (ROLAP)Relational OLAP (ROLAP) Traditional relational representationTraditional relational representation

Multidimensional OLAP (MOLAP)Multidimensional OLAP (MOLAP) Cube structureCube structure

OLAP OperationsOLAP Operations Cube slicingCube slicing–come up with 2-D view of data–come up with 2-D view of data Drill-downDrill-down–going from summary to more detailed –going from summary to more detailed

viewsviews

Chapter 11 2727

Figure 11-21 Slicing a data cube

Chapter 11 2828

Figure 11-22 Example of drill-down

Summary report

Drill-down with color added

Starting with summary data, users can obtain details for particular cells

Chapter 11 2929

Data Mining and Data Mining and VisualizationVisualization

Knowledge discovery using a blend of statistical, AI, and Knowledge discovery using a blend of statistical, AI, and computer graphics techniquescomputer graphics techniques

Goals:Goals: Explain observed events or conditionsExplain observed events or conditions Confirm hypothesesConfirm hypotheses Explore data for new or unexpected relationshipsExplore data for new or unexpected relationships

TechniquesTechniques Statistical regressionStatistical regression Decision treeDecision tree ClusteringClustering Association ruleAssociation rule Sequence associationSequence association … … and so on.and so on.

Data visualization–representing data in graphical/multimedia Data visualization–representing data in graphical/multimedia formats for analysisformats for analysis