View
220
Download
4
Category
Preview:
Citation preview
2014 Fall2014 Fall 11
Chapter 11:Chapter 11: Data Warehousing Data Warehousing
楊立偉教授台灣大學工管系
註 註 :: 於於 1111 版為版為 Chapter 10Chapter 10
Chapter 11 22
DefinitionDefinition Data WarehouseData Warehouse: :
A subject-oriented, integrated, time-variant, non-A subject-oriented, integrated, time-variant, non-updatable collection of data used in support of updatable collection of data used in support of management decision-making processesmanagement decision-making processes
Subject-oriented:Subject-oriented: e.g. customers, patients, e.g. customers, patients, students, productsstudents, products
Integrated: Integrated: Consistent naming conventions, formats, Consistent naming conventions, formats, encoding structures; from multiple data sourcesencoding structures; from multiple data sources
Time-variant: Time-variant: Can study trends and changesCan study trends and changes Non-updatable: Non-updatable: Read-only, periodically refreshedRead-only, periodically refreshed
Data MartData Mart:: A data warehouse that is limited in scope A data warehouse that is limited in scope Ex. Corporate (Data Warehouse) v.s. Department (Data Mart)Ex. Corporate (Data Warehouse) v.s. Department (Data Mart)
Chapter 11 33
History Leading to Data History Leading to Data WarehousingWarehousing
Improvement in database technologies, Improvement in database technologies, especially relational DBMSs especially relational DBMSs 資料庫技術的進步資料庫技術的進步
Advances in computer hardware, including Advances in computer hardware, including mass storage and parallel architecturesmass storage and parallel architectures配合大量儲存與平行運算架構的進步配合大量儲存與平行運算架構的進步
Emergence of end-user computing with Emergence of end-user computing with powerful interfaces and tools powerful interfaces and tools 使用者自行操作使用者自行操作
Advances in middleware, enabling Advances in middleware, enabling heterogeneous database connectivity heterogeneous database connectivity 相互整合相互整合
Recognition of difference between operational Recognition of difference between operational and informational systems and informational systems
體認到日常作業與資訊分析是不同的體認到日常作業與資訊分析是不同的
Chapter 11 44
Need for Data WarehousingNeed for Data Warehousing Integrated, company-wide view of high-quality Integrated, company-wide view of high-quality
information (from disparate databases) information (from disparate databases) 整合各種資訊整合各種資訊 Separation of Separation of operationaloperational and and informationalinformational systems and systems and
data (for improved performance) data (for improved performance) 獨立於日常作業外獨立於日常作業外
Table 11-1 – Comparison of Operational and Informational Systems
Chapter 11 55
Issues with Company-Wide Issues with Company-Wide ViewView
Inconsistent key structures Inconsistent key structures 不一致的不一致的PKPK
Synonyms Synonyms 同義字問題同義字問題 Free-form vs. structured fieldsFree-form vs. structured fields Inconsistent data values Inconsistent data values 資料值不一致資料值不一致 Missing data Missing data 缺值問題缺值問題
See figure 11-1 for exampleSee figure 11-1 for example
Chapter 11 66
Figure 11-1 Examples of heterogeneous data
Chapter 11 77
Organizational Trends Organizational Trends Motivating Data Motivating Data
WarehousesWarehouses No single system of records No single system of records Multiple systems not synchronizedMultiple systems not synchronized Organizational need to analyze activities in a Organizational need to analyze activities in a
balanced waybalanced way
從組織角度從組織角度 , , 看資料需要整體來看看資料需要整體來看 for Customer relationship management for Customer relationship management
(CRM) and Supplier relationship (CRM) and Supplier relationship
management (SRM)management (SRM)
Chapter 11 88
Data Warehouse Data Warehouse ArchitecturesArchitectures
Independent Data MartIndependent Data Mart Dependent Data Mart and Dependent Data Mart and
Operational Data StoreOperational Data Store Logical Data Mart and Real-Logical Data Mart and Real-
Time Data WarehouseTime Data Warehouse Three-Layer architectureThree-Layer architecture
All involve some form of extraction, transformation and loading (ETLETL)
Chapter 11 99
Figure 11-2 Independent data mart data warehousing architecture
Data marts:Data marts:Mini-warehouses, limited in scope
E
T
L
Separate ETL for each independent data mart
Data access complexity due to multiple data marts
Chapter 11 1010
Source: adapted from Strange (1997).
Table 11-2 – Data Warehouse Versus Data Mart
Chapter 11 1111
Data CharacteristicsData CharacteristicsStatus vs. Event DataStatus vs. Event Data
Status
Status
Event = a database action (create/update/delete) that results from a transaction
Figure 11-6 Example of DBMS
log entry
Chapter 11 1212
Other Data Warehouse Other Data Warehouse ChangesChanges
加新欄位加新欄位 New descriptive attributesNew descriptive attributes New business activity attributesNew business activity attributes New classes of descriptive attributesNew classes of descriptive attributes Descriptive attributes become more Descriptive attributes become more
refinedrefined 加描述資料加描述資料 Descriptive data are related Descriptive data are related
to one anotherto one another 加新資料源加新資料源 New source of dataNew source of data
Chapter 11 1313
Derived Data Derived Data 加入衍生性資加入衍生性資料料
ObjectivesObjectives Ease of use for decision support applicationsEase of use for decision support applications Fast response to predefined user queriesFast response to predefined user queries Customized data for particular target audiencesCustomized data for particular target audiences Ad-hoc query supportAd-hoc query support Data mining capabilitiesData mining capabilities
CharacteristicsCharacteristics Detailed (mostly periodic) dataDetailed (mostly periodic) data Aggregate (for summary)Aggregate (for summary) Distributed (to departmental servers)Distributed (to departmental servers)
Most common data model = star schemastar schema(also called “dimensional model”)
與一般資料庫設計準則略有不同
Chapter 11 1414
Figure 11-9 Components of a star schemastar schemaFact tables contain factual or quantitative data
Dimension tables contain descriptions about the subjects of the business
1:N relationship between dimension tables and fact tables
Excellent for ad-hoc queries, but bad for online transaction processing
Dimension tables are denormalized to maximize performance
→日常作業(有寫入或修改)還是要用正規化後的表格
Chapter 11 1515
Figure 11-10 Star schema example
Fact table provides statistics for sales broken down by product, period and store dimensions
Chapter 11 1616
Figure 11-11 Star schema with sample data
Chapter 11 1717
Issues Regarding Star Schema Issues Regarding Star Schema (1)(1)
Dimension table keys must be Dimension table keys must be surrogatesurrogate (non-(non-intelligent and non-business related),intelligent and non-business related), because: because: Keys may change over timeKeys may change over time Length/format consistencyLength/format consistency
資料的精細度資料的精細度 Granularity of Fact Table–what Granularity of Fact Table–what level of detail do you want? level of detail do you want? Transactional grain–finest levelTransactional grain–finest level Aggregated grain–more summarizedAggregated grain–more summarized Finer grains Finer grains better better market basket analysismarket basket analysis capability capability Finer grain Finer grain more dimension tables, more rows in fact table more dimension tables, more rows in fact table
Chapter 11 1818
Issues Regarding Star Schema Issues Regarding Star Schema (2)(2)
資料期間 資料期間 Duration of the database–how much Duration of the database–how much history should be kept?history should be kept? Natural duration–13 months or 5 quartersNatural duration–13 months or 5 quarters Financial institutions may need longer durationFinancial institutions may need longer duration Older data is more difficult to source and cleanseOlder data is more difficult to source and cleanse
Chapter 11
Size of Fact TableSize of Fact Table Depends on the number of dimensions and the grain of the fact tableDepends on the number of dimensions and the grain of the fact table Number of rows = product of number of possible values for each dimension Number of rows = product of number of possible values for each dimension
associated with the fact tableassociated with the fact table Example: Assume the following for the next Figure:Example: Assume the following for the next Figure:
Total rows calculated as follows (assuming only half the products record Total rows calculated as follows (assuming only half the products record
sales for a given month):sales for a given month):
1919
Chapter 112020
(new) Figure 9-12 Modeling dates
Fact tables contain time-period data Date dimensions are important
Chapter 11
Slowly Changing Dimensions Slowly Changing Dimensions (SCD)(SCD)
How to maintain knowledge of the past How to maintain knowledge of the past Kimble’s approaches:Kimble’s approaches:
Type 1: just replace old data with new (lose Type 1: just replace old data with new (lose historical data)historical data)
Type 2: for each changing attribute, create a Type 2: for each changing attribute, create a current value field and several old-valued fields current value field and several old-valued fields (multivalued)(multivalued)
Type 3: create a new dimension table row each Type 3: create a new dimension table row each time the dimension object changes, with all time the dimension object changes, with all dimension characteristics at the time of change. dimension characteristics at the time of change. Most common approachMost common approach
2121
Chapter 112222
(new) Figure 9-18 Example of Type 2 SCD Customer dimension table
The dimension table contains several records for the same customer. The specific customer record to use depends on the key and the date of the fact, which should be between start and end dates of the SCD customer record.
Chapter 112323
(new) Figure 9-19 Dimension segmentation
For rapidly changing attributes (hot attributes), Type 2 SCD approach creates too many rows and too much redundant data. Use segmentation instead.
Chapter 11
10 Essential Rules for Dimensional 10 Essential Rules for Dimensional ModelingModeling
Use atomic factsUse atomic facts Create single-process Create single-process
fact tablesfact tables Include a date Include a date
dimension for each dimension for each fact tablefact table
Enforce consistent Enforce consistent graingrain
Disallow null keys in Disallow null keys in fact tablesfact tables
Honor hierarchiesHonor hierarchies Decode dimension Decode dimension
tablestables Use surrogate keysUse surrogate keys Conform dimensionsConform dimensions Balance requirements Balance requirements
with actual datawith actual data
2424
Chapter 11
Columnar databasesColumnar databases for Big Data (huge volume, often unstructured)for Big Data (huge volume, often unstructured) Columnar databases optimize storage for Columnar databases optimize storage for
summary data of few columns (different need summary data of few columns (different need than OLTP)than OLTP)
Data compression Data compression NoSQLNoSQL
““Not only SQL”Not only SQL” Deals with unstructured dataDeals with unstructured data Hadoop HBase, Cassandra, MongoDBHadoop HBase, Cassandra, MongoDB
2525
Other Data Warehouse Other Data Warehouse AdvancesAdvances
Chapter 11 2626
Online Analytical Processing (OLAP) Online Analytical Processing (OLAP) ToolsTools
The use of a set of graphical tools that provides The use of a set of graphical tools that provides users with multidimensional views of their data users with multidimensional views of their data and allows them to analyze the data using and allows them to analyze the data using simple windowing techniquessimple windowing techniques
Relational OLAP (ROLAP)Relational OLAP (ROLAP) Traditional relational representationTraditional relational representation
Multidimensional OLAP (MOLAP)Multidimensional OLAP (MOLAP) Cube structureCube structure
OLAP OperationsOLAP Operations Cube slicingCube slicing–come up with 2-D view of data–come up with 2-D view of data Drill-downDrill-down–going from summary to more detailed –going from summary to more detailed
viewsviews
Chapter 11 2727
Figure 11-21 Slicing a data cube
Chapter 11 2828
Figure 11-22 Example of drill-down
Summary report
Drill-down with color added
Starting with summary data, users can obtain details for particular cells
Chapter 11 2929
Data Mining and Data Mining and VisualizationVisualization
Knowledge discovery using a blend of statistical, AI, and Knowledge discovery using a blend of statistical, AI, and computer graphics techniquescomputer graphics techniques
Goals:Goals: Explain observed events or conditionsExplain observed events or conditions Confirm hypothesesConfirm hypotheses Explore data for new or unexpected relationshipsExplore data for new or unexpected relationships
TechniquesTechniques Statistical regressionStatistical regression Decision treeDecision tree ClusteringClustering Association ruleAssociation rule Sequence associationSequence association … … and so on.and so on.
Data visualization–representing data in graphical/multimedia Data visualization–representing data in graphical/multimedia formats for analysisformats for analysis
Recommended