Upload
christopher-gibbs
View
305
Download
4
Embed Size (px)
Citation preview
多表连接查询优化的相关研究吕 彬
2009.3.5
motivation星型连接效果对比图
0
500
1000
1500
2000
2500
8. 00E+04 5. 00E+05 1. 05E+06三元组数
(ms)
时间
Star- j oi n 优化前 Star- j oi n 优化后
' Z' 型连接效果对比图
0
5000
10000
15000
20000
8. 00E+04 5. 00E+05 1. 05E+06三元组数
(ms)
时间
z- j oi n 优化前 z- j oi n 优化后
图中表示连接顺序对查询效率的影响:准确估计选择度要考虑属性间的相关性
问题关键:高效地计算属性间的相关度
AgendaMulti-table join overview
•Heuristic and randomized optimization for the join ordering problemMichael Steinbrunn , et al, The VLDB Journal (1997) 6: 191–208
Attribute correlation detection•BHUNT: Automatic Discovery of Fuzzy Algebraic Constraints in Relational Data
Paul G. Brown Peter J. Haas , Proceedings of the 29th VLDB Conference, 2003
•CORDS: Automatic Discovery of Correlations and Soft Functional DependenciesIhab F. Ilyas , Volker Markl, et al, SIGMOD 2004, June 13–18, 2004,
•COCA: More Accurate Multidimensional Histograms out of More Accurate Correlations Detection
CAO Wei1, QIN Xiongpai, WANG Shan,WAIM2008
Star join•Star Gazing from atop your DB2 z/OS Database Server
Terry Purcell, et al, Intelligent Optimizer
•Star join revisited: Performance internals for cluster architecturesJosep Aguilar-Saborit, Data & Knowledge Engineering 63 (2007) 995–1013
Heuristic and randomized optimization for the join ordering problem
•Choosing join type based on cost
•Solution space for the join ordering problem
•Join ordering strategies
•Quantitative analysis
•Conclusion
Multi-table join overview
Choosing join type based on cost•Cost models
•Nested loop join
•Sort-merge join
•Hash join
Multi-table join overview
IO Cost
Elapsed Time
Row Cost
CPU Cost
Base Cost
Scan CostPage Cost
Solution space for the join ordering problem•Left-deep trees
n! ways to allocate n base relations
to the tree’s leavesgood solutions because of
exploiting the cost-reducing
pipelining technique
•Bushy treesan adaptable plan
enumeration strategylinear graphs
(n3 − n)/6 star graphs
(n−1) 2n−2
Multi-table join overview
Join ordering strategies•Deterministic algorithms
heuristic or exhaustive search
•Randomized algorithmsDefine a set of moves which constitute edges between the different solutions of the solution spaceperforms a random walk along the edges according to certain rules,terminating as soon as no more applicable moves exist or a time limit is exceeded
•Genetic algorithmsmake use of a randomized search strategy very similar to biological evolution
•Hybrid algorithmscombine the strategies of pure deterministic and pure randomized algorithms
Multi-table join overview
Quantitative analysis•Class of the join graph
•Relation cardinalities and domain sizes
Multi-table join overview
Quantitative analysis•Solution spaces
Multi-table join overview
Quantitative analysis•deterministic algorithms
Multi-table join overview
Quantitative analysis•deterministic algorithms
Multi-table join overview
Quantitative analysis•Randomized and
genetic algorithms
Multi-table join overview
Quantitative analysis•Randomized and
genetic algorithms
Multi-table join overview
Quantitative analysis•Randomized and
genetic algorithms
Multi-table join overview
Quantitative analysis•Randomized and
genetic algorithms
Multi-table join overview
Quantitative analysis•Total running time
•Find final solution time
Multi-table join overview
Conclusion•Heuristic algorithms Vs Randomized and genetic algorithms
compute quickly, but far from the optimum. better suited for join optimizations; although require a longer running time
•Solution spaceExcept the star join graph, the bushy tree is preferable than left-deep
processing trees.
•The extensibility of randomized and genetic algorithms
Multi-table join overview
Query-Driven approaches•Query workload
•Query feedback
•Multidimensional histogram
•SASH algorithm
•Advantage and disadvantage
Data-Driven approaches•Statistical
Chi-square test, Log-linear model
•ProbabilityBayesian network, Markov network
•Full scan vs. Sample
•hard FDs, soft FDs, Correlation
Attribute correlation detection
BHUNT: Automatic Discovery of Fuzzy Algebraic Constraints in Relational Data
•Example
Attribute correlation detection
Overview of BHUNT•Algebraic Constraints
•Example a1 : deliveries.deliveryDate, a2 : orders.shipDate, ⊕:as the subtraction operator, P : orders.orderID = deliveries.orderID,I = { 2, 3, 4, 5} {∪ 12, 13, . . . , 19 } {∪ 31, 32, 33, 34, 35 } .
Attribute correlation detection
Overview of BHUNT•Generating Candidates C = (a1, a2, P,⊕).
Generating Pairing RulesTurning Pairing Rules Into Candidates
•Identifying Fuzzy ConstraintsConstructing Bump Intervals, by applying statistical histogramming, segmentation, or clustering techniques to a sample of the column valuesChoosing the Sample Size, The sample size is selected to control the number of “exception” records that fail to satisfy the constraint.
•Exploiting the ConstraintsIdentify the most useful set of constraints, and create “exception tables” to hold all of the exception records.
•Modify the queries to incorporate the constraints the optimizer uses the constraints to identify new, more efficient access paths.
Attribute correlation detection
Experimental results of BHUNT
Attribute correlation detection
CORDS: Automatic Discovery of Correlations and Soft Functional Dependencies
•.
Attribute correlation detection
CORDS: Automatic Discovery of Correlations and Soft Functional Dependencies
•soft functional dependencies•C1 => C2, the value of C1 determines the value of C2 not with certainty, but merely with high probability.
•hard functional dependencies•the value of C1 completely determines the value of C2.
Attribute correlation detection
CORDS overview•First: Enumerating and Pruning•Pruning rule
Type constraintStatistical constraintParing constraintWorkload constraintIdentify trivial cases
Attribute correlation detection
CORDS overview
Attribute correlation detection
Analyze distinct value of sampled
column (to test for SOFT FD)
Soft FD?
Chi-squared analysis (to test
for statistical dependency)
Correlated?
Column Group
(Top-k pair)
YES
NO
YES
CORDS discovers three property and relationship•Trivial cases
“soft” keys“trivial” column
•Soft FDs C1 => C2 |C1|/|C1, C2| CORDS estimates the |C1| and |C1, C2| using sample
•CorrelationsDetect statistical dependence using the sample-based chi-squared analysis
Attribute correlation detection
CORDS and query optimization
a query having a selection predicate p1 p2, p1 = “Make = Honda” ∧and p2 = “Model = Accord”. true selectivity : σp1 p2 = 1/10. ∧naïve estimate : Sp1 p2 = 1/|Make|∧ ・ 1/|Model| = 1/7 ・ 1/8 = 1/56,adjusted estimate : naïve estimate *|Make| |Model|/|Make, Model| = 1/56 * 56/9 = 1/9,
Attribute correlation detection
CORDS experimental result•Synthetic Data:
The Accidents database
•Benchmarking Data:The TPC-H benchmark
•Real-world Data: include a subset of the Census database and the Auto database,
Attribute correlation detection
CORDS experimental result
Attribute correlation detection
CORDS experimental result
Attribute correlation detection
COCA: More Accurate Multidimensional Histograms out of More Accurate Correlations Detection
robust and informative metric
—— entropy correlation coefficients
a novel yet simple kind of multi-dimensional synopses
—— COCA-Hist to cope with different correlations
Attribute correlation detection
Entropy correlation coefficients
Attribute correlation detection
Merits of COCA•Simple and straightforward
•Accurate and robust
•Informative and unified
Attribute correlation detection
Different histograms for different degrees of correlations•Histograms for value sparsity
MHIST+squeezing
•Histograms for mutual independenceAVI assumption
•Histograms for other situationsMHist-MaxDiff histograms
•Other improvementsdiscarding empty buckets
Attribute correlation detection
Experiments
CORDS: #1(0.39, 0.29) ,#3 (0.45, 0.58)
Attribute correlation detection
Experiments
Attribute correlation detection
Star Gazing from atop your DB2 z/OS Database Server•Index key feedback
•The problems are:• The cartesian join
•difficult to create suitable multi-column indexes unless the combination of filtering dimensions that are included in the queries are known.
Star join
Star Gazing from atop your DB2 z/OS Database Server•PAIR-WISE JOIN
Star join
Star Gazing from atop your DB2 z/OS Database Server•Performance
Star join
Star join revisited: Performance internals for cluster architectures
Star join
Cluster architectures and horizontal partitioning•partitioning schemes:
round robin,range partitioninghash partitioning
•Collocated and non-collocated
•Repartitioning Operator
Star join
Star join processing techniques•Bitmap join (BJ)
size ||T|| · ||S|| bitslarge size of the indicesthe maintenance of these indices
•Multi hierarchical clustering (MHC)reducing the number of I/O accesses to the fact table. a chunk-based file system specific to OLAP cubes is proposed. star join queries are transformed into range queries
•Star hash join (SHJ)Push down bit filters technique
Star join
Analysis
Star join
Analysis
Star join
Analysis
Star join
Thanks!Q & A
卡方检验 (Chi - square test) 作为非参数检验的一种 ,是常用的检验方法 , 通过比较两项或多项频数 , 旨在检测在一定显著性水平上实际频数与以某种理论模型或分布特征假设为基础的期望频数的差异度 [3 ] 。卡方检验第一步 , 建立原假设 H0 和备择假设 H1 ; 第二步 , 根据理论经验或理论分布计算期望频数 ; 第三步 , 根据实际频数和期望频数计算样本卡方值。如果运算出的卡方值大于卡方临界值 , 接受原假设 , 反之 , 接受备择假设[4 ,5 ,6 ]
Agenda