Surrogate-Assisted Tuning for Computer … Tuning for Computer Experimentswith Qualitative and Quantitative Parameters Ray-Bing Chen Department of Statistics, National Cheng Kung

Surrogate-Assisted Tuning for Computer Experiments with Qualitative and Quantitative Parameters

Ray-Bing Chen

Department of Statistics,

National Cheng Kung University

11

ATAT2018 in NCKU

• Tainan City

• 國立成功大學 (National Cheng Kung University)

44

� ��

� ��

�� !��

�"��#�$��!��

%�"��

&��$��!��

�� "��

�� '�� (

Collaborators

Outlines

55

Tuning as an optimization problem

Surrogate Assisted Tuning (SAT)

Surrogate construction

Infill criteria

Numerical results

Conclusion

Optimization Problem

66

Motivation Problem

Find quantitative and qualitative factors to minimize run time of Algebraic Multigrid (AMG) solver

Search set: possible combinations 15*7*2*3=63077

Algebraic Multigrid Solver FactorsData type Factor Range Levels

Quantitative(continuous)

θ 0.02~0.11 [15]ω 0.6~1.8 [7]

Qualitative(category)

Smoother GS, SGS 2Cycle V, F, W 3

Motivation Problem

88

Minimize AMG execution time

Problem Domain

Auto-Tuning as an Optimization Problem

99

Target function

Minimal total time

Unknown response value

Expensive evaluation

Q&Q Input factors

x : Quantitative

z : Qualitative

Derivative Free Optimization

• Here the responses are generated via a “black-box function”.

• There is no close form of the objective function. • Derivative-free optimization approach:

• Grid Search Method• Direct Search and Pattern Search

–Kolda et al. (2003)–No gradient (derivative-free)–Iterative optimization algorithm–At each iteration, search the next point from a

pattern

)1,9.0(with |1)23(||12)23(|),(min

0

3/73/7

xxyyyxxyxf

Surrogate Assistant Approach

• Useful statistical tools:– Experimental designs– Model building

• Two well-known methods:– Response Surface Methodology (RSM)– Design and Analysis of Computer Experiment

(DACE)

• Response Surface Methodology– Box and Wilson (1951)– Noise response: y = f (x) + – Experimental Design + Regression model– Approximate f by a lower-order polynomial (Taylor

expansion)– Central Composite Design

• Design and Analysis of Computer Experiments (DACE)– Sacks et al. (1989) and Jones et al. (1998)– Model by a Gaussian Process:

– Kriging method or radial bases

– Space-filling designs

)),(exp())(),(( and )()()(

2121 xxdxzxzCorrxzxfxy T

))(),(()( ),(ˆ)()(ˆ iii iiT xzxzCorrxrxrcxfxy

Surrogate Assisted Tuning (SAT)

1616

Surrogate Assisted Tuning

1717

Initial DesignInitial Design

Surrogate Construction


Optimal?Optimal?Identify OptimaIdentify Optima

Function EvaluationFunction

Evaluation

New Design

New Design

Infill CriteriaInfill Criteria

Y






Evaluation

New Design

New Design


Y


1818






Evaluation

New Design

New Design


Y


1919






Evaluation

New Design

New Design


Y


2020






Evaluation

New Design

New Design


Y


2121






Evaluation

New Design

New Design


Y


2222






Evaluation

New Design

New Design


Y


2323






Evaluation

New Design

New Design


Y


2424






Evaluation

New Design

New Design


Y


2525






Evaluation

New Design

New Design


Y


2626

Surrogate construction

Statistical models:

Gaussian Process (Kriging) (Sacks et al., 1989)

Overcomplete basis surrogate method(Wang, Chen, 2008, Chen, Wang, Wu, 2011)

Infill criterion

Expected Improvement (EI)(Jones et al., 1998)

Works well for stationary data with quantitative factors only

How about Q&Q?






Evaluation

New Design

New Design


Y

Q & Q Input Factors

2727

Quantitative and Qualitative (Q&Q) factors

Statistical model

Q&Q Gaussian ProcessReferences: Qian, Wu, & Wu. (2008); Zhou, Qian, & Zhou (2011)

Infill criterion

propose here

Focus

2828






Evaluation

New Design

New Design


Y






Evaluation

New Design

New Design


Y

SAT- Initial Design

2929

Qualitative sub-domains

Whole domain

Whole Domain

3030

Individual LHD

Latin Hypercube Design (LHD)

Individual LHD

LHD on each qualitative sub-domain

Chen, Hsieh, Hung, Wang (2013) 3131

Bad design LHD Space-filling LHD

Q & Q Design

Peter Z. G. Qian and C. F. Jeff Wu. Sliced space-filling designs. Biometrika, 96(4):945–956, 2009.

Jian-Feng Yang, C. Devon Lin, Peter Z.G. Qian, and Dennis K.J. Lin. Construction of sliced orthogonal Latin hypercube designs. Stat. Sin., 23(3):1117–1130, 2013

3232






Evaluation

New Design

New Design


Y

SAT- Surrogate Construction

3333

“Divide-and-Conquer” approach

more smaller search space: 6 independent dim

strong relations for QN factors; no relations for QL factors

“All-in-One” approach

one large search space

relations for all variables

Ideas

3434

AMG qualitative factors: (smoother, cycle)

Qualitative sub-domains are

Divide-and-Conquer

3535

Smoother

SGS

GS

Cycle V F W

“Divide-and-Conquer” approachIndependent Gaussian Process

Sub-domain GP (SDGP)

“All-in-One” approachQualitative and Quantitative Gaussian Process

Whole domain QQGP (WDQQ)

Ideas

3636

SDGP: Independent Sub-Domain Surrogates

An surrogate in each sub-domain is constructed

Gaussian Process (GP) is applied

Statistical surrogate3737

SDGP construction - Step 1

3838

Assume observed data follows Gaussian Process

SDGP construction - Step 2

3939

Estimate R, β, σ2 from observations

WDQQ: Global Whole-Domain Q&Q Surrogate

Main idea:

To estimate of the correlations between QL factors

To share information between QL factors

To reduce the size of initial experiment design

Qian, Wu, & Wu. (2008); Zhou, Qian, & Zhou (2011)

4040

Summary on Surrogate construction

4141

SDGP WDQQ

Approach Divide-and-Conquer All-in-One

Domain Local domain Global domain

Estimation of R Cheap Expensive

Ill-condition of R Grows slower Grows faster

Size of initial design (may be reduced)






Evaluation

New Design

New Design


Y

SAT - Infill Criterion

4242

Infill Criteria

Classical problem (quantitative only)

Classical Expected Improvement (Classical EI)

Q & Q Problem

Minimum-shared EI (MSEI)

Minimal Prediction (MP)

4343

Classical Expected Improvement (EI)

GP-based Prediction

Improvement function

Expected Improvement

4444

Classical EI

Expected Improvement

4545Figures taken/modified from Forrester: Engineering Design via surrogate Modelling: A Practical Guide (2008)

EI Criterion

EI criterion

Take x with the largest EI(x) as the next point

Theorem

EI(x) > 0 for all unexplored points

EI(x) = 0 for all explored points

4646

EI Formula

Balance between

Prediction-based local exploitation

Error-based global exploration

4747

Minimum-shared Expected Improvement

Motivation

Extend the EI to across all sub-domains

Add new design via maximizing EI among all sub-domains

Improvement function

WDEI

4848

Whole-Domain Expected Improvement

WDEI criterion

Take w with largest as the next point

4949

Sub-Domain Expected Improvement (SDEI)

Motivation

Minimize the target function is our main goal

Approach

Step 1: Choose the sub-domain with minimum prediction

Step 2: Add an new design in that domain with EI criterion

5050

Minimal Prediction

Summary on Infill Criteria

5151

SDEI WDEI

Idea Filter sub-domain by minimal prediction Extend classical EI

Search Domain Local Global

Exploitation/Exploration Exploit. > Explore. Balanced

Features Cheaper; may stick to local minimum

More expensive; globally converge

SAT Framework

5252






Evaluation

New Design

New Design


Y

SDGPWDQQSDGPWDQQ

SDEIWDEISDEIWDEI

Numerical Results

5353

Testing Examples

AMG solver (real data)

Oscillation-2D (gabor)

5454

Experiment Settings

Stopping criterion

M: maximum iteration number

Repetition for N times

Different initial designs

Robustness w.r.t. initial designs

5555

Metrics & Quantile Curve

5656

Metrics & Quantile curve

5757

Comparison

Robustness

Efficiency

Two surrogates

SDGP / WDQQ

Two infill criteria

SDEI / WDEI

5858

P1: AMG Solver

Find quantitative and qualitative factors to minimize run time of Algebraic Multigrid solver

Search set: possible combinations 15*7*2*3=630

5959

Algebraic Multigrid Solver FactorsData type Factor Range Levels

Quantitative(continuous)

θ 0.02~0.11 [15]ω 0.6~1.8 [7]

Qualitative(category)

Smoother GS, SGS 2Cycle V, F, W 3

P1: AMG True Surface (Six 15x7 Subdomain)(amg_ani_cg)

6060

V F W

SGS

GS

MinMin

P1: AMG Results (5%, 50%, 95% quantile, median) (amg_ani_cg)

6161

SDGP

WDQQ

WDEISDEI

Divide-and-Conquer

All-in-One

P1: AMG results

6262

P2: Oscillation (3 QL, 2 QN)

6363


6464

SDGP

WDQQ

WDEISDEI

Divide-and-Conquer

All-in-One


6565

Conclusion

6666

Guidelines

Surrogates

SDGP for simple surface

WDQQ for complicated surface

SDGP, if modeling resource is limited

Infill criteria

WDEI is suggested in general

SDEI is better if surrogates differs significantly

6767

Surrogate selection

6868

SDGP WDQQ

Surface Simple Complicated

Cost Cheap Expensive

Pros and Cons for surrogates

Infill Criterion selection

6969

SDEI WDEI

Simple Fast Slow

Oscillating Stick to local min Convergent globally

Convergence w.r.t. surface property

For large size of the qualitative factors:

Simplify the correlation structures for QQ-GP.

Treed Gaussian Process (tGP, Gramacy and Lee, 2008)

Other GP model structures:

Branching and Nested Factors

Performance Tuning of Next‐Generation Sequencing Assembly

Other models: Radial basis function (RBF)






Evaluation

New Design

New Design


Y

Thank you.

7171

Questions/comments/collaborations are welcome!

Documents

Surrogate-Assisted Tuning for Computer … Tuning for Computer Experimentswith Qualitative and Quantitative Parameters Ray-Bing Chen Department of Statistics, National Cheng Kung