Automation Test Framework

Continue Improvement

Welcome Presentation

Presented by Christina Geng

AGENDA

❶ Case Introduction

❷ Testing Strategy Evolution

❸ Case ROI Analysis

❹ Success factor

❺ Revelation

Title：7 Times Evolution of Test Automation Framework under JS

Product for Efficiency Improvement

Target：

❶ Build a robust test automation framework

❷ Manage test cases efficiently

❸ Reduce test case running time and maintenance cost

❹ Shared framework under scrum development

CASE INTRODUCTION

WHO AND WHAT?

Data Analysis Visualization

• reporting, online analytical processing • analytics • data mining • process mining • event processing • business performance management • benchmarking • text mining • predictive analytics

WHO AND WHAT?

Data Analysis Visualization

• reporting, online analytical processing • analytics • data mining • process mining • event processing • business performance management • benchmarking • text mining • predictive analytics

Programming language: Javascript/Java

Provide service:

The Visualization Service is a software module which provides various type of visualization objects that can be used in different applications on different platforms (Web, Mobile and Desktop).

Visualization types:

• A basic visual component like chart (Bar, Column Pie, Line, etc ), Geo Map, Gauge, Table or Grid. • An advanced visual component like (Tree map, Candlestick, Tagcloud)

• Above visual component with interactions and animations • Combination of multiple visual components, it could be composed by same type of visual components or different type

of component.

AGENDA

❶ Case Introduction

❷ Testing Strategy Evolution

❸ Case ROI Analysis

❹ Success factor

❺ Revelation

We h

ave it!

Seleniu

m

Data D

riven

Jenkin

s

Test Link

Full A

uto

matio

n

Develo

per en

viron

men

t

Wh

at’s mo

re…

2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

VERSION-1

Test page

SilkTest

TC: test page Scheduler: ASTEC Test tool: SilkTest TC Management: Excel

VERSION-1

Test page

SilkTest

+ -

HTML Test

page

1. First automation framework setup 2. Overview functional test case

coverage 3. Reduce testing overlap for shared

component

writing test case

maintenance

the relation between TC and record

resource

We h

ave it!

Seleniu

m

Data D

riven

Jenkin

s

Test Link

Full A

uto

matio

n

Develo

per en

viron

men

t

Wh

at’s mo

re…

2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

VERSION-2

DIV

DIV DIV

DIV Test page

Selenium

Scheduler: ASTEC TC Management: Excel/XML (test manager) TC: test page Test tool: Selenium Test Driver

Other tools • Auto generate test plan from Excel to xml • Test Manager

DIV

DIV DIV

DIV Test page

VERSION-2

+ - 1. Reduce test pages 2. Reduce resource running cost 3. Test steps turns to be keyword driven in xml 4. Write test case in .xls, auto generate .xml

Selenium

DIV

DIV DIV

DIV Test page

test driver

test case sync

case unit

test data redundancy

We h

ave it!

Seleniu

m

Data D

riven

Jenkin

s

Test Link

Full A

uto

matio

n

Develo

per en

viron

men

t

Wh

at’s mo

re…

2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

VERSION-3

Wrapper API

Unified Test Driver

Abstract Test Data

Eliminate Test Page

And How?

VERSION-3

Wrapper API

backgroud color

visible

legend position

verticalOffset horizontalOffset

visible labelFont

labelColor formatString

layout padding

tooltip enable

customization text

mainValue subValue

title main sub

Selectability mode

deSelectale defaultSelectedItems

xAxisMember yAxisMember

bar barMarkerSize

measureMember

initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);

VERSION-3

Wrapper API

backgroud color

visible

legend position

verticalOffset horizontalOffset

visible labelFont

labelColor formatString

layout padding

tooltip enable

customization text

mainValue subValue

title main sub

Selectability mode

deSelectale defaultSelectedItems

xAxisMember yAxisMember

bar barMarkerSize

measureMember

updateChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);

initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);

VERSION-3

Wrapper API

backgroud color

visible

legend position

verticalOffset horizontalOffset

visible labelFont

labelColor formatString

layout padding

tooltip enable

customization text

mainValue subValue

title main sub

Selectability mode

deSelectale defaultSelectedItems

xAxisMember yAxisMember

bar barMarkerSize

measureMember

initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);

updateChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);

testUtil.generateChart(viz/line; TestData.a1a1m1m1)

VERSION-3

Wrapper API

backgroud color

visible

legend position

verticalOffset horizontalOffset

visible labelFont

labelColor formatString

layout padding

tooltip enable

customization text

mainValue subValue

title main sub

Selectability mode

deSelectale defaultSelectedItems

xAxisMember yAxisMember

bar barMarkerSize

measureMember

initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]); testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])

VERSION-3

Abstract Test Data

public

private

testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])

mySQL

testUtil.generateChart(viz/line; TestData.a1a1m1m1; legendVisibleTrue)

VERSION-3

Eliminate Test Page

VERSION-3

Unified Test Driver

selenium1

selenium2 {StartBrowser}

IE, Chrome, Firefox (selenium2)

RemoteWebDriver webDriver = null; webDriver = new FirefoxDriver();

Robot()

ImageComparer

VERSION-3

Unified Test Driver

selenium1

selenium2

{StartBrowser}

Safari (selenium1)

SeleniumServer seleniumServer = new SeleniumServer(); seleniumServer.boot(); selenium = new DefaultSelenium("localhost", 4444, "*safariproxy "+ safariLocation,urlRoot); CommandExecutor executor = new SeleneseCommandExecutor(selenium); DesiredCapabilities dc = new DesiredCapabilities(); webDriver = new RemoteWebDriver(executor, dc);

Robot()

ImageComparer

VERSION-3

Unified Test Driver

selenium1

selenium2 {TakeSnapshot}

Point p = this.getElementOffset(targetElement); Dimension d = this.getElementSize(targetElement);

BufferedImage bi = new Robot().createScreenCapture(p.x, p.y, d.width, d.height);

Robot()

ImageComparer

VERSION-3

Unified Test Driver

selenium1

selenium2

ImageComparer

{FileCompare} cmd.exe ImageComparer.exe –file filepath1 filepath2 – threshhold 100

Robot()

We h

ave it!

Seleniu

m

Data D

riven

Head

less

Test Link

Full A

uto

matio

n

Develo

per en

viron

men

t

Wh

at’s mo

re…

2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

VERSION-4

Tooltip Selection

Gesture

CSS Button

Link

Dropdown

Checkbox

Table Text

Scroll

Hover

Animation

Theme

Upload

Download

Tooltip Selection

Gesture

VERSION-4

Selection

VERSION-4

Gesture

VERSION-4

var eventObj = {

type : originalEvent.type,

timeStamp : originalEvent.timeStamp,

pageX : originalEvent.pageX,

pageY : originalEvent.pageY

};

eventRecord.push(eventObj);

var handler = function(event){

var clickEvt = document.createEvent('MouseEvents');

clickEvt.initMouseEvent(event.type, event.pageX, event.pageY…);

d3.select('.main .plot')[0][0].dispatchEvent(clickEvt)

};

testAPI_lasso_heatmap = [

{"type":"mouseover","timeStamp":1348739963751,"pageX":126,"pageY":94},

{"type":"mouseover","timeStamp":1348739963751,"pageX":485,"pageY":94},

{"type":"mousemove","timeStamp":1348739963766,"pageX":485,"pageY":208},

{"type":"mousemove","timeStamp":1348739963773,"pageX":126,"pageY":208}

]

We h

ave it!

Seleniu

m

Data D

riven

Head

less

Test Link, Jen

kins

Full A

uto

matio

n

Develo

per en

viron

men

t

Wh

at’s mo

re…

2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

VERSION-5

How to arrange test case?

VERSION-5

Test

Link

TC (.xls/.xml)

Executable TP

import

export

Result log save

VERSION-5

Continuous Integration

1. Trigger

CI Server Test Executor

Build Machine Code repository

Manual Auto

TC mngmt

Test Case Pool

Test Driver

Selenium2 Java/C++

Web Server

2.TP/Platform/build info

resources

4. get assets

libs

4. get test plan 3. Build info 5. result write back

5. result log

.xml

.xls

VERSION-5

Where we are.

1. Trigger

CI Server Test Executor

Build Machine Code repository

Manual Auto

TC mngmt

Test Case Pool

Test Driver

Selenium2 Java/C++

Web Server

2.TP/Platform/build info

resources

4. get assets

libs

4. get test plan 3. Build info 5. result write back

5. result log

.xml

.xls

VERSION-5

Test Link

Jenkins Java Tool

Perforce

Where we are.

We h

ave it!

Seleniu

m

Data D

riven

Head

less

Test Link, Jen

kins

Full A

uto

matio

n

Develo

per en

viron

men

t

Wh

at’s mo

re…

2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

Engineering

VERSION-6

Paint

Layout Display

http://phantomjs.org PhantomJS (Headless Webkit)

VERSION-6

Test driver inner logic

CasperJS IE/Safari/FF/Chrome

config Browser?

Open browser

RunJS (test step)

….

Take Snapshot

Result validation

Save result log

Read JS list TP

PhantomJS: 1. open webkit 2. runJS 3. Take snapshot 4. close

not complete not complete

Reporting

And that’s OK

We h

ave it!

Seleniu

m

Data D

riven

Head

less

Test Link, Jen

kins

Full A

uto

matio

n

Develo

per en

viron

men

t

Wh

at’s mo

re…

2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

trigger

CI Server Test Executor

Build Machine

P4 server

TC Management

Test Case Pool

Test Driver

Selenium2 Java/C++

get resources

sync up

.java

per check-in

local

Test Executor

result log

VERSION-7

Casper/PhantomJS

trigger

CI Server Test Executor

Build Machine

P4 server

TC Management

Test Case Pool

Test Driver

Selenium2 Java/C++

get resources

sync up

.java

per check-in

local

Test Executor

result log

VERSION-7

Casper/PhantomJS

Eclipse Plug-in

Perforce

Jenkins

AGENDA

❶ Case Introduction

❷ Testing Strategy Evolution

❸ Case ROI Analysis

❹ Success factor

❺ Revelation

1. Set up environment

2. Test case execution

3. Test result reporting

4. Maintenance

CASE ROI ANALYSIS

1) Test machine 2) Assets (util, BM…) 3) TP 4) Libs

1) Browser compatibility 2) Run in headless

1) Email sending 2) Sonar integrated with CI

1) BM update 2) TC management

Functional Test: BAT (nightly), FA

1000+ cases

4.5h

1. Set up environment

2. Test case execution

3. Test result reporting

4. Maintenance

CASE ROI ANALYSIS

1) Test machine 2) Assets (util, BM…) 3) TP 4) Libs

1) Browser compatibility 2) Run in headless

1) Email sending 2) Sonar integrated with CI

1) BM update 2) TC management

Functional Test: BAT (nightly), FA

1000+ cases

47min

AGENDA

❶ Case Introduction

❷ Testing Strategy Evolution

❸ Case ROI Analysis

❹ Success factor

❺ Revelation

SUCCESS FACTOR ❶ ❷❸❹

Focus

SUCCESS FACTOR ❶ ❷❸❹

Team work

SUCCESS FACTOR ❶ ❷❸❹

Continuous Improvement

SUCCESS FACTOR ❶ ❷❸❹

Learn 360°

AGENDA

❶ Case Introduction

❷ Testing Strategy Evolution

❸ Case ROI Analysis

❹ Success factor

❺ Revelation

OUR REVELATION

不被传统的测试方法束缚；从产品用到的技术找到突破；持续改进的思想；不断尝试的理念

CONTRIBUTOR Xiaolu, Ye

Christina, Geng Helen, Jiang Chen, Zhou Peipei, Jiang

Coco, Dai Grace, Zhao Minghao, Lu Kent, Wang Izzie, Wang

Phil, Xu

SUPPORTED FROM Daoyu, Wei

Li, Zhao Jimmy, Yang Alex, Feng Alex, Su Jiwei, Li ... …

THE END NO

Email: Christina.Geng@sap.com Sina Weibo: 耿晓儿de测试