Upload
dionny-santiago
View
63
Download
0
Embed Size (px)
Citation preview
May 4th, 2016, Orlando, Florida Analyze, Diagnose, and Prevent Test FlakinessMay 5th, 2016, Orlando, FL
Dionny Santiago
2Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Agenda1. Automation Benefits & Challenges2. Metric Analysis3. Owning Full-Stack Testing4. UI Testing Recommendations5. Coding Recommendations
Demo:
3Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Testing Landscape• 200 engineers• 70 test engineers• 30+ million lines of code• 400,000 documented tests• Highly complex business rules• Sensitive domain
4Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Why automate?• Hands-off
regression testing• Focus on
exploratory• Reduce testing cost• Find defects
5Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Consistent
Meaningful
Quick Feedback
Ideal Automation
Detects Defects
Maintainable
Evolves with System
6Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
UI Automation Challenges
• End-to-end Complexity• Asynchronous Content• Dynamic Content• DOM Structure Evolution
7Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
UI Automation Challenges
• High test volume• Multiple platforms• Multiple browsers
8Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Inconsistent
Unstable
Delayed Feedback
“Flaky” Automation
False Negatives
Unmaintainable
9Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
“Flaky” Automation
10Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Analyze and Prevent Flakiness: MetricsPerformance Metrics Complexity Metrics
11Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Execution Performance
Failure % Over Time
Average Duration
Bug Detection Accuracy
12Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Failure % Over Time
Useful?
Failure Percentage
# Fi
xtur
es
13Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
ComplexityMetric Indicator ofTest Fixture Size Test organizationTest Method Size Test case designAssertion Count Over-testing and coupling
14Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Inheritance Complexity
0 1
2
2
6*
Depth within the inheritance hierarchy
*Difficult to refactor. May encounter unintended behavior.
15Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Cyclomatic Complexity
Start
X > 10
Y < 10
callA() callB()
callC() callD()
Edges - Nodes + 2
Edges = 9
Nodes = 8
CC = 3
Linearly independent paths through a program
*Indicator of branch/path complexity and non-determinism.
16Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Tracking and Diagnosing
Continuous Integration Tracking
Combine continuous execution with tracking software
17Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Tracking Metrics
18Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Tracking Metrics – Run Totals
19Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Unit
System
Integration
10%
20%
70%
Complexity
Runtime
Excessive UI AutomationResults in tests that are error-prone and difficult to maintain
20Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Unit
Integration
System
70%
20%
10%Complexi
ty
Runtime
Owning Full-Stack TestingQA must be more involved in lower-level testing
21Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Unit
Component
System
UI Test Breakdown• Reduce UI tests via unit tests• HTTP Mocking
[End-to-end]
22Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
23Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Demo
24Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
UI Testing Recommendations
• End-to-end Complexity• Asynchronous Content• Dynamic Content• DOM Structure Evolution
25Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
UI Testing Architecture
Test Abstraction*
Application Under Test
Application Model*Specifies tests against
Executes tests against
*Reduces end-to-end complexity*Encapsulates DOM Structure and handles dynamic content*Delegates to various platforms and browsers*Handles asynchronous content
26Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Application Modeling
Page Objects ElementsControl Objects
• Atomic• Modeled using
IDs, CSS selectors, XPath
• Runtime execution
• Containers• Promote
organization• Compile-time
• Containers• Promote DRY• Compile-time
27Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Control
Page ObjectControl
Elements
28Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
A note on element selectors…
• CSS selectors are faster than XPath on Chrome/Firefox
• Avoid presentation elements (class names)
• Avoid hierarchical paths (div > div > tr > div…)
• Avoid structural elements (div, p, table, span)
29Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Element Selectors
30Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Coding RecommendationsMaintainability Stability
31Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
MaintainabilityOrganization of tests promotes readability
Excessive nestingToo flat
32Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
MaintainabilityEstablish and adhere to a coding style standard
33Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Maintainability
Not focused and difficult to debug
Strive for short, focused test cases
34Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
MaintainabilityFavor composition over inheritance for code reuse
35Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
MaintainabilityFavor composition over inheritance for code reuse
36Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityExplicit delays may result in non-determinism and wasted execution time
37Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityFavor implicit delays over explicit delays
38Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Stability• Implicit vs. explicit delays
39Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityRaw exception handling introduces logic and affects readability
40Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityInstead, use built-in unit testing constructs
41Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
StabilityAvoid conditionals and loops
*Recall cyclomatic complexity
42Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Summary• Collect and analyze key metrics• Own full stack testing• Implement best practices• Prevent flakiness• Reduce testing cost• Find defects, faster
43Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201
6
Thank you!
Tariq KingGabriel NunezAdam CandoEduardo Pena
Robert Vanderwall
Troy ThomasJorge Martinez
Michael Mattera
Acknowledgements
Sample Project Availablegithub.com/dionny