Why Classic Software Testing Doesn’t Work Anymore

nt Session

Presented by:

Regg Po re

Brought to you by:

340 Corporate Way, Suite Orange Park, FL 32073 888‐2

W5 Concurre4/9/2014 12:45 PM

“Why Classic Software Testing Doesn’t Work Anymore”

Struyk

larion Softwa

300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

http://starcanada.techwell.com/�

Regg Struyk Polarion Software

With twenty years of commercial software development and testing experience, Regg Struyk has held many different positions ranging from the head of technical product management for Agfa HealthCare to, most recently, product evangelist for Polarion QA. Regg has developed for several software testing tools including test integrity, iTest, and Polarion QA. Dedicated to the domain of test management, Regg is continually analyzing testing trends and their potential impact on the discipline of software testing.

http://members.techwell.com/sme-profiles/regg-struyk�

2/11/2014

1

QA Evolves

20 plus years software dev experience

Who am I?

Senior Product Manager @ MKS (PTC)

Group Product Manager @ Agfa Healthcare

Product Manager @ Polarion for QA Software

Polarion Software® Polarion® QA™ - www.polarion.com

g @ Q

2

2/11/2014

2

•Introduction•Company Profile•Speed of Light•Big Data•Mobile•Tools•Tools•Automation•Resources

“We produce all significant documents for specifications, QA and test plans easily and rapidly with Polarion QA”

Andreas Deuter , Phoenix Contact Electronics

Waterfall– Test Specifications

Traditional QA

– Test Cases – Test Execution

Polarion Software® Polarion® QA™ - www.polarion.com 4

2/11/2014

3

Agile– Get ready to test

The Speed of Light

– More time constraints– Originated from development

Hybrids/Spin-Offs– Scrum, Agile-Fall– Does not means getting rid of documentation


Market Pressures– Regulatory– Quick check-in

5

The Speed of Light

“Testing is not a phase on Agile teams, testing is a way of life. Agile teams should test continuously. It’s the only way to ensure that the feautures implemented during a given iteration or sprint are actually done”


2/11/2014

4

Time Constraints - Deliver Faster

Pressure to test more with Limited Resources

Focus becomes task completion vs coverageQuality may suffer (may miss critical bugs)

Examples: Madden Football 2008 – prematurely releasedXbox – hardware crashes = LAWSUIT

Lack of Planning – Still need to plan not a free for all

What‘s going on QA – Pitfalls

– More time constraintsLack of Documentation

– Key component of Agile is removing clunky documentation– Does not means getting rid

Daily Meeting Hassle– Waste of time?????


– Quick check-in

“I’m surprised at the name Agile —it should be called couch potato. There are too many meetings. It’s ironic that there are all these books explaining how simple it is.”

8

2/11/2014

5

The real goal of testing should be to improve the software– A continuous search for information

What‘s going on QA

– Instead of pass or fail think in terms of whether or not tests provide valuable information

Understanding project and goals– Define coverage requirements– Understand/establish goals of a sprint

O f %


“Out of 200 participants, 64 % said that switching to Agile Development was harder than it initially seemed.”

9

•Risk-based testing accommodates change (focus on main areas)– Reduce the scope of testing based on risk

What‘s going on QA -Solutions

•Careful automated testing can accomodate change*

•Good unit testing is essential (limited bug-finding 30%)

•System Testing will cover more on average


Combine risk-based testing with maintainable automated regression testing at the system test level

10

2/11/2014

6

The big data construct

What is Big Data?

2/11/2014

7

“Data sets whose size is beyond the ability of commonly used tools to process it within acceptable time”.

Big Data – What is it?

Transformation in how society processes information.

Explosion of big data is just as transformative as the internet was.

According to IBM, “Every day, we create 2.5 quintillion bytes of data —so much that 90% of the data in the world today has been created in


so much that 90% of the data in the world today has been created in the last two years alone”.

Gartner believes Big Data will drive a total IT spend of $34billion in 2013 rising to $232billion by 2016!

13

Big Data – Sources


2/11/2014

8

What is Big Data?

The big data construct theory

Source: Gartner

To gain the right insights, big data is typically broken down by three characteristics:

Big Data – 3Vs?

•Volume: How much data•Velocity: How fast data is processed•Variety: The various types of data

Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.


processing for enhanced insight and decision making.

16

2/11/2014

9

Data Volume

"I have too much data to analyze" “We live in a world that is getting more digitized and more connected

Big Data Volume - Challenges

every day.”Data is now created by employees, customers, 3rd partyExtracting information from large data sets Massive, complex and semi-structuredRegulatory demands increase pressure on testingConsolidation of data from systems, apps, internetMi i d t & d t i t it


Missing data & data integrity

18

2/11/2014

10

Data Velocity

Speed at which new data is being created

Big Data – Velocity

The need for real-time analyticsacquisition of data vs. actionable insight


“Every business is now a digital business”

20

2/11/2014

11

The ability to implement a data driven testing process

Big Data – Velocity Challenges

Improving response rates for decision making

Expectations – when fast isn’t fast enough


Data Variety

2/11/2014

12

structured/unstructured datatext

Big Data – Variety

textsensor dataaudiovideoclick streamslog files


g

23

80% data growth in images & videoNumber of data types Unstructured/semi

Big Data – Variety Challenges

Number of data types Unstructured/semiVolatility of changing dataSources of data

Mobile devicesInternetEmail


2/11/2014

13

Traditional methods not enough

Big Data – Challenges

Traditional tools are inadequate

Lack of knowledge as to what to test and how much data to test


Test strategies for structured vs unstructured data

25

Test early – Test fastData driven testing

Big Data – Solutions

Data driven testing Develop a strong testing environmentMultiple testing methodologies

functional/non-functionalData Sampling

Identify suitable sampling technique


y p g qInclude correct uses casesRight test data sets

26

2/11/2014

14

Mobile Testing

Mobility application testers, face many critical challenges compared to conventional software testingchallenges compared to conventional software testing

Number of mobile phones will exceed world population by 2014 (7.3 billion)

Mobile – Landscape

y ( )


2/11/2014

15

BYODComplexity added due to combination of multiple

Mobile– Challenges

Complexity added due to combination of multiple platforms each having numerous devices and various OS versions

Various versions of each platform and deviceMobile user: range from tech savvy to newbieTesting on various networks and network vendors


CDMA and GSM with their 2G, 3G and 4G versions

29

Rapid Roll-out ScheduleExpectations for apps

Mobile– Challenges

Expectations for appsUltra competitive to release new apps

Complexity added due to combination of multiple platforms each having numerous devices and various OS versions


2/11/2014

16

Emulation Software (Zap, Keynote)

Mobile– Solutions

Cloud Computing (Perfecto, Soasta)Crowd sourcing

Performance Testing (Loadster, Neotys)


Tools

2/11/2014

17

Tools

Unavailability of right tools– Budget, availability and funding– Type of Tools:

• DEFFECT TRACKING• TEST CASE CREATION & Execution• Test Automation

Integrations: lack there of– Integrations: lack there of – MS Excel & MS Word equation

• Time consuming to manage and maintain• Metrics • Integrations

Tools - Solution

Test Management Software– Combines Test Cases, Test Executions,

Defects, Test Automation– Replace or leverage MS Excel/Word– Customizable workflow– Automation– Automation– Integrations

2/11/2014

18

Tools - Solution

Network with other testers to find information about l k llesser-known test tools. Online QA/Testing forumsInteroperable tools

Automation

2/11/2014

19

Test Automation

1. Test Automation Will Eliminate Human Testers1. Test Automation Will Eliminate Human TestersFALSEFALSE

2. What tests should be automated first?2. What tests should be automated first?Automate your testing procedure when you have lot of regression work.

Test Automation - Implementation

– Commitmentcan take time to set p & maintain• can take time to set-up & maintain

• If not implemented correctly = $$$ Desire to automate tests across apps & platformsMost automated test tools do not tell you what to testInitial cost can be high

• purchase and set-up

100% Automation does not exist!

2/11/2014

20

Test Automation - Solutions

The right automation – Right skill sets (programing scripts)– Ensure we create reusable tests

• can take time to set-up & maintain– Unit test will provide highest ROI

There is a large gap between the people who ownautomated test tools and the people who actually realize the benefits from test automation.

5 Lessons Learned

2/11/2014

21

Lessons

To handle the volume and ensure live data i t ti Bi D t t ti d t integration, Big Data testing needs to empower developers, testers and enterprises with freedom to experiment and innovate.

Lessons

Successful software test automation is possible if f d t l i dd d d d fundamental issues are addressed and managed. Automated software testing is truly a different way of testing and requires adjustments to current test methods and organizational structures. However, thepayback from test automation can far outweigh the costs.

2/11/2014

22

Lessons

With nearly a third of all page views now coming f bil d t bl t t t d t b from mobile and tablets, testers need to embrace mobile. Mobile is no longer an area that a few UX people specialize in, and we need to start designing and testing everything for smartphones and tablets as well as computers.

Lessons

Development cycles will continue to accelerate b d i f t ( k t based on various factors (market, process, technology), and quality expectations will need to meet the challenge through increased visibility, new ways of thinking combined with advances in software.

2/11/2014

23

Lessons

What did you learn????

Resources“Game Testing Evolves” whitepaper @ Polarion

“Big Data Testing” whitepaper @ Polarion Big Data Testing whitepaper @ Polarion

http://www.polarion.com/products/qa/index.php

[email protected]

T itt R St k1Twitter: ReggStruyk1

LinkedIn: Software Test Case Management

Technology

Why Classic Software Testing Doesn’t Work Anymore