omslag-TMapNEXT.indd 1 18-05-2011 15:30:21 · Alfonso López de Arenosa Anantharaman Iyer Dan Hannigan A special thank you goes to my Sogeti colleagues Leo van der Aalst, Nicolas

omslag-TMapNEXT.indd 1 18-05-2011 15:30:21

TMap NEXT® Testing Clouds

Ewald Roodenrijs

Sogeti Netherlands 2011

Colofon

©2011 Sogeti Netherlands B.V.

BookandePub production LINE UP boek en media, Groningen Coverdesign Andréas Prins Bookdesign Jan Faber

isbn 978-90-75414-00-4 (book), 978-90-75414-36-3 (ePub)

Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,withoutthepublisher’spriorconsent.

3

Content

Preface 5Management Summary 7

1 An Introduction to TMap NEXT Testing Clouds: Testing in the Cloud Era 11A Move to the Cloud Era 11TMap NEXT Testing Clouds: Testing in the Cloud Era 13A Step Back: The History of TMap 15Testing Evolves: TMap in Steps 16Test Approach Guide 20Reading Guide 22

2 Framework and Importance of Testing: Even in the Cloud 23What is Testing? 23Pitfalls: What Testing is Not 26Why Test: The Risks and … 26Benefits: What Does Testing Deliver? 28The Role of Testing: Who Tests 29What is Structured Testing? 41

3 The Business in Charge of IT: The Cloud 45The Moment of Transformation for IT 45Product to “Whatever as a Service” 47Information Technology to Business Technology 47The Cloud Era 47The Creation of Business Technology 55Business Technology = IT as a Commodity? 55

4 “Whatever” as a Service: Cloud-Enabled Software Testing as a Service 57Cloud Service Models: “Whatever” as a Service 59Drivers for STaaS Adoption 60How Does That Work: The STaaS Process 61STaaS Provider Process Model 69Cloud-Enabled STaaS: The Conclusion 74


4

5 Testing Cloud Strategy: A Move to 3D 77Creating a Cloud Test Strategy: A Move to 3D 78Business in Control: Business Driven Test Management 78Business in the Driver’s Seat: BDTM for Cloud 79The Second Step: An Analysis of the Cloud Risks 86The Next Steps: A Cloud Test Strategy 94

6 Testing the Cloud: In, On or With… 97Testing the Cloud Itself: Cloud Infrastructure 97Functional Testing of the Cloud Infrastructure 105Quality of the Cloud Infrastructure Using “Agile” 107Cloud Applications: Testing on the Cloud (SaaS) 108Instant Deployable Test Infrastructure: Testing on the Cloud 117

7 Cloud Risks: Worth Testing for… 121Compliance, Data Privacy and Security: A Need for Insight 123Control: Private vs. Public Cloud in Security 140Reliability: Cloud Recovery Testing 141Performance of the Cloud: Test It? 147All These Risks: Is the Business Even Ready for the Cloud? 151

References 153Index 155

5

Preface

I’m one from “Generation Now”: I want my news, social contacts, phone calls, messages, mail and other information now! Not in a day, 5 minutes or a year: now. Services are avail-able so that I can have it now. The cloud supports me on that, and it supports my ability to test that out. I can have testing services available now, with all the opportunities to get testing running now. So the cloud lets me be GenerationNow.

So in November 2010 when the idea came up for a book about cloud testing, my first thought was that I had to do it: now!

This book is different from its TMap predecessors. Whereas the books in the TMap series are handbooks with step-by-step information, this book is an innovation, about testing clouds for the early adopter. It describes the cloud business model for testing, Business Technology and steps we took in cloud projects. It can be seen as a successor of the SeizetheCloud book Sogeti published in late March of 2010. Where that book lacks discussion of testing, this book carries on to focus on testing.

Cloud, at its simplest, is Internet-based computing with the use of shared resources and software provided on demand with reduced management effort. As a service it is still at an early stage, but the growth of cloud-based computing is outstripping even the most optimistic predictions. It’s early 2011 and almost all forecasts of “the” most important IT technologies name cloud computing in their Top 3.

That growth is based on a compelling value proposition: speed to market, agility to bring for-ward or retire service, and the chance to move expenditure from CapEx into OpEx. Although the cloud is still in its infancy, it is increasingly clear that the cloud model will supplement, if not entirely replace, mainframe and client/server installations in the years to come.

In my view, the cloud is a business model or platform on which testing must be carried out just like any other service. It enables convenient, on-demand network access to a shared pool of configurable computing resources. Those resources, from networks, servers, stor-age, applications and services, can be rapidly provisioned and released—thereby drastically reducing management effort and service provider interaction.

The cloud is not only an IT opportunity, but a strategicbusinessopportunity; it creates the ability to get the business in charge of IT and change from Information Technology


6

(IT) to BusinessTechnology (BT). But the cloud market is still in a very early stage and will continue to grow and evolve. And as the Cloud Era emerges, testing will change! Not only for information systems, but also for testing the infrastructure, cloud-enabled applica-tions, and the ability to have instant deployable test infrastructure. Testing applications on the cloud is the same as testing applications on a traditional infrastructure. Only what is tested is different.

Books like this are seldom written exclusively by the author. So I would like to take this oppor-tunity to thank all of the people who helped me in creating the content of this book:

Andréas Prins Rob Baarda Michiel BoreelMark Buenen Richard Ammerlaan Pierre BedardNick Lloyd Erik Smit Michiel RigterinkRamanathan Iyer Flavien Boucher Karl SniderDirkjan Kaper Kanchan Apte John BloedjesAlfonso López de Arenosa Anantharaman Iyer Dan Hannigan

A special thank you goes to my Sogeti colleagues Leo van der Aalst, Nicolas Claudon and Clare Argent. Leo for having the vision for STaaS in 2008, Nicolas for helping me out with the security risks in the cloud and Clare was of great help in indicating improvements to the English language.

This book could not have been published without the assistance of Sogeti Nederland B.V. Software Control management team and the IBM Alliance; I am particularly grateful to Nijs Blokland, Marc Valkier, Marco Kortman and Jean-Marc Gaultier.

I have written this book with great enthusiasm. And I feel that using the cloud for testing has a lot of potential for how we use testing services in the future. It will give us testers the ability to be the GenerationNow. I believe this book will interest those people who are part of that generation, and that it will help them in using the cloud for the benefit of testing.

EwaldRoodenrijs “Somewhere in the Clouds,” April 2011

7

Management Summary

What is the cloud? The cloud at it simplest is internet-based computing, with the use of shared resources and software provided on demand with reduced management effort. The cloud it is still at an early stage, but the growth of cloud-based computing is outstripping even the most optimistic predictions. It’s early 2011 and in almost all forecasts of “the” most important IT technologies name cloud computing in their Top 3.

That growth is based on a compelling value proposition: speed to market, agility to bring forward or retire service, and the chance to move expenditure from Capital Expenditure (CapEx) into Operational Expenditure (OpEx). Although the cloud is still in its infancy, it is increasingly clear that the cloud model will supplement, if not entirely replace, mainframe and client/server installations in the years to come.

Am I saying that on my own? No, all the mayor analysts have their opinion on and are looking into the cloud. Gartner predicts that, as soon as 2012, one in five businesses will have no IT assets at all. They will simply leverage the capabilities of the cloud as computing becomes available to businesses in much the same way as a utility like electricity [Pettey, 2010]. IDC expects the server revenues from the public and private cloud to increase by 29% and 62% respectively—a disparity explained by the heightened security considerations, lower appetite for risk, and lower responsiveness to financial drivers in the public sector [IDC, 2009].

In 2010 they forecast a growth in cloud service adoption, fuelled by the “Pay-as-you-Go” model that means you pay only for what you use. Forrester goes further still, explicitly refer-ring to cloud as not an adjunct, but a successor to traditional approaches; “Cloud computing is a sustainable, long-term IT paradigm, and the successor to previous mainframe, client/server, and network computing eras” [Ried, 2010].

As cloud computing evolves, and cloud service adoption becomes ever more wide-ranging, a new global infrastructure is being created; this infrastructure can easily be connected to traditional infrastructure (including legacy systems). But it is not just for business IT assets that clouds remove previous limitations. It does the same from a software or application testing perspective, removing the typical constraints presented by having to test on client-owned or internal resources. A cloud infrastructure creates significant new opportunities for software quality assurance and testing.


8

This book describes the two aspects of the cloud, the business model aspect and the cloud platform. On both which testing must be carried out just like any other service. It enables convenient, on-demand network access to a shared pool of configurable comput-ing resources. Those resources, from networks, servers, storage, applications and services, can be rapidly provisioned and released—thereby drastically reducing management effort and service provider interaction.

Naturally, there is an understandable nervousness about this new approach and questions are being asked around integration, security and implementation. But, in our view, these challenges are outweighed by the advantages, as cloud breaks down limitations caused by testing on internal resources, and gives test teams the opportunity to free themselves from issues relating to the internal availability of hardware, applications and services, and enable a more effective way to collaborate.

We have identified a number of test cloud models, simply based on cloud vendor solutions. These are:

•Private cloud – A cloud owned by the client on which a private cloud is implemented.

•Public cloud – A public cloud that is available to the client on demand.

•Hybrid cloud – A group of clouds composed out of two or more clouds (private, com-munity or public).

•Community cloud – A group of clouds composed out of two or more public or hybrid clouds to form a community.

This cloud hype is still too many times perceived as an IT matter, a matter that is best left to infrastructure technicians. The cloud is not an IT opportunity, but a strategicbusinessopportunity; it creates the ability to get the business in charge of IT and change from Information Technology (IT) to BusinessTechnology(BT). But the cloud market is still in a very early stage and will continue to grow and evolve.

As the Cloud Era emerges testing will change! This consist of several aspects; testing the infrastructure, cloud-enabled applications, and the ability to have instant deployable test infrastructure. This all has its impact on the way we do testing in the future. As other types of applications will not disappear, it doesn’t replace what we test, but provides another addition to software testing. Testing applications on the cloud is the same as testing appli-cations on a traditional infrastructure. Only what is tested is different.

9

Management Summary

With testing clouds there are a lot more parties involved in testing. Not only the client and the stakeholders, the business, but also 3rd party suppliers of standard or SoftwareasaService (SaaS) applications, new quality attributes due to the cloud infrastructure, and the growing importance of non-functional requirements.

These new items to test for make testing the cloud different, not even to think about the consequences of the cloud business model. The on-demand use of resourcing, testing tools and infrastructure provides the opportunity to create a pay-per-use test service; SoftwareTestingasaService (STaaS). With STaaS the benefits of the cloud as a business model are used to provide a testing services to clients.

The cloud also delivers us more risks on security, data integrity, privacy issues, data recovery and performance. Is the Business even ready for IT? All these risks can be attended to by testing for them. But that’ll be a costly exercise and the cloud offers a reduction in costs. Other counter measures need to be taken to create a trusted cloud solution. Measures that decrease the risks de cloud created, but also increase the quality of the solution. These actions help create a better solution, better in use and more efficient.

We are not yet fully realizing the opportunities that exist today. It’s easy to be blinded by the small everyday issues so that the larger goal stays out of reach. Now that businesses and public organizations are “wired” and the cloud has emerged as a collection of Inter-net platforms and tools for connecting, integration and sharing of data and processes, it has become possible to think about the bigger issues that are going unaddressed; ITwillbecomeacommoditythatcanbeturnedonandoffwhenneeded!

The cloud will keep the promise IT has done the business, it’ll support the business and create Business Technology. But with all these risks is the business even ready for the cloud? IT needs to support the business with the usage of the cloud. And with support not only the creating of a cloud is meant, but a change in its mindset. As also testers can help the business with looking for some actions and measures than can support in creating a higher quality of the solution.1

1 These actions can also help in traditional applications.

11

1 An Introduction to TMap NEXT Testing Clouds: Testing in the Cloud Era

The growth of the cloud-based market is outstripping even the most optimistic predictions. That growth is based on a compelling value proposition: speed to market, agility to bring forward or retire service, and the chance to move expenditure from CapEx into OpEx. For soft-ware testing, cloud also offers a range of opportunities.

A Move to the Cloud Era

When the business asks, “What is the cloud, when should I consider a cloud service model, when not and how can it be deployed?” this is a great question for IT to give a thorough answer. An answer that can help the business in understanding the cloud better and make a correct decision on implementing it. When looking at cloud service and deployment models the cloud comes back to its original starting point; cloud computing, as this laid the way for the cloud to be created.


12

Cloud computing has been an up and coming market since 2006, but the underlying concept of cloud computing is old, even “prehistorically” for IT services; it dates back to 1961. In those years, Prof. John McCarthy said, “Computation may someday be organized as a public utility.” He was the first to publicly suggest (in a speech given to celebrate MIT’s centennial) that computer time-sharing technology might lead to a future in which computing power and even specific applications could be sold through the utility business model (like water or electricity). Almost all the modern-day characteristics of cloud computing—elastic—as a utility, online and the illusion of an infinite supply the comparison to the electricity industry, and the use of public, private, government, and community forms, have been thoroughly explored by Douglas Parkhill in 1966 [Parkhill, 1966]. This idea of a computer or information utility was very popular in the late 1960s, but faded by the mid-1970s as it became clear that the hardware, software and telecommunications technologies of the time were simply not ready. However, since 2000 the idea has resurfaced in new forms.

Figure 1.1 Interest in the cloud throughout the years

The term “cloud” is borrowed from telephony in that telecommunications companies, who until the 1990s primarily offered dedicated point-to-point data circuits, began offering Virtual Private Network (VPN) services with comparable quality of service, but at a much lower cost. By switching traffic to balance utilization as they saw fit, telecom companies were able to utilize their overall network bandwidth more effectively. The cloud symbol was used to denote the demarcation point between the responsibilities of the provider from that of the user. Cloud computing extends this boundary to cover servers as well as the network infrastructure. The first scholarly use of the term “cloud computing” was in a 1997 lecture by Ramnath Chellappa.

1 An Introduction to TMap NEXT Testing Clouds

13

Amazon played a key role in the development of cloud computing by modernizing their data centers after the dot-com bubble which, like most computer networks, would use 10-20% of their capacity at any one time while leaving room for seasonal spikes. Having found that the new cloud architecture resulted in significant internal efficiency improvements whereby small, fast-moving “two-pizza teams” could add new features faster and more easily. Amazon initiated a new product development effort to provide cloud computing to external customers, and launched Amazon Web Service (AWS) on a utility computing basis in 2006. In 2007, Google, IBM and a number of universities embarked on a large scale cloud computing research project.

TMap NEXT Testing Clouds: Testing in the Cloud Era

TMap is Sogeti’s prominent Test Management approach to the structured testing of infor-mation systems. The approach is described generically, since the specific makeup of this, the best fitting test approach, depends on the situation in which it is applied.

TMap can be summarized in four essentials (see Figure 1.2):

TMap is based on a business-driven test management approach.1 TMap describes a structured test approach.2 TMap contains a complete tool box.3 TMap is an adaptive test method.4

The first mentioned essential is directly related to the fact that the importance of the busi-ness in IT; the IT-business case (the justification of a project) for organizations is continuously growing. For testing this means the choices on what risks to cover with testing, what results are to be delivered and how much time and money to spend need to be made based on rational and economic grounds. For cloud this means testing is done on-demand according to the business’ needs. This is why TMap has developed the business driven test manage-ment approach, which can be seen as the “leading thread” of the structured TMap test process. This essential is explained in this book from three perspectives:

From testing Information Technology to testing Business Technology (Chapter 3)•How to create a cloud test strategy (Chapter 4)•Testing the cloud itself (Chapter 5)•The cloud risks for testers (Chapter 6)•The cloud business model for testing: Software Testing as a Service (Chapter 7)•

By describing a structured test process (essential 2) and giving a complete tool box, TMap answers the classic questions what/when, how, what with and who. With the description of test process use has been made of the TMap life cycle model: a development cycle


14

related description of the test cycle. The life cycle model describes what/when should be carried out.

Besides this, to be able to execute the test process properly, several issues in the field of infrastructure (whatwith), techniques (how) and organization (who) should be implemented. TMap provides a lot of applicable information in the shape of examples, checklists, tech-nique descriptions, procedures, test organization structures and test environment/tools (essential 3).

Furthermore, TMap has a flexible design so that it can be applied to several system devel-opment situations: i.e. new development as well as maintenance of information systems, development in-house or a purchased package and with outsourcing of (parts of) the test-ing (essential 4).

In this chapter, a sequential overview is given of how TMap became a standard approach to structured testing, the reasons for a new version of TMap, the key points of TMap and a number of suggestions concerning which chapters are of interest to which target groups.

Figure 1.2 TMap NEXT model of essentials


15

A Step Back: The History of TMap

In the international testing world, TMap is a familiar concept, as this approach to testing has been in existence since 1995. While it is not necessary to know the history of TMap in order to understand or apply the approach, this section invites you to take a glimpse behind the scenes.

Standard Approach to Structured Testing

This book was preceded by the Dutch book TestenvolgensTMap (= Testing according to TMap) in 1995, SoftwareTesting (a guide to the TMap approach) in 2002 (both books by Pol, Teunissen and Van Veenendaal) and TMapNEXTforresult-driventesting in 2006 (Koomen, Van der Aalst, Broekman and Vroon).1 The books turned out to be, and still are, bestsellers. TMap has evolved over recent years to become a standard for testing information systems. It is currently applied in hundreds of companies and organizations, including many banks, insurance companies, pension funds and government organizations. The fact that TMap is seen to be a prominent standard approach to structured testing is demonstrated by facts as, for example, suppliers of test tools advertising with the words “applicable in combina-tion with the TMap techniques”; test professionals who ensure that TMap experience is prominent on their CV or, increasingly, that they are TMap-certificated; recruitment adver-tisements in which test professionals with knowledge of TMap are sought, and independent training institutions that provide TMap courses.

Figure 1.3 History of TMap

The Strength of TMap

The strength of TMap can largely be attributed to the considerable practical experience that is the basis for the method. This experience comes from hundreds of professional testers in as many projects over the last twenty years! The aim of this book is to be a valuable aid in coping with most, if not all, challenges in the area of testing now and in the near future.

1 Up-to-date information about the translations in other languages can be found at www.tmap.net.


16

The disadvantage of a book is that the content, by definition, is static, while in the field of IT new insights, system development methods, etc., are created with great regularity.

It would be commercially irresponsible to compile and publish a new version of the TMap book with every new development in IT. To enable TMap to keep up with current develop-ments, an expansion mechanism is created. An expansion describes the way TMap has to be applied in a new development. Recent expansions, in the form of white papers, are included in the book TMapTestTopics[TestTopics, 2005]. To keep the TMap users updated about the new expansions, different means of communication are used. For example, the large number of presentations and workshops that are given at test conferences, the popular TMap Test Topics sessions (in which current test themes are presented) and the many articles in the various specialist publications. All of this makes TMap what it is now: “A complete test approach, with which any organization can successfully take on any test challenge, now and in the future!”

TipTake a look at www.tmap.net. You will find there, among other things:

downloads (including white papers for expansions, checklists, test-design tech- –niques and a glossary)published TMap articles –TMap newsletters (if you wish to receive these automatically, you can register for –this on the site).

Testing Evolves: TMap in Steps

“Why all these new versions of TMap,” you may ask, “Was there something wrong with the previous version?” No, however, since it the first version appeared on the market, various new developments have taken place both in IT and testing. And in order to keep TMap as complete and up to date as possible, these have been incorporated in the new version and add-ons have been created. This concerns developments such as the increasingimportanceofITtoorganizations, a number of innovationsintheareaofsystemdevelopment and the transformationfromInformationTechnologytoBusinessTechnology. Besides this, the tester appears to be better served by a test approach written as a guide, rather than as a testing manual. An explanation of this is given in the following sections.

Increasing Importance of IT to Organizations

Since the end of the nineties, the use of software or Information Technology in general, has become increasingly important to organizations, so that IT projects are more often


17

initiated and managed from within a user organization. This is prompted by the following developments:

•Cost reductions of IT development and management. IT is required to be cheaper and the business case (the “why” in combination with costs and profits) formulated more clearly.

•Growing automation of business processes. More and more business processes within organizations are either automated or strongly dependent on other automated processes.

•Quicker deployment of automation. With the growth of automation, IT has grown from being a company support resource to one that differentiates the company from the competition. This means that the flexibility and speed with which this resource can be deployed is of crucial importance in beating the competition.

•Quality of automation is becoming more important (see the practical example “Con-sequences of Software Failures”). The fact that IT end users currently make sure they have their say, combined with the fact that CEO’s, CFO’s and CIO’s are made personally responsible for the accuracy of the company’s financial information leads to (renewed) interest in quality of IT.

These four aspects are summed up as “more for less, faster and better.” A consequence of this is that IT projects are becoming increasingly dynamic and chaotic in nature. This can put great pressure on the testers and increase the relative share of testing within IT. In order to make the test process manageable and to keep it that way, the “business-driven test management” (BDTM) approach was developed for TMap, and has been used in this book. With this, the creation of a test strategy is directly related to the business risks. It enables the client to make responsible, risk-based choices in the test process. By making these choices, the client has significant influence on the timeline and the costs of the test process.

In More DetailConsequences of Software FailuresBecause IT has become more important within organizations, the impact of any software problems is increased.

Some examples are:

revenue loss; –brand/reputation loss; –compensation claims, and productivity loss. –


18

The fact that this can involve considerable losses is demonstrated by the following real-world examples:

A sporting goods manufacturer suffered a 24% drop in turnover (€100 million) in a –single quarter because of software failures in the stock administration. In the days that followed, after the company had announced that they had problems with the software, the share value decreased by more than 20%.Through a failure in their encryption software, a financial advisory organization –displayed their customers’ social security numbers and passwords in legible text on their website. This caused distress among the customers and led to a sizeable loss of business.After a pharmaceutical wholesale company had gone bankrupt, the parent –company submitted a claim for damages of €500 million to the software supplier, claiming that the “enterprise software” had been faulty.

Innovations in the System Development Area

The test principles on which TMap is based, came into being in the eighties, and, of course, they relate to the system development methods of that time. Nowadays, these methods are often referred to as waterfall methods. Their most important characteristic is the purely sequential execution of the various development activities. Today there is a strong growing interest in other methods. The most important characteristic of the new genera-tion of system development methods is incremental or iterative development, with the test process increasingly being integrated in the development process. With the Agile approach as the most profound example of this. For more information on testing in Agile Software Development Environments (ASDE) please look into “Testing in Agile Software Development Environments with TMap NEXT®” [Davis, 2010].

Since, in the course of various activities, testing touches on the chosen system develop-ment method, it is inevitable that a test approach like TMap should evolve in step with changes in the area of system development. How to apply TMap in certain situations is often already laid down in the previously mentioned expansions. The key points of the expansions have now been integrated in this new version of TMap, so that the book again provides a current and as complete as possible overview of how TMap can be applied in a variety of situations.

Transformation from Information Technology to Business Technology

Today cloud is the hype in IT. It tells us that it can fulfill the promise IT has been making to the business for years. It will move Information Technology over to Business Technology


19

as this has an impact on the business itself; business decisions are increasingly becom-ing technology decisions. Many business processes are so technologically enhanced that businessandtechnologyareinterwoven. Today technological innovation and business innovation are synonymous. This brings out a lot of opportunities to the business that we now just have to realize!

The cloud, among others, makes possible the move to Business Technology. However, as the cloud is IT, this needs to be tested. With IT so interwoven with business and other parts of our everyday life, the risks of encountering problems are extremely high. TMap can be applied to test the cloud: it can help test the cloud infrastructure, create test infrastructures, test cloud-enabled applications and it can be used on testing services.

Integrating Quality Assurance with PointZERO®

It is commonly known that our IT landscape is too expensive. Too much time and effort is going into delivering new solutions and even more into maintaining the existing ones. The landscape is becoming increasingly complex to manage and poor quality negatively impacts the competitive position of enterprises. High quality, timely and cost efficient IT solutions that are completely aligned with business demands are required to be able to compete in today’s business world, and agility is crucial. This requires testing and quality assurance to evolve as well. Aligning the quality of (IT) solutions with the business requirements and reaching the desired level of quality faster and at a reduced cost requires continuous change and improvement.

Figure 1.4 Testing and quality assurance are part of the development process at PointZERO®


20

We need to change the existing ways of working and move from classical functional and acceptance testing, to even more adequate and efficient solutions. Important elements are:

The effort to increase and improve the collaboration between different IT disciplines such •as developers, designers and testers to reach a better standard of quality jointly Taking quality measures much earlier in the development lifecycle•And finally, the ever increasing need to align all IT activities, including testing, with the •demands of the business. We need to make sure that each euro that is invested, adds value to your company.

To achieve these goals, it is necessary to start thinking of quality right from the start—by testing ideas, concepts and documents. As soon as a new initiative arises, testing and quality assurance need to become part of the total development process at PointZERO®. However, our present way of working does not always accommodate this concept. In most cases, it will require some time to develop and grow. The concept of PointZERO® provides a framework and growth model to support the realization of better, cheaper and faster quality. Some elements of this concept have been around for quite some time and are considered “proven technology.” Other parts are relatively new or still in the development stages.

Test Approach Guide

The previously mentioned four developments, i.e. the increasing importance of IT within organizations, the innovations in system development, the transformation from Information Technology to Business Technology, and integrating quality assurance with PointZERO®, indicate the increased dynamic of the various development and test environments. In a situation like this, a manual is often seen as being too rigid. Furthermore, it appears that a broader public with a wide range of competencies is increasingly carrying out testing, so there is now a greater need for more comprehensive descriptions of how specific TMap activities can be carried out.

This book, TMapNEXTTestingClouds, has been revised in response to the transformation from Information Technology to Business Technology. In short, TMapNEXTTestingClouds offers the tester aguidetodeliveringresultstotheclientinBusinessTechnology.

Testing the Cloud: What Can TMap NEXT Testing Clouds Offer?

But what exactly does TMapNEXTTestingClouds offer—what can you do with it? You will find the answer to these questions in this section, in which a general overview is given of the assistance that TMapNEXTTestingClouds can supply and where TMapNEXTTesting


21

Clouds can be applied. In the book TMapNEXT®[TMap NEXT, 2006], we go into this in more detail and more attention is paid to the way in which TMap can be applied in vari-ous situations.

Where Can TMap NEXT Testing Clouds Help?

In order to assist the tester in his work, TMapNEXTTestingClouds explains how to carry out certain activities, or how these are supported by TMap. This concerns help with:

the translation of the clients (cloud) demands and requirements into a concrete test •strategy and management of the execution;assisting the test manager, testing coordinator and/or tester to deal with cloud infra-•structure, cloud-enabled applications and software testing as a service;the execution of, among other things: •

a cloud risk analysis; –a test cloud strategy, and –a (non-)functional test –

the setting up and the management of test infrastructure for the current and other •projects;the execution of the test activities with real-world examples, tips and also detailed expla-•nations of certain aspects, andconsidering the test process as much as possible from an exterior vantage point, by •answering, for example, practical questions (what does testing actually deliver?) and making use of general project information.

Where Can TMap NEXT Testing Clouds Be Applied?

TMapNEXTTestingClouds addresses the following possibilities of applying TMap:

where there is either a demand-supplier relationship (e.g. outsourcing) between client, •developer and tester (each with their own responsibilities), or a collective interactive approach;with iterative, incremental, waterfall and agile approaches;•with new development in the cloud and migration of information systems to the cloud;•in situations with combinations of development approaches, such as in-house, reuse-•based, use of Software as a Service and assembling of purchased modules, all within a single IT architecture;with coverage of non-functional requirements of the information system in the test •approach, andin situations where the cloud infrastructure needs to be tested.•


22

Reading Guide

The book TMapNEXTTestingClouds has been written to give answer on the following business problems:

You need to have a test executed in a short period of time and only for a short time frame •(test manager, program manager, business manager or manager of IT department).You are requested to test a cloud infrastructure (test managers, test coordinators, infra-•structure testers and testers).You are requested to test cloud applications and/or the integration of them in the enter-•prise architecture (test managers, test coordinators and testers).

The book contains a short introduction to “TMap Next for result-driven testing”: Chapter 1 “An Introduction to TMap NEXT Testing Clouds” and Chapter 2 “Framework and Importance of Testing: Even in the Cloud.” These chapters are interesting for all the target groups.

Chapter 3 “The Business in Charge of IT: The Cloud” and Chapter 4 “Whatever” as a Service: Cloud-Enabled Software Testing as a Service” are of interest for the test manager, program manager and business manager and manager of the IT department.

Chapter 5 “Testing Cloud Strategy: A Move to 3D,” Chapter 6 “Testing the Cloud: In, On or With…” and Chapter 7 “Cloud Risks: Worth Testing for…” are mainly interesting for the target groups of test managers, test coordinators, infrastructure testers and testers.

At various places in the book, definitions, practical examples, tips and more detailed expla-nations are provided. These can be recognized by title, box and a green background.

23

2 Framework and Importance of Testing: Even in the Cloud

This chapter provides an introduction to testing in general and focuses on structured testing. No specific (prior) knowledge of TMap or clouds is required in order to understand this. In sequence, an explanation is given of: what is understood by testing, why testing is necessary (and what it delivers), what the role of testing is and what structured testing involves.

What is Testing?

While many definitions of the concept of testing exist, one way or another they all contain comparable aspects. Each of the definitions centres on the comparison of the test object against a standard (e.g. expectation, correct operation, requirement). With this, it is impor-tant to know exactly what you are going to test (the test object), against what you are going to compare it to (the test basis) and how you are going to test it (the test methods and techniques).


24

The International Standardisation Organisation (ISO) and the International Electrotechnical Commission (IEC) apply the following definition [ISO/IEC, 1991]:

DefinitionTechnical operation that consists of the determination of one or more characteris-tics of a given product, process or service according to a specified procedure.

This definition is more an IT definition, but testing is not an IT concern anymore; it’s a busi-ness concern. Testing supplies insight in the difference between the actual and the required status of an object. Where quality is roughly to be described as “meeting the requirements and expectations,” testing delivers information on the quality. It provides insight into, for example, the risks that are involved in accepting lesser quality. For that is the main aim of testing. Testing is one of the means of detection used within a quality control system. It is related to reviewing, simulation, inspection, auditing, desk-checking, walkthrough, etc. The various instruments of detection are spread across the groups of evaluation and testing:1

Evaluations: assessment of interim products. •Testing: assessment of the end products. •

Put bluntly, the main aim of testing is to find defects: testing aims to bring to light the lack in quality, which reveals itself in defects. Put formally: it aims to establish the difference between the product and the previously set requirements. Put positively: it aims to create faith in the product.

The level of product quality bears a relationship to the risks that an organisation takes when these products are put into operation. Therefore, in this book we define testing as follows:

DefinitionTesting is a process that provides insight into, and advice on, quality and the related risks. This to allow the business to appreciate and understand the risks of a software implementation.

Advice on the quality of what? Before an answer to this can be given, the concept of quality requires further explanation. What, in fact, is quality?

1 The theory also refers to verification and validation. Verification involves the evaluation of (part of) a system to determine whether the products of a development phase meet the conditions that were set at the beginning of that phase. Validation is understood to mean determining whether the products of the system development meet the user needs and requirements. [IEEE, 1998]

2 Framework and Importance of Testing

25

DefinitionThe totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs [ISO, 1994].

In aiming to convert “implied needs” into “stated needs” we soon discover the difficulty of subjecting the quality of an information system to discussion. The language for discussing quality is lacking. However, since 1977, when McCall [McCall, 1977] came up with the pro-posal to divide the concept of quality into a number of different properties, the so-called quality characteristics, much progress has been made in this area.

DefinitionA quality characteristic describes a property of an information system.

A well-known set of quality characteristics was issued by the ISO and IEC [ISO 9126-1, 1999]. In addition, organisations often create their own variation of the above set. For TMap, a set of quality characteristics specifically suited to testing has been compiled, and these are listed and explained in TMap NEXT®. The cloud, and specifically the cloud infrastructure, has its own quality characteristics, these are explained in Chapter 6.

What, then, is the answer to the question: “Advice on the quality of what?”

Since, where quality is concerned, the issue is usually the correct operation of the software, testing can be summed up as being seen by many to mean: establishing that the software functions correctly. While this may be a good answer in certain cases, it should be realised that testing is more than that. Apart from the software, other test objects exist, the quality of which can be established. That which is tested, and upon which quality recommenda-tions are subsequently given, is referred to as a test object.

DefinitionThe test object is the information system (or part thereof) to be tested.

A test object consists of hardware, system software, application software, organisation, procedures, documentation or implementation. Advising on the quality of these can involve—apart from functionality—quality characteristics such as security, user friendliness, perfor-mance, maintainability, portability and testability.


26

Pitfalls: What Testing is Not

In practice, it is by no means clear to everyone what testing is and what could or should be tested. Here are a few examples of what testing is not:

Testing is not a matter of releasing or accepting something. Testing supplies advice on •the quality. The decision as regards release is up to others (stakeholders), usually the commissioner of the test.Testing is not a post-development phase. It covers a series of activities that should be •carried out in parallel to development. Testing is something other than the implementation of an information system. Test results •are rather more inclined to hinder the implementation plans. And it is important to have these—often closely related—activities well accommodated organisationally.Testing is not intended initially to establish whether the • requested functionality has been implemented, but to play an important part in establishing whether the required functionality has been implemented. While the test should of course not be discounted, the judgement of whether the right solution has been specified is another issue. Testing is not cheap. However, a good, timely executed test will have a positive influence •on the development process and a qualitatively better system can be delivered, so that fewer disruptions will occur during production. Boehm demonstrated long ago that the reworking of defects costs increasing effort, time and money in proportion to the length of time between the first moment of their existence and the moment of their detection [Boehm, 1981]. See also “What does testing deliver?” in the next section.Testing is not training for operation and management. Because a test process generally •lends itself very well to this purpose, this aspect is often too easily included as a second-ary request. Solid agreements should see to it that both the test and the training would be qualitatively adequate. A budget and time should be made exclusively available for the training, and agreements made as regards priorities, since at certain times choices will have to be made.

It is the task of the test manager, among others, to see that these pitfalls are avoided and to make it clear to the client exactly what testing involves.

Why Test: The Risks and …

In Chapter 1 it is explained that IT has been increasing in importance to organisations since the end of the nineties. But with this, many organisations are plagued by projects getting out of hand in terms of both budget and time, owing to software defects during the opera-tion of the developed information systems. This shows that organisations are accepting, or having to accept, systems without having real insight into their quality. In many cases,


27

too, there is a lack of good management information upon release. This often results in big risks to the company operations: high reworking costs, productivity loss, brand/reputation loss, compensation claims and loss of competitiveness through late availability of the new product (revenue loss) may be the consequences.

Figure 2.1 Cloud risks that can be tested for

Before an information system goes into production, the organisation will have to ask itself explicitly whether all requirements are met. Have all the parts and aspects of the system been explored in sufficient depth? Besides the functionality, have checks been carried out on, for example, the effectivity, performance and security aspects? Or, as ISO puts it: has it been established whether the product possesses the characteristics and features necessary to meet the states or (even more difficult) implied needs? What is self-evidently implied to one may be a revelation to another.

Have all the errors been reworked, and have any new errors been introduced in the course of reworking them? Can the company operations depend on this system? Does the system really provide the reliable solution to the information issue for which it was designed?

The real question is: what risks are we taking and what measures have been taken to reduce those risks? In order to avoid obtaining answers to these crucial questions only at the opera-tional phase, a good, reliable testing process is required. That demands a structured test approach, organisation and infrastructure, with which continuous insight may be obtained in a controlled manner into the status of the system and the risks involved.


28

Benefits: What Does Testing Deliver?

While a structured test approach is considered to be of great importance, the question “What do you think of the test process?” is generally answered with “Expensive!” This response can seldom be quantified, since it is often a gut reaction, unsupported by figures. Testing is expensive. Yes, that is true if you only look at the test costs and disregard the test benefits. Test costs are based on, among other things:

the costs of the test infrastructure, and•the hours worked by the testers and their fees. •

Test benefits are [Black, 2002]:

The prevention of (high) reworking costs and consequential damage to the production •situation, thanks to defects being found during testing and rectified within the system development process. Examples of consequential damage are: revenue loss, brand/reputation loss, compensation claims and productivity loss.The prevention of damage in production, thanks to errors being found during testing, •and, while not being solved, being flagged as “known errors.”Having/gaining faith in the product.•Facilitating good project management through the supply of (progress and quality) •information.

If there is a way of expressing test benefits in money, the answer to the question “What does testing deliver” may be, from a test-economic perspective:

Test Yield = Test Benefits – Test Costs

Although this appears to be a simple calculation, in practice it is very difficult to estimate how much damage would have been caused by failures that were found during testing, had they occurred at the production stage. And anyway, how do you translate, for example, potential loss of image into money? In the literature, some attempts have nevertheless been made at making this calculation, for example [Aalst, 1999].

However, it remains difficult to establish exactly what having faith in the quality of a product, or gaining (progressive) information really delivers. Despite that, within the world of testing there are more and more tips and tricks to be found that make it possible to observe one or more of the following defects and to operate accordingly:

too much testing is carried out, so that the costs of finding a defect during testing no longer •offset the damage that this defect would potentially cause if it occurred in production


29

too little testing is done, so that more issues occur in production and the reworking •costs of these are proportionately higher than the test costs would have been to find the defects during testingtesting is carried out ineffectively, so that the test investment is not recouped.•

The Role of Testing: Who Tests

This section explains both the significance and role of certain test concepts in their environ-ment. Spread across the following subjects, the associated concepts are explained:

Testing and quality management;•Testing: how and by whom;•Test and system development process;•Test levels and responsibilities, and•Test types.•

Testing and Quality Management

Quality was, is and remains a challenge within IT (see also examples in the section “Testing Evolves: TMap in Steps” in Chapter 1) and if IT wants to move to Business Technology quality is of greatest importance to get this right! Testing is not the sole solution to this. After all, quality has to be built in, not tested in! Testing is the instrument that can provide insight into the quality of information systems, so that test results—provided that they are accurately interpreted—deliver a contribution to the improvement of the quality of information systems. Testing should be embedded in a system of measures in order to arrive at quality. In other words, testing should be embedded in the quality management of the organisation.

The definition of quality as expressed by the ISO (see section “What is Testing?”) strongly hints at its elusiveness. What is clearly implied to one is anything but to another. Implicit-ness is very much subjective. An important aspect of quality management is therefore the minimisation of implied requirements, by converting them into specified requirements and making the degree visible to which the specified requirements are met. The structural improvement of quality should take place top-down. To this end, measures should be taken to establish those requirements and to render the development process manageable.

DefinitionQuality assurance covers all the planned and systematic activities necessary to provide adequate confidence that a product or service meets the requirements for quality [ISO, 1994].


30

These measures should lead to a situation whereby:

There are measurement points and units that provide an indication of the quality of the •processes (standardisation);It is clear to the individual employee which requirements his work must meet and also •that he can evaluate them on the basis of the above-mentioned standards;It is possible for an independent party to evaluate the products/services on the basis of •the above-mentioned standards, andThe management can trace the causes of weaknesses in products or services, and con-•sider how they can be prevented in future.

These measures may be divided into preventive, detective and corrective measures:

Preventive measures are aimed at preventing a lack in quality. They can be, for example, •documentation standards, methods, techniques, training, etc.;Detective measures are aimed at discovering a lack of quality, for example by evaluation •(including inspections, reviews, walkthroughs) and testing, andCorrective measures are aimed at rectifying the lack of quality, such as the reworking •of defects that have been exposed by means of testing.

It is of essential importance that the various measures are cohesive. Testing is not an independent activity; it is only a small cog in the quality management wheel. It is only one of the forms of quality control that can be employed. Quality control is in turn only one of the activities aimed at guaranteeing quality. And quality assurance is, in the end, only one dimension of quality management.

Testing: How and by Whom

Testing often attracts little attention until the moment the test execution is about to begin. Then suddenly a large number of interested parties ask the test manager about the status. This section demonstrates, however, that testing is more than just the execution of tests. We then explain the ways of testing and who can carry out the testing.

There Is More to TestingTesting is more than a matter of taking measurements—crucially, it involves the right plan-ning and preparation. Testing is the tip of the iceberg, the bigger part of which is hidden from view (see Figure 2.2).


31

Figure 2.2 Measurements are only the tip of the testing iceberg

In this analogy, the actual execution of the tests is the visible part, but on average, it only covers 40% of the test activities. The other activities—planning and preparation—take up on average 20% and 40% of the testing effort respectively. This part is not usually recog-


32

nised as such by the organisation, while in fact it is where the biggest benefit, not least regarding time, is to be gained. And, significantly, by carrying out these activities as much as possible in advance of the actual test execution, the testing is on the critical path of the system development programme as briefly as possible. It is even possible, because of technical developments (test automation), to see a decreasing line in the percentage of test executions regarding preparation and planning.

Ways of TestingThere are various ways of testing (in this case, executing tests). For example, is the testing being done by running the software, or precisely by not running it? And is a characteristic of the system being tested using test cases specially designed for it, or precisely not? A number of ways of testing are:

Dynamic explicit testing•Dynamic implicit testing•Static testing•

Dynamic Explicit TestingWith dynamic explicit testing, the test cases are explicitly designed to obtain information on the relevant quality characteristic. With the execution of the test, or the running of software, the actual result is compared against the expected result in order to determine whether the system is behaving according to requirements. This is the most usual way of testing.

Dynamic Implicit TestingDuring dynamic testing, information can also be gleaned concerning other quality charac-teristics, for which no explicit test cases have been designed. This is called dynamic implicit testing. Judgements can be made, for example, on the user-friendliness or performance of a system based on experience gained without the specific test cases being present. This can be planned if there has been a prior agreement to provide findings on it, but it can also take place without being planned. For example, if breakdowns occur regularly during the testing. In that case, a judgement can be made concerning the security of company operations.

Static TestingWith static testing, the end products are assessed without software being run. This test usually consists of the inspection of documentation, such as security procedures, training, manuals, etc., and is often supported by checklists.

In More DetailA Paper End-to-End TestEnd-2-end testing is one of the most difficult tests there is. Functional everything should work fine separately and with the end-2-end test all the systems in the com-


33

plete process should be linked with each other. A small mistake in one of the sub-stantive systems and everything goes wrong. Therefore, everything should really be good. Unfortunately some systems (read “departments”) sometimes speak a different language with each other. Because when “System X” delivers something else compared to what “System Y” expects there is a mismatch on the content. Thus, the (functional) specifications are not correctly discussed with each other, communication!

Now how do we solve this? One way is to run a paper end-2-end test as a static test in preparation for your end-2-end test. What is a paper end-2-end test? All parties sit down with each other and talk through the process that takes place within the systems. By creating a transparent flow of the process with all parties present this yields to a number of advantages, namely:

Possible mismatches in the specification are found; –These mismatches are transparent and can directly be discussed; –The test specifications with attention to the end-2-end test are a lot simpler be- –cause the actual process has been made visible to/by all parties, andEveryone has the same mindset with the end-2-end test, making communication –better and easier.

Who Tests?Anyone can do testing. Who actually does the testing is partly determined by the role or responsibility held by someone at a given time. This often concerns representatives from development, users and/or management departments. Besides these, testing is carried out by professional testers, who are trained in testing and who often bring a different perspec-tive to testing. Where, for example, a developer wants to demonstrate that the software works well (“Surely I’m capable of programming?”), the test professional will go in search of defects in the software. Moreover, a test professional is involved full-time in testing, while the aforementioned department representatives in many cases carry out the testing as a side issue. In practice, the mix of well-trained test professionals and representatives from the various departments leads to fruitful interaction, with one being strong in testing knowledge and the other contributing much subject or system knowledge.

Test and System Development Process

The test and system development processes are closely intertwined. One delivers the prod-ucts, which are evaluated and tested by the other. A common way of visualising the relation-ship between these processes is the so-called V model. A widely held misunderstanding is that the V model is suited only for a waterfall method. But that misrepresents the intention behind the model. It is also eminently usable with an iterative and incremental system devel-


34

opment method. Therefore, with such a method, a V model can be drawn, for example, for each increment. Many situations are conceivable that influence the shape and the specific parts of the V model. A few situations are shown in the box “Influences on the V Model.”

With the help of the V model, the correlation between test basis, evaluation and testing (test levels) is explained in this and the following subsection.

In More DetailInfluences on the V ModelThe form and specific parts of a V model can vary through, for example:

The place of the testing within the system development approach. –Using a waterfall development method with characteristics including: construc- –tion of the system in one go, phased with clear transfer points, often a lengthy cyclical process (SDM, among others).Using an incremental and iterative development method with the following pos- –sible characteristics: constructing the system in parts, phased with clear trans-fer points; short cyclical process (DSDM and RUP, among others).Using an agile development method characterised by the four principles: –individuals and interaction over processes and tools, working software over extensive system documentation, user’s input over contract negotiation, reacting to changes over following a plan (extreme programming and SCRUM, among others).

The place of testing within the life cycle of the information system. –Are we looking at new development or the maintenance of a system? –Does this involve the conversion or migration of a system? –

A self-developed system, a purchased package, purchased components, or distrib- –uted systems.The situation whereby (parts of) the system development and/or (parts of) the test- –ing are outsourced (outsourcing and off-/near shoring, among other things).

Left Side of the V ModelIn Figure 2.3 the left-hand sideshows the phases in which the system is built or converted from wish, legislation, policy, opportunity and/or problem into the realised solution. In this case, the left-hand side shows the concepts of requirements, functional and technical designs and realisation. While the exact naming of these concepts is dependent on the selected development method, it is not required in order to indicate the relationship between the system development and test process at a general level.


35

Figure 2.3 The left side of the V model

EvaluationsDuring the system development process, various interim products are developed. Depend-ing on the selected method, these take a particular form, content and relationship with each other and can be tested on these.

DefinitionEvaluation is assessing the intermediary products in the system development pro-cess.

In the V model, the left-hand side shows which interim products can be evaluated (against each other). In evaluation, the result can be compared with:

The preceding interim product. For example, is the functional design consistent with the •technical design?The requirements from the succeeding phase. For example, can the builder realise the •given design unambiguously and are the specifications testable?


36

Other interim products at the same level. For example, is the functional design consistent •internally and with functional designs related to it?The agreed product standard. For example, are there use cases present?•The expectations of the client (see box “Realised Requirements”). Is the interim product •still consistent with the expectations of the acceptors?

With this, various techniques are available for the evaluation: reviews, inspections and walkthroughs (see also “TMap NEXT” [TMapNEXT, 2006]).

In More DetailRealised RequirementsWhat about the trajectory of wish, legislation, etc., to product? Will, for example, all the requirements be realised, or will something be lost along the way? A survey carried out by the Standish Group unfortunately shows a less than encouraging picture. The findings of the survey (see Figure 2.4), in which the percentage of realised requirements was determined, shows that, of the original defined require-ments, only 42% to 74% are actually realised by the project [The Standish Group, 2011].

Besides normal evaluation results (the finding of defects) a well organized and executed evaluation process can deliver a contribution to a higher realisation per-centage in respect of the original defined requirements.

Figure 2.4 Realized requirements


37

Test Levels and Responsibilities

In a system development phase, a separation can be made between the responsibilities of client, user, manager and system administrator on the one hand and system developer and supplier on the other. In the context of testing, the first group is collectively known as the accepting (requesting) party and the second group as the supplying party. Other concepts that are also mentioned in this connection are the demand and supply organisations. At a general level, there are two possible aims in testing:

The supplying party demonstrates that what should be supplied actually is supplied.•The accepting party establishes whether what has been requested has actually been •received and whether they can do with the product what they want to/need to do.

Right Side of the V ModelIn Figure 2.5, a horizontal dotted line indicates this (formal) separation. In practice, the sepa-ration is less concrete and system developers will be called in for the information analysis and the setting up of the specifications, and will also supply support with the acceptance test. The expertise of users and administrators will also be employed in the building activi-ties. It is important to define the various responsibilities clearly. This certainly applies to the testing. Who acts as the client of a test, who accepts it, who wants advice on the quality and who will test what, and when?

Testing takes place at the right-hand side of the V model. With this, a distinction is often made within the supplying party between testing by the developer and testing by the project/supplier:

Testing by developer. For example, by a programmer, technical designer or engineer.•Testing by project/supplier. For example, by a project or supplier of software or package •supplier, or maintenance organisation.

In practice, this distinction in (test) responsibilities is translated into the grouping of test activities into test levels.

DefinitionA test level is a group of test activities that are managed and executed collectively.

For every phase of construction, there are one or more test levels. A misconception in this is that the test level rectangles in the V model are seen as phases of the system develop-ment process. However, these represent the test execution (the measuring phase) of a test level.


38

Figure 2.5 places the development and system tests under the responsibility of the supplying party and the acceptance tests under the responsibility of the accepting party:

Development • tests: tests in which the developing party demonstrates that the product meets the technical specifications, among other things.System • tests: tests in which the supplying party demonstrates that the product meets the functional and non-functional specifications and the technical design, among other things.Acceptance • tests: tests in which the accepting party establishes that the product meets expectations, among other things.

Figure 2.5 The right side of the V model

Although test level is a much-used concept, in practice people often have difficulty in sub-stantiating it. It does not appear to be possible to designate the test level set. Even within one company, it is often impossible to define one set that should be used in every project.

In this book, we refer to the three test levels mentioned. In incidental cases, in order to describe a certain case appropriately, these test levels could be further subdivided.

As mentioned previously, there is no such thing as one standard set of test levels. This is simply because it strongly depends on the organisation, the project and even the indi-vidual. But of course, there are some indications available for arriving at a relevant test


39

level categorisation in a particular situation. You can see these indications in TMap NEXT® [TMapNEXT, 2006].

Test Basis and Test LevelsA test level is aimed at demonstrating the degree to which the product meets certain expectations, requirements, functional specifications or technical specifications. System documentation is often used here for reference. In certain situations, usually in cases of migration operations, the current production system may also serve for reference. If there is little, no, or only obsolete system documentation available, the knowledge of, for example, the end users and product owners may be used for reference. There are many sources of information that can be used for reference in testing. The general term for this is “test basis.”

DefinitionThe test basis is the information that defines the required system behaviour.

This is used as a basis for creating test cases. In the event that a test basis can only be changed via the formal change procedure, this is referred to as a “fixed test basis.”

Figure 2.6 uses arrows with the text “Input for” to indicate which information sources can be used in which test level as a basis in order to derive test cases. From the model it also appears that it is possible that the same test basis is being used in two test levels. This often happens when there are two different parties carrying out a test according to their

Figure 2.6 Test basis in the V model


40

individual responsibilities. In the illustrated model, a functional design is used as test basis by the supplying party to demonstrate to the accepting party, for example, that the system complies with this design. However, the accepting party uses the same functional design as a test basis in order to check whether the system supplied actually complies with this design.

It is obvious that in such a situation, there is a chance of duplicate testing being carried out. This can be a conscious and perfectly valid decision, but it can be equally justifiable to combine certain test levels.

Test Types

During the testing of a system, various types of properties can be looked at—the so-called quality characteristics. Examples of these are functionality, performance and continuity.

At detail level, it may be, however, that a certain quality characteristic is too general, making this difficult to maintain in practice. The quality characteristic should then be cast in a form that is easier for the tester to use. In the example of functionality, risks are conceivable in the area of interfaces, in- and output validations, relationship checks, or just the processing. With performance we could look to load and/or stress risks. And in the example of continu-ity, there is the matter of the importance of backup, restore and/or failover facilities. This commonly used form of quality characteristics is called the “test type.”

DefinitionA test type is a group of test activities with the intention of checking the informa-tion system in respect of a number of correlated (part aspects of) quality charac-teristics.

On the website http://www.tmap.net, you will find a list of a number of common test types. This list is not exhaustive and will vary from test project to test project, and from organisa-tion to organisation.

A strange fish among these is the “regression” test type: a test type was, after all, intended to provide detailed information on the specific risks with regard to the relevant quality characteristics, while regression, on the contrary, is a rather general term and in fact cites a specific risk in itself.

DefinitionRegression is the phenomenon that the quality of a system deteriorates as a whole as a result of individual amendments.


41

DefinitionA regression test is aimed at verifying that all the unchanged parts of a system still function correctly after the implementation of a change.

Often, the establishment of whether a regression has taken place is an aim in itself. It is therefore better to pay some attention to it here, where the distribution of quality charac-teristics across test levels is being considered in general terms.

When filling in the detail, thought could be given to what is meant by “correct function-ing” in the above definition. Does this concern, for example, functionality, performance or failover facilities? In fact, all the quality characteristics and test types can be part of a regression test.

What is Structured Testing?

In practice, it seems that testing is still being carried out in an unstructured manner in many projects. This section, besides citing a number of disadvantages of unstructured test-ing and advantages of structured testing, also cites a few characteristics of the structured approach.

Disadvantages of Unstructured Testing

Unstructured testing is typified by a disorderly situation, in which it is impossible to pre-dict the test effort, to execute tests feasibly or to measure results effectively. This is often referred to as “ad hoc testing.” Such an approach employs no quality criteria in order to, for example, determine and prioritise risks and test activities. Neither is a test-design technique employed for the creation of test cases. Some of the findings that have resulted from the various studies of structured and unstructured testing are:

Time pressures owing to: •absence of a good test plan and budgeting method –absence of an approach in which it is stated which test activities are to be carried –out in which phase, and by whomabsence of solid agreements on terms and procedures for delivery and reworking –of the applications.

No insight in or ability to supply advice on the quality of the system due to: •absence of a risk strategy –absence of a test strategy –test design techniques not being used, therefore both quality and quantity of the –test cases are inadequate.


42

Inefficiency and ineffectiveness owing to: •lack of co-ordination between the various test parties, so that objects are potentially –tested more than once, or even worse: not tested at alllack of agreements in the area of configuration and change management for both –test and system development productsthe incorrect or non-use of the—often available—testing tools –lack of prioritisation, so that less important parts are often tested before more –risk-related parts.

Figure 2.7 A structured test approach

Advantages of a Structured Testing Approach

So what are the advantages, then, of structured testing? A simple, but correct, answer to that is that in a structured approach, the aforementioned disadvantages are absent. Or, put positively, a structured testing approach offers the following advantages:

it can be used in any situation, regardless of who the client is or which system develop-•ment approach is used;it delivers insight into, and advice on, any risks in respect of the quality of the tested •system;it finds defects at an early stage;•it prevents defects;•the testing is on the critical path of the total development as briefly as possible, so that •the total lead time of the development is shortened;the test products (e.g. test cases) are reusable, and•the test process is comprehensible and manageable.•


43

Features of the Structured Testing Approach

What does the structured testing approach look like? In general, it can be said that a structured testing approach is typified by:

Providing a structure, so that it is clear exactly • what, by whom, when and in whatsequence has to be done.Covering the full scope and describing the complete range of relevant aspects.•Providing concrete footholds, so that the wheel needn’t be reinvented repeatedly.•Managing test activities in the context of time, money and quality.•

45

3 The Business in Charge of IT: The Cloud

The growth of cloud-based computing is outstripping even the most optimistic predictions. It’s early 2011 and almost all forecasts of “the” most important IT technologies name cloud computing in their Top 3. These forecasts and studies have contributed to hype around cloud computing that is reaching C-level executives. Regrettably, too many times cloud is perceived as an IT matter, a matter that is best left to infrastructure technicians. The truth is that the cloud is not an IT oppor-tunity, but a strategic business opportunity. It creates the ability for the business to be in complete charge of IT and change from Information Technology (IT) to Business Technology (BT). But the cloud market is still at very early stages and will continue to grow and evolve.

The Moment of Transformation for IT

The latest recession has shown that a crisis can hit on a global scale and companies have to react. Receiving money was an issue during this financial crisis [Hoenig, 2009]; it was triggered by a liquidity shortfall in the United States’ banking system and resulted in the


46

collapse of large financial institutions, the bailout of banks by national governments, and downturns in stock markets aroundtheworld. It contributed to the failure of key businesses, declines in consumer wealth, estimated in the trillions of Euros, substantial financial com-mitments incurred by governments, and a significant decline in economic activity.

Within this context, businesses needed to save on costs, wherever possible! And they had to do this in the new world of business—characterized as international, connected and chaotic. As a result, company CxOs are seeking alternatives to reduce costs, improve service, and manage risks. In short, they are looking at the cloud.

Figure 3.1 From LAN applications to cloud services [Hoenig, 2009]

IT has changed over the last 20 years, as Figure 3.1 shows, transforming from LAN applica-tions to cloud services. The cloud growth is based on a compelling value proposition: speed to market, agility to bring forward or retire services, the chance to move expenditure from capital expenditures (CapEx) to operational expenditures (OpEx), and a range of opportuni-ties for reducing costs—the last mentioned is likely to be a key incentive for cloud growth!

In this context, the cloud is not only changing the way people do technology, it is changing the way people do business. It is cloud computing that is changing business, changing products and services, changing markets and even changing innovation itself. It all needs to be flexible!

3 The Business in Charge of IT

47

Product to “Whatever as a Service”

As a result of this flexibility “Whatever as a Service” solutions are created, not just products, but also a complete service. A service based on continuity, recurrence and trust, not a “sell once” principle. As vendors want to keep their clients, they need to sell a service that is of added value to them. These services are to be integrated into the current available enterprise architecture, and, as these services are systems by themselves, they become a systemofsystems [SeizeTheCloud, 2011]. All systems are connected to each other to perform a service to the business.

Information Technology to Business Technology

It must be clear that the cloud should not be a goal by itself; it enables IT to be part of the business and do what it was created for—to support the business when needed, ensur-ing that technology aides the business. Business decisions are becoming more and more dependent on technology. What happens when an email system goes down? Or the inquiry database is corrupted? A lot of business processes, maybe even all, are so technology-enhanced that business and technology are interwoven.

In the future companies will move from Information Technology to Business Technology, where technological innovation and business innovation are synonymous. Technology can bring direct business opportunities. This is not a long-term view, it’s something that is happening now. People and companies are becoming smarter in using technology; for example using social media has completely changed marketing, advertising and even news and public opinion. Other companies have outsourced part of their IT or have no internal IT at all.

The Cloud Era

What is the Cloud Era? To understand this, it’s important first of all to know what the cloud is. According to the National Institute of Standards and Technology (NIST), cloud comput-ing is defined as:

DefinitionCloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, stor-age, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model pro-motes availability and is composed of five essential characteristics, three service models, and four deployment models [NIST, 2009b].


48

Cloud services are derived from cloud computing. Where cloud computing is broadly a technology infrastructure model, cloud services are more of a utility business model. Cloud services are enabledby cloud computing; cloud computing is hidden underneath the busi-ness or consumer service, and most of the time is maintained by the party that offers the service.

IDC provides the following framework to distinguish between cloud computing and cloud services:

DefinitionCloud computing is an emerging IT development, deployment and delivery model, enabling real-time delivery of products, services and solutions over the Internet (i.e., enabling cloud services) [IDC, 2010].

DefinitionCloud services are consumer and business products, services and solutions that are delivered and consumed in real-time over the Internet [IDC, 2010].

In short, a cloud service is virtually any business or consumer service that is delivered and consumed with the use of Internet technology in real-time. Cloud computing, an important, but much narrower term, is the IT environment—encompassing all elements of the full “stack” of IT and network products (and supporting services)—that enables the develop-ment, delivery and consumption of cloud services

In this book, we are referring to the business model where we use the term “cloud services” and the infrastructure model where we use the term “cloud computing.” And the cloud computing definition from NIST is also used in this book.

The Cloud Era will emerge over the coming years as a separate business model on howwedobusiness, not IT. In the Cloud Era, IT will become more of a commodity, like tap water that can be turned on and off when needed.

What Makes a Cloud a Cloud?

It is the characteristics of the cloud which make it possible to create Business Technology. But what are these characteristics? In other words—what makes a cloud a cloud? To answer this question, it’s first of all important to recognize that no one expert agrees on all the characteristics. Figure 3.2 shows the most used characteristics, which are based on the five characteristics of the National Institute of Standards and Technology [NIST, 2009b] and three added characteristics. These are outlined below.


49

On-demand self-service. Cloud computing resources can be procured and disposed of by the consumer without human interaction with the cloud service provider. This automated process reduces the personnel overhead of the cloud provider, cutting costs and lowering the price at which the services can be offered.

Resource pooling. By using a technique called “virtualization,” the cloud provider pools his computing resources. This resource pool enables the sharing of virtual and physical resources by multiple consumers, “dynamically assigning and releasing resources accord-ing to consumer demand” [NIST, 2009b]. The consumer has no explicit knowledge of the physical location of the resources being used, except when the consumer requests to limit the physical location of his data to meet legal requirements.

Broad network access. Cloud services are accessible over the network via standardized interfaces, enabling access to the service not only by complex devices such as personal computers, but also by light weight devices such as smart phones.

Rapid elasticity. The available cloud computing resources are rapidly matched to the actual demand, quickly increasing the cloud capabilities for a service if the demand rises, and quickly releasing the capabilities when the need drops. This automated process decreases the procurement time for new computing capabilities when the need is there, while prevent-ing an abundance of unused computing power when the need has subsided.

Measured service. Cloud computing enables the measuring of used resources, as is the case in utility computing. The measurements can be used to provide resource efficiency information to the cloud provider, and can be used to provide the consumer a payment model based on “pay-per-use.” For example, the consumer may be billed for the data trans-fer volumes, the number of hours a service is running, or the volume of the data stored per month.

3rd party integration. As cloud services are only created with one focus and are highly standardized one single service doesn’t cover the full service portfolio of any consumer. Specialized services are created to cover that one part and integrate with other providers. This 3rd party integration through standardized methods leads to an integrated solution based on multiple services.


50

Figure 3.2 Characteristics of the cloud and cloud services, based on NIST definitions and Sogeti research

Multi-tenancy. Enables sharing of resources and costs across a large pool of users thus allowing for centralization of infrastructure in locations with lower costs (such as real estate, electricity, etc.), peak-load capacity increases (users need not engineer for highest possible load-levels), and utilization and efficiency improvements for systems that are often only 10–20% utilized.

Device and location independence. This feature enables users to access systems using a web browser, regardless of their location or what device they are using (e.g., PC, mobile). As infrastructure is off-site (typically provided by a third-party) and accessed via the Internet, users can connect from anywhere.

What Is Not a Cloud?

While a degree of hype surrounding cloud has been mostly within IT, soon it will jump over to the wider business, with some concerns around the use of the cloud. These concerns refer to cloudwashing in the industry, whereby companies re-label their existing products as cloud computing resulting in a lot of marketing innovation on top of real innovation. The result: more overblown hype surrounding cloud. The cloud may be seen as over-hyped and misunderstood in the short term, but it is revolutionary in the long term, representing more of a gradual shift in our thinking about computer systems and not a sudden trans-formational change. This is explained by Forrester Research VP Frank Gillett in an interview he did in 2008 [Gillett, 2008].

There is even a “Cloud Washing 101” [Glauser, 2010] available for “wannabe cloud wash-lings,” which shows the following playbook:


51

Create a new consulting service to help cloud customers to feel assured.•Dust off old CD boxes, apply white-out liberally, change words like “network”, “storage” •and “software” to “cloud” and “SaaS.” Give no thought to the actual meaning of any terms.Form an industry consortium for the purpose of gathering luddites in the spirit of creat-•ing standards that will save the dinosaurs.Announce cloudy partnerships simply because a fusion of synergy always means •1 + 1 = 3.Call everything a cloud service just for effect, even if it is simply a basic IT service. Cloud •is a much bigger tent than any of us ever realized.Convince customers they need cloud management tools in addition to the manage-•ment tools they already have. Apparently the cloud broke the old tools, or was that the Internet’s fault?

Adopting the Cloud: A New Global Infrastructure

As the cloud evolves, and cloud service adoption becomes ever more wide-ranging, a new global infrastructure is being created. This infrastructure can easily be connected to tra-ditional infrastructure (including legacy systems), as shown in Figure 3.3. But it is not just for business IT assets that the cloud removes previous limitations. It does the same from a software or application testing perspective, removing the typical constraints presented by having to test on client-owned or internal resources. A cloud infrastructure creates signifi-cant new opportunities for software quality assurance and testing. It enables convenient, on-demand network access to a shared pool of configurable computing resources. Its great-

Figure 3.3 Connecting cloud environments with traditional environments


52

est benefits lie in the scalability, the ability to reduce costs, greener IT, location-independent access and an instantly deployable infrastructure.

How is a cloud scalable? It’s a bit technical, but clouds utilize the option to distribute the usage over multiple servers (evenly). This enables the infrastructure, platforms and soft-ware to be distributed over these different servers, thus enabling the opportunity to create scalable services. And the business is waking up to the fact that scalability is as important to their success as access to capital and markets. They usually don’t know how important scalability is until their systems fail. When customers and revenue are lost, they then look into the underlying causes and find their systems don’t really have the performance capability they thought they did. Unfortunately cloud platforms are not a magicsolutiontoscalability, as they are not infinite, but the cloud makes it feasible and affordable to rescale by powers of ten rather than multiples of two, which obviously helps to overcome performance bottlenecks for applications.

Lowering Total Cost of Ownership: No More Maintenance

The cloud promises cost reductions by lowering maintenance costs and the Total Cost of Ownership (TCO). But how? The use of standardized services can create lower maintenance costs. Multiple versions or types of software generates the need for maintenance for all these versions or types. When only one standard application or service is used to provide the needed functionality, the amount of maintenance will decrease significantly. The instal-lation of cloud applications is easier, since they don’t have to be installed on each user’s computer. And as only one version of the application is used, it is also easier to support and to improve, since the changes reach the clients instantly.

Other opportunities that reduce the maintenance costs are a more effectiveutilization of resources. Most systems are only utilized for 10-20%, and when multi-tenancy is applied on resources, their efficiency rate can go up. As Figure 3.4 shows, instead of using separate environments for each project, they can all be done on one environment with peak utilization of around 13 and the least at 5; this compared to more (or less) availability than needed. Multi-tenancy can also provide centralization for infrastructure in locations where the costs, such as real estate, electricity, etc, are lower.


53

Figure 3.4 Environment utilization rate ‘traditional’ vs. cloud

When all costs for the service, used on a per-need basis, are shared with many other sub-scribers in the cloud, this creates the flexibility to lower TCO of the services compared to traditional usage of a product.

The higher efficiency ratio of resource utilization with cloud computing offers another ben-efit: it makes a greater contribution to the environment than normal data centers. By sharing cloud resources for their test infrastructure, businesses will use IT resources “on demand”


54

and therefore eliminate waste. In addition, clients using cloud data centers can minimize energy use and deliver environmental savings in CO2 of up to 55%.

As working with cloud services is an online service, it generates a location-independent access to these services, for example the use of cloud-based storage providers like Dropbox or “Office” applications with Google Apps and Microsoft Office 365. This enables working with “The New World of Work.” You can login to these services wherever and, as they are available 24/7, whenever. Please make sure you have an Internet connection!

Cloud computing is Internet-based computing. This generates an infrastructure in the cloud and this infrastructure can be one like in Figure 3.5. And all the major cloud service provid-ers, like Amazon, Rackspace Cloud, Salesforce, Skytap, IBM, Microsoft and Google offer this as a service to clients, see Figure 3.5. This provides us with the opportunity to instantly deploy needed (test) environments. When virtualized, parts of the infrastructure are incor-porated in the cloud, they can be used on-demand. This not only makes the implementation easier but also the execution of the different instances in the cloud. The virtualized parts of the infrastructure can be added or removed from the cloud whenever needed, creating a flexible option to create infrastructure (including the needed configuration).

Figure 3.5 Who are offering the cloud?

Naturally, there is an understandable nervousness about this new approach and questions are being asked about integration, security and implementation. But these challenges are outweighed by the advantages, as cloud gives business the opportunity to free themselves from issues relating to the limitations caused by internal availability of IT resources (hard-ware, applications and services), and enables a more effective way to collaborate.


55

The Creation of Business Technology

Cloud computing has evolved to cloud services and into “the Cloud.” But why has an IT devel-opment led to a business service? This is because IT has not lived up to its role in supporting the business. The last decade it has mostly been a cost to the business, slow to react to the business needs and seemingly trying to shape technology execution to what IT wanted or what was good enough, rather than what the business needed. Today the characteristics of the cloud can make cloud an aid to business and turn IT into BT. Business Technology is the conjunction between IT, business models and the consumers, see Figure 3.6. The technology aids the business to manage its relationship with the market.

Figure 3.6 Business Technology: where IT, business and consumers congregate join

Business Technology = IT as a Commodity?

As the cloud promises in part that technology will become more business-driven, it is still based on the interaction between business and technology. The cloud shifts IT from a separate entity to something embedded in everyday business.

Technology is part of everyday life, in business as much as in our personal life. Take away software, networks and the Internet from any organization and it would come to a standstill almost immediately. This is more so now than ever before. In the recent past, we could prob-


56

ably manage without technology for a day or two. But today it’s more than just a greater dependency: we are no longer doing the same things. Supported by technology, we are starting to do newthings that we simply could not do without that technology.

But we have also got used to software as a normal and even standard part of daily life. Some types of software are so intrinsic to everyday life that we cannot function without it, like email. As these software products are standardized applications and have found their way into all parts of our organization, the minute they fail, they show us how much we need them, and CIOs can get fired as a result. Looking closer at what we now demand of Information Technology, it becomes clear that cloud is a good model for provisioning and paying for technology. It will give a boost to whatever we are doing.

However despite all these developments, we are not yet fully realizing the opportunities that exist today. It is easy to be blinded by the small everyday issues, so that the larger goal stays out of reach. Now that businesses and public organizations are “wired” and the cloud has emerged as a collection of Internet platforms and tools for connecting, integra-tion and sharing of data and processes, it has become possible to think about the bigger issues that are going unaddressed. ITwillbecomeacommoditythatcanbeturnedonandoffwhenneeded!

57

4 “Whatever” as a Service: Cloud-Enabled Software Testing as a Service

As the importance of IT testing is growing, demand for services is also growing. Businesses want a service to provide them with on-demand, pay per use testing, but still against the correct quality. Some impor-tant drivers for this are:

Business demands are higher and expectations are for “first time right” •software launches;Legislation and regulations (e.g. SOX, SAS70, Basel II/III act and Clinger •Cohen act); put stronger the business demands on quality assurance and test processes;Mergers, chain integrations, globalization and technological develop-•ments lead to more complex IT chains, andBusiness demands swift, flexible, high quality and cost effective IT ser-•vices that contribute to the business processes.

IT has become a utility. The business departments’ demand guarantees from IT services that IT implementations will not threaten business continuity. The business department demands


58

a test process that clearly demonstrates that requirements have been sufficiently met, and that risks for deployment are acceptable. Testing will become a utility also.

Test service providers who can offer the test process as demanded by the business depart-ments will be very successful, especially if these providers always: make the client’s objec-tives highest priority and commit to focusing primarily on the success of the client’s business; making the testing of IT move into the testing of BT. A robust and successful collaboration with the client is founded on the skills of the providers’ test professionals, highly industrial-ized test processes, open communication, flexible use of tooling and infrastructure, and full transparency regarding objectives, measurable results, responsibility, operation procedures and costs. The model to support this all is called: SoftwareTestingasaService.

The cloud is divided into three types of layers, named IaaS, PaaS and SaaS. Software Test-ing as a Service is executed as part of all three of these layers as it can be used to test the infrastructure, platform and software, or all combined, see Figure 4.1.

Figure 4.1 STaaS and its effect on the three different cloud layers

4 “Whatever”as a Service

59

Cloud Service Models: “Whatever” as a Service

Infrastructure as a Service (IaaS)

This is the foundation of cloud computing as we now understand it—the remote provision of process capacity, storage, networking and other basic computing resources. An orga-nization simply contracts for the necessary amount of virtual infrastructure and installs its operating system, applications and data. Depending on the cloud provider and com-mercial model, these resources can be highly flexible and scalable, responding and billing the client, according to demand and load. On one hand, this approach gives the client an extraordinary amount of freedom, but on the other hand, the client needs to exercise close management. Availability and reliability of the rented infrastructure, including “fall over” and data protection, is the responsibility of the cloud provider.

Platform as a Service (PaaS)

For this service, the cloud provider offers a programming platform and tools for the client to develop its own applications. This development platform is mostly based on .NET or Java, extended with cloud-specific services. As such, it has great appeal for organizations who want to build their own software, without the need or costs of building and running their own data centers. In addition, they benefit from effectively outsourcing the manage-ment and maintenance of the underlying cloud infrastructure (network, servers, operating systems and storage).

Software as a Service (SaaS)

Here the cloud provider offers complete applications for end users, created on PaaS. Only the service is paid for, normally on a unit cost basis. The software does not run locally on a machine but in the cloud and is delivered via a browser. The most well-known example of this would be Google Apps and Salesforce.com. The client only needs to very lightly configure the application before its immediate use.

The downside of this simplicity is that the client has very little influence on how the applica-tion works or on its future development life cycle. The amount of control on the development life cycle is governed by the SaaS provider, acting on behalf of a massive client base, passing on to those clients the economies of scale and investment efficiencies. Here, as elsewhere, clients benefit from moving spend out of CapEx and into OpEx, while retaining an ability to move as fast as the marketplace in which the SaaS sits.


60

Software Testing as a Service (STaaS)

Software Testing as a Service (STaaS) is a model for software testing used to test an appli-cation as a service provided to clients across the Internet. By eliminating the need to test the application on the client’s own infrastructure and equipment with testers on site, STaaS alleviates the client’s burden of installing and maintaining test infrastructure, test tooling, sourcing, and (test) support. Using STaaS can also reduce the costs of testing through less costly, on-demand pricing.

From the STaaS provider’s standpoint, STaaS has the attraction of providing stronger protec-tion of its test approach and establishing an ongoing revenue stream. The STaaS provider can test the application on its own server or, even better, on a cloud infrastructure. This way, the client can reduce their investment in server hardware too.

Drivers for STaaS Adoption

The traditional rationale for test outsourcing is that by applying economies of scale to the testing of applications, a test service provider can test better, cheaper and faster than com-panies can themselves. STaaS could be the next step in test outsourcing. Several important changes made to the way we work could make a rapid acceptance of STaaS possible:

•The testing industry has matured into a standard practice. In the past, executives viewed corporate test centers as strategic investments. Today, people consider testing to be a cost center and, as such, it is suitable for cost reduction and outsourcing. has become commodity, so hastesting!

•Testing by companies themselves is expensive. In-house testing activities require expensive overhead including, for example, salaries, health care, liability, and physical building space.

•Standard test approaches are available. With some exceptions, testers can use a standard test approach to test any application. For instance TMap® Next [TMapNEXT, 2006] can be used for multiple types of applications.

•A specialized testing provider can target global markets. A testing provider special-ized in testing widespread applications (packages) can more easily reach the entire user base.

•Security is sufficiently well trusted and transparent. With the broad adoption of SSL, VPN and Citrix, testing providers have a secure way of accessing the applications under test. This still allows the environments to remain isolated from each other.

•Wide Area Network’s bandwidth has grown drastically. In addition to network qual-ity of service improvement, it is now possible for testing providers to successfully and consistently access remote locations and applications with low latencies and acceptable speeds.


61

•The cloud creates the availability of computing power is available when needed. Cloud computing enables us to create on-demand test infrastructure. The client is reas-sured that (test) infrastructure is no longer scarce and is not a mysterious environment that must be carefully managed. Test infrastructure has the capacity and flexibility to be quickly increased and decreased without upfront investment, moving expenditure from CapEx to OpEx.

•Testing tools become available on an on-demand, pay-per-use basis. Cloud provides the opportunity and flexibility to use those tools as and when needed. Tests that need specific tools can be executed at the moment that they are needed and only paid for when being used.

How Does That Work: The STaaS Process

Software Testing as a Service is a model of software testing in which an application is tested as a service provided to clients across the Internet. The process is that a client has a test demand, which is sent through the Internet to a STaaS provider. After the testing is completed, the STaaS provider sends the client a test report (see Figure 4.2).

Figure 4.2 The interaction between the client and STaaS provider: the STaaS process


62

What happened in between? How did the provider deal with the test demand? Did the provider use a “real-timeSTaaS” or a “real-enough-timeSTaaS”? In addition, how did the provider deal with other challenges like test infrastructure, 24/7 availability and the client/provider communication?

Real-Time and Real-Enough-Time STaaS: Using a Web Interface

In a real-timeSTaaS (Figure 4.3), the test demand is implemented without human interven-tion by the provider. In the ultimate form of the real-time STaaS, a test object (e.g. applica-tion software), including test basis (e.g. requirements, use cases, set of heuristics), design and architecture model, is offered to the STaaS provider. Without human intervention this is implemented in a (cloud-based) test environment. All the testing is performed by human simulators against the model and neural network forecasting, and a test report is sent to the client. Is the scenario above real-time STaaS science fiction? No, with the enablement of cloud environments and the use of pay-per-use tooling real-time, STaaS is a reality.

Figure 4.3 The real-time STaaS interface

Optional services that are already available are:

Performance Testing as a Service (PTaaS): The periodical execution of performance mea-•surements or tests of applications from various worldwide locations;Security Scanning as a Service (SSaaS): Self-service scanning of applications with various •forms of reporting available;Test Cloud Load Testing: The periodical execution of usage loads for various applications •around the world, using the cloud as a load model;Regression subscription to periodically check the external and internal links on a web •site. Are the links for instance still working correctly and not broken?Regression subscription for application interfaces in a suite of applications. Monitoring •the health and functionality of the application landscape.Test Infrastructure in the Cloud: The provisioning of underutilized test infrastructure in •a cloud environment, enabling the creation of multiple environments as needed.


63

In a real-enough-timeSTaaS (Figure 4.4), the test demand often requires human interven-tion in the workflow. The demand is carried out behind the “scenes” by many humans, although it appears as if the test demand is carried out by computers. By its very nature, this introduces a latency and unpredictability to the STaaS process.

Figure 4.4 The real-enough-time STaaS interface

Available services of existing real-enough-time STaaS:

•Work Package broker. Through a formal test demand mechanism, everything about the assignment is specified in a work package. This includes items such as: what should be tested, how and when, what criteria should be used, and what knowledge is necessary. The work packages are stored in a kind of virtual (digital) cupboard. The work package can be pushed by the Work Package broker to, or pulled by, a tester or work package team that possesses sufficient means to carry out the work package. The work package serves as the contract between client and provider.

•Managed Testing Services (MTS). MTS is the structured form of a Work Package broker that is specialized for a particular client or application. Through MTS, the provider takes full responsibility for test assignments, with clear commitments expressed in KPIs on qual-ity, cost level and time to market. MTS is organized in so-called test lines—an operational organization to provide test services to one or more clients. A test line has a fixed team of testers, infrastructure, test tools and standardized work procedures. Every test line has a permanent key team of testers who ensure continuity and knowledge retention. There is also a flex team; when the work available on their test line is insufficient, the flex team is temporarily assigned to other test lines. It is a flexible pool of testers deployed to test lines with the most work pressure.

Test Infrastructure: In the Cloud

With STaaS it should be possible to test an application from anywhere in the world, regard-less of the location of the tester and the client. This requires special attention to the test


64

infrastructure. Figure 4.5 contains an existing and operational infrastructure used by a provider.

Figure 4.5 An example of Test infrastructure in the cloud

The tester, either from home or the provider’s office, has a remote connection tool on his computer by which he or she can establish a connection to a public, private or hybrid cloud infrastructure where the System under Test (SuT) is hosted. Based on a certain clas-sification, the tester can log onto the cloud environment just as any other environment. If a connection with the client’s intranet (or private cloud) is needed, a secure tunnel is set up. For example, cloud-based virtual workstations connect via a VPN connection to access the client’s intranet or private cloud. In this way, multiple clients can be connected on the cloud, each with its own virtual machines, and security is guaranteed.

Other possibilities are:

hosting the test infrastructure by the STaaS provider, and•outsourcing the test infrastructure to a third party hosting provider.•

Availability: 24/7

When it is possible to test an application from all over the world through the Internet, the provider and his testers should be available 24/7. In this situation, a test demand is not


65

rejected just because it is the night where the provider is situated. The provider just needs a broad network of testers spread across the different time zones, or needs testers avail-able 24/7 in a specific time zone. Because the demand for testing services will fluctuate, it is recommended that the provider has a fixed pool of testers and a pool of flex testers (Figure 4.6). In practice, students have proven to be very suitable as flex testers; they like to work in virtual environments, are time-independent and location-independent and can be paid per assignment.

Figure 4.6 STaaS resourcing model in a fixed pool of testers and flex testers

In addition to the capacity of the fixed pool, the flex testers provide the required flexibility to cope with peaks and variation of required workload. Assignment of the flex testers is based on the planned test capacity demand and agreed reaction times. The learning time of flex testers is usually relatively short due to the provider’s standard working practices. In principle the flex testers leave the fixed pool at the end of the demand peak.

Of course, using test tools and test automation could also support a STaaS provider’s 24/7 availability.

The STaaS Governance Model: Test Demand vs. Test Supply

The STaaS provider has to distinguish the various interaction points where the client and provider can interact and communicate. Figure 4.7 gives an overview of the generic gover-nance model used by a STaaS provider.


66

•Client contract manager vs. Provider delivery manager. Agreement on a strategic level regarding contracts and SLA. At this level there is a responsibility for setting up the contracts and SLA.

•Client manager test services vs. Provider test line manager. Set up and maintain the standard procedures and KPIs. Initiate and start work packages.

•Client project manage vs. Provider test manager. Planning and monitoring progress of test activities. Progress reporting, defect reporting and management. Delivering the conclusive test report after finishing the test.

•Client development teams vs. Provider test coordinator. Findings after the testability review are reported. If required, testers and designers meet in a session to clarify the findings. The result is a clear unambiguous test basis.

•Client development teams vs. Provider test coordinator. Intake of delivered software. The initial test is performed on the basis of agreed entry criteria. Issues regarding the initial test are reported to project management and development. If required, a meet-ing is set up to clarify issues, and the result is a system with sufficient quality to start test execution.

•Client development teams vs. Provider test coordinator/test engineer. Re-test of resolved defects. Through a delivery document, the developer lists which defects are resolved in the new build. This document is the basis of the re-tests.

Figure 4.7 Governence model of a test line


67

•3rd party SaaS supplier vs. Provider test coordinator/test manager. Integration of the SaaS software in the SuT. Through a delivery document the supplier lists how to install and integrate the SaaS software in the system. This document is the basis of the system integration testing.

STaaS Provider ServicesA service item is a certain element of the test process offered to the client for which the STaaS provider is responsible, and can be highly varied. Moreover, the established service offering can be modified when new services are proposed or existing ones are eliminated. The STaaS provider must deliver a result based on the demand. The delivery must occur within the pre-defined timeframe, at pre-defined costs, and at a pre-defined quality level. The provider is responsible for guaranteeing continuity in delivering the result.

Some typical service items are, as shown in Figure 4.8:

Usability Testing;•End-2-End Testing;•Performance Testing;•Load Testing;•Assessments, like TPI NEXT• ® or other scans;Test Infrastructure Management;•Security Scanning;•Test Data Management;•Cloud Quality Assurance or Cloud Assessment Testing;•Structured Evaluations;•Regression Testing, and•Test Script creation, using Model Driven Quality Engineering.•

But also:

Security Testing;•Defect Management, and•Testability Reviews.•


68

Figure 4.8 Cloud-enabled Testing Services that can be delivered by a STaaS provider

In More DetailPerformance Testing on the CloudsA performance test is usually done at the end of the test phase. This because of the need of a well enough performing environment. The environment that is most production-like is the acceptance environment. This creates a risk in applications that are highly dependent on a high performance, because the defects are found late in the process and are therefore expensive to fix. But how can we help this?

With a test cloud this can be done in every environment. Whenever the infrastruc-ture needs to be upgraded to a production-like infrastructure this can be achieved in the cloud. After the needed performance tests the environment can again be decreased.

With the use of the needed infrastructure from a cloud a more “real” load can be generated than the virtual load from arduous tools. A cloud enabled performance test tool, like for example Cloud Test from SOASTA (see Figure 4.9) works with the cloud to generate the needed load and stress to test an application (which is or isn’t in a cloud environment).


69

Figure 4.9 SOASTA architecture model for a cloud-based performance test using an IBM Cloud

STaaS Provider Process Model

A number of processes have to be set up by the STaaS provider to offer these services. The STaaS provider process model (Figure 4.10) consists of two parallel primary processes:

The process for the actual execution of the service in an assignment;•The process that supports and monitors the execution.•

The processes serve to support the assigned employees’ collaboration needed to accomplish the contracted services. The processes are described in detail below.


70

Figure 4.10 STaaS provider process model

Initiation

This is the first phase for execution of the assignment. The assignment always comprises one or more service tailored to the client’s specific demand. The initiation phase serves to describe accurately the scope of the assignment. This can be done by creating a so-called assignmentdescription and as an option asking the client to approve it. An assignment description concretely describes:

the STaaS service asked for, including preconditions and basic assumptions;•clear commitments expressed in KPIs on quality, cost level and time to market;•the infrastructure to be tested on/where the SuT is installed;•the agreements on monitoring by the STaaS provider in relation to communication lines, •progress reporting and consultation, andthe deliverables.•

Furthermore, the initiation phase is used to identify what is available in the provider’s orga-nization for (re)use on behalf of the assignment. This may include templates, standards, existing test scripts, test models or test design patterns from previous assignments, test environments and tooling.


71

In More DetailTest Design PatternA test design pattern is a generic set up test structure and/or test strategy, which solves a specific common type of test design problem. Test design patterns are generically described, offering the advantage of a recognizable solution pattern, regardless of the implementation details. Using test design patterns accelerates the communication of a test assignment because the solution of a common test design problem has, in fact, been given a “name.”

Execution

In this phase, the assignment is executed, conforming with the agreements with the client as described in the assignment description. Furthermore, the parties communicate via the agreed communication lines on the results, progress, risks and bottlenecks in the execution of the assignment.

Completion

Reuse of resources is one of the success factors of the provider. In this phase, the assignment is assessed and a satisfaction measurement is agreed with the client. The lessons learned from the assessment are fed back into the provider’s organization and incorporated into the new version of the service. This results in formal process improvement embedded into the processes of the provider’s organization.

Support and Monitoring

The provider continuously supports and monitors the assignment process as described above. The progress, risks and bottlenecks involved in the execution of the assignment are monitored. Where and when necessary, the involved parties reach new agreements on the assignment.

Delivery Management

This process covers activities that aim to acquire assignments for the provider and man-age (long-term) relationships. Examples are maintaining the test environment,1 repeated testing of releases, and live monitoring of applications. A contract is created to govern how both parties will handle the assignment. It specifies agreements on the service level provided by the provider.

1 When the test environment is located in the cloud, this can be turned off to save costs.


72

Security and Compliancy

The test environment is created on a cloud-based infrastructure. The well-publicized nervous-ness around security in the cloud is an issue that all service providers are working hard to answer. Fundamentally the nature of cloud computing means the data of one consumer is often stored alongside the data of another. To some extent that challenge is being met through encryption, which is often used to segregate data-at-rest, but this is not a cure-all and a thorough evaluation of the encryption systems used should always be undertaken.

Test data is frequently sensitive and its location is therefore important, since data entering or exiting national borders can contravene national and international regulations, such as the EU Data Protection Directive. To address this, the STaaS solution is transparent about the geographic location where data and services are stored, and it allows clients to keep data on their own servers, using a VPN connection.

Planning

The planning process ensures that the “right” tester is deployed to each assignment. “Right” in this context means that the knowledge and competencies of the tester match the knowl-edge and competencies required for the assignment. Other planning aspects are:

required availability of the tester (during working hours, weekends, 24/7);•required or available location (office, home, offshore, nearshore , onshore);•required or available (cloud) infrastructure (public, private or hybrid cloud);•required or available test tooling, and•technology (bandwidth and processing power availability).•

Service Management

The range of services provided by the STaaS provider is not set in stone—it may grow or recede. To this end, it must be determined periodically whether the current service offering is in line with the requested services. In addition, services must be known (to the client, assignment management and tester) and the products for the services must be up-to-date and in line with the latest developments.

Human Resource Management

The process of human resource management aims, among other things, to continuously develop the skills and career of the provider’s testers. This requires matters like defined job positions with associated competencies, continued training and remuneration levels.


73

Financial and Operational Management

Financial management is a continuous process based on budgeting (what are the expected costs and benefits) and monitoring (what are the actual costs and benefits). Operational management can be executed based on many factors. Examples of these factors are:

the percentage of assignments completed within the agreed key performance indicators •(KPIs) on quality, cost level and time to market;pay-per-use of the needed infrastructure, tools and resources per agreed timeframe, •andthe percentage of test services acquired as compared to test services acquired by com-•petitors.

The test cost reduction was achieved through the following measures:

Measure Short explanation Level of cost reduction

Resource rational-ization

Assign tasks to employees with matching seniority level. Focus on healthy ratio for test management vs. test coordination vs. senior test engineers vs. junior test engineers.

Combined these three measures have led in practice to cost reductions of 5-15%.

Sufficiently lean core team

The size of the key team is adjusted to the highs and lows of the mid-term forecast in such a way that the average level of occupation for the key team is > 95%.

Alternatives for idle time

Within testing dealing with idle time is a common phenomenon. Idle time for a test team can rise to 20% of the test effort. By using the economies of scale of the test line a very flexible process is set up that allows prompt re-assignment of testers to other projects in the case of idle time. In practice the test line has proven to reduce idle time to a level below 5%.

Uniform process Install a uniform test process, with standardized test products and procedures. Maintain a key team to use and re-use the process and test deliver-ables in multiple projects.

By installing a uniform test process team cost reductions up to 5% of original test costs have been achieved.


74

Measure Short explanation Level of cost reduction

Test automation Proper use of test automating con-tributes to test cost reduction.

The use of test automation and the needed test tools on demand has resulted in a com-bined cost reduction up to 25% of original test costs.

Test Tools (as a Service)

The test tools were only used when needed, resulting in a more flex-ible use of the tools and even the integration of new test tooling to accelerate the testing process.

Near shoring and off shoring

Through off shoring and near shor-ing testing activities are transferred to regions with lower cost rates. In practice the amount of off/near shoring depends on certain condi-tions and varies from 0% to 70% of all testing activities.

Taking into account the invest-ment cost (translations, remote connections, extra QA and communication effort) test off shoring has resulted in cost reductions varying from 10% to 30%.

Cloud infrastruc-ture

By moving the existing test in-frastructure to a (public) cloud infra-structure the costs for stand-by time and management were reduced. Only when the environment was needed it was turned “on.”

No CapEx was needed and only OpEx for the use of the cloud infrastructure, this resulted in a cost reduction varying from 25% to 50%.

Table 4.1 Measures taken by the STaaS provider to enable cost reduction

Cloud-Enabled STaaS: The Conclusion

The Benefits

Thanks to survival of the fittest, and increasing competition, STaaS providers are constantly looking to upgrade their services to ensure that they provide their clients with the best full test service solution available. Therefore, these providers have to make sure they:

Use the scarce expertise on structured testing, infrastructure and tools optimally to •create a complete service;Work on improving the test processes continuously, by using new insights, automated •test case execution and design, and reusing the available information;Use the different qualities and skills of an international and professional test capacity •from multiple locations around the world to provide an constant available workforce to prepare and execute the testing;Create an on-demand test service, either as a single service or embedded in long term •service contract;


75

Industrialize the test services (“test factory”), by using not only tools for test manage-•ment, but also for test design and test execution;Produce reliable test product quality to provide a thorough insight in the applications •risks;Have an available cloud-based test infrastructure with on-demand tooling to perform •the testing;Have a competitive pricing model on a pay-per-use and fixed-price basis;•Give advance insight into costs and lead time.•

With all this, the STaaS provider takes full responsibility for test assignments, with clear commitments expressed in KPIs on quality, cost level and time to market. A solution should be available for single applications, full projects, and portfolios. STaaS leads to cost opti-mization as well as demonstrable improvements in quality of testing, test process, test deliverables, test results and flexibility of test operations.

The Challenges

A STaaS provider must devote continuous attention to a number of challenges. Managing these challenges is critical for the provider’s long-term success:

The provider must determine on a continuous basis whether the services offered still •match the client’s demand. The client, not the provider, determines the required quality level;The professionalism of the provider is based on the knowledge and competency of the •testers on the one hand, and the stability of the tester population on the other. If there is a continuous inflow and outflow of people in the “pool of testers,” there is no stability and no solid basis for knowledge building;Often an important client objective is cost savings. One way to achieve this is by archiving •test ware, test data, and (cloud-based) test infrastructure for reuse;Continuous attention to optimizing, organizing and refining test objects as an intellectual •property is critical;As a provider, it is important to render an objective assessment of the delivered software •or hardware, independently of the client. On the other hand, the client may have other interests (less costs, short time-to-market). This is an important challenge that may pose contradictions;The test basis (e.g. requirements, use cases, design specifications, heuristics) should be •available in English or translatable into a language that is understood by the testers, preferably in a clear and simple form using tools such as HP QualityCenter Requirements or IBM Rational Requisite Pro;If the quality of the test basis is inadequate, an alternative way to gain domain knowl-•edge is needed;


76

The test infrastructure must comply with rules and regulations stated by the client, but •also the country or union the test data resides in. The use of a cloud deployment model must take into account the security and local compliance;And last but not least, test environments should be accessible from various locations.•

77

5 Testing Cloud Strategy: A Move to 3D

Is testing in the Cloud Era so different from how we have been doing our testing? Yes and no. Testing applications on the cloud is the same as testing applications on a traditional infrastructure. Only what is tested is different. With cloud there are a lot more parties involved in testing: not only the client and the stakeholders, the business, but also 3rd party suppliers of standard or SaaS applications. There are also additional quality attributes due to the cloud infrastructure, and non-functional requirements become more important.

These new items to test make testing the cloud different, not even taking into account the consequences of the cloud business model. The on-demand use of resources, testing tools and infrastructure provides the opportunity to create a pay-per-use test service: SoftwareTestingasaService (STaaS). With STaaS the benefits of the cloud as a business model are used to provide a testing service to clients. There will be more on STaaS in Chapter 6.


78

Creating a Cloud Test Strategy: A Move to 3D

Cloud is not only technology and infrastructure that drives change and provides opportuni-ties for change, but also a business model that thrives on change. When discussing clouds it is therefore important to start on a strategic level, as it is for testing clouds.

Important questions that we need to ask ourselves are, among others, do we have a test strategy that helps us evaluate the potential of the cloud, does the cloud provide oppor-tunities to change the test strategy, how can we assure the compliance of the cloud, but also what are the strategic areas where cloud could be an enabler, and how do we utilize the opportunities the cloud gives us? These are questions that are all valid and that all need to be answered, while keeping in mind the business aspect as we move to BusinessTechnology.

Business in Control: Business Driven Test Management

The exploration of the cloud has to start with the business strategy. Businessdriventestmanagement[BDTM, 2008] puts the business or business case in the driving seat regard-ing a testing project. With that in mind, together with the business interests (client, service owners, users of services and other stakeholders) a (cloud) test strategy can be created. As with testing of traditional applications, the testing of cloud infrastructure and applications is done by covering the most important risks, but with a deeper attention to the three cloud service types of Infrastructure, Platform and Software.

Figure 5.1 The cloud test strategy: leveraging the business, BDTM aspects and the cloud layers

5 Testing Cloud Strategy

79

Why not test all the cloud services as thoroughly as possible? If an organization has an infinite number of resources available, then this is an option. But in real life, no organization has indefinite resources, and as the time-to-market with cloud is important, no organization has the time for that, either. Therefore choices have to be made concerning what must be tested and how thoroughly it must be tested. The core of the test strategy is that what really needs to be tested is tested, to cover the product risks. Neither too little testing, nor too much testing. The business must be certain that the testing covers all potential risks and what might go wrong if the requirements are not met. This means that a risk analysis must be made before a single test case is considered, specified or executed.

Keep in mind that the business also wants to keep the costs of the tests as low as possible and wants to understand the quality of the product as soon as possible. The theory of IT governance classifies project control into four aspects: result, risk, time and costs. Busi-ness Driven Test Management (BDTM) uses these four aspects as the basis for the test strategy. Business driven test management therefore leads the way to the test strategy of cloud applications, it helps create the cloud test strategy by leveraging the business, the three cloud layers in IaaS, PaaS and SaaS, and the four BDTM aspects of result, risk, time and costs (see Figure 5.1).

Business in the Driver’s Seat: BDTM for Cloud

The cloud starts and finishes with the goals and identity of the business. Business driven test management connects seamlessly with this approach. All investments and choices with the cloud, therefore also with cloud testing, are inferred from the interests of the business: the business case.

The business driven test management approach of Sogeti has four aspects [BDTM, 2008]. First BDTM sets the client, the business, at the core. The test manager gives the business influence over the four control aspects: result, risk, time and costs. Second, the test manager communicates the testgoals in the language of the business. Third, with BDTM all tests are based on product risks; the cloud test strategy is based on cloud risk analysis. The fourth and final aspect of BDTM is making the results of tests visible for the business. The steps of BDTM aim to achieve this (see Figure 5.2).


80

Figure 5.2 Summary of the BDTM steps

1 Establishing the assignment and collecting test goals The business formulates the assignment, taking account the four BDTM aspects: result, risk, time and cost. Establishing the test goals aims to determine what, in the eyes of the business, are the desired results of the testing activities for the cloud services. A test goal is a success criterion for the test assignment as specified in the language of the business. The result of this step is the formulationoftheassignment and a testgoaltable.


81

2 Determining the risk class Based on a risk analysis, it is established what must be tested (object parts), on what cloud layer and whatmustbe examined (characteristics): this is the cloud riskanalysis (CRA). The risk class related to the test goals and the relevant characteristics is established for each object part. If multipletest levels are involved, the test levels that must be set up are determined in a plan that covers all test levels (the Master Test Plan). The cloud test strategy also defines, for each combination of a characteristic/cloud layer/object part, an indication of the relative intensity of testing in a specific test level. This step results in a cloudrisktable.

What follows is an iterativeprocess—formulating the test strategy:1

3 Determining the test intensity Deciding whether a combination of characteristics and object parts must be tested lightly, moderately or intensively determines the test intensity. The risk class for each object part, as defined in the previous step, is used as a starting point to determine the test intensity. The initial principle followed here is the greater the risk, the more thorough the required testing.

4 Estimation, planning and feedback An overall budget is established for the test, which is plotted in a planning stage. This budget is confirmed with the business and, depending on their needs, adjusted if nec-essary. In the latter case, steps 3 and 4 are repeated. This gives the business ongoing direct control of the test process, enabling the business to manage it on the basis of the balance between result and risk versus time and cost.

Endofiteration

These steps result in a strategytable.

5 Allocating test design techniques If the business agrees to the budget and planning, the test intensity is translated into concrete statements regarding the desired coverage. This involves allocating test design techniques to the combinations of characteristic/cloud layer/object part, taking circum-stances into account such as the available test basis, projections and experience of the test team. The techniques are used to design the test cases at a later stage.

1 From this point the cloud is like other software and is not other then described in Business Driven Test Management. These steps will not be elaborated further.


82

Figure 5.3 Allocating test design techniques: from test goals to test cases

Figure 5.4 BDTM tables for the cloud


83

This step results in a testdesigntable for each test level.

6 Providing insight and control options Throughoutthetestprocess, the test manager provides the client and other stakeholders with adequate insight into and control options for the IT Governance aspects:

result: the test goals achieved•risk: the risks covered•time: whether the end date—or deadline—is realized•cost: whether the test project remains within the agreed budget.•

The First Step: Establishing the Assignment and Collecting Test GoalsThe testing activities do not always easily correspond to the result that the business wants to achieve. This is why there needs to be a translation of the desired business result into the testgoals. Examples include result definitions expressed in terms of realized/operational critical success factors, business processes, enterprise architecture, time-to-market, agility of the service; other examples involve the contracts, the service level agreement (SLA) and so on. The term “test goal” is used as a collective term.

DefinitionA test goal is a success criterion for the test assignment formulated in the lan-guage of the business (or other stakeholders).

Establishing the assignment and identifying test goals covers step 1 of the BDTM steps. It creates a clear understanding for all stakeholders and the business like of what needs to be done and what does not. Establishing the assignment requires an understanding of the objective of the project, the (project) organization, the setup of the development process, the (cloud) system to be tested, and the requirements with which the cloud must comply. By clarifying the goals and expectations at the start of the process, uncertainty, miscommunication and disappointment at a later stage are prevented. These objectives and expectations are derived directly from the business case.

DefinitionThe business case provides the justification for the project and answers the ques-tions: Why do we need to do this project? What investments are needed? What objectives does the customer want to achieve by using the result of the project?

A (cloud) service can exist on all cloud service types or layers, or two of them, or even just one. See Figure 5.5. It’s important to define the test goals for the correct cloud service type. A SaaS implementation will have other test goals than that of a PaaS implementation. Keeping in mind the connection and the interusage of the three service types and each


84

other’s services, the test goals will have an overlap over the different layers. Naming all of them separately creates a logical list of all the test goals for IT, but for the business people they need to be combined, as these stakeholders have no knowledge of nor interest in that part of the cloud.

Figure 5.5 Cloud services can exist over all cloud layers

Whether they are explicit or not, a few things stand out when looking at the importance of the business case for testing. The why is wholly or mainly irrelevant for testing purposes using traditional applications, but for the cloud it matters. Not only in the cloud, the business and IT are becoming more interwoven with each other, and IT is expected to participate in the creation of a cloud service, a cloud application or an infrastructure in the cloud.

Good test goals are as important as good communication—and formulating them is just as hard! In addition to this important role in communication, the test goals are an important input for the cloudriskanalysis (CRA).

The following parts of the definition require additional attention and clarification:

•Successcriterionofthetestassignment implies that as long as the reports show that “all is well” with the success criteria, execution of the test assignment can proceed as planned. However, as soon as there is a risk that a success criterion might not be real-ized, adjustment by management is necessary.

•Inthelanguageofthebusiness implies that a test goal must reflect the perception of the business and is therefore by preference not formulated in IT terminology.


85

Collecting the test goals is not a clearly delineated, separate activity; it occurs simultane-ously with establishing the assignment. The test goals can only be established correctly if the test manager is able to consult with the business. Clarifying the test goals correctly at the start of the process prevents uncertainty, miscommunication and disappointment at a later stage. The result of collecting the test goals is an inventory of the test goals in a test goal table.

ExampleA test goal is a success factor of the test process, specified in the language of the business. Test goals are presented in the test goal table, like this example. The indicated cloud test goals and associated relevant characteristics are based on experience.

No. Test Goal Service Type

1 Business process A and service 1 work according to specifications PaaS SaaS

1.1 Business process A is a new process SaaS

1.2 Service 1 is a new service to develop with very complex business rules (for example, a calculation module to calculate output and cost deductions of insurance)

PaaS

1.3 After a fault in a financial service, a rollback must be possible (maximum independence between services)

SaaS

2 Business process B and service 2 work according to specifications IaaS PaaS SaaS

2.1 Business process B is an existing process and is not modified SaaS

2.2 Service 2 does not disrupt the existing and new business processes, as it is an existing, much used service that is not modified

IaaS PaaS

3 Non-cloud architecture can connect to cloud architecture (for example, out-of-the-box applications such as SAP and Siebel, legacy applications on mainframe, existing databases)

IaaS PaaS

4 Standardization of services IaaS PaaS SaaS

4.1 All services and business processes are re-usable, currently and in the future

PaaS SaaS

4.2 Standards are correctly applied to new services (for example XML, SOAP, technical services such as security, authorizations, transfor-mations and logging)

IaaS PaaS


86

No. Test Goal Service Type

5 No unauthorized access is allowed to the infrastructure, platform and services

IaaS PaaS SaaS

5.1 Unauthorized access to commercial information is not possible, the security is guaranteed at server level (for example LDAP), service level (error running) and transport level (for example https)

IaaS PaaS

5.2 Architecture and software are implemented according to the requirements of the Project Start Architecture

IaaS SaaS

6 Existing legacy data is correctly accessed by services, opening documents up in the cloud has no consequences for the existing legacy functionality

IaaS PaaS

7 The performance is scalable and non-cloud systems perform as usual

IaaS PaaS SaaS

7.1 The performance of the existing legacy material is not affected by being accessed through the cloud

IaaS PaaS

7.2 Performance is scalable for up to 72 times standard production performance levels

IaaS

7.3 The client has an offer for a car insurance policy on the screen within 4 seconds

SaaS

7.4 High volumes of requests over the Internet to the client database do not have an adverse influence on the performance of the data-base responding to requests by employees of the business

IaaS PaaS SaaS

Table 5.1 An example test goal table

Services play a major role in the test goal table. The business will not frequently designate a specific service as a test goal, because the business is more concerned about the busi-ness process as a whole. A stakeholder that does want to make services test goals is the enterprise architect. It is the responsibility of the enterprise architect to standardize ser-vices so they are reusable in future and to implement services and architecture in general according to standards imposed by the enterprise architect.

The Second Step: An Analysis of the Cloud Risks

Establishing the risks of a cloud service, application or infrastructure covers step 2 of the BDTM steps. It creates a cloud risk analysis from the formulated assignment and test goals. With that in place, the focus in the product risk analysis is on the product risks; in other words, the risk for the organization if the product does not have the expected quality.


87

DefinitionA cloud risk is the chance that the cloud will fail as measured in relation to the expected damage if it does.

Cloud risk = Chance of failure * Damage

where Chance of failure = Chance of faults * Use frequency

What are the risks of clouds? The first thing to know is that with clouds, testing should be applied on three layers: the infrastructure, platform and software layer. These layers are aligned with the three service models of cloud computing (see Figure 5.6). For more infor-mation on the cloud computing service models, see Chapter 6. Additionally, the testing has to be performed by two different parties: namely the cloud provider, offering Software as a Service (SaaS), and the cloud consumer, consuming and developing cloud-enabled applications inside the cloud environments. Not only tests against the software layer, but also Platform as a Service (PaaS) and Infrastructure as a Services (IaaS) have different test goals and therefore need to be tested (separately) by both parties, as the cloud provider cannot assure the performance of the developed applications.

Figure 5.6 Service types of the cloud and their suppliers


88

Determining the cloud risks can partly be derived from what a cloud is made up of (as ServiceOrientedArchitecture (SOA) is composed of layers), the characteristics of the cloud (on-demand self-service, resource pooling, broad network access, rapid elasticity, measured service and multi-tenancy) and security. As every service type and every application has a specific source, they often introduce their own risks, which have to be targeted by the developers and testers alike. See Figure 5.7.

Figure 5.7 The cloud with its service models, deployment models and characteristics

All this should be considered as a motivating argument for aggressively testing and con-sequently mitigating all the risks to the infrastructure itself and the internal interactions it is affected by (like operating system calls, network communication, file system access, and the like), to guarantee the correct and secure operation of applications. No matter what segment or layer of a cloud a developer is testing, assuring its quality through sophisticated testing plays a crucial role.

As a helping hand a non-exhaustive list of risks of failure and damage relevant to the cloud has been established. The list in Table 5.2 can help when determining the product risks of the cloud.

1 The damage that a service can cause increases if this service is related to various other services and is part of various business processes.

2 Damage to reputation and loss of client confidence increase if errors arise in the cloud affecting external cloud consumers. This includes the use of web interfaces for consum-ers, such as Internet banking, but also online interfaces, such as for intermediaries who want to have an offer approved in real time.


89

3 The chance of failure of future cloud services increases if development standards are not taken into account when developing these services. It is therefore difficult to reuse the cloud service in the future, meaning that more software adjustments are necessary.

4 The chance of failure of a service does increase if non-functional system requirements are not implemented at the service level. Think in particular of performance and security requirements.

5 The chance of failure of a business process increases if the underlying services have not been aligned with the business process.

6 The chance of failure of a business process increases as new technology is used more. For many companies, the passage to SOA means that a new administration organiza-tion must be arranged to be able to support the new technology.

7 The chance of failure of a business process increases if the required changes in chain applications (ERP application, legacy applications) for SOA are relatively large.

8 The chance of failure of a business process increases if data errors can be contributed to external service providers. The effect of faulty information provision by external service providers is more difficult to control.

9 The chance of failure of a business process increases with the implementation of a rollback.

10 The chance of failure on authorization increases if the release of authorizations is regu-lated by a central security service for a large number of users and a large number of data collections.

11 The multiple layers of the cloud and the use of application-specific code often introduce their own risks, especially for security testing. Each layer could have a security breach.

12 The mythofinfinitescalability: the scaling ability is solely determined by the applica-tion—the technologies it uses and its internal architecture—not the infrastructure.

13 As cloud computing delivers us “hosted” infrastructure, the Service Level Agreement (SLA) determines its continuity. Looking into these boundaries is the ultimate test.

14 System boundaries within the cloud should be clearly stated. When these boundaries are not clear or not enforced, the system can behave differently than expected.

15 Cloud environments and applications are hosted using a variety of methods and loca-tions. Different laws and regulations around the hosting of data can affect these differ-ent locations.

Table 5.2 List of potential chances of failure and damages of the cloud

Knowledge of the cloud infrastructure, platforms, services, applications, organization, pos-sible damage and chance of failure is required to execute the cloud risk analysis (CRA). Such knowledge is nearly always distributed across multiple parties and people inside and outside the organization. In practice, the test manager is often the facilitator and organizer of the CRA, approaching various people who can contribute knowledge about the cloud risks.


90

When executing the CRA, the test manager must keep the purpose of the CRA in mind: anunderstanding,sharedbyallstakeholders,ofthecloudriskscorrespondingtothechar-acteristicsandobjectpartsofthecloud. He must devote his attention to the fact that, in addition to the cloud risks, the CRA also brings process risks (relating to the test process), product risks (related to the applications in the cloud), new product requirements and test goals to light.

The cloud risk analysis (CRA) consists of the seven steps as shown in Figure 5.8.

Figure 5.8 The seven steps of the cloud risk analysis (CRA)

1 Preparation. In the first step, the test manager creates an overview of damage and chance-of-failure elements that may be relevant to the CRA. This is done on the basis of existing information, such as the requirements for standard SaaS solutions, designs, enterprise architecture design, or similar documents.

2 Determining relevant elements (per cloud service type). Based on the test goals, the participants of the CRA determine the set of damage and chance-of-failure elements on which the CRA must focus. These elements are divided into the three cloud service types, although every element can be on every service type or layer. The CRA is then executed for the sub collection of the damage and chance-of-failure elements collected in step 1.

3 Agreement standard services. All participants establish a mutual agreement on which levels of the cloud or its applications will employ standard services. These services are then integrated into several statementsofwork for the supplier of these standard services.


91

4 Determining damage. The participants determine the damage level for each part of the cloud based on the cloud architecture and service types. These are separated by the processes within the cloud and not standard services.

5 Determining chance of failure. For those parts that are customized or part of the end-to-end tests (chain risks) the participants establish the chance of failure per characteristic based on the object parts that constitute the cloud system (test object).

6 Determining risk class. The participants determine the risk class for the combination of characteristics, service types and object parts on the basis of the risk of damage and chance of failure.

7 Completeness check. A completeness check is the last step in this process.

The Product Risk Analysis vs. the Cloud Risk Analysis

What is the difference between a cloud risk analysis and a product risk analysis? The dif-ferences are around:

The result of a cloud risk analysis is a 3D model of the risks (see Figure 5.9). It gives insight •in the damages and chance of failure per characteristic, object part and service type.The larger amount of stakeholders (step 1), like for IT the Enterprise architects, owner •of the cloud layer, 3rd party service suppliers, and for the business Marketing and end services users.Within clouds, a service is the relevant object part as a part of a business process •(step 2). Functionality, for example, is no longer formed by a number of subsystems but by services. The characteristic functionality can be subdivided into the various services and the totality of the object parts is the business process. The same reasoning applies to the remaining characteristics. To get a complete overview of all services and business processes, which fall within the scope of the cloud project, the object parts are arranged by characteristic in a table, see Table 5.3.Agreements on what are and what aren’t standard services (step 3). These standard •services are not tested separately, but only in the end-to-end test. The 3rd party service supplier can be enforced to comply with a StatementofWork(SoW) where the expected quality of the service is agreed upon. The use of Quality Gates can help in getting trans-parency in the quality of the service.Functional testing is of lesser importance. As their supplier approves the functional •requirements of the standard services, functionality is of lesser risk. But non-functionalrequirements are not sufficiently allocated in the tests of the supplier. Integration of the standard services in the cloud has the priority of test, for example performance, secu-rity and integration testing. Non-functional requirements should get a higher risk class compared with functional requirements.Chain risks are always determined in a cloud project (step 5), as a cloud consists of •multiple service types they should always be tested at least once in an end-to-end test.


92

Because of the greater complexity and dependence of standard services the risk classes •of High, Medium and Low are not always sufficient. A more empirical method of risk classes is preferred, like for example numbers.

Figure 5.9 CRA: a 3D end result

In More DetailBesides a traditional performance test, performance and continuity can be tested per service as with security, (suitability of) infrastructure is related to cloud infra-structure components. A large number are standard software components that get their specific configuration for the business.

Character-istic

Service Type

Objects

Functionality SaaS Screen app 1/GUI Service A Business process 1

PaaS Screen app 1/GUI IIS –

IaaS ESB Private interface –

Performance SaaS Service A, B, C, E All business pro-cesses

–

PaaS Screen app 3/GUI WAP –

IaaS ESB – –


93

Character-istic

Service Type

Objects

Security SaaS All services All business pro-cesses

–

PaaS All apps All Windows All ERP

IaaS All VPN ESB Private interface

Continuity SaaS Screen app 5/GUI All business pro-cesses

–

PaaS Windows 3 ERP 2 Legacy 1

IaaS VPN to Legacy 1 ESB –

Suitability SaaS Business process 2 Business process 5 –

PaaS All Windows, ESB IIS WAS

IaaS – – –

Reusability SaaS Screen app 1/GUI Screen app 2/GUI Non-legacy ser-vices

PaaS – – –

IaaS ESB – –

Infra-structure

SaaS All business processes, ERP

– –

PaaS ESB WAP WAS

IaaS – VPN’s –

Scalability SaaS Non-legacy apps Non-legacy services –

PaaS IIS, ESB Windows 1, 2 & 5 WAP

IaaS – VPN’s –

Table 5.3 An example for determining object parts for clouds

Which services as object parts are involved in which characteristic follows from the analysis of the test goal table, the cloud design and the PSA. If, for example, a test goal is connected with making an offer in which characteristics such as rel-evant functionality and performance are mentioned, then the services, which are involved in making an offer, are classified by both the characteristic functionality and the characteristic performance.

For cloud projects, it is important that the mode of classification is standardized. The risk class allocated to a (part of the) cloud can be found from the service registry, with an explanation of why the relevant risk class has been added. With a standardized mode of


94

classification, it is possible later to understand why the specific risk class is provided. More-over the risk class for the cloud can easily be reconsidered with the standardized mode of classification if the relevant service is to be reused.

The Next Steps: A Cloud Test Strategy

The last step of the cloud risk analysis is verification that the result is as complete as pos-sible. For this purpose, the separate risk tables are merged into a single overview showing the risk class for each combination of characteristic and object part.

The test goals are also incorporated into the overview to make executing the completeness check easier for the business. Based on the risk table, as shown in Figure 5.4, the cloud test strategy is formed. This is transmitted to the participants and business. They are asked to approve the result. The business grants definitive approval and makes a decision in the event any points of discussion arise.

The final result obtained is an overview showing the risk class per combination of charac-teristic and object part, like the example shown in Table 5.4. It is the basis for selecting the test intensity for each combination of characteristic and object part in the cloud testing strategy. The test strategy around cloud implementations gives insight into the quality and compliance of the cloud by setting up the correct tests, checks and acceptance criteria.

Characteristic/ Object Part

Service Type

Test Goal Risk Class

Functionality C

Application 1 PaaS SaaS

Business process A and service 1 work according to specifications

C

Application 2 IaaS PaaS SaaS

Business process B and service 2 work according to specifications

C

Application 3 PaaS SaaS

Business process A and service 1 work according to specifications

C

Business pro-cesses

IaaS PaaS SaaS

Business process A and service 1 work according to specifications Standardization of services

B

Security A

All services IaaS PaaS SaaS

No unauthorized access is allowed to the infrastruc-ture, platform and services Standardization of services

A


95


Service Type


Business processes

IaaS PaaS SaaS

Business process A and service 1 work according to specifications No unauthorized access is allowed to the infrastruc-ture, platform and services

A

Performance B

Applications IaaS PaaS SaaS

The performance is scalable and non-cloud systems perform as usual

A

All services IaaS PaaS SaaS

The performance is scalable and non-cloud systems perform as usual Existing legacy is correctly opened up by services, the opening-up in the cloud has no consequences for the existing legacy functionality

C

Business processes

IaaS PaaS SaaS

Business process A and service 1 work according to specifications Business process B and service 2 work according to specifications

B

Stability C

Non-legacy applications

IaaS PaaS

Existing legacy is correctly opened up by services, the opening-up in the cloud has no consequences for the existing legacy functionality The performance is scalable and non-cloud systems perform as usual

C

Continuity A

Application 5 SaaS Standardization of services B

Business processes

IaaS PaaS

Business process A and service 1 work according to specifications Business process B and service 2 work according to specifications

A

Suitability B

Business process PaaS Business process A and service 1 work according to specifications Business process B and service 2 work according to specifications

B

Reusability C

Non-legacy services

SaaS Standardization of services C


96


Service Type


Infrastructure B

All infrastructure IaaS PaaS

Existing legacy is correctly opened up by services, the opening-up in the cloud has no consequences for the existing legacy functionality Non-cloud architecture can connect to cloud architecture (for example, out of the box applica-tions such as SAP and Siebel, legacy applications on mainframe, existing databases)

B

Table 5.4 An example of a risk table for a cloud implementation

97

6 Testing the Cloud: In, On or With…

As the Cloud Era emerges, testing will change! Changes are ahead not only for testing information systems, but also for testing the infra-structure and cloud-enabled applications, even with the ability to have instant deployable test infrastructure. What are the changes that come along with the cloud? It enables the move to testing the Business Technology. This all has an impact on the way we will do testing in the future. As the current types of applications will not disappear due to cloud applications, it doesn’t replace what we test, but it provides an addition to software testing.

Testing the Cloud Itself: Cloud Infrastructure

Why test the (cloud) infrastructure? Nowadays, no one cares about where his or her text messages and e-mails are stored. There is little or no realization that there is a whole world that lies behind these everyday things. Everyone just assumes that they are there when they are requested. In order to facilitate and especially to guarantee this, a cloud provider needs to take significant measures relating to the cloud infrastructure. The temporary


98

unavailability of these services or the loss of those messages would undoubtedly cost the cloud provider existing and potential customers.

Infrastructure solutions often consist of hardware and/or software components. They are generally the standard of the larger parties and suppliers. In addition, the solutions belong to the category of proven technology. More and more infrastructure components are com-mercial off-the-shelf (COTS) products such as appliances. But cloud solutions only exist involving software components.

In More DetailTraditional InfrastructureUnfortunately traditional infrastructure is tested with minimal or no attention to the testing process. Standard requirements when starting an infrastructure project are a number of general requirements, like 99.9% availability, at least the same performance as in the present situation, et cetera. Rarely are these requirements made SMART (Specific, Measurable, Achievable, Realistic and Timely).

During the construction phase the infrastructure is configured, migrated or directly deployed. Normally after the design and construction phase testing should com-mence, but as the testing is conducted under pressure nobody tests the (test) infra-structure. What happens is that the infrastructure is implemented like a pilot, and after successful completion the pilot (the infrastructure) will go into production.

Regarding the implementation of cloud infrastructure, as in cloud computing solutions, it cannot be sufficient to just verify that itworks. In addition to a utility function there are other important aspects, like availability, security, performance, scalability and adaptability. These also must be tested before the whole cloud system can go into production.

Business Demands a Working Infrastructure, Even in the Cloud

As not only infrastructure, but all of IT, becomes more and more a commodity, the various infrastructure components become “invisible” to the users. Only workstations, laptops and printers are visible. And as these have become normal they doesn’t remind people of infrastructure, only appliances. With the cloud, on-demand and “Anywhere, Anytime, Anyplace” come to mind. Although this is already closer to reality than some think, it still needs some work. For example, whereas traditional infrastructure has a maximal 99.9% availability, cloud infrastructure has promise of 100% availability.1

1 Google Apps makes a new promise: No downtime.

6 Testing the Cloud

99

The underlying cloud infrastructure forms the foundation for cloud solutions. For instance, there is an enormous variety of (mobile) devices that needs to be supported so that infor-mation can be ported to the underlying systems. The demands of such collaboration, “The New World of Work,” Voice over IP (VoIP), use of smart phones and apps sets high demands on the infrastructure. In the past, e-mail was an application with moderate use. Nowadays e-mail is inextricably linked to the business processes of an organization. And if the e-mail system fails the CIO can be fired, as e-mail is a standard part of business and facilitates other applications and solutions. It’s already a commodity (see Figure 6.1). The infrastructure, like any utility, has become less visible while the world around it has become more dynamic and change is a given.

Figure 6.1 E-mail is already a commodity to the business: it’s Business Technology

Cloud Solutions Not Only “Need to Work,” But Work “Well Enough” for Business

With the passing of time the IT infrastructure has become increasingly complex. Fifteen years ago, an e-mail solution was a PC on a table in the corner of the room, connected to the Internet via a modem. Today, IT infrastructure items are end-to-end solutions consisting of integrated (and therefore) complex components. A typical e-mail architecture consists of clustered and/or virtualized servers, dual front-end servers, webmail functionality, spam filters, anti-virus solutions, firewalls in a demilitarized zone2 environment, and a huge variety of devices (fixed and mobile). The necessary network facilities are still not fully available. Changing one of these components can have far-reaching consequences for the entire chain, resulting in production disruptions of the mail application, but also of business applications that use the underlying infrastructure.

Now with cloud the infrastructure becomes even more complex. Not only do we have vir-tualized parts of the infrastructure in the cloud, but also the platforms are virtualized in

2 In computer security, a DMZ, or demilitarized zone is a physical or logical subnetwork that contains and exposes an organization’s external services to a larger untrusted network, usually the Internet [Wikipedia].


100

the cloud. And to add to this, third parties can even host the infrastructure and platforms. As organizations are increasingly dependent on technology, the number and complexity of infrastructure changes is increasing. Often enough something about failing systems appears in the press. Although not all of these problems can be traced to infrastructure components, it is important to realize that (cloud) infrastructure is a link in the IT chain and a chain is only as strong as its weakest link. So it needs to be tested, but how thoroughly, against what, and what to look for?

How thorough the test is depends on the test strategy: the higher the risk, the more thorough the test. Regarding the implementation of cloud services, it’s not sufficient to verify that “it works,” it also needs to be available, scalable, reliable, adaptable, secure and accountable. All of these are non-functional quality attributes of the cloud (see Figure 6.2). But not all of them need to work perfectly correctly; they only need to work in the way the business would like them to work. IT moves into the business!

Figure 6.2 The Cloud infrastructure quality attributes

6 Testing the Cloud

101

Non-Functional Quality Attributes of the Cloud InfrastructureThese non-functional quality attributes are all very nice, but what do they mean and how can they be tested? Note that when testing cloud infrastructure there are always three lay-ers to test for, and these non-functional quality attributes only target the IaaS.

ScalabilityA cloud in the first place should be sufficiently scalable to meet the variable demand of the business. Scalability focuses on the dynamic capacity of the cloud infrastructure, the sup-porting organization, and the processes of growing and shrinking the volume of services. It’s a required property of the cloud, which indicates its ability to handle growing amounts of work in a graceful manner. It also means the ease with which a system can be expanded/upgraded when there is an increase in users and the need for more speed, processing and storage capacity, and downgraded when there is a decrease in users and less need for speed, processing and storage capacity.

Testing for scalability is a difficult task, as you need to know what to test for. That is why the cloud provider mostly does this, but a cloud consumer might want to test it in their integration tests. With the cloud the scalability can be enormous. It’s possible to use a huge amount of computing power and storage space if needed. An agile load test can test the boundaries of the scalability. It creates a (high) volume of load when needed to check for the maximum load allowance of the infrastructure.3 For example, the scalability can be tested with an increasing performance demand on the infrastructure; more and more virtu-alized machines are added to the infrastructure to see if it can cope with the performance demands. Testing the shrinkage of the infrastructure cannot be done, but this quality can be tested within the PaaS or SaaS layer of the system as a whole.

AvailabilityThe cloud infrastructure is available in sufficient quantities to meet the (often implicit) demands of the organization. The availability is partly determined by business requirements, both explicit and implicit, such as confidence and expectation on the part of the organiza-tion (which is only made explicit in the event of a disruption). Availability in this context relates primarily to the continuous operation of systems in accordance with SLA, and it is the degree to which a system is available for the users (at the desired times).

For IaaS it’s important to test the availability, as it is one of the pillars of the cloud. If a cloud interface isn’t available, its benefits will disappear. Public clouds should have at least 99.99%4 availability, as the service of the cloud provider enables the movement of data to an online server if another one fails. Currently Google has an availability of 100% for its enterprise mail solution. The cloud availability can be tested like normal availability

3 Theoretically the cloud has an unlimited amount of computing power and storage space. But in practice there are always boundaries.

4 Private clouds can have lower availability, but that’s determined in its design.


102

of infrastructure systems. A test therefore should be conducted to test the ability of the system to recover from an individual hardware component failure, such as a network or disk problem.

In More DetailAvailability Testing vs. Reliability TestingTraditional testing for availability means running an application for a planned period of time, collecting failure events and repair times, and comparing the avail-ability percentage to the original service level agreement (SLA).

Where reliability testing is about finding defects and reducing the number of fail-ures, availability testing is primarily concerned with measuring and minimizing the actual repair time. That may seem odd at first, but take another look at the formula for calculating percentage availability [Wikipedia]:

(Mean Time between Failures / (Mean Time between Failures + Mean Time to Recovery)) × 100

Notice that as Mean Time to Recovery (MTTR) trends towards zero, the percentage availability trends towards 100%. This idea becomes the essential focus of avail-ability testing: reduce and eliminate downtime.

The closer the testing is to real-world situations, the better the test confidence. Some organizations are reluctant to allocate fully configured server machines and isolated network environments to a long battery of availability testing. Just remem-ber that a software defect found after deployment costs ten times more to fix than if found before deployment.

Availability is typically specified in nines notation. For example, 3-nines availability corresponds to 99.9% availability. A 5-nines availability corresponds to 99.999% availability. Downtime per year is a more intuitive way of understanding the avail-ability. See Table 6.1, which compares the availability and the corresponding down-time.

Availability Downtime

90% (1-nine) 36.5 days/year

99% (2-nines) 3.65 days/year

99.9% (3-nines) 8.76 hours/year

Availability Downtime

99.99% (4-nines) 52 minutes/year

99.999% (5-nines) 5 minutes/year

99.9999% (6-nines) 31 seconds/year

Table 6.1 The availability vs. the corresponding downtime

6 Testing the Cloud

103

ReliabilityThe cloud supports the business while protecting against damage from unwanted inter-ruptions to supply facilities or theft/loss of data. It’s a set of attributes that bears on the capability of software to maintain its level of performance under stated conditions for a stated period of time [ISO 9126-1, 1999], and it consists of the following sub-characteristics: Maturity, Fault Tolerance, Recoverability, and Reliability Compliance. If one of these sub-characteristics fails, the business will notice it immediately and the system will not respond as expected or as required! A cloud solution needs to be reliable, as the business will depend on it.

All these sub-characteristics need to be tested for, especially with a cloud infrastructure, as an “as a service” infrastructure solution must be reliable under the required circumstances. Testing is done by using a security-like test approach. The test cases need to be setup by negative testing, and not only do the requirements need to be checked, but they also need to be challenged by approaching them negatively. Negative testing, or abusecases, can define if the requirements are met. A test mindset of “is the infrastructure reliable?” will determine the quality of the Infrastructure as a Service. See the boxed text above on “Avail-ability vs. reliability testing.”

AdaptabilityA cloud is easily adaptable to changing organizational needs. These include the ability for updates/upgrades to take effect and for new applications to be made available, but also for connections with other clouds and even organizations (community clouds). The infrastructure needs to have the ability to work in a different environment without having to execute extra actions to enable this. A cloud is characteristically always available and able to work on and with different platforms. The PaaS layer is built on the IaaS layer and needs to support this. Standardization of the different versions of the infrastructure improves the adaptability of any piece of hardware and software. A cloud infrastructure needs to support the business processes that the service provides.

When testing adaptability, timeliness is usually an important factor. Timeliness is defined as the degree to which the information is available in time in order to take the measures the information was meant for. Adaptability can be tested dynamically and explicitly with a real-life test (RLT) [TMapNEXT, 2006].

In More DetailReal-Life Test [TMapNext, 2006]With the real-life test (RLT), it is not the intention to test the system behavior in sep-arate situations, but to simulate the realistic usage of the system in a statistically responsible way. This test mainly focuses on characteristics such as effectiveness,


104

connectivity, continuity and performance of the system being tested. Many defects that are found with a real-life test are connected with a system’s use of resources:

crashing of transactions following lengthy use; –crashing of transactions that are carried out in a particular sequence; –inadequate response times and speed of processing; –insufficient memory or storage space available; –insufficient capacity of peripherals and data-communication network, and –unavailability of system components after an update. –

To be able to test whether a system can handle realistic usage of it, that usage should be somehow specified. This also serves as the basis for the test basis and, in this context, is often referred to as the profile. The two most common types are:

Operational profile. Simulation of the realistic usage of the system, by carrying out –a sequence of transactions, which is compiled in a statistically responsible way.Load profile. Simulation of a realistic loading of the system in terms of numbers of –users and/or transactions.

SecurityA reliable cloud should provide adequate security for the required level of integrity and confidentiality. Security is a very broad term, but at a minimum it should involve user identification and the screening of systems and data residing within the cloud. Security is important in the cloud infrastructure, with a direct link to the multi-tenancy characteristic of the cloud. This multi-tenancy and the elasticity of cloud solutions create a security risk, which can be a great threat and isn’t effectively solved by the cloud providers.

In More DetailMulti-Tenancy and ElasticityThe elasticity characteristic of the cloud enables the opportunity to grow and shrink in infrastructure performance and capacity. The allocated computing power is enlarged (or reduced) when needed. However, when the infrastructure is located in a certain part of the cloud, it’s unknown what part of the cloud will be allocated. This could be in certain “risk areas” within the cloud.

These risk areas are allocated to infrastructure managed by unknown or obscure parties. One of the most known risks of multi-tenancy solutions is the unknown “neighbor”: it might be a criminal or hacker that wants to have access to the stored information.

6 Testing the Cloud

105

When the elasticity creates an enlarged allocation of the infrastructure next to a risk area, that can generate a security risk. Mutual arrangement with the cloud provider could determine what will happen as the allocation grow. For example, an IP range could be set, or the physical server that the enlargement is executed on could be determined.

The security of the infrastructure is also an attribute that, partly, needs to be addressed in a negative manner, using abuse cases. How is the infrastructure secured and how “easy” is it to get access to this? Using special companies to execute hacker-like tests as part of an integrated test solution for the infrastructure is of added value.

AccountabilityCalculating resources, tooling and infrastructure costs associated with its use on the cloud is maybe the most important reason to move to a cloud, so accountability is a main charac-teristic of the cloud infrastructure. This covers audit trails and specific accountant demands (e.g. Sarbanes-Oxley). The pay-per-use requires an optimal accountability of cloud services and applications.

Accountability can be tested statically with a checklist composed of the setup of certain measures. The realization of these measures in the system can be tested in a dynamic and explicit way.

The qualities and characteristics shown above are based on generic quality attributes for cloud infrastructure. These quality attributes are not all critical for every IaaS solution, and in some cases there could be more needed for testing purposes. It is possible to deviate from this list if the situation requires. But it is recommended that the initial quality attributes use this fixedset of quality attributes as a basis, and only after identification of all risks to decide on the inclusion and/or removal of certain quality attributes.

Functional Testing of the Cloud Infrastructure

As with all the non-functional characteristics of the infrastructure layer of the cloud, it’s only tested on this level. But the functional requirements need to be tested while integrat-ing the different pieces with each other. In other words, all three layers need to be tested and the integration between these layers is also part of the test. An integration test over the complete infrastructure is the way to start this.

All different parts of the IaaS need to be connected with each other, either physically or using simulation techniques, and while using data to communicate between these parts.


106

All data should be able to go from A to B. To test this, a Program Interface Test5 is the best test technique to use [TMap, 2002].

In More DetailProgram Interface Test: For the Integration of IaaS [TMap, 2002]The program interface test (PIT) is a test design technique that is used to test the interfaces between the various programs, infrastructure components and/or mod-ules. After testing the independent components of the cloud, this technique is used to verify whether the components still function correctly after integration with real data flows.

When two components in an infrastructure work with each other, the integration of these two needs to be tested. Executing test cases created with PIT permits verification of whether component 1 and 2 function correctly together. If there is a component 3 (or more), these are replaced with stubs and drivers. The integration between component 1 and 2 is done with real data.

The goal of the PIT is to test the interfaces between the various components, programs and/or modules, and thus check if the various components interpret the data flows in the same way. This is because at these points in the programs many defects occur, produced by differences between the two components. These types of misinterpretations and incorrect execution are dealt with during this test. The program interface test focuses especially on the interfaces and not on the cor-rectness and completeness of processing. It’s end result is a test script, like, for example, that in Table 6.2.

Test case 1 2 3

Object Site 1 Other 3 Number 8

Geo Sketch Final Sketch

Mutation GIDS BAG Total BAG Total

Status Not measured Demolition Not measured

Geo note Y Y N

Date Today + 4 Today - 5 Today

Document# 34576 23 7685

Document date Today + 2 Today - 7 Today + 2

Expected Result Measured Deleted Not measured

Table 6.2 The result: an example of a PIT test script

5 For more information see the explanation on Test Design Technique – Program Interface Test at http://www.testingthefuture.net/2009/08/test-design-technique-%E2%80%93-program-interface-test/.

6 Testing the Cloud

107

Quality of the Cloud Infrastructure Using “Agile”

It is important to know which quality the cloud infrastructure has to meet. The mostly implicit demands should be made explicit in the cloud requirements. The business is in the driver’s seat, but it needs to specify clear requirements or use an agileapproach in creating and checking the cloud infrastructure.

An infrastructure is set up with the usage of virtual infrastructure components; virtual machines emulate the physical infrastructure. When these components are created from standard components, such as IaaS, it’s possible to quickly connect the components and establish the correct high-level functioning.

An infrastructure engineer can setup part of the cloud infrastructure and check within minutes to see if it is functioning correctly. The system is built like a LEGO® building where every component, either a system, an interface or any other infrastructure component, is checked to see if it works. When it doesn’t function as needed, a direct change can correct the situation. When it is shown to work, another component can be added to the infrastruc-ture; every component is integrated as needed in the infrastructure. This agile approach is based on iterative deployment in which requirements and solutions evolve in combination and change is embraced, which would therefore appear to offer a potential solution to the problems in a traditional approach. The setup of an agile framework consists of designing the infrastructure, defining the process and assigning people to run the process.

Figure 6.3 The deployment of an agile framework consists of infrastructure, process and people

The incremental deployment processes have been specifically developed to increase both speed and flexibility. The use of highly iterative, frequently repeated and incremental process


108

steps and the focus on business involvement and interaction theoretically supports early delivery of value to the business. But the infrastructure basis is in defining the requirements based on Availability, Scalability, Reliability, Adaptability, Security, and Accountability, as described earlier.

Cloud Applications: Testing on the Cloud (SaaS)

Cloud applications are still few compared to traditional applications, but they are the future. But what are cloud applications? When the question is asked, “can you name a few cloud applications?” most people answer Salesforce.com, Facebook, Google Apps and even Microsoft Azure.6 Four hits (where one isn’t actually a cloud application), as the best-known examples. Are there more? Yes and they are growing in numbers!

But how do we test these cloud applications? What’s so special about them that they need a different type of testing than traditional applications? Cloud applications are applications that are created to leverage the opportunities the cloud gives them, but they also work with the disadvantages the cloud offers, like, for example, standardization.

Figure 6.4 The cloud is defined by its service model, deployment model and usage

6 Microsoft Azure is not a cloud application, but an infrastructure and platform.

6 Testing the Cloud

109

Mostly cloud applications are based in the third cloud layer: SaaS. They are Software as a Service solutions that run completely on cloud infrastructure and platforms. And that is exactly the reason the testing of SaaS applications is different from traditional applica-tions. When they are integrated in the current architecture they need to be tested on three levels: namely the infrastructure, the platform and the application itself (see Figure 6.4). The usage of standard services of applications also means a change for system testing. Functional testing will be executed at a minimum, as the standard applications are already tested and approved by the supplier. But that doesn’t say anything about how it integrates into the client’s cloud.

Testing the Business: Testing Business Technology

As we said in Chapter 3, the cloud moves Information Technology to Business Technology. Testing cloud applications is in line with that. With software testing we are not testing the IT anymore, but the BT. But how do you test BT? As the “B” in BT stands for business, a business approach should be used with testing. In Chapter 5, see the section “Creating a Cloud Test Strategy.” The business is put in the driver’s seat with BDTM. But is that enough for testing SaaS solutions?

In More DetailIs IT Already a Commodity?Technology is part of everyday life, in business as much as in our personal lives. Take away the applications, e-mail, networks and internet from any organization and it would come to a standstill almost immediately. This is more so now than ever before: in the days of phone and fax, we could probably do fine without tech-nology for a day or two.

The difference today is more than just greater dependency: we are no longer doing the same things. Supported with technology we are starting to do new things that we simply could not do without that technology. And now that technology has found its way into all parts of our organization, we are asking new things of it.

Looking closer at what we now demand of Information Technology, it becomes clear that cloud computing is a good model for provisioning and paying for tech-nology. It will give a boost to whatever we are doing.

When testing for BT, the business scenarios are the most important part of what to test with. The application should perfectly fit the business needs and demands in doing its work. So setting up the test using the user scenarios or using the Real Life Test technique is what is needed while testing SaaS applications. The user scenarios are worked into test scripts, which are easy to read for the business and are, preferably, executed automatically.


110

To create these scripts multiple options are valid, but the two most widely available are using the test technique real-life Test and Process-Cycle Test (PCT) and creating a (test) model that derives test scripts from user scenarios using Model Driven Quality Improve-ment (MDQI). As the Business Technology is tested, the testing should connect with the business and connect with the business processes.

ExampleProcess-Cycle Test (PCT) [TMapNEXT, 2006]The process-cycle test is a technique that is applied in particular to testing the characteristic of Suitability (integration between the administrative organization and the automated information system). The basis of the testing should include structured information on the required system behavior in the form of paths and decision points. The process-cycle test digresses on a number of points from most other test design techniques:

The process-cycle test is not a design test, but a structure test: the test cases issue –from the structure of the procedure flow and not from the design specifications.The predicted result in the process-cycle test is simple: the physical test case –should be executable. This checks implicitly that the individual actions can be car-ried out. In contrast to other test design techniques, no explicit prediction is made of the result, and so this does not have to be checked.

In testing the Business Technology, the business plays a key role in determining how effec-tive the testing is. If the business doesn’t take the time to look at a system and help with setting up and executing the test scripts, it’ll be very difficult for testers do an adequate assessment of the system’s quality. Although the business can only help with checking and approving the system, it plays a vital part in the final outcome of the tests.

To help the business, testers also should focus on the business aspects of the system. That can be done by using test design techniques that are appropriate for creating test scripts from a business standpoint. Test design techniques like the Process-Cycle Test (PCT), Use Case Test (UCT) and Real-Life Test (RLT) [TMapNEXT, 2006] are of great use when creating the test scripts. These test scripts focus more on the business use of the system than oth-ers. This focus provides the testers with the tools they need to execute tests of the Business Technology because they are in line with the needs of the business.

Therefore the testing can start at the moment the project starts. When the business require-ments or scenarios are defined, the testing can start creating the test scripts from these scenarios; the testers don’t have to wait until the code is written. The code is already writ-ten and tested functionally by the supplier, but the application needs to be successfully integrated with the system. That integration starts by determining how business is planning

6 Testing the Cloud

111

to use the system. As that is defined at the start of the project, testing can also start at that moment. It is even necessary to start thinking of quality right from the start by test-ing ideas, concepts and documents. The principle is to start at PointZERO®. However, our current way of working does not always accommodate this concept, and in most cases it will require some time to grow into it.

Non-Functionals of SaaS Don’t Need to Be Tested?!?

As stated earlier, the SaaS functional requirements are covered by the supplier’s tests, but the non-functional requirements are only tested by the supplier in the supplier’s own environ-ment: an environment that can differ from the eventual environment.7 Cloud applications run over three layers and a change from the supplier’s environment is fundamental.

ExampleTesting Salesforce.comAt a delivery company they are testing a Salesforce.com implementation. To test the application, the test team is using the business requirements and user experi-ences to build the required test scenarios. The business is consulted and “used” throughout the process in both an iterative and an incremental manner.

These test scenarios are tested in the beginning of each (two week) sprint. After the last sprint, the test scenarios are complete, and they can be used for testing the system, testing system integration and acceptance testing.

What differs in testing a SaaS implementation from an on premise or bespoke development project is the reduced testing that is done of the functional aspects. The supplier has a well-documented set of functional limitations. Therefore it’s a matter of revealing and clarifying these limitations to the business. A lot more test-ing is done on the business-related functions.

With traditional applications, any way you look at the (functional) design documents you will notice the lack of non-functionals. They are hardly mentioned. The design describes what the application needs to do to work and why this is needed, but nothing further… For instance, take an online store. What would we like to do in an online store? We order an item, shop around, then before buying, we log in. This is a simple but accurate scenario for an online store. The business expects that it is properly constructed and working. A functional tester will use the design document and use a system test to ensure that the product meets the specifications, most of the time. The tester will make recommendations and report on the

7 With cloud infrastructure, even the IaaS is always a separate part of the test object.


112

quality of the product. This can be positive when the specifications are met, but even after a positive report from this tester a customer might still not want to shop.

Why? The reason is that some things, like security, usability or performance are not regu-lated. What if you have a purchase in the cart and when you click on it, to see it, it takes 30 seconds to show up? What if you have to click on an item 5 times to put that item into your cart because you have to go through 4 screens? What if you can order products for free? Good for you, but maybe not for the store owner, and maybe you are even breaking a law?

A lot of people call these things non-functionals. But is it a non-functional? It can be reversed! Does a shop function when customers stand in a line for the cashier for 3 hours? If products are for free? Or if they have to look for toothpaste under “make-up”? No! The shop will close down! So, part of the function of a shop is how the shop operates.

“Within a cloud application, functional requirements are met!” Why? Because they are tested and confirmed by the supplier. But the non-functional requirements are not. For testing a cloud application it is necessary to look at the operation of the software. Testing from a user or a business perspective!

Some non-functional specification will be in the design documentation, for example, about security and the loading times of screens. If they are in the design documentation, then test cases can be created to cover them. But if not, then test cases have to be defined to cover the full functioning of the application. Together with the business needs, these requirements can be integrated into the test cases. And don’t forget to put them in the documentation! The quality of the application will be greatly enhanced because these requirements are met.

Here are a few questions that can help uncover the non-functional requirements. See also TMap NEXT [TMapNEXT, 2006]:

What requirements are listed in the field of security, performance and usability?•How are the requirements translated into the design documentation? •Are the requirements created with the SMART criteria [SMART, 1981] in mind (Specific, •Measurable, Attainable, Relevant and Time-bound), for example “screen should appear within 5 seconds” instead of “the screen must be displayed within a reasonable time”?Are the requirements explained in only one way?•

In More DetailA checklist and a real-life test are excellent test design techniques for this. An alter-native is that it is arranged with the supplier of the software that specific software is delivered that can help to cover the non-functional requirements.

6 Testing the Cloud

113

Using Models for Business Approval: Model Driven Quality ImprovementThe functional test cases for cloud applications have to be derived from the business, so that the application is created/bought to match the need of the business. One of the ways to create those functional test scripts is by using Model-Based Testing (MBT). With MBT a (test) model is used, derived from the business requirements, to automatically generate test cases. Most of the time the challenge with this approach is to include the creation of a formal model in which the operation of (part of) the application is represented. Creating this model is done by hand. When this model is finished, it can be read by a tool that enables the creation of test cases. An additional advantage is that MBT allows the creation of standard test cases, something that is an added value for the ability to offshore or outsource test execution. MBT also claims it has the opportunity to check the quality of the requirements in an early stage in the development process.

Unfortunately this is where implementing Model-Based Test design will fail in a lot of projects. Because if a team has never learned to do and use modeling, if a test team is not able to create a complex model, how can they do it? And besides, the organization around them isn’t ready to work with these types of complex test models. The acceptance of these models isn’t easy where they cannot be verified because the business doesn’t understand the complexity.

MBT only covers one part of the possibilities models have for testing. See the boxed test, “Model Driven Quality Improvement or Model-Based Testing?” MDQI doesn’t always need a complex model. A simple and understandable model can be used, a model that testers can understand and work with, and that represents a part of the behavior of the application. And start with simple parts the business can also understand and approve.

For example, using a simple process flow can be the start of a project. Maybe this flow doesn’t hold all the information, but it is complex enough to generate the test cases8 that cover the total businessprocess. By starting test activities with Model Driven Design Improvement, it’s possible to make models that are understandable for the business part of the organization.

In More DetailModel-Based Testing ExperienceWhen testers are asked about their experience with Model-Based Testing, the answers are often not very clear or the testers don’t have any experience with this type of testing. When you ask the same question to the business, you hear, “That is an old approach that doesn’t work for us.” And that is correct. MBT has already been very promising for at least 15 years.

8 The test cases generated from a process flow via the test design technique Process-Cycle Test with test-depth 1 or 2.


114

Business and testers who have tried in the past to use MBT have become a bit skeptical. And that is one reason why Model Driven Quality Improvement is a bet-ter name for this type of testing. It just takes away the concerns the business and testers have about MBT and encourages them to look at something that can really improve the quality of software.

Model Driven Quality Improvement is not just a simple new term, it’s a combination of techniques that offer a type of testing to the business that not only generates test cases but, more importantly, delivers added value to improve the quality of the final result.

Definition“Model Driven Quality Improvement is the acceleration of the development pro-cess and the improvement of the quality of the software, by using and with validat-ing and verifying models.”

With MDQI, specific test models are still needed in high-risk environments with a high com-plexity. These test-specific models can be successful where a formal approach is needed. The business can specify the detail level, and if a high-level model is approved by the business, from there the detail levels are filled in.

MDQI offers testers the opportunity to create functional test cases in line with the business by using a simple model, but sometimes more functional test cases are needed, even for cloud-based applications. This depends on the coverage, risks and the type of model, and what types of tests are needed.

In More DetailModel Driven Quality Improvement or Model-Based Testing?The last couple of years people have been talking about integrating testing with other parts of the development lifecycle (ALM). A good thing, but if testers strive to a perfect test approach without involving other parties in the ALM this is just a sub optimization.

If you look at the definition of Model-Based Testing (MBT) the one most used is: “Automation of test design,” by Utting and Legeard [MBT, 2006]. This definition is quite “small” when you compare it with the MDQI definition by Andréas Prins: “Model Driven Quality Improvement, is the acceleration of the development pro-cess and the improvement of the quality of the software, by using and with validat-ing and verifying models.”

6 Testing the Cloud

115

Model-Based Testing is only focused at the testing part of the ALM and not at the other disciplines; it’s not focused at improving the designs, doing evaluations of the designs, giving feedback, and helping the business with old legacy systems to cre-ate test sets, like for example regression test sets. Model Driven Quality Improve-ment is a better approach. An approach that really gives answers to the business, project and testing problems.

Model Driven Quality Improvement is the complete set of doing different things with models. Model-Based Test Design (according to the definitions above) is just one of them.

Figure 6.5 MBT as part of MDQI

With the MDQI approach testers are not only focused making the model for test-ing purposes, but making models to improve the total quality of the product and process.

To be successful and integrate with the other parties you should use a model used by the other parties. This doesn’t mean per definition that you should use exactly the same models, but there should be one model that everybody in the project agrees upon, especially the business.


116

Model Driven Quality Improvement can only be effective if all parties in the ALM work together! For example a business analyst who makes the requirements can do this in MS Excel, or in a tool for designing requirements. But if this is only text it isn’t very clear and there is room for assumptions. If the business analyst creates a model that is in a format good enough to be checked by the (test) team there is the first quick win; aligning the tasks.

ExampleAn Example of MDQIAt a telecom company testers received documentation or information from business analysts that was not good enough to create sufficiently detailed and structured test cases. This was because there was still room for assumptions and interpretations. To compensate for this, they used models (like flow diagrams) to verify the designs. In making these models, they found a lot of inaccuracy and incompleteness in the documentation.

Modeling was the first step to discovering these flaws. And in making these models the assumptions and interpretations disappeared. By doing this at the very start of the project they improved at the beginning what was able to lead to higher quality testing at the next step (development). In short, they did an evaluation focused at completeness by making a model of the design!

Integrating SaaS Services: Using the SupplierWhen creating a test strategy for the cloud standardSaaSservices are used to integrate into the application landscape of the business. These SaaS services or applications are a direct result of the rationalization behind the application portfolio: use standardized tools when possible. But how to handle these services when testing? Do they need to be tested? Or what do you or don’t you test?

For testing purposes, these SaaS applications make it somewhat easy. Because it’s a SaaS application the software the supplier offers is a complete application for end users. The software doesn’t run on a local machine, but in the cloud. These SaaS applications can also be “apps” that are installed locally but run on another service, like a cloud. Only light configuration is needed on the business side. These SaaS applications are created to support a business service and only that. For example, within Google Apps the Agenda only offers agenda functions like a calendar and tasks, unlike Microsoft Exchange that has a whole lot more to offer. The downside of course is that this standardization and simplicity offers the client very little influence over how the application works and its future development.

6 Testing the Cloud

117

Integrating these SaaS applications into the business architecture requires some skills. But more and more SaaS applications come with multiple integration options. As standards develop around this, better integration will be available with multiple other applications. But these applications do need to be tested if the integration is done correctly and the software can be accepted. Only integration and acceptance testing, not system testing? As it’s a complete application delivered by its supplier, it’s an application that should function correctly. It’s a full product.

If the SaaS application is (newly) developed to be integrated into the current architecture, some checks on the quality could be appropriate. Regular checks maybe agreed upon with the supplier to confirm that the quality is as expected, so that there are no surprises when the application is delivered. Quality Gates can help with that. Traditionally, a Quality Gate could consist of the sign off by all stakeholders of a finalized functional design before starting the technical design.

In cloud or SaaS projects the “gates” consist of checks to confirm the project is still on schedule and delivering the expected quality. These checks are agreed upon by the sup-plier and the business and integrated in the test cloud strategy. In this test cloud strategy, determine who will conduct the acceptance test. In general it’s agreed upon that a third party, for instance the test team, will execute this acceptance test for the business. The SaaS supplier is responsible for the QA of its software, the business for the acceptance and integration.

In More DetailChecking, Exploring and AcceptingThe business needs to accept that a SaaS application is suitable. But how does “accepting” correspond to testing? Testing is developed out of three things, namely:

“Check” is the determination whether a created software product complies with 1 the specifications.“Explore” is an (in depth) investigation of non-specified parts of a (created) soft-2 ware product, looking for errors.“Accept” is whether the described specification or product is the correct solution 3 to the (user’s) problem.

Instant Deployable Test Infrastructure: Testing on the Cloud

A fitting test infrastructure is required for (dynamic) testing of a test object. And so, tes-ters are dependent of the test infrastructure, because without a test infrastructure no test


118

can be executed! To ensure a good quality test object and as preparation for production, it’s recommended to have an infrastructure that makes this available. The DTAP model is a technical aid to support testing, DTAP stands for Development, Test, Acceptance and Production.

Figure 6.6 The best practice in test infrastructure: separate environments for development, test, acceptance and production

The principle of this model is that every user of the infrastructure has a type of environ-ment safe from intrusion by others. The environment types are equal to the four stages in software development: development, test, acceptance and production. This model is also of high value in the Cloud Era.

Development Environment

A developmentenvironment is the environment where the application will be developed and/or modified. During development, the application is tested for any errors by the devel-oper in the development environment. The unit test is also performed in this type of envi-ronment. The design and related test activities of this environment are conducted as part of the development process. When it’s necessary to do a test, the developer uses a part of the area to be used for these tests. Ideally the development platform offers standard facilities for testing, such as test data, test tools and procedures for versioning, transfer, defects and error recovery. When this is the case, this provides the developers with enough opportunities to manage their testing process.

An important aspect that developers face is the manageability of their environment. In practice, all too often a programmer has five or more versions of a program on its hands. Preserving the relationship between test cases, test results and the test object then demands

6 Testing the Cloud

119

a lot of attention. Once the developer agrees on the application, it is transferred to the next environment: the test environment.

The applicability of cloud solutions is ideal for development environments. A development environment is often relatively small in size and thus lends itself to relatively easy introduc-tion in cloud solutions (IaaS and/or PaaS).

Test Environment

The testenvironment exists to test (parts of) the whole system in both technical and func-tional testing. These tests should be performed in a manageable environment. Manageable means that resources like included software, documentation, test files and test ware are available and can be managed. In the transfer of new or modified software, the tester must be verifiable and tests must be reproducible.

The individual tests from one (sub) system should be able to take place separately from the tests of other (sub) systems. The simultaneous use of the same test data, especially in this context, can be responsible for many problems.

In this type of environment tools can be used to give an insight on a technical level about different events to test. Examples are the use of SQL to watch the database directly, having direct access to the system logs and the possibility to start and stop batches.

The use of cloud solutions for test environments strongly depends on the nature of the applications, the information system (administrative or technical), and the implementa-tion of test levels and test types. But criteria like (sensitivity) test data, dependencies, and interfaces determine whether a test environment can be offered using a cloud solution. The diversity of a test environment is, unlike the development environment, larger, but generally at an IAAS level.

Acceptance Environment

The acceptanceenvironment offers the future users and administrators the possibility of testing the test object in an environment that is as much as possible production-like. It happens that this type of environment is divided into a user-acceptance test (UAT) and production-acceptance test (PAT). The user-acceptance test is also regularly executed in the test environment).

Often organizations find the acceptance environment (for the PAT) expensive. This is not surprising, since for the PAT, it is important that the environment is not only functional but also technically the same as the production environment. This means, logically, that a PAT


120

environment must have the same hardware as the production environment. A PAT environ-ment is actually a second production environment: it’s production-like.

All levels of testing can be executed in cloud environments, but for some types of tests a test environment in the cloud delivers even more added value. Separate test levels can now be executed on separate environments, as desired. Testers don’t have to wait until the end of the testing phase to move to a “production-like” environment to do performance, load and stress tests. A production-like environment can be created as needed.

An end-to-end test might be set up in the cloud. All the necessary servers and images can be added to the cloud to create that end-to-end environment. When all the different parts are integrated with each other, a full end-to-end test can be executed. Even end-to-end tests that transcend a client can be done in the cloud. All necessary components can be published in the cloud to create the whole chain of systems. Thus the business processes can also be tested.

The cloud gives (testers) the opportunity to instantly deploy needed (test) infrastructure. When virtualized parts of the infrastructure are incorporated into the cloud they can be used on-demand. This enables us to create and delete infrastructure with the click of a mouse!

121

7 Cloud Risks: Worth Testing for…

Clouds can help IT. The cloud is scalable, less expensive to maintain, enables Green IT, provides instantly deployable (test) infrastructure, reduces Total Cost of Ownership (TCO), utilizes servers more effec-tively, uses independent locations, and promotes Business Technology. But this comes with a price tag. The cloud also delivers more risks for security, data integrity, privacy issues, data recovery and performance. Besides, is the business even ready for IT?

All of these risks can be addressed by testing for them. But that’ll be a costly exercise, and the attraction of cloud is the reduction in costs. Other counter measures need to be taken to create a trusted cloud solution: measures that decrease the risks the cloud created, but also increase the quality of the solution. These actions help create a better solution, better in practice and more efficient.

Testers can help the business by looking for functions and measures that can increase the quality of the cloud solution.1 These risks are some of the most common for cloud solutions, but some are more likely to occur than others in different cases where cloud solutions are

1 These actions can also help in traditional applications.


122

used. Major concerns2 where testers can support the business to integrate the cloud can be categorized as follows:

Compliance: regulations may prohibit the use of clouds for certain workloads and •data.Data privacy: a shared, multi-tenant infrastructure increases the potential for unauthor-•ized exposure, especially in the case of public clouds.Less control: some businesses are uncomfortable with the idea of their information on •systems they do not own in-house.Security management: how will today’s enterprise security controls be represented in •the cloud?Reliability: there are worries about service disruptions affecting the business.•

Figure 7.1 Cloud risks where testers can help the business

2 Other concerns are the performance and the ability of the cloud to scale up and down.

7 Cloud Risks

123

Even though the cloud enables the creation of Business Technology, the business still needs to be ready to approve and, more importantly, to use the cloud solution.

Compliance, Data Privacy and Security: A Need for Insight

Although the cloud provides tangible business benefits for organizations, the security, com-pliance (data integrity) and privacy challenges associated with the cloud require special consideration. Security is always named as the biggest issue for cloud adoption. And that is, of course, correct. But why is this so important in the cloud? Isn’t it important with tradi-tional systems? Yes, it’s just as important, but as the cloud is the hot topic of the moment, its deficiencies are receiving part of that attention.

Cloud Security Issues: All around Us

The cloud also increases risks for security and compliance, so it poses a question that needs to be asked: “is the business even ready for IT to use a cloud?” Of course it is, but the busi-ness cannot just sit back and relax. The business has to proactively work with IT to create a cloud solution that will enable the move to BT.

The cloud allows organizations to use services and store data outside their own control. This development raises security questions and should produce a degree of wariness about using cloud services. Think about these risks[Brodkin, 2008]:

•Privileged user access. Data stored and processed outside the enterprise’s direct control brings with it an inherent level of risk, because outsourced services bypass the physical, logical and personnel controls IT exerts over in-house programs.

•Regulatory compliance. Data owners are responsible for the integrity and confiden-tiality of their data, even when the data is outside their direct control, which is the case with external service providers such as cloud providers. Where traditional ser-vice providers are forced to comply with external audits and obtain security certifi-cations, so should cloud providers. Cloud providers who refuse to undergo this scru-tiny are signaling that customers can only use them for the most trivial functions. Most, if not all, of the leading cloud providers do not support on-site external audits at the request of customers. As a result, some compliance cannot be achieved because on-site auditing is a requirement that cannot be satisfied: for example, the Payment Card Industry level 1 compliance.

•Data location. The exact location of data in the cloud is often unknown. Data may be located in systems in other countries, which may conflict with regulations prohibiting data from leaving the country or state. It is advisable to investigate whether cloud providers will commit to keeping data in specific jurisdictions and whether the providers will make con-tractual commitments to obey local privacy requirements on behalf of their clients.


124

For example, the EU Data Protection Directive places restrictions on the export of per-sonal data from the EU to countries whose data protection laws are not judged as “adequate” by EU standards [EuropeanCommission, 1995]. Without proper attention, European personal data might be moved outside the EU in violation of the directive.

Figure 7.2 The data is subject to the laws of the country where the data is stored

Using the Data Protection Directive or SafeHarborPrinciples[EuropeanCommission, 1995] can help in regulating the processing of personal data within and outside of the European Union. These Safe Harbor Principles3 allow personal data to be stored outside the EU when it upholds these principles. The Safe Harbor Privacy Principles allow US companies to register if they meet the European Union requirements and obtain proper certification.

These Principles provide:

•Notice. Individuals must be informed that their data is being collected and told how it will be used.

•Choice. Individuals must have the ability to opt out of any collection and transfer of the data to third parties.

•Onward Transfer. Transfers of data to third parties may only occur where the third parties are organizations that follow adequate data-protection principles.

•Security. Reasonable efforts must be made to prevent loss of collected information.

•Data Integrity. Data must be relevant and reliable for the purpose it was collected for.

•Access. Individuals must be able to access information held about them, and correct or delete it if it is inaccurate.

•Enforcement. There must be an effective means of enforcing these rules.

3 Safe Harbor Arrangement Official Site.

7 Cloud Risks

125

In More DetailTransfer of Personal Data to Third Countries under the Data Protection Direc-tiveThird countries is the term used in EU legislation to designate countries outside the European Union. Personal data may only be transferred to third countries if that country provides an adequate level of protection. Some exceptions to this rule are provided: for instance, when the controller can personally guarantee that the recipient will comply with the data protection rules.

A European “Working Party” negotiated with the US about the protection of per-sonal data, and as a result the Safe Harbor Principles were crafted.

The United States prefers a “sectoral” approach to data protection legislation, which relies on a combination of legislation, regulation, and self-regulation, rather than governmental regulation alone. The US recommended in the Framework for Global Electronic Commerce that this approach should be led by the private sector, and companies should implement self-regulation in reaction to issues brought on by Internet technology.

To date, the US has no single data protection law comparable to the EU’s Data Protection Directive. Privacy legislation in the United States tends to be adopted on an ad hoc basis, with legislation arising when certain sectors and circumstances require (e.g., the Video Privacy Protection Act of 1988, the Cable Television Protec-tion and Competition Act of 1992, the Fair Credit Reporting Act, and the 2010 Mas-sachusetts Data Privacy Regulations). Therefore, while certain sectors may already satisfy the EU Directive (at least in part), most do not.

•Data segregation. The shared nature and massive scale characteristic of the cloud make it likely that one’s data is stored alongside data of other consumers. Encryption is often used to segregate “data-at-rest,” but it’s not a cure-all. It’s advisable to do a thorough evaluation of the encryption systems used by the cloud provider. A properly built but poorly managed encryption scheme may be just as devastating as no encryption at all, because although the confidentiality of data may be preserved, availability of data may be at risk if it is not guaranteed.

•Recovery. Cloud providers should have recovery mechanisms in place in case of a disas-ter. According to Gartner: “Any offering that does not replicate the data and application infrastructure across multiple sites is vulnerable to a total failure.” Cloud providers should have guidelines concerning business continuity planning, detailing how long it will take for services to be fully restored. For example, Zmanda, an open source backup provider,


126

offers a cloud backup solution named Amanda Enterprise.4 Figure 7.3 shows that the cloud provides an excellent means to radically simplify the data recovery process.

Figure 7.3 The cloud also offers excellent ways to help in data recovery

•Investigative support. Gartner warns that “investigating inappropriate or illegal activity may be impossible in cloud computing, because logging and data may be co-located and spread across ever-changing sets of hosts and data centers.” If cloud providers cannot provide customers with a contractual statement specifying support for incorruptible log-ging and investigation, Gartner says that “the only safe assumption is that investigation and discovery requests will be impossible” [Gartner, 2008]. Clients need to have trust in the service, and audits or third-party investigations are not always an option.

•Data lock-in. Availability of client data may be at risk if a cloud provider becomes insolvent or is acquired by another organization. Providers should have and disclose procedures whereby customers can retrieve their data as needed, and as importantly, in the data format of their choice. If the data is presented in a format proprietary to the cloud provider, it may be unusable by any other provider. The use of open standards by providers to prevent data lock-in is recommended, but not always supported.

In his paper, Guido Kok proposes a Cloud Computing Confidentiality Framework (CCCF) that will enable companies to review the possibilities for engaging in cloud-based services,

4 Amanda Enterprise for Cloud Based Data Recovery.

7 Cloud Risks

127

based on the confidentiality of the data used within the company [Kok, 2010]. The goal of the Framework is to explain the differences between security in cloud environments in the vari-ous cloud deployment models (private, public, hybrid and collaboration), and the security in present-day information security practices. As it is a good practice for every enterprise to follow such a risk management strategy to secure their data and information systems, the framework we present here will be relevant to every entity interested in working with cloud-based information systems.

In More DetailCloud Deployment ModelsThe different deployment models of the cloud are mainly Private, Hybrid and Public Clouds, as well as Community and Hosted Private Clouds.

– Private clouds. Private clouds run in the service of a single organization, where resources are not shared by other entities. “The physical infrastructure may be owned by and/or physically located in the organization’s datacenters (on-premise) or that of a designated service provider (off-premise) with an extension of manage-ment and security control planes controlled by the organization or designated service provider respectively” [Bardin, 2009]. Private cloud users are considered to be trusted by the organization, in which they are either employees, or have con-tractual agreements. – Public clouds. Public clouds are based on massive-scale offerings to the general public. The infrastructure is located on the premises of the provider, who also owns and manages the cloud infrastructure. Public cloud users are considered to be un-trusted, which means they are not tied to the organization as employees and that the user has no contractual agreements with the provider. – Hybrid clouds. Hybrid clouds are a combination of public, private, and community clouds. Hybrid clouds leverage the capabilities of each cloud deployment model. Each part of a hybrid cloud is connected to the other by a gateway, controlling the applications and data that flow from each part to the other. Where private and community clouds are characteristically either managed, owned, and located on the organization’s or a third-party provider’s site, hybrid clouds combine the char-acteristics of being both on the organization’s and third-party provider’s site. The users of hybrid clouds can be considered as both trusted and untrusted. Untrusted users are prevented from accessing the resources of the private and the commu-nity parts of the hybrid cloud. – Community clouds. Community clouds run in the service of a community of orga-nizations, having the same deployment characteristics as private clouds. Commu-nity users are also considered as trusted by the organizations that are part of the community.


128

Regardless of which delivery model is utilized, cloud offerings can be deployed in four primary ways, each with its own characteristics. The characteristics of the deployment models are:

who owns the infrastructure;1 who manages the infrastructure;2 where is the infrastructure located, and3 who accesses the cloud services.4

Figure 7.4 The basic cloud model: public, private and hybrid (including a community cloud)

Responsibility for Security

A big issue in cloud surrounding security is “who has what responsibility?” This is outside the level of the cloud, IaaS, PaaS or SaaS. In , and the breakdown of this responsibility is shown using Responsibility Matrices. For each cloud layer, the responsibility differs between the cloud provider and the client.

Security Type IaaS Provider Client

Infrastructure Security

Network level Redundancy of the network layer Implement strict default security groups IDS Logs, audits Implement DOS & DDOS filters

Secure connection (firewall + encryption)

7 Cloud Risks

129


Host level Ensure prevention and detection controls Virtualization security Restrict physical & logical access to the VM hypervisor Data storage separation Control reports (login activities)

Responsible for what data is in the cloud Should ask for security informa-tion under NDA Virtualization Software Security OS image should be hardened and running only minimum ser-vices for your application Assume your virtual server will be available to anyone online, restrict access, or keep it current Run a host firewall and limit ac-cess Install a hosted IDS Install a log server to centralize each log with a higher security protection Protect access to the hardened image and verify the integrity Require private key to access the hosts and safeguard them Restrict physical & logical access to the VM hypervisor Do not keep decryption key in the cloud Do not allow password-based authentication for shell access Require a Sudo password for super-user privileges If security is compromised, shut down the affected sector and make a snapshot for forensics

Data security

Data in-house Classify data Be aware of compliance regulation

Data transfer Encrypted connection Encrypted connection

Data storage Encrypted data Provide integrity file Track the data

Encrypted data for storage Key encryption management

Data processing Data separation Application security

As an application cannot use en-crypted data, ensure data could be used unencrypted in the cloud

Data deletion Clear and sanitize with respect to NIST recommendations

Meta data Collect Audit and Archive infra-structure logs


130


IAM

Identity provisioning

Add, Modify, Delete user capabili-ties Support for file provisioning Ideally SPML support Publish account management policies, provisioning methods, role definitions Support for developers, testers, end users and administrators API support for provisioning

Provisioning and deprovisioning Definition of virtual infrastructure roles and associated privileges Automated provisioning via an organization-wide standard such as SPML based on roles (Business and privileged) API support for provisioning users Configure virtual machines to use the LDAP/ AD when possible over an encrypted connection Configure virtual machine images with pre-populated users and groups of people who need to ac-cess the VM At first login, credentials should be changed Audit VM and remove unneces-sary users

Federated identity SSO

SAML support and how to connect it to the service identity provider (use case examples…)

Authentication management

Login and static password SSL support Delegated authentication

For IT personnel, establish a VPN connection If possible, use the LDAP or active directory or SSO via VPN Whenever possible, a VPN should be used for an application user, otherwise the application should accept authentication requests in a standard format (SAML, WS Federation, etc.)

Compliance management

Custom profile abilities Define personal user attributes Respect provisioning & deprovi-sioning policies to deny revoked authorization

7 Cloud Risks

131


Security Management

Availability management

Provide service health dashboard Redundant architecture Measure availability Set & explain maximum quota trig-gers (access to API, % of CPU, % memory …) Include features to let administra-tors set some quota trigger

Be aware of SLA Monitoring tools to check virtual server health Redundant and reliable Internet connection & network services DNS routing services and authen-tication services Availability of the virtual servers and attached storage (persistent and ephemeral) Availability of virtual storage

Access control Provide network filtering Federation support Responsible for access to the network servers and application platform infrastructure Managing the access control to administrative processes (backup, hosts, hypervisors, network mainte-nance, firewall…) through a strong authentication and role-based process to support provisioning and revocation of administrative privileges

Network access with virtual fire-wall Virtual server access Inside customer’s office, access to computers to manage cloud services should be restricted

VPC management Responsible for measuring the vulnerabilities, patching them and configuring systems owned by the CSP The scope is:

networks,•hardware,•hosts•hypervisors,•applications•management console used by •the customer to managed their virtual infrastructure

Responsible for personal comput-ers and in particular the browser Applications or services interfac-ing with the SaaS Responsible for the VPC of the vir-tual infrastructure, which includesactive VM and host imageshardening the standardized im-age, with a minimal approach to privilege configurationvirtual network configuration and virtual firewall policies

Intrusion detection

Monitoring intrusion of the cloud layer: hypervisor, application, log events, DOS & EDOS and web management console

Monitoring virtual network inter-faces Monitor log activities (VM, data-bases, applications) Monitor services delivered by third parties that you use (data encryption, storage usage…)


132


Incident response Track incident regarding host avail-ability

Track the incident Inform the user Help the CSP to remediate

Data Privacy

Collection Custom SLA capabilities Be aware of compliance & regula-tion and decide what can be stored online

Usage Specify data will be stored on a cloud Data governance to ensure data collected is only used in the con-text for which it was collected

Retention & destruction

Erase and sanitize when space is reallocated Destruction of the media when it is replaced

Destroy encryption key for en-crypted data Destroy data according to compli-ance & regulation

Location & transfer

Need to specify the data location Data cannot be transferred to third parties without notice to the data owner

Table 7.1 IaaS Responsibility Matrix showing the differentiation between the IaaS provider and the client

Security Type PaaS Provider Client


Network level Redundancy of the network layer Implement strict default security groups IDS Logs, audits Implement DOS & DDOS filters Monitor & measure

Redundant Internet connection Secure connection (firewall + encryption) Monitor & measure

Host level Ensure prevention and detection controls Virtualization security Restrict physical & logical access to the VM hypervisor Data storage separation Provide activity reports

Responsible for what data is in the cloud Should ask for security informa-tion under NDA

7 Cloud Risks

133


Application level Availability Data confidentiality & integrity Assess the platform software Provide activity reports Secure the runtime engine Sandbox architecture Publish API for authentication and authorization control SSL & federation support

Responsible for operation secu-rity management:

authentication and authoriza-•tion managementstrong password policies•implement strong authentica-•tion if supported by CSPencrypt data if possible•audit reports•

Understand applications upon which third parties depend and assess that they are secured, if applicable Ask for containment and isola-tion architecture information

Data Security

Data in-house Classify data Be aware of compliance regula-tion


Data storage Encrypted data Provide integrity file Track the data

Encrypted data for storagekey encryption management


As an application cannot use encrypted data, ensure data could be used unencrypted in the cloud


Meta data Collect Audit and Archive infra-structure logs


134


IAM


Add, Modify, Delete user capabili-ties Support for file provisioning Ideally SPML support Publish account management policies, provisioning methods, role definitions Support for developers, testers, end users and administrators API support for provisioning

Provisioning Deprovisioning Profile management Administrative management User provisioning should be done over an encrypted channel If the CSP does not support SPML, try to look for an SPML gateway to connect to the CSP Audit accounts and deauthorize unnecessary accounts




Login & static password SSL support Delegated authentication

Password policies

Authorization management

Support administrator and user roles (developers, testers, end us-ers) Support trusted networks (to con-nect to) Give access to logs & audits Ideally XACML & Oauth support Monitor creation and removal of users and who performed the ac-tion (to monitor rogue acts by cloud employees) Control access to the customer’s code repository and development environment Special rights for developers to access database, directory services, file repository … Control privileges for moving applications from development environment to test environment to production

Assignment of user privileges Logs and Audit management


Custom profile abilities Define personal user attributes Respect provisioning & deprovi-sioning policies to deny revoked authorization

7 Cloud Risks

135


Security Management


Provide service health dashboard Redundant architecture Measure availability Set & explain maximum quota trig-gers (access to API, % of CPU, % memory,…) Include features to let administra-tors set some quota trigger

Be aware of SLA Measure availability / month Redundant and reliable Internet connection

Access control Provide network filtering Federation support Provisioning support Responsible for the access to the network servers and application platform infrastructure

Provisioning Deprovisioning Profile management Administrative management Responsible for access to the ap-plication deployed, provisioning and authentication of end users

VPC management Responsible for measuring the vulnerabilities, patching them and configuring systems owned by the CSP the scope is:

networks,•hardware,•hosts,•hypervisors,•applications,•management console used by •the customers to managed their virtual infrastructure

Responsible for personal computers and in particular the browser Applications or services interfac-ing with the SaaS Responsible for the deployed application VPC domains cover:

analyze source code•black box testing•penetration testing•

Intrusion detection

Monitoring intrusion of the cloud layer: hypervisor, application, log events, DOS & EDOS, web manage-ment console and privilege escala-tion attack

Monitoring intrusion of deployed application using the PaaS platform

Incident response Track the incident Inform the user Help the CSP to remediate

Data Privacy

Collection Custom SLA capabilities Be aware of compliance & regu-lation and decide what can be stored online


136


Usage Specify data will be stored on a cloud Data governance to ensure data collected is only used in the con-text for which it was collected



Destroy encryption key for en-crypted data

Table 7.2 PaaS Responsibility Matrix showing the differentiation between the PaaS provider and the client

Security Type SaaS Provider Client


Network level Redundancy of the network layer Implement strict default security groups IDS Logs, audits Implement DOS & DDOS filters Monitor & measure

Redundant Internet connection Secure connection (firewall + encryption) Monitor & measure

Host level Ensure prevention and detection controls Virtualization security Restrict physical & logical access to the VM hypervisor Provide activity reports

Responsible for what data is in the cloud Should ask for security informa-tion under NDA Monitor & measure

Application level Availability Data confidentiality & integrity Assess the platform software Provide activity reports

Responsible for operation secu-rity management:

authentication and authoriza-•tion managementstrong password policies•implement strong authentica-•tion if supported by CSPencrypt data if possible•audit reports•

Data Security

Data in-house Classify data Be aware of compliance regula-tion

7 Cloud Risks

137



Data storage Encrypted data Provide integrity file Track the data Data storage separation

Encrypted data for storagekey encryption management


As an application cannot use encrypted data, ensure data could be used unencrypted in the cloud


Verify that data has correctly been erased with provider

Meta data Collect audit and archive infrastruc-ture logs

Log access

IAM


Add, Modify, Delete user capabili-ties Support for file provisioning Ideally SPML support Publish account management poli-cies, provisioning methods and role definitions User provisioning should be done over an encrypted channel

User provisioning and deprovi-sioning Role awarding User provisioning should be done over an encrypted channel If the CSP does not support SPML, try to look for an SPML gateway to connect to the CSP Audit accounts and deauthorize unnecessary accounts




Credential management SSL support Delegated authentication

Password policies

Authorization management

Support administrator and user roles Support trusted networks (to con-nect to) Give access to logs & audits Ideally XACML & Oauth support

Assignments of user privileges Logs and audit management


Custom profile feature Define personal user attributes Respect provisioning & depro-visioning processes to deny revoked authorization


138


Security Management


Provide service health dashboard Redundant architecture Measure availability

Be aware of SLA Measure availability / month to report Redundant and reliable Internet connection

Access control Provide network filtering Federation support Provisioning support

Provisioning & deprovisioning processes Profile management Administrative management processes

VPC management Responsible for networks, hard-ware, hosts, applications and stor-age owned by the CSP

Responsible for personal computers and in particular the browser Applications or services inter-facing with the SaaS

Intrusion detection Monitoring intrusion of the cloud layer: hypervisor, application, log events, DOS & EDOS, web manage-ment console and privilege escala-tion attack

NA

Incident response Track incident Troubleshoot Keep customer informed

Track the incident Inform the use Help the CSP to remediate

Data Privacy

Collection Custom SLA capabilities Be aware of compliance & regulation and decide what can be stored online

Usage Specify data will be stored on a cloud Data governance to ensure data collected is only used in the context for which it was collected



Destroy encryption key for encrypted data

Table 7.3 SaaS Responsibility Matrix showing the differentiation between the SaaS provider and the client

7 Cloud Risks

139

By using these tables, a matrix is created that enables the business, the cloud provider and the (security) test team to understand who does what to maintain security in the cloud.

Static or Dynamic Security Testing: Do Both!

To determine how secure the cloud is, the Responsibility Matrix will help, but another option is to directly test the security of the cloud. This can be done by doing a static security test to test the application or service from the inside out—by examining its source code, byte code or application binaries for conditions indicative of a security vulnerability. Another option is a dynamic security test of the application to test the application from the outside in—by examining the application in its running state and trying to poke it and prod it in unexpected ways in order to discover security vulnerabilities.

Both static and dynamic security testing look at the application by itself in the SaaS layer. In a cloud environment, security testing should be applied on threelayers, namely the service, the infrastructure and the platform layers. Additionally, security testing has to be performed by two different parties, on the one side by the cloud providers, offering Software as a Service, and on the other side, by cloud consumers, developing custom applications to be executed inside cloud environments. As the latter more or less make use of Platform and Infrastructure as a Service, the cloud provider itself cannot assure the security of a cli-ent’s application, as application-specific code often introduces its own risks, which have to be addressed by the developers or a dedicated security testing team. However, this should not be an excuse for cloud providers to disregard testing of the security of the cloud’s infrastructure (PaaS and SaaS). Moreover, this highlights a good reason to aggressively test and consequently mitigate risks to the infrastructure itself and its internal interactions (like operating system calls, network communication, file system access, and the like), to guarantee that application run smoothly. This is due to the fact that such a secure cloud infrastructure boosts the overall security of the environment and results in even more secure systems. Hence, no matter which side or layer of a cloud a developer or tester is operating from, assuring security through sophisticated testing plays a crucial role.

Many vendors of security testing tools provide static and dynamic (application) security testing capabilities. The ability to test an application both statically and dynamically will become increasingly important, for these reasons:

Some vulnerabilities can be found only with static security testing, others with dynamic •security testing; testing in both ways yields the most comprehensive testing.Many web applications that would traditionally be scanned with dynamic security testing •tools also use a significant amount of client-side code, in the form of JavaScript, Flash, Flex and Silverlight. This code must also be analyzed for security vulnerabilities, typically using static analysis.


140

This is true for traditional applications and even more for cloud applications where we have three different layers with security issues to test.

Control: Private vs. Public Cloud in Security

The common view of private versus public cloud computing suggests that it is prudent to permit only public information to enter public cloud environments. With the process described in the cloud computing confidentiality framework, an organization can assess security controls that need to be in place to protect the information in question.

When an organization considers a cloud service offering an operational environment for the information in question, both parties can perform a gap analysis to determine which security controls are required to ensure the integrity of the information, and which security controls the cloud service provider supports. The difference between the required controls and the supported controls is called the security gap. To reduce the organizational risk that the security gap imposes, the NIST recommends the following three options to close the gap between what security is needed and what security external service providers offer:

“Use the existing contractual vehicle to require the external provider to meet the addi-•tional security control requirements established by the organization” [NIST, 2009a].“Negotiate with the provider for additional security controls (including compensating •controls) if the existing contractual vehicle does not provide for such added require-ments” [NIST, 2009a].“Employ alternative risk mitigation measures within the organizational information sys-•tem when a contract either does not exist or the contract does not provide the necessary leverage for the organization to obtain needed security controls” [NIST, 2009a].

If the cloud provider can implement the additional controls demanded by the organization, the public cloud environment of the provider meets the security requirements set by the organization.

The issue here is that giving up control makes it hard to keep overall complexity under control. IT groups end up purchasing services, business users provision their own solutions, and so forth (see Figure 7.5). Redundancy and cost explosion are real risks. The risk is often not truly acknowledged by business users who just want quick solutions.

A communal business and IT approach, like enterprise architecture, provides guidelines on how to use cloud services and ensure an active role for the IT department in brokering services. Business and IT governance processes should include strong controls that focus on continuously reducing complexity [SeizeTheCloud, 2011].

7 Cloud Risks

141

Figure 7.5 A risk for the cloud: sharing control

Reliability: Cloud Recovery Testing

The cloud is often considered unreliable, but is that accurate? Fortunately, the reliability of the cloud can be tested, as testing is an essential part of disaster recovery planning and a key component of the Disaster Recovery Plan (DRP). The DRP shows the processes, policies and procedures involved in preparing for recovery or continuation of technological infra-structure critical to an organization after a natural or human-induced disaster. Disaster recovery is a subset of businesscontinuity. While business continuity involves planning to keep all aspects of a business functioning in the midst of disruptive events, disaster recovery focuses on the IT or cloud systems that support business functions.

Frequently the creation of a DRP results in a false sense of security. Management has the tendency to relax and think they have that part of the system in order. But if you don’t test your disaster recovery plan, there’s a risk that it won’t work as expected when it’s really needed. The Disaster Recovery Test as part of businesscontinuity testing is becoming an annual event for most IT departments, but with the cloud it should become an even more regular event so that the business can check whether their services are sufficient regulated. As more trust is imparted to the cloud it’s a very healthy thing to do, and it is mandated by a lot of regulators and almost insisted upon by internal audit teams.

With testing business continuity, the implicit risks in the business process are revealed. The most missioncritical processes need to be tested more thoroughly and the taskcriti-cal processes less thoroughly. As Figure 7.6 shows, the criticality of the business process is calculated by the chance of failure and the cost of potential damage if, in this case,up-time and businesscapability is lost.


142

Figure 7.6 Business continuity: the importance of maintaining the business processes

Whenever possible, choose a remote site recovery strategy that can be tested as frequently as possible. Testing the disaster recovery process has the following benefits:

showing that the disaster recovery plan works;•discovering problems, mistakes, and errors, and resolving them before they are relied •upon to work;educating staff in executing tests and managing disaster recovery situations;•reminding the members of the organization, IT and business, of the necessity of such a •disaster recovery plan and the importance of planning accordingly; andincreasing awareness of the disaster recovery strategy.•

After each test, use the detailed logs and schedules to identify any errors in your procedures, and eliminate them. Retest the changed procedures, and then incorporate them into your recovery plan. After changing the recovery plan, completely revise all existing disaster recovery documents.

Steps to Take in a DRP Test

First, performing a DRP test without proper risk management can put an organization at significant risk. To put things into perspective, let’s analyze the steps, risks and (counter)measures of a disaster recovery test; see Table 7.4 [Spirovski, 2010].

7 Cloud Risks

143

DRP Test Step

Activity Risks Measures

1 Failure of primary systems

In order to per-form a disaster situation, the primary systems need to be caused to fail on some level

Databases damaged or not saved properly due to forced shutdown or forced power failureHardware components failing due to forced shutdown or power failureSpilt-brain cluster due to uncontrolled sequence of server and storage failures

Perform full backup prior to the initiation of the DRP testBackup components and vendor support are on hand during the entire testNot performing a direct forced shutdown but forcing a network-level isolation at the routers

2 Activation of Disaster Recovery systems

Severing any relation between the DR and the primary systems and running the DR systems as pri-mary, temporarily

Actual failure of primary system during the testFailure of the primary system while the DR system is found to be non-functional

Every interested party must be fully aware of the test, including business custodians, directors of divisions and top management who would initi-ate the real Business Continu-ity PlanFull backup prior to the initia-tion of the DRP test at DRP site, and full vendor support.

3 Reconfig-uring the user envi-ronment

Intervening in the end-user environ-ment in a way that will make them use the DR system

Configuration error that might cause the end-user to input test data into the primary systemsConfiguration error that might cause the primary system to stop functioning

Scripted and documented steps of reconfiguration, where all steps should be performed by 2 persons: one observing the other’s actions

4 Revert-ing to the primary systems

Resuming the primary systems at some level and reestablishing the relation between the DR and the primary systems

Configuration error that might cause the primary system to stop functioningCopying of test data that was input into the DR test system back into the primary loca-tionFailure of primary sys-tems during resumption

Scripted and documented steps of reconfiguration, where 2 persons should perform all steps: one observing the other’s actions.Fully controlled and docu-mented process of resumption, which guarantees that only the primary system is data masterFull backup prior to the initia-tion of the DRP test, Backup components and vendor sup-port on hand during the entire test

Table 7.4 Steps, risks and measures of a disaster recovery test


144

With all these risks, is it more prudent not to perform a DRP test? Absolutely not! Performing the DRP test actually confirms that things are running. And if something breaks, you can be better prepared for it next time. Not performing the test will just make the business think everything is great, until a problem occurs. And that problem is certain to arise sometime. Execute frequent tests early during cloud implementation or when the disaster recovery plan is prepared. Once all the major problems have been removed, less frequent testing is possible. The test frequency will depend on:

the interval between major changes in the cloud infrastructure, platform or software;•how current the business wants to keep the recovery plan; and•how critical and sensitive the business processes are (which is an issue with all testing, •but the more critical the processes are, the more frequently testing may be required).

So, perform the DRP test regularly, but with a whole set of countermeasures for the possible problems that could arise during the test.

In More DetailCloud Cracks: Break Down or SupportIn April 2011 the Eastern US data center of Amazon’s Web Services (AWS) failed of. This can be the start on some negative talking about the cloud. The cloud is certainly starting to show its cracks.

But is this a bad thing? Is it a not good thing for the cloud’s image that it wasn’t reacting more so than multiple sites and services, like Reddit and Foursquare, that went down? No, it’s not! For the first time we’ve seen what can happen when the cloud fails. And we can now keep it in mind when creating these services and we’ll know what to do when it happens again.

To anticipate on this cloud users should “design with failure in mind.” A cloud is also software and software can fail. But in early 2011 everybody was on “Cloud 9” with the cloud and only the pessimists were talking about cloud failures. Now they’ve seen what can happen and they can prepare for it. Not only the cloud pro-viders like AWS, but also the owners of those cloud services.

Cloud providers and brokers should have recovery mechanisms in place in case of a disaster. According to Gartner: “Any offering that does not replicate the data and application infrastructure across multiple sites is vulnerable to a total failure.” Cloud providers should have guidelines concerning business continuity planning, detailing how long it will take for services to be fully restored.

7 Cloud Risks

145

According to the research firm Gartner, the cloud market will grow to €71 billion next year. Companies will continue to weigh the benefits of the cloud, its massive cost savings and easy scalability, against the relatively small risk and annoyance of outages—risks and annoyances that could also have happened with a failure in their own non-cloud data centers. The level of cost savings gained from moving to the cloud depends on all kinds of variables. The study by Booz Allen Hamilton “The Economics of Cloud Computing” [BoozAllenHamilton, 2009] found that “the benefit-to-cost ratio of a non-virtualized 1,000-server data center could reach 15.4:1 after implementation, and total life cycle cost may be 66% lower than maintaining a traditional data center.”

In the AWS case, luckily nothing catastrophic happened; no private data was lost or “left on the street.” So everybody can sleep well. However, this failure helps to build the hype around the cloud into a more supported use of its services. The fail-ure doesn’t make cracks in the cloud so it’ll break. But it will make cloud construc-tion better and we will be able to enjoy the benefits even more in the near future.

Amazon, other cloud providers and their cloud customers will learn from this out-age, because these savings from cloud are simply too great to dismiss, and they will keep the customers coming.

Specifying a Disaster Recovery Test

In order to determine which business processes to test, testers must keep the core of the planning process in mind. Each business-critical process defined in the DRP should be completely reassessed and prioritized based on the Cloud Risk Analysis (see Chapter 5) of threats, vulnerabilities and safeguards. Performing mandatory recovery testing on pro-cesses with a high CR is a no-brainer and easily defensible to the business. It is the less obviously critical processes that will require decisions from the business as to what levels they deem acceptable

This process can be simplified by using the damage ratio of the CRA. The CRA is then used as input for a ranking system with which the business can make decisions based on empirical data as opposed to subjective evaluations. The damage level that was determined in step four of the Cloud Risk Analysis is the input for a ranking system; when using numbers while determining the damage level these numbers can be directly used to determine the DRT.

In More DetailWhen another method is used to classify the damage level, one that isn’t easily quantifiable, those values can be transformed into numerical data.


146

For example, the standard TMap NEXT® uses the values High, Medium, and Low. These values need to be transferred to numerical values. This can be done easily by replacing “3” for High, “2” for Medium, “1,” and “0” for Low or None. But a method that provide for better distinctions is preferred.I prefer to use the numerical range “0, 1, 3, 5, 9.” The differentiation in the numbers reflects a wider difference between the risks. The “0” and “1” are directly exchange-able with None and Low, but Medium can now be dispersed between “3” and “5,” and High between “5” and “9.”

When reassessing the risk values, the business should also keep in mind the levels of damage. By doing so, it will be clear what effects risk can have on different levels of damage.

Characteristic: Functionality

Process Sub-process

Product requirement Damage Arguments

Sales ---- Compliance with the func-tional requirements

9 Loss of revenue if the sales process breaks down

Sales Advice With an eye to the legal duty of care, the advice given and how the client decides to devi-ate from the advice must be recorded

5 High fines and negative press will result for the company if this func-tionality does not work (correctly)

Sales Offer The offer must contain the correct price

3 An incorrect price may result in loss of revenue

Booking Offer The accepted offer must be filed with the date in the filename

1 The date is automati-cally stored in the data of the offer

Table 7.5 Damage table for the characteristic Functionality

In the example provided in Table 7.5, processes with a damage level score of 9, 5 and 3 will be tested, but a process with a score of 1 will not.

The key to this method is that the business has provided information based on current analysis, and therefore has set the boundaries for testing; the business is in charge of the testing.

7 Cloud Risks

147

Performance of the Cloud: Test It?

Another common concern about the cloud is its performance. Does it meet the enormous expectations held out, including the hype that its performance is scalable to infinity? The answer is “no.” There is no infinite amount of servers in cloud centers. But there is enough to serve the scale of performance needs. There are not many applications (including web-sites) that need a load of more than 100,000 virtual users, and even fewer need a load of more than 1,000,000.

When using traditional testing to create a load for these applications it can be a very expensive experience. A license for 100,000 virtual users with the two most often-used performance test tools (HP Performance Center and IBM Rational Performance Tester) will cost a lot of money. The cloud can leverage this load, which is only actually needed a few times a year. It can create the load for the few minutes that it is needed, reducing the costs overall!

But the cloud also provides performance. It may not be indefinite, but it is large. What is large and what is enough? There are two ways to look at this. First, one could do a bench-mark test of the performance of the application and repeat that performance in a cloud. The problem is that the application will only perform as it does now. But performance requests can also exceed the business capacity. For example, seasonal variations at Christmas might even generate requests far in excess of maximum capacity. Not every system has the same pattern of performance usage. Cloud centers are equipped to support the need for performance on an overall average rating; they can handle the performance requests as needed, giving the appearance of infinite capacity.

But cloud providers cannot have a multitude of services running to satisfy the same per-formance requests. For example, if a cloud has multiple clients all with salary calculations, that creates a performance problem. That performance problem will be shown at the end of the month, when the calculations are generally done. When all of those services are executed at the same time, they put a larger demand on the performance capacity, which will lead to the same issues that exist today when computing power is needed.

A cloud client needs to have insight into the cloud’s usage volumes and practices, either through a Service Level Agreement (SLA) or through guarantees by the cloud provider. When selecting a cloud provider, this risk needs to be taken into account. Cloud providers are reluctant to give information about their usage volumes, but if a client wants to be sure the cloud can leverage its performance requests, this information is going to be necessary.


148

Elasticity of the Cloud Service: Use the Cloud for Load Testing

The same principles that exist for testing traditional applications also exist on cloud ser-vices. These services can also be tested for performance issues. However, when testing the performance of cloud services, they’ll be more related to the elasticity of the cloud, or, in other words, the on-demand scaling up of the service. The usage of (automatic) load pat-terns helps test the cloud performance.

For example, IBM Rational Performance Tester and SOASTA Cloud Testing have the option to test an application with standardized load patterns. These patterns represent several ways in which cloud applications are used. Patterns like those shown in Figure 7.7 show the different patterns that emerge:

•Unpredictable burst. When the usage volume of a service is not yet known, or the service is known to fluctuate in an unpredictable manner, this pattern can test whether the service can handle that fluctuation.

•Predictable burst. When the usage of a service is predictable, like for example seasonal variations, the load patterns can be adjusted to provide the necessary elasticity of the service.

Figure 7.7 Automatic load patterns to help test the elasticity of the cloud service

7 Cloud Risks

149

•Periodic usage. Services that are used at specific times in a certain period have special load patterns. When, for example, a service runs twice a month and the rest of the month the usage rate is nearly nil, it only needs the computing power at those moments. The load pattern should be adapted to test this situation.

•Hyped usage. Sometimes there is a lot of hype creating frenzied requests for informa-tion. Information requests or news feeds can be subject to enormous demand. Testing the maximum load of these services can provide insight in the risks that are associated with these issues.

In More DetailAt the end of 2010 WikiLeaks was in the news about the unveiling of US Foreign Office diplomatic documents and a criminal investigation of Julian Assange, the founder of WikiLeaks. After the news broke that PayPal, MasterCard and Visa were not allowing any money transfers to WikiLeaks, these organizations were hit by hackers with DDoS attacks. DDoS stands for distributed denial-of-service, and it is an attack on a website to make it unavailable to its potential users. As a result, the sites of MasterCard, Visa and PayPal were unavailable. They went “down.”

Commonly a DDoS attack saturates the target machine (the website) with external communication requests. As a result, the target cannot respond to the traffic, or responds so slowly as to be rendered effectively unavailable. It consumes the tar-get’s resources so that it can no longer provide its intended service, or obstructs the communication media between the intended users and the victim so that they can no longer communicate adequately.

WikiLeaks also gets hit a lot with DDoS attacks, but they have thought of an answer to these attacks: they moved their website into Amazon’s EC2 cloud. As only the website itself has been transferred to Amazon, for now the data is hosted in France, so the US government cannot make any claim to the data. With this solu-tion, WikiLeaks hopes to fend off any attacks on its website. As the cloud is elastic, it can increase its saturation by adding extra services from the cloud. As a result, it won’t be totally swamped and will function normally. The cloud helps in security, instead of being a security risk!


150

Figure 7.8 A graphical explanation of a DDoS attack [Wikipedia]

In More DetailA Winter CloudIn the last few days the Netherlands was seized by Father Winter. Global warming at its best? Or just a cold end of 2010? The fact is that transport systems have had problems with the snow and ice. I myself got stuck in Dublin at the start of this month because of the winter weather. Not only flights were cancelled, public trans-port came to a halt and we all got stuck on the freeway for hours. The websites of these transport organizations were unavailable, due to heavy traffic. As a result

7 Cloud Risks

151

people were unable to get information about their travel and didn’t know what to do.

The websites were overloaded with traffic from people looking to see if their flight was still going, rebooking their cancelled flight, checking the times of delayed trains, looking for traffic jams before they left or seeing what roads were closed down due to ice and snow. All this information became unavailable. But what could help find a solution to this problem?

The websites were unavailable because they were not prepared for that amount of traffic. They didn’t have enough bandwidth and servers available; they were not scaled to that large amount of requests. If the websites had been in a cloud envi-ronment they could have had that scalability, which would have allowed adding more services when the available amount of services came to a critical low.

This not only would have helped this past winter, but also during the ash cloud in Western Europe when airline web traffic was overloaded due to the eruption of the Eyjafjallajökull volcano. A cloud offers websites the needed scalability!

All These Risks: Is the Business Even Ready for the Cloud?

The cloud will keep the promise IT has made to the business. It will support the business and create Business Technology. But with all these risks, is the business even ready for the cloud? The cloud is not the magic solution IT needs to help them achieve this BT. The busi-ness needs to support the move to using the cloud. And support means not only permitting cloud use, but also actively embracing it as a change in the business mindset.

Services will be standardized with fewer unnecessary or less commonly used features. There should be a strategy as to which data and services are stored in the cloud, what type of cloud deployment model is used for which services or data, and what are the non-functional requirements of the services. As business manager you need to know what the requirements and usage capacity of the cloud is. You’re not going to the cloud just because there’s a lot of hype about it, but because it serves a need: a need that is greater than merely reducing the Total Cost of Ownership. The cloud serves the larger demand for greater flexibility and faster time-to-market!

153

References

[Aalst, 1999] L. van der Aalst,and C. de Koning, Testing expensive? Not testing is more expen-sive!, Informatie, October 1999, www.tmap.net

[Bardin, 2009] J. Bardin, J. Callas, S. Chaput, P. Fusco, F. Gilbert, C. Hoff, D. Hurst, S. Kuma-raswamy, L. Lynch, S. Matsumoto, B. O’Higgins, J. Pawluk, G. Reese, J. Reich, J. Ritter, J. Spi-vey, and J. Viega. Security guidance for critical areas of focus in cloud computing. Technical report, Cloud Security Alliance, April 2009

[BDTM, 2008] TMap NEXT: Business Driven Test Management, L. van der Aalst, R. Baarda, E. Roodenrijs, J. Vink and B. Visser, 2008

[Black, 2002] R. Black, Keynote at Eurostar2002, Copenhagen

[Boehm, 1981] B.W. Boehm, (1981), Software Engineering Economics, Prentice-Hall Inc., Engle-wood Cliffs, isbn 0-13-822122-7

[BoozAllenHamilton, 2009] T. Afford and G. Morton. The Economics of Cloud Computing, 2009, http://www.boozallen.com/media/file/Economics-of-Cloud-Computing.pdf

[Brodkin, 2008] J. Brodkin, Gartner: Seven cloud-computing security risks, Retrieved September 23, 2009, from Network World, from http://www.networkworld.com/news/2008/070208-cloud.html

[Davis, 2010] Testing in Agile Software Development Environments with TMap NEXT®, C. Davis and L. van der Aalst, 2010 (http://www.tmap.net/Images/Testing_in_Agile_Software_Deve-lopment_Environments_with_TMap_NEXT_sept%202010_tcm8-59700.pdf)

[EuropeanCommission, 1995] 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data. Official Journal of the EC. 23

[Gartner, 2008] Assessing the Security Risks of Cloud Computing, Retrieved December 5, 2009, http://www.gartner.com/DisplayDocument?id=685308

[Gillett, 2008] Cloud Computing is Hyped and Overblown, Forrester’s Frank Gillett.....Big Tech Companies Have “Cloud Envy”, 2008

[Glauser, 2010] Cloud Washing 101 - The cloud marketing playbook, R. Glauser, http://service-now.web5.hubspot.com/Blog/bid/42570/Cloud-Washing-101-The-cloud-marketing-playbook

[Hoenig, 2009] Connecting to the Cloud, G. Hoenig, 2009 (http://blogs.i365.com/cloud-connec-ted-recovery/connecting-to-the-cloud/)

[IDC, 2009] IDC Cloud Services Forecast, F. Gens, October 5, 2009

[IDC, 2010] Defining “Cloud Services” and “Cloud Computing,” F. Gens, 2008

[IEEE, 1998] The Institute of Electrical and Electronics Engineers, Inc. (1998), IEEE Std 1028-1997


154

Standard for Software Reviews, 345 East 47th Street, New York, NY 10017-2394, USA, isbn 1-55937-987-1

[ISO, 1994] ISO 8402 (1994), Quality Management and Quality Assurance: Vocabulary, Internati-onal Organization of Standardization

[ISO 9126-1, 1999] ISO/IEC 9126 part 1 (1999), Information Technology: Software Product Quality – Part 1: Quality Model, International Organization of Standardization

[ISO/IEC, 1991] ISO/IEC Guide 2 (1991), General terms and definitions concerning standardiza-tion and related activities, International Organization of Standardization

[Kok, 2010] G. Kok, Cloud Computing & Confidentality (2010), home.student.utwente.nl/g.r.kok/.../CloudComputingAndConfidentiality.pdf

[MBT, 2006] Practical Model-Based Testing: A Tools Approach, M. Utting and B. Legeard, 2006, Morgan Kaufmann; isbn-13: 978-0123725011

[McCall, 1977] J.A. McCall, P.K. Richards and G.F. Walters (1977), Factors in software quality, RADC-TR-77-363 Rome Air Development Center, Griffis Air Force, Rome (New York, USA)

[NIST, 2009a] NIST/SEMATECH, e-Handbook of Statistical Methods, 2009, http://www.itl.nist.gov/div898/handbook/eda/section2/eda23.htm

[NIST, 2009b] The NIST Definition of Cloud Computing, P. Mell and T. Grance, 2009

[Parkhill, 1966] The Challenge of the Computer Utility, D. Parkhill, 1966

[Pettey, 2010] Gartner Highlights Key Predictions for IT Organizations and Users in 2010 and Beyond, Christy Pettey and Holly Stevens, January 23, 2010

[Ried, 2010] The Evolution Of Cloud Computing Markets, Stefan Ried, Ph.D., Holger Kisker, Ph.D., Pascal Matzke, July 6, 2010

[SeizeTheCloud, 2011] Seize the Cloud, M. van den Berg and E. van Ommeren, March 2011, Sogeti Group

[SMART, 1981] There’s a S.M.A.R.T. way to write management’s goals and objectives, G.T. Doran, (1981). Management Review, Volume 70, Issue 11 (AMA FORUM), pp. 35-36

[Spirovski, 2010] Mitigating Risks of the IT Disaster Recovery Test, Retrieved April 1, 2010, from http://www.shortinfosec.net/2010/03/mitigating-risks-of-it-disaster.html

[TestTopics, 2005] TMap Test Topics, T. Koomen and R. Baarda, 2005

[The Standish Group, 2011] The Standish Group, (2011), CHAOS MAnifeStO: The Laws of CHAOS and the CHAOS 100 Best PM Practices, The Standish Group International, Boston, MA

[TMap, 2002] Software testing: a guide to the TMap approach, M. Pol, R. Teunissen and E. van Veenendaal; isbn 0-201-74571-2

[TMapNEXT, 2006] T. Koomen, L. van der Aalst, B. Broekman and M. Vroon (2006), TMap Next: for resultdriven testing, ’s-Hertogenbosch: Tutein Nolthenius Publishers, isbn 90-72194-80-2

[TMapSOA, 2008] TMap SOA Model, S. Hoevenaars, J. van Lieshout, J. Berends and J. Door-man, 2008

[Wikipedia] Wikipedia, http://en.wikipedia.org/wiki/Main_Page

[Zech, 2010] Risk-Based Security Testing in Cloud Computing Environments, P. Zech, 2010

155

Index

Aacceptance environment 119acceptance test 38accountability 105adaptability 103agile 18, 107availability 101availability testing 102

BBDTM 17, 78, 79business case 13, 83business driven test management (BDTM) 78, 79Business Technology 18, 47, 55, 109

Ccloud:

characteristics 48cost reductions 52layers 58performance 147

cloud applications 108cloud business model 48, 78cloud computing:

definition 47, 48history 12

cloud deployment models 127Cloud Era 11, 13, 47cloud infrastructure 51, 97

accountability 105adaptability 103agile approach 107availability 101functional testing 105non-functional quality attributes 101


156

reliability 103scalability 101security 104

cloud provider 59, 87, 97, 101, 123, 139, 147cloud risk 87, 121, 123cloud risk analysis (CRA) 86

product risk analysis vs. ~ 91steps 90

cloud service models 59cloud services 48

definition 48cloud test strategy 78, 94cloud washing 50community cloud 127compliance 123

Ddata privacy 123Data Protection Directive 124, 125development environment 118development test 38Disaster Recovery Plan (DRP) 141, 142Disaster Recovery Test 145DTAP model 118

Eelasticity 49, 104, 148environment 52evaluation 24, 35

GGreen IT 52, 121

Hhybrid cloud 127

IIaaS 58, 59infrastructure 51, 63, 97

traditional 51, 98Infrastructure as a Service (IaaS) 59

Index

157

Lload testing 148location-independent access 52, 54, 65

MManaged Testing Services (MTS) 63Model Driven Quality Improvement 113, 114

Nnon-functional requirements 111, 112

Oon-demand 49, 51, 54, 61, 77

PPaaS 58, 59pay-per-use 49, 61, 77, 105performance test 68, 92, 147Platform as a Service (PaaS) 59PointZERO® 19, 111private cloud 127, 140process-cycle test 110program interface test 106public cloud 127, 140

Qquality assurance 29quality attributes:

non-functional 101Quality Gate 117quality management 29

Rreal-life test 103recovery testing 141regression 40regression test 40, 41reliability 103, 141reliability testing 102requirements:

non-functional 111, 112resource pooling 49


158

Responsibility Matrix 128risks 26, 86, 121, 123

SSaaS 58, 59

integrating SaaS services 116Safe Harbor Principles 124scalability 101security 104, 123

responsibility for ~ 128security issues 123security testing 139self-service 49service 47, 57Software as a Service (SaaS) 59Software Testing as a Service (STaaS) 58, 60, 61STaaS 58, 60

benefits 74challenges 75drivers for adoption 60governance model 65process 61real-enough-time 62, 63real-time 62

STaaS provider:process model 69services 67

system test 38

Ttest basis 39test design pattern 71test environment 119test goal 83test goal table 85test infrastructure 63, 117testing:

~ and quality management 29

~ and system development process 33benefits 28definition 23execution 30

Index

159

levels and responsibilities 37reasons for ~ 26role of ~ 29structured ~ 41ways of ~ 32

test level 37, 39test strategy 78, 94test type 40TMap 13, 15, 16

essentials 13history 15

traditional application 84, 111traditional infrastructure 51, 98

Vvirtualization 49V model 33, 34, 37

WWork Package broker 63

omslag-TMapNEXT.indd 1 18-05-2011 15:30:21

Documents

omslag-TMapNEXT.indd 1 18-05-2011 15:30:21 · Alfonso López de Arenosa Anantharaman Iyer Dan Hannigan A special thank you goes to my Sogeti colleagues Leo van der Aalst, Nicolas