Website Evaluation in the Tourism Industry Evaluating Quality of Customer Service in Airline B2C Systems Prof. Niv Ahituv, Gil Amoray כנס תיירות ואינטרנט

Website Evaluation in the Tourism IndustryEvaluating Quality of Customer Service in Airline B2C Systems

Prof. Niv Ahituv, Gil Amoray

' כנס , ן נטוויז מכון הגלובלי בכפר ואינטרנט , 013תיירות האינטרנט 8.6.09לחקר

Based on a Research performed for M.Sc. Thesis in Management Sciences - Technology and Information Systems, 2006

2

Research Introduction, Objectives and Scope

Assumptions and Hypotheses

Methodology

Results and Best Practices

Agenda

3

WWW introduced new method of communication:


In-Store Service/Sales

Phone Service/Sales

Web-Based Service/Sales

Companies and non-profit organizations have recognized the

benefits and values of online communication channels

Reducing operating costs

Increasing sales opportunities

Improving customer service by:

Reducing wait times and expanding service hours (24/7/365)

Enabling self-sufficiency of customers

Research Introduction

4

Define criteria for the evaluation and measurement of the

Quality of Service (QoS) in airline B2C systems

Demonstrate the applicability of those criteria through an

empirical study

Substantiate applicability of evaluation tool (AWET-- Airline

Website Evaluation Tool) in other industries

Research Objective


5

Website Evaluation

Highlights of Literature Review

Highlights of Literature Review

Interface & Design

18 evaluation criteria:

• Graphic design and multimedia

• Style & text

• Flexibility

20 evaluation criteria

• Logical structure

• Ease of use

• Search engine

• Help functions

• Navigation necessities

Navigation


• Prod/Service-related content

• Company info

• Information quality

• Interactivity

Reliability

28 evaluation criteria:

• Customer profile

• Order process

• After-sale support

• Customer service

Technical Aspects


• Speed

• Security

• Software and database

Content

6

Assumptions: tested using non-statistical sample

A1 – Streamlined process and improved website design are the

main reasons for users to favor a website

User group had to identify dominant criteria in favoring websites:

Website design, Process, Content, Technical Aspects, Other

A2 – Users do not consider technical aspects to be a significant

factor in the quality of service provided by airline websites

Technical aspects were not mentioned as a contributor for an award-winning

website in previous studies

Research Assumptions


7

Hypotheses: tested by evaluating 33 airline websites

H1 – Web-based QoS level in the airline sector is higher in N.A

compared to Europe and Israel

Percentage of online sales: 40% in N.A vs. 14% in Europe and lower rate in IL

Purchasing patterns are expected to drive higher standards of service

QoS incurs costs, therefore investment decisions seek value and ROI

H2 – Web-based QoS of competitors working in the same region and

industry will be similar

High competition is expected to result in resemblance of websites w/i industry

H3 – QoS in websites of “low-fare” airlines is superior comparing to

QoS of “Major” Airlines’ websites

Direct sales (e.g.: online) are expected to impel outstanding service levels

Research Hypotheses


8

Hypotheses Analysis

Results & Best Practices

H1 – Web-based QoS level in the airline sector is higher in N.A

compared to Europe and Israel

Validated: Average total score of 67% vs. 64% vs. 59% respectively

H2 – Web-based QoS of competitors working in the same

region and industry will be similar

Could not be categorically validated or refuted: airlines differ by category

H3 – QoS in websites of “low-fare” airlines in superior

comparing to QoS of “Major” Airlines’ websites

Rejected: Average total score of 67% for major airlines vs. 63% for low-fare

Possible explanation: better capabilities to make significant investment by

major airlines

X

9

Innovative multi-phased approach, embracing various techniques:

Methodology

Methodology

Phase 1 Phase 2 Phase 3

Implementation• Literature review

• Identification of evaluation criteria

• Identification of participating websites

• Preliminary questionnaire

• Industry characteristic mapping

• Development of AWET, v1

• SME review and feedback

• Development of AWET, v2

Phase 4

Analysis• Self evaluation of

33 websites• Results analysis

• Hypotheses validation

Evaluation Tool DevelopmentPreparations

10

Interface and Design

Navigation

Content

Technical

Reservation Process

Usability

Customer Support

Evaluation Categories in AWET

Methodology

Total of 73 Questions in 7 Categories

30 Yes/No Questions

43 Descriptive Questions

11

Implementation

Self evaluation of 33 airline websites

Methodology

Major Airlines Continental

United

American Airlines

Delta

U.S. Airways

Air Canada

Northwest Airlines

Alaska Airlines

Hawaiian

Alitalia

British Airways

Air France

Lufthansa

Iberia

Olympic

Swiss

El Al

“Low-Fare” Airlines

America West

JetBlue

Southwest

Spirit

ATA

Ted

Song

USA3000

Frontier

Independence Air

EasyJet

Sky Europe

Virgin Atlantic

Air One

Arkia

Israir

12

AWET DemonstrationEnter Search Parameters (Reservation Process)

Methodology

1

2 3

4

5 6

7

8

9

10

• 2 adults, 1 child (age: 12)• NY (LGA) to San Diego• Prefer non-stop• Date and time specified

• Flexible plans for better deal• Assign seats together• Obtain flight info

11

12

13

13

19 questionnaires sent to non-random B2C system users

Return rate = 68% (13 surveys)

Total of 26 evaluations, for 21 airline websites:

12 American airlines (46%)

6 European airlines (23%)

5 Israeli airlines (19%)

3 Canadian airlines (12%)

User group members average 11 trips per year

Performing 58% of airline reservations online

Results: Preliminary QuestionnaireUser Group Demographics


14

Process and Website Design are most important for online QoS

Average number of evaluation criteria mentioned by users per group:

Results: Preliminary QuestionnaireDominant Criteria Groups and Validation of Assumptions

AirlinesWebsite design 2.38Navigation 1.69Process 3.08

Contents 1.31Technical 0.54Avg # of criteria mentioned 9.00

Assumptions A1 & A2 have been validated:

A1 – Streamlined process and improved website design are the

main reasons for users to favor a website

A2 – Users do not consider technical aspects to be a significant

factor in a quality of service provided by airline websites


15

Results: Self-Evaluation of 33 Airline WebsitesTotal Score

Total Score - All Websites(Max Score = 365)

180

190

200

210

220

230

240

250

260

270

280

290

300

310

Overall Avg = 237.27

(65%)

N.A Major Carriers

Group Avg = 252

Group Avg = 236

N.A Low-Fare Carriers

Group Avg = 233 Group

Avg = 229

Group Avg = 214

EU Major Carriers

EU Low-Fare Carriers

Israeli Carriers

81% - American Airlines

52% - Arkia


Average Total Score:

237 Points = 65%Best Performer

Worst Performer

69% 65% 64% 63% 59% Avg GroupScore

16

Evaluation Results: Relative Ranking

Relative Score

60%

65%

70%

75%

80%

85%

90%

95%

100%

AA

Ha

wa

iian

Jet B

lue

No

rth

we

st

Am

eri

ca W

est

Co

ntin

en

tal

Vir

gin

Sw

iss

Sp

irit

Alit

alia

So

ng

Un

ited

Air

Fra

nce

US

Air

Ala

ska

Ind

ep

en

de

nce

El-

Al

Fro

ntie

r

Ibe

ria

Bri

tish

Ea

syJe

t

Sky

Eu

rop

e

So

uth

we

st

AT

A

De

lta

Air

Ca

na

da

Oly

mp

ic

Isra

ir

Te

d

Lu

ftha

nsa

US

A 3

00

0

Air

On

e

Ark

ia

% o

f B

est

Sco

re

Results reveal significant performance gaps:

17

Evaluation Results: Distribution

Results are in proximity to Normal Distribution:

0.1%

13.6%

34.1%

2.1%

0.1%

2.1%

13.6%

34.1%

0.0%

3.0%

15.2%

30.3%

0.0%

3.0%

12.1%

36.4%

Normal Distribution

Study

-2s-3s -1s 0s 1s 2s 3s

18

Based on evaluation results, best performance values have been

established for every question within AWET

Best Practices


Question Criterion Measurement Best Practice By1 Fast home page loading

speedSeconds to fully load homepage 3 seconds Air France

2 Fun and matching colors Best dominant colors Blue 79% of websites3 Number of photos Could not be clearly

determinedN/A

Type of photos Hawaiian beach Hawaiian Airlines4 Font type Arial 45% of websites

Font size (headers) Above 8 88% of websitesFont size (main text) Above 8 58% of websitesNumber of words in home page 132-226Number of pictures and banners 2-5Spacing score (1-to-5) 3-5

6 Site avoids vertical and horizontal scrolling

Percentage of home-page presented on screen (w/o scrolling)

100% 18% of websites

7 Amount of menu options and links

Number of menu option and links 34-35 Spirit, British Airways

8 Yes 85% of websites9 Yes 39% of websites

Easy to read typefaces (on homepage)

Photos that inspire vacation atmosphere

United, AA, Spirit, Alitalia

5 Page load

One-click to most menu and sub-menu options, by rolling over the text

Featuring reservations from homepage

Example of Best Practices for Questions 1-9

19

ההקשבה על תודה

Appendix:

AWET: Airline Website Evaluation Tool

21

AWET (1/12)


Website:

Evaluation Date:

Section 1 - Website Entry Type the address of the website you wish to evaluate.Please measure time from clicking “go” to full loading of home page (in seconds).

Not at all

Very Much

1 2 3 4 5 Question Answer

1 Fast home page loading speed Time to fully load home page (seconds)

Section 2 - Home Page BrowsingLook at the home page, and answer the following questions:

Not at all

Very Much


2 How fun and matching are the colors? What are the dominant colors?

Number of photos

Describe photos

Font type (1)

Font size – headers (1)

Font size - main text (1)

Number of words in home page (1), (2)

If you rated 1 or 2 - do you think amount of home page text is lacking or excessive?

Number of pictures and banners

Is the spacing sufficient?

5

To what extent the photos used inspire vacation atmosphere?

How easy to read are the typefaces on the home page?

3

Comments

Comments# Criterion

RatingDescription

# Criterion

RatingDescription

4

How appropriate is the page-load (amount of text and pictures, spacing)

22

AWET (2/12)


6Site avoids vertical and horizontal scrolling?

Estimated percentage of home page presented on screen (w/o scrolling)

Number of menu option and links (including banners)

If you rated 1 or 2 - do you think amount of home page text is lacking or excessive?

(1) Copy sample of text from home page and paste to Word; If page features more than one size / type, refer to the dominant size / type(2) Count number of words by looking at “File >> Properties >> Statistics”.

# Yes No Comments8

9

10

11

Section 3 - Site Structure & Sub-Pages Browse through the website and answer the following questions:

# Yes No Comments12

13

14

15

16

17

18

19

20

21

Criterion

Site using consistent font size across pages

Site featuring easily accessible site map

Home page icon is consistent across all site pages

Is the location of menus consistent across all pages?

Did the site feature easily-accessible flight status information (departure, arrival, gate)?

Site using consistent colors across pages

All the following contact methods are available: email, fax, phone, mail

Availability of Frequently Asked Questions (FAQ)

Site available in foreign languages

7To what extent is the amount of menu options and links appropriate?

"Contact Us" information easily accessible from the home page

Site featuring reservations from home page

Home page features specials on the home page

One-click to most menu and sub-menu options, by rolling over the text

Criterion

Site featuring consistent frame

23

AWET (3/12)


Not at all

Very Much


22Site structure is clear and straightforward

What are the main menu options?

23How interactive is the route map(1 = not existing, 5 = interactive)

What type of map is featured?

24

Does the site offer cars, hotels and packages?(1 = none, 3 = just one, 4 = both separately, 5 = packages)

What add-on service is featured?

25 Fast sub-pages loading speed (3) Time to fully load sub-page (3)

(3) Average loading time of three random pages

Section 4 - Reservation

Note: Please read instructions & questions to all phases prior to starting the reservation process.

You wish to make flight reservations for your family trip to San Diego. - Your wife, 10 year-old daughter and you would like to leave NY on Thursday, August 11, 2005 and return on Thursday, August 18, 2005. - Price is a major consideration in your plans.- You prefer leaving NY between 7 and 10 am, but would consider leaving earlier or later if the price difference is significant (over $50 per tickets). - Preferred return time would be anytime after 12:00 noon, but could be earlier or later if necessary.- You prefer flying out of LaGuardia airport. Second priority is JFK airport. - You prefer non-stop flights, but would consider maximum of one connection if cheaper by $50 or more. However, you would only agree to one connecting flight as long as the stop-over is not longer than 3 hours in each direction.- You wish to assign all three seats together.- You would like to know what meals are served during the flights.- You would like to know how many miles you can earn for the round-trip.

DescriptionCommentsCriterion

Rating

#

This reservation section is comprised of seven phases Basic search, View results, Modify search, Selecting flights, Reservation support, Error handling and Reservation summary

24

AWET (4/12)


Phase I - Enter Basic Search RequirementsPlease return to the website home page. Record the current time: Start entering your flight requirements. Count the number of mouse clicks / fields filled. If needed, continue to the reservation page

Time to complete phase I : 26 Total duration of phase I (in seconds): 27 Number of clicks (phase I):

Please answer the following questions:

Not at all

Very Much

1 2 3 4 5 Question AnswerDid the site feature calendar for date search?

Did the site feature type-in fields for date search?

Did the site feature day and month fields for date search?

Default search dates are in the future?

Return date automatically updated based on departing flight?

Site allowing part-of-day entry for time (e.g.: morning, evening)

DescriptionComments

28

# Criterion

Rating

How easy to operate did you find the entry for dates and hours of travel?

Stop this phase upon completion of your basic flight requirements - just before you click on "Search Flight" (or any equivalent button).

25

AWET (5/12)


Did the site feature type-in fields for airports?

Did the site feature drop down list of airports?

Did the site feature search for airport codes?

Did the site feature an interactive map to select the departure/desination airports?

Did the provide search of nearby airports?

# Yes No Comments30

31

32

33

34

35 Website offers ability to navigate across fields using the tab and arrow keys

Search entry differentiates passengers: adult, children and seniors

Upon moving to reservation page, could you avoid data re-entry?

Criterion

Website offers ability to search flights based on number of connectionsWebsite offers ability to search flights based on price or schedule

Could you complete the reservations from the home page (without continuing to the reservation page)?

29How easy to operate did you find the entry for airport or route?

26

AWET (6/12)


Phase II - View Search ResultsRecord the current time: Click on "Search Flight" (or "Go", or any equivalent button).

Time to load search results (in seconds) (4)

Please view results and answer the following questions:

Not at all

Very Much

1 2 3 4 5 Question AnswerOutbound and inbound flight presented on one page?

Outbound and inbound flights are paired (as opposed to showing separately)?

Did site feature matrix for combinations of departing and returning flight?

37 Fast loading speed (4) Time to present flight search results (as

recorded above) (4)

Total number of itineraries presented (5)

Number of itineraries fully accommodating your request

Percentage of matching itineraries (calculated from the data above)

Did the site clearly indicate when the no matches were available?

(4) If inbound and outbound flights presented on separate pages, consider the first search only(5) If inbound and outbound flights presented on separate pages, count the number of possible combinations (Outbound x Inbound)

CommentsCriterion

RatingDescription

#

Presentation of all possible itineraries was clear and comfortable

36

38Abundance of flights was sufficient and appropriate

27

AWET (7/12)


# Yes No Comments39

40

41

42

43

44

45

Identify the least expensive flights accommodating your request (please do not select flight yet)Total Price (two adults and one child):

Pricing information available in U.S. dollars

Site features ability to filter results based on number of connections

Site features ability to sort results by price

Did the site feature clear presentation of departure and arrival times?

CriterionRequested itinerary (route, time) showing next to search results

Did the site feature clear child pricing for your flight?

Did the site feature meal information during the flight?

28

AWET (8/12)


Phase III - Modify Search

Not at all

Very Much

1 2 3 4 5 Question AnswerSearch results page features a link for flexible flights

Flexible search suggested alternative dates and time of day

Site providing fields to enter range of possible travel dates (e.g.: +/- 3 days)

Did you have to browse back and forth in order to search your flights?

Did you have to re-enter airports (from/to)?

Did you have to re-enter number of passengers?

Identify the least expensive flights accommodating your request Total Price (two adults and one child):

46

47Site prevented unnecessary data re-entry

You are still not sure you got the best deal, and would like to search for cheaper flights…You might be more flexible with your travel dates if you would get better prices.Please try to find cheaper flights for a 7-day vacation on the same route

# Criterion

Rating

Search for cheaper flights for flexible dates was easy and comfortable

DescriptionComments

29

AWET (9/12)


Phase IV - Selecting Your Flights

(6) If the website does not allow placing reservation on hold, please write "N/A" in question 52 and other relevant questions


Not at all

Very Much


48 Comfortable entry of passenger details Number of field requiring entry (per person, not including optional fields)

Price information shows breakdown by person and total

Price information shows breakdown of price, surcharges and taxes?

50Clear and straightforward presentation of Terms & Conditions

Terms & conditions clearly presented (not hidden) prior to purchase?

51Site allows alternative methods of payment

Number of credit cards accepted

# Yes No Comments52

53

54

55

56

Seat selection available online

Site allows you to return to previous process step (without an error)

Criterion

Help available for process-related questions

Description

Stop this phase upon placing your flights on hold, or prior to entering credit card information (6)

Comments#

Rating

Seat availability clearly presented prior to purchase

Site allow “Hold” on tickets prior to purchaseCriterion

Clear presentation of all order charges 49

30

AWET (10/12)


Phase V - Reservation Support

Select your preferred flights. Record the current time: Try to contact a customer service / sales representative online, to ask them if you could change your reservation within 2 hours

# Yes No Comments57

58

Send an email regarding your question via the websiteDate and time email was sent:

Time you received response to your question (7):

(7) Refers to "final" response to your question (not to immediate response promising future follow-up)

Not at all

Very Much


User-friendly contact form (via email) Number of required fields

Are all required fields are applicable for your enquiry?

Did you receive immediate response informing you that your enquiry has been received (within 5 minutes from sending your question)?

Did the website (or immediate email response) specify expected length to reply to your question?

Time to reply to your email enquiry (in hours)

Business hours to reply (assuming 8 hours/day)

Did final response meet time specified?

Were you satisfied with the answer provided to you?

Reponse was courteous?

61

59

# Criterion

60

Site offering online, in-person support?

Quick reply to email enquiries

Email response was courteous, accurate and relevant

DescriptionComments

Rating

Looking up contact info was available without losing my reservation info?

Criteria

31

AWET (11/12)


Phase VI - Error Handling

Return to the reservation page / home page. Enter departure date of Aug 11, 2005 and return date of Aug 8, 2005 (same route)

# Yes No Comments

62

63

Return to the reservation page / home page. Enter departure date of Aug 11 (morning) and return date of Aug 11 (afternoon)

# Yes No Comments64

Return to the reservation page / home page. Enter departure city "Nwe York" (deliberately misspelled)

# Yes No Comments65

CriterionSystem alerts for possible error when return time is adjacent to departure time.

System provides error warning message when return date is earlier than departure date.

System provides instructions for fixing the error

Criterion

CriterionSystem suggests ways to handle spelling mistakes?

32

AWET (12/12)


Phase VII - Reservation Summary

Please answer the following questions regarding the reservation process you just experienced

Not at all

Very Much


66The reservation process was transparent, easy and clear

Was progress of the reservation process indicated?

67 Task was fully completed

Percentage of the completion, based on the following five tasks: - Found desired flight? - Found seat assignment? - Found child pricing? - Found mileage information? - Found meal information?

68Task completed without errors and broken links

Number of error messages

69 Is the transaction perceived as secure?Was a security alert or message containing security presented prior to entering credit card info?

70Avoidance of unnecessary security alerts

Number of security alerts during visit

Section 5 - Post Sale and Summary


# Yes No Comments7172

73

(8) Assuming use of same computer, to eliminate impact of different settings.

Site allows the user to change an existing reservation

Site allows the user to cancel reservation

Criterion

RatingDescription

Comments

You were able to avoid pop-ups during the visit (8)

Criterion

#

Documents

Website Evaluation in the Tourism Industry Evaluating Quality of Customer Service in Airline B2C Systems Prof. Niv Ahituv, Gil Amoray כנס תיירות ואינטרנט