Yoshida presentation

EFL Student’s Ability for Website Information Comprehension and Perceptions on Website Usability

s1170173 Makoto Yoshida

Supervised by Prof. Debopriyo Roy

Overview

•  Abstract •  Research Workflow •  Question what We Set (to Participants) •  Results and Discussion (1-9) •  Conclusion and Future Work

Abstract The purpose of this study is to observe and consider the impression of L2 readers during website analysis in target language (English). At first, participants completed an extensive website analysis using established web design models with a closed-ended website information comprehension questionnaire (testing readers’ understanding of the text content in the website). In the second stage, they answered three usability questionnaires (QUIS, CSUQ and MPRC questionnaires), recording their impression about the interface, navigational efficiency, content comprehension, learning aspects of the interface, basic comfort level with text-graphics content etc. I used Pearson Correlation and non-parametric Friedman Test to analyze user data. The self-reports on the questionnaires (QUIS, CSUQ and MPRC) provided a general outlook about the website content and navigation and might not have been specific to the accuracy scores (related to the text content of the website). This exploratory analysis as discussed in this article could help us obtain initial data on how EFL readers in a typical context as this would perform with English website information and the type of impression they have about the website.

Research Workflow • 59 participants completed the website information comprehension questionnaire. Accuracy scores on the questionnaire were measured. •  Following the interaction with the website, readers completed the three questionnaires on self-reporting.

• Collection their answers

•  Data Analysis •  Consider the Results

Questionnaires The website information comprehension questionnaire focused on the efficiency with which L2 readers are able to search through information from the Belize tourism website. The focus at this stage was on readers’ ability to navigate through the pages, based on cues from the questionnaire. The questionnaire was designed on the basis of the following: 1.  multiple-choice questions asking readers to accurately pinpoint the

information available in the website. 2.  readers’ ability to sequence order steps in the correct order when

searching for an information from the webpage. (Q1 with 8 steps to be correctly ordered).

10 questions were asked and each question (except Q1) could be scored as 1(correct) or 0 (incorrect) (binary scale). The total accuracy scores for each student were measured. QUIS, CSUQ and MPRC questionnaires are standard software usability questionnaires used for self-reporting users’ preference for the website.

Results and Discussion Ques%on N Minimum Maximum Mean Std. Devia%on

Q1

59

0 10 8.66 2.496 Q2 0 2 1.97 .260 Q3 2 2 2.00 .000 Q4 0 2 1.19 .991 Q5 0 2 1.19 .991 Q6 0 2 1.73 .691 Q7 0 2 1.90 .443 Q8 0 2 1.64 .663

Valid N (List wise)

Table 1: Descriptive Statistics for the 8 Website Information Comprehension Questions

0

0.5

1

1.5

2

2.5

1 2 3 4 5 6 7

Mea

n A

ccu

rate

Sco

re

Question Number (1 = Q2; 2 = Q3; 3 = Q4; 4 = Q5; 5 = Q6; 6 = Q7; 7 = Q8)

Mean Accuracy Score (Maxium Value = 2)

Mean Accuracy Score (Maxium Value = 2)

Figure 1. Mean Accuracy Scores for Q2 – Q8 (Wrong Answer = 0; Correct Answer = 2)

Results and Discussion 2

Ques%on Pearson

Correla%on Sig.(2-‐tailed) N=59

Q2-‐Q7 .567** .000

59 Q2-‐Q8 .328* .011 Q7-‐Q4 .280* .032 Q8-‐Q7 .344* .008

**. Correla%on is significant at the 0.01 level (2-‐tailed) *Correla%on is significant at the 0.05 level (2-‐tailed).

Table 2: Pearson Correlation Values (Statistically Significant Values only) among the Website Information Comprehension Questions

Results and Discussion 3 Ques%on N Minimum Maximum Mean Std. Devia%on

Q1 59 0 7 4.10 1.658 Q2 59 1 7 4.29 1.521 Q3 59 0 7 3.81 1.503 Q4 59 0 7 3.61 1.576 Q5 59 0 7 3.69 1.664 Q6 59 1 7 4.03 1.575 Q7 59 1 7 3.97 1.553 Q8 59 0 7 3.32 1.824 Q9 59 0 7 3.29 2.205 Q10 59 0 7 2.92 2.020 Q11 59 0 7 4.36 1.689 Q12 59 1 7 4.37 1.376 Q13 59 1 7 4.20 1.412 Q14 59 0 7 3.95 1.665 Q15 59 1 7 4.63 1.484 Q16 59 0 7 4.05 1.726 Q17 59 1 7 3.63 1.809 Q18 59 1 7 3.49 1.696 Q19 59 1 7 4.17 1.652 Valid N (listwise) 59

Table 3. Descriptive Statistics for Self-Reports on the CSUQ Questionnaire

Results and Discussion 4 Question Category Number Significant Correlation with other questions

Q1 All questions Q2 All questions Q3 All questions except Q9 Q4 All questions Q5 All questions Q6 All questions Q7 All questions Q8 All questions except Q9 Q9 All questions except Q3 and Q8 and Q15 Q10 All questions except Q12 Q11 All questions except Q12 Q12 All questions except Q9, Q10, Q11, Q12 Q13 All questions Q14 All questions Q15 All questions except Q9 Q16 All questions Q17 All questions Q18 All questions Q19 All questions

Table 4: Significant Correlation between Self-Reports on CSUQ Questionnaire (N = 59)

Results and Discussion 5 Ques&ons Minimum Maximum Mean Std. Devia&on Q1 1 9 6.12 1.848 Q2 0 9 5.64 1.989 Q3 0 9 5.25 2.411 Q4 0 9 4.75 2.279 Q5 0 9 4.97 2.304 Q6 0 9 5.03 2.573 Q7 0 9 6.24 2.254 Q8 0 9 6.25 1.944 Q9 0 9 5.73 2.148 Q10 0 9 4.42 2.183 Q11 0 9 5.69 2.053 Q12 0 9 5.71 2.407 Q13 0 9 3.93 2.684 Q14 0 9 3.81 2.549 Q15 0 9 4.73 2.211 Q16 0 9 4.64 2.517 Q17 0 9 4.27 2.180 Q18 0 9 4.37 2.189 Q19 0 9 4.14 2.583 Q20 0 9 4.81 2.467 Q21 0 9 5.51 2.176 Q22 0 9 2.12 2.841 Q23 0 9 2.76 2.654 Q24 0 9 4.69 2.541

Table 5: Descriptive Statistics of the Self-reports on the QUIS Questionnaire

59.4 64.8 54.2 49.2 44.2

0 10 20 30 40 50 60 70

Per

cen

tag

e o

f A

gre

emen

t w

ith

th

e S

tate

men

t Different Categories on the Questionnaire

Category 1 = Overall Reaction to the Website; Category 2 = Web Page; Category 3 = Terminology and Website

Information; Category 4 = Learning; Category 5 =

Percentage of Positive Responses on Different

Categories in QUIS Questionnaire

Percentage of Positive Response on Different Categories

Figure 2. Percentage Agreement with the Different Categories in QUIS Questionnaire

Results and Discussion 6 Question Number Significant Correlation with other questions

Q1 All questions except Q2, Q10, Q14, Q16, Q23, Q24 Q2 No questions except Q22 Q3 All questions except Q2, Q10, Q11, Q17, Q22, Q23 Q4 All questions except Q2, Q7, Q8, Q9, Q10, Q11, Q12, Q15, Q16, Q17, Q18, Q20,

Q24 Q5 All questions except QQ2, Q5, Q11, Q12, Q19, Q20, Q22, Q23

Q6 All questions except Q2, Q6, Q10, Q13, Q14, Q15, Q16, Q17, Q22, Q23, Q24

Q7 All questions except Q2, Q4, Q10, Q13, Q16, Q17, Q22, Q23, Q24 Q8 All questions except Q2, Q4, Q13, Q14, Q16, Q19, Q20, Q22, Q23, Q24

Q9 All questions except Q2, Q4, Q13, Q14, Q19, Q20, Q22, Q23 Q10

All questions except Q1, Q2, Q3, Q4, Q6, Q7, Q12, Q13, Q14, Q19, Q20, Q22, Q23 Q11 All questions except Q2, Q3, Q4, Q5, Q11, Q13, Q21, Q22, Q23, Q24 Q12 All questions except Q2, Q4, Q5, Q10, Q12, Q13, Q15, Q16, Q19 Q13 All questions except Q2, Q6, Q7, Q8, Q9, Q10, Q11, Q12, Q13, Q15, Q16, Q17, Q18,

Q20, Q21, Q24 Q14 All questions except Q1, Q2, Q6, Q8, Q9, Q10, Q14, Q15, Q16, Q17, Q19, Q20, Q22,

Q23, Q24 Q15

All questions except Q1, Q2, Q4, Q6, Q12, Q13, Q14, Q15, Q19, Q20, Q22, Q23 Q16 All questions except Q1, Q2, Q4, Q6, Q7, Q8, Q12, Q13, Q14, Q16, Q20, Q21, Q22,

Q23 Q17

All questions except Q2, Q3, Q4, Q6, Q7, Q13, Q14, Q17, Q19, Q20, Q22, Q23, Q24 Q18 All questions except Q2, Q4, Q13, Q18, Q22 Q19

All questions except Q2, Q5, Q8, Q9, Q10, Q12, Q14, Q15, Q17, Q19, Q24 Q20

All questions except Q2, Q4, Q5, Q8, Q9, Q10, Q13, Q14, Q15, Q16, Q17, Q20, Q24 Q21 All questions except Q2, Q11, Q13, Q16, Q21, Q23 Q22 All questions except Q1, Q3, Q5, Q6, Q7, Q8, Q9, Q10, Q11, Q12, Q14, Q15, Q16,

Q17, Q18, Q22, Q24 Q23 No questions except Q4, Q13, Q18, Q19, Q20, Q22, Q24 Q24 No questions except Q3, Q5, Q9, Q10, Q15, Q16, Q18, Q21, Q23

Table 6: Significant Correlation between Self-Reports on QUIS Questionnaire (N = 59)


Mean Rank

Chi-‐Square

Asymp. Sig.

Q1 11.19

126.626 .000

Q2 11.62 Q3 9.54 Q4 8.3 Q5 8.88 Q6 10.4 Q7 10.14 Q8 7.2 Q9 8.14 Q10 7.08 Q11 12.19 Q12 11.83 Q13 11.12 Q14 10.4 Q15 13 Q16 10.95 Q17 8.69 Q18 7.71 Q19 11.62

Table 7: Friedman Test Values for the 19 Questions in CSUQ Questionnaire

Ques%on Categories Mean Rank Chi-‐Square Asymp. Sig. Q1

Comprehensive evalua%on

3.690

21.532 .000 Q2 3.070 Q3 2.960 Q4 2.690 Q5 2.590 Q6

Web Page Design 1.580

24.818 .000 Q7 2.260 Q8 2.160 Q9

Terminology and

Website Informa%on

4.260

60.670 .000

Q10 2.920 Q11 4.110 Q12 4.220 Q13 2.860 Q14 2.640 Q15

Learning

3.210

5.769 .217 Q16 3.210 Q17 2.760 Q18 2.810 Q19 3.000 Q20

Website Capabili%es

3.490

75.683 .000 Q21 3.900 Q22 1.950 Q23 2.350 Q24 3.310


Table 8: Friedman Test Statistics for the 5 Different Categories in the QUIS Questionnaire


Words with Maximum Frequency (Top 10) Frequency Convenient 32

Clean 29 Dull 29 Slow 28 Helpful 27 Useful 26

Accessible 24 Engaging 23

Fun 23 Usable 23

Table 9. Words chosen with Maximum Frequency

Conclusion and Future Work

•  Production -L2 readers have wide variability in the efficiency scale, when analyzing an English website. •  Future Work - This will help us judge their levels of proficiency and the types of English websites they could be exposed to for various kinds of assignments, task-based language learning etc.

Thank you

Technology

Yoshida presentation