Kwan Paper

8/19/2019 Kwan Paper

1/14

Paper to be presented at the 7th Quality in Higher Education International Seminar,

Transforming Quality , RMIT, Melbourne, October !!The paper is as submitted by the author and has not been proof read or edited by the Seminar organisers

__________________________________________________________________________

Independent Benchmarking in Higher Education: Reflections on an

implementation

Kam-Por KWAN,

Educational Development Centre, The Hong Kong Polytechnic University,

Yuk Choi Road, Hunghom, Kowloon, Hong Kong SAR, China.

E-mail: [email protected]

Abstract

There has been an increasing interest in benchmarking as a referencing process for assuring

and enhancing quality and outcome of higher education provisions. A number of maor

collaborative benchmarking proects have been carried out in the !"A# Australia# !$ and

other European countries in the areas of library and information services# facilities

t d i i fi i l i d i ti d i t


2/14

management admissions financial services academic practice academic management

benchmarking activities have impacted upon the organisation# or the issues and practical

problems that may arise in the actual implementation process.

This paper reports the implementation of an independent benchmarking study recently

conducted by a university in &ong $ong to identify the key performance indicators for

educational development support services in higher education institutions. 't describes the

obectives of the study and the methodology adopted# reports the outcomes of the study# and

reflects on the benefits# problems and issues in the benchmarking e(ercise.

Benchmarking and continuous quality improvement (CQI)

There are many definitions of benchmarking )for e(ample# 5amp# 0121/ 5amp + Tweet#

011alea-5urmi and &awkins )011?3 make an important distinction between 9benchmark% and

9benchmarking% The arg e that a benchmark is simpl a 9defined standard a pre

- 2 -


3/14

three linked but rather different approachesF capable of providing different levels of

understanding and potential benefits but requiring different levels of effort and costs. Betric

benchmarking focuses on the collection of quantitative performance data# and comparing the performance across institutions or units on the basis of the information collected. iagnostic

benchmarking# on the other hand# involves institutions comparing its practices# process and

outcomes to those of other organiGations. =rocess benchmarking is the highest level of

benchmarking in which best practices are identified# learned# adapted and put to practice in

the home organisation.

Context of the benchmarking study

The unit for which the benchmarking study was carried out is the Educational evelopment

5entre of a university in &ong $ong. The 5entre plays a unique and central role in

supporting all teaching and learning enhancement initiatives of the university through various

educational development support services# including:

provision of short courses# workshops# and seminars on teaching and learning for

different categories of teaching staff/ coordinating and supporting different forms of teaching evaluation services/

initiating# co-ordinating # and managing of teaching development proects/

providing consultation for staff on teaching-related matters/

developing and disseminating resources for promoting effective teaching and

learning/

supporting the work of# and involvement in# various university# faculty# and

de t e t l ittee te hi d le i

- 3 -


4/14

5onducting the research: collecting primary andCor secondary data about other

organisations being studied.

Analysing the data: analysing and interpreting the data collected for developing

recommendations for self-improvement.

Adapting the findings to the home institution: identifying and adapting the process for

improvement in the home institution.

The present study adopted a similar four-step method for benchmarking. 'n the absence of

any e(isting database or collaborative consortium for benchmarking educational development

activities in universities# it was decided that an independent benchmarking was carried out. 't

was further decided that to gain the most from the e(ercise# the benchmarking study would

collect both quantitative and qualitative data# and focus on two maor themes:

the quality assurance process and mechanism for educational development in

local universities/

the input# process# output and outcomes of educational development activities

of comparable units in overseas universities.

To collect data from local universities# a letter was written to the heads of corresponding units)4L?3 asking for information about their respective quality assurance procedures.

'nformation concerning input# process# output and outcomes of similar units in overseas

university was obtained by a systematic search of published annual activity reports of

comparable units via the 'nternet. 'nformation collected was collated and analysed to derive

the benchmarks for various categories of educational development support services. The

results were then presented to the members of the 5entre in its annual staff retreat# to inform

d f ilit t lf l ti d ti l i f th 5 t % ti iti d i f th

- 4 -


5/14

the 5entre to roughly compare the scope# coverage# and efficiency of its activities and

services with reference to outside organisations.

Impact on Quality Improvement

'nstitutional self-evaluation is not new. 't is often prescribed as the first step in assuring and

enhancing quality. =rior to the present benchmarking study# the 5entre concerned has already

carried out a number of self-evaluations over the years for various internal and e(ternal

quality review purposes. &owever# without any e(ternal referencing point# it is difficult for

the 5entre to obectively evaluate its own strengths and inadequacies. Thus# the so-called

self-evaluations were often based on beliefs or subective feelings rather than empirical

evidence# and ended up as either self-ustifications or self-complacency for its e(isting

practices. 'mprovement goals were mainly derived from simple e(trapolation of past year%s

performance.

Three maor benefits have been observed in conducting the independent benchmarking study

and evaluating the 5entre%s activities against the tentative benchmarks derived:

't helps to meet the demand and requirement of the funding agency for public

accountability purposes through the establishment of tentative international benchmarks

against which the 5entre can assess its own performance.

6y providing a tentative e(ternal yardstick for assessing performance# the benchmarks

encourage the 5entre to 9look outside the bo(% by comparing its own performance and

practices with what other units of similar nature are doing# and how.

't bl th 5 t t id tif l ti t th ll i it ti iti d

- 5 -


6/14

systematic attempt to benchmark their activities and services amidst heavy workload without

any pressure from outside# particularly in times of shrinking resources. 'n this respect# the

e(ternal quality assurance procedure has stimulated and encouraged institutions to take thefirst step in searching for and development of best practices. &owever# whether institutions

will be able to really gain from the effort is another matter# which is likely to be dependent on

many factors.

A number of conceptual and practical problems have also arisen in the process of the

independent benchmarking study. They are discussed in greater detail below.

=roblems in defining and identifying best practices and 9best-practice% organisations

in higher education

The lack of a universally accepted definition and measure of quality of higher education

in general and quality of educational development support services in particular has led

to problems in defining and identifying best practices and 9best-practice% organisations in

the benchmarking process. These problems apply not only to the case of an independent

benchmarking e(ercise like ours. Even if we were able to engage a number of local

andCor international peer organisations to participate voluntarily in a collaborative

benchmarking e(ercise# the problem would still e(ist because of the inherent difficulty of

defining high performance for organisations where profitability is not a performance

indicator )>alea-5urmi and &awkins# 011?# p. 0;3. As a matter of fact# the target

organisations for this study were selected not because they were the 9best-practice%

organisations# but mainly because of their comparability in nature and siGe with the

benchmarking organisation and readily available performance data. As such# they

t i l th th t l t ti l f 9b t

- 6 -


7/14

standardised template for reporting performance# it is highly probable that the centres

have been selective in reporting their work. 't is likely that they tend to report part of

their work that is more visible and quantifiable# but omit the ones that are less visible ormore qualitative in nature. These observations raise significant doubts about the e(tent to

which the data are directly comparable# and whether it is possible to have a meaningful

comparison of the performance across the organisations on the basis of such data.

ifficulties in benchmarking 9process% and 9quality%

Mriters on benchmarking has repeatedly pointed out that establishing benchmarks is

only the first step in a true benchmarking e(ercise. To facilitate continuous qualityimprovements# it is pertinent that the high-performing organisations are identified and

the processes they employ to achieve high quality be learned and adapted. !nfortunately#

while inputs and outputs are relatively easy to quantify and compare# processes and

quality are much more difficult to document and benchmark. 'n fact# only one of the si(

universities surveyed has included in its report a rough and incomplete measure of the

quality of its services in the form of an overall mean participants% satisfaction rating of

the courses and workshops run by the unit. 4one attempted to report the impact of their

services on teaching and learning of the institution# which is the ultimate performanceindicator of the quality of their activities. Iurthermore# while all of the si( universities

reported on their maor outputs# none spelt out in their reports how they were achieved.

Thus# it is difficult for any interested reader to distil from the reports the 9best practices%

that have been employed to achieve high quality# not to mention adapting them to their

own organisations. 't appears that individual benchmarking based on published

performance data# while viable for 9metric% benchmarking# is rather inadequate in

i ldi i f ti f th f 9di ti % 9 % b h ki

- 7 -


8/14

#uture development planned

Hecognising that a one-shot independent benchmarking e(ercise has a number of pitfalls andlimitations# the 5entre has formulated the following plans to ma(imise the potential gains

from benchmarking for future development:

To undertake benchmarking of its provisions and services in a regular on-going

manner.

To e(plore the possibility of setting up a local network for collaborative

benchmarking of educational development support services in higher education.

To encourage and support staff to organise site visits to other universities tounderstand in greater depth the quality and process of their educational development

activities and services# and to identify and benchmark good practices in educational

development in higher education that might be worth adopting by the 5entre.

Conclusions

The ability of institutions to critically evaluate their own performance is vital to the processof continuous quality improvement. &owever# institutions may have a natural tendency to

protect their self-interests and thus# be reluctant to admit their inadequacies unless challenged

by empirical evidence. 6enchmarking# even for a small-scale independent e(ercise like the

one reported here# can have the impact of forcing institutions to 9look outside the bo(% and

obectively compare their own performance against that of their peers# through the

establishment of appropriate benchmarks. 't can provide a useful 9wake-up% call for

institutions which have perhaps been too complacent about their performance.

- 8 -


9/14

- 9 -


10/14

Bibliography

Alstete# *.M. )01183. Benchmarking in Higher Education: Adapting Best Practices to

Improe !ua"ity. EH'5 igest. EH'5 5learinghouse on &igher Education. Available

at http:CCwww.ed.govCdatabasesCEH'5PigestsCedunther and I. &awkins

)eds.3 Tota" !ua"ity &anagement in Human Serice 'rgani(ations. 4ew Dork:

"pringer =ublishing 5ompany 'nc.

* k 4 )03 6 h ki i !$ &E A i ! "it A i

- 10 -


11/14

"chofield# A. )01123. An introduction to benchmarking in higher education. 'n: A. "chofield

)ed.3 Benchmarking in Higher Education: An Internationa" )eiew. ,ondon: The

5ommonwealth &igher Education Banagement "ervice.

Moolf# &. )01113. 'nstitutional comparisons Q Heality or illusionsO 'n: &. "mith# B.

Armstrong# + ". 6rown )eds.3 Benchmarking and Thresho"d Standards in Higher

Education. ,ondon: $ogan =age.

Darrow# . )01113. The business approach to benchmarking Q An e(ploration of the issues as

a background for its use in higher education. 'n: &. "mith# B. Armstrong# + ".6rown )eds.3 Benchmarking and Thresho"d Standards in Higher Education. ,ondon:

$ogan =age.

- 11 -


12/14

Table 1: Key input and output/outcome indicators of educational development activities of target institutions

Centre/Institution University A (USA) University B (USA) University C (Canada) University D (Australia) University E (Australia)

Year of activities 0111C 0112C0111 0111C C0 0111C

Student size Rraduate 5ourse on T+, in

&E

>raduate 5ertificate in

Educational "tudies )&E3

Teahin! Evaluation

! #dvice on teacingevaluation

5onsultation on re-design of

student evaluation system

eveloped a new student

evaluation system

eveloped and monitored !

teaching evaluation system

'ndividual and small group

consultation

! )niversity instructorevaluation

4ot directly involved 1? classes

J0 forms

0


13/14

feedbac& sessions for staff on formativeevaluation

! 'nterpretation ofresults

5onsultation on interpretation Hating 'nterpretation >uide

! (ter teacingevaluation services

Teaching ocumentation

=rogram: J8 participants

Teaching portfolio handbook

Evaluation of online courses

and technology-enhanced

learning

Kideo-taped J8 classes# 0J?

self taping by staff

Annual graduate survey

A guide on =reparing a

Teaching ossier

source books on teaching

evaluation

Hegular evaluation of all

taught units through "E

Evaluation of Hesearch

"tudent E(perience

E(perience of 5ommencing

"tudents

"tudent course e(perience

survey

"tudent satisfaction survey

for departmental strategic

planning

Support "or teahin! !rants# initiatives or pro$ets

! Teacing grants/initiatives

8 funded proect

1 mini-grants

; mini-grants C mini-proects Averaged 8 mini-proects

"upport in-department 5

initiatives

J teaching development

initiatives

? new collaborative T

proects

! Funded pro+ects < ; J

Consultation

! $onsultation for staff 'ndividual teachingconsultation

Iocused consultation

E-mail

J comprehensive

consultations


14/14

- 14 -

Documents

Kwan Paper