Overcoming the Paradox of Standards in Higher Education Through the Covenant between Higher Education and Society Martin Carroll Keynote Address for Oman

Overcoming the Paradox Overcoming the Paradox of Standards in Higher of Standards in Higher

EducationEducationThrough the Covenant between Through the Covenant between

Higher Education and SocietyHigher Education and Society

Martin CarrollMartin CarrollKeynote Address for Oman National Quality Conference, October Keynote Address for Oman National Quality Conference, October

20082008

KNOWKNOW DON’T KNOWDON’T KNOW

UN

AW

AR

EU

NA

WA

RE

AW

AR

EA

WA

RE

KNOWLEDGEKNOWLEDGEA

WA

RE

NE

SS

AW

AR

EN

ES

S

This is the main goal of higher education…This is the main goal of higher education…

We are awareof what we

know

We are awareof what wedon’t know

We aren’t aware of what we

know

We aren’t awareof what wedon’t know

OutlineOutline

• The CovenantThe Covenant

• The Paradox of StandardsThe Paradox of Standards

• A Standards TaxonomyA Standards Taxonomy

• Who Influences the Standards?Who Influences the Standards?

• Proposed Way Forward for OmanProposed Way Forward for Oman

The CovenantThe Covenant

MARKET CONTRACTMARKET CONTRACT SOCIAL COVENANTSOCIAL COVENANT

Customer receiveswhat we provide

Student participatesin what we provide

By transacting resources we exchange them

By sharing knowledgewe multiply it

Serves shareholders, even if this happens by serving society at large

Serves society at large, even if this happens by serving stakeholders

Motivated by profit Motivated by social vision

Bound by contracts Bound by peers

Customer receivesCustomer receiveswhat we providewhat we provide

Student participatesStudent participatesin what we providein what we provide

By transacting resources By transacting resources we exchange themwe exchange them

By sharing knowledgeBy sharing knowledgewe multiply itwe multiply it

Serves shareholders, Serves shareholders, even if this happens by even if this happens by serving society at largeserving society at large

Serves society at large, Serves society at large, even if this happens by even if this happens by serving stakeholdersserving stakeholders

Motivated by profitMotivated by profit Motivated by social visionMotivated by social vision

Bound by contractsBound by contracts Bound by peersBound by peers

Key PointsKey Points• Higher Education is does not follow the Higher Education is does not follow the

rules of the competitive market place.rules of the competitive market place.

• So we should not use a quality assurance So we should not use a quality assurance system designed for the market.system designed for the market.

• Higher Education forms a covenant with Higher Education forms a covenant with society.society.

• We should use a quality assurance system We should use a quality assurance system commensurate with that covenant.commensurate with that covenant.

• How do we do that? Through the careful How do we do that? Through the careful use of appropriate standards.use of appropriate standards.

The Paradox of The Paradox of StandardsStandards

Knowledge about theKnowledge about thedesired state of higher desired state of higher education quality is often education quality is often expressed as standards. expressed as standards. How can standards helpHow can standards helpan organisation? an organisation?

Carroll, M.I. (June 2003), “Does Carroll, M.I. (June 2003), “Does Auditing Against HE Standards Auditing Against HE Standards Encourage Masterpieces or Paint-by-Encourage Masterpieces or Paint-by-Numbers?”, Numbers?”, Assessment & Evaluation Assessment & Evaluation in Higher Educationin Higher Education, Vol.28, No.3., Vol.28, No.3.

From the start, the From the start, the standards provide you with standards provide you with a reasonable idea of what a reasonable idea of what you will look like as an you will look like as an organisation if you comply.organisation if you comply.

From the start, the From the start, the standards provide you with standards provide you with a reasonable idea of what a reasonable idea of what you will look like as an you will look like as an organisation if you comply.organisation if you comply.

The more detailed the The more detailed the standards, the more standards, the more particular the outcome of particular the outcome of compliance.compliance.

1 1

2

92

22

3

3

33

3

445

5

6

7

7

7

7

7

7

877

7

7

7

6

Implementation of standards Implementation of standards can be planned and can be planned and prioritised. prioritised.

1 1

2

92

22

3

3

33

3

445

5

6

7

7

7

7

7

7

8

1 2 3 4 5 6 7 8 9

77

7

7

7

6

Implementation of standards Implementation of standards can be planned and can be planned and prioritised.prioritised.

As you successfully achieve As you successfully achieve each standard, the assessor each standard, the assessor gives you a tick in the gives you a tick in the appropriate box. appropriate box.

Assessor’s Report CardAssessor’s Report Card

2

92

22

3

3

33

3

445

5

6

7

7

7

7

7

7

8

1 2 3 4 5 6 7 8 9

77

7

7

7

6


9

3

3

33

3

445

5

6

7

7

7

7

7

7

8

1 2 3 4 5 6 7 8 9

77

7

7

7

6


9

445

5

6

7

7

7

7

7

7

8

1 2 3 4 5 6 7 8 9

77

7

7

7

6


9

5

5

6

7

7

7

7

7

7

8

1 2 3 4 5 6 7 8 9

77

7

7

7

6


9

6

7

7

7

7

7

7

8

1 2 3 4 5 6 7 8 9

77

7

7

7

6


9

7

7

7

7

7

7

8

1 2 3 4 5 6 7 8 9

77

7

7

7


9

8

1 2 3 4 5 6 7 8 9Assessor’s Report CardAssessor’s Report Card

9


1 2 3 4 5 6 7 8 9

All standards met. The All standards met. The picture is clearly picture is clearly recognisable and shows recognisable and shows what was intended.what was intended.


Imagine if one or more of Imagine if one or more of the standards was not met the standards was not met as a result of one or more of as a result of one or more of the set criteria being the set criteria being breached. The result would breached. The result would not be acceptable.not be acceptable.


1 2 3 4 5 6 7 8 9

Imagine if one or more of Imagine if one or more of the standards was not met the standards was not met as a result of one or more of as a result of one or more of the set criteria being the set criteria being breached. The result would breached. The result would not be acceptable.not be acceptable.

The assessor can help The assessor can help identify such breaches. identify such breaches. Although sometimes Although sometimes uncomfortable for the HEI, uncomfortable for the HEI, this is a value-adding role.this is a value-adding role.

Colour Shape


1 2 3 4 5 6 7 8 9

And if all the standards are And if all the standards are met, the picture is clearly met, the picture is clearly recognisable as the Mona recognisable as the Mona Lisa, as intended.Lisa, as intended.

The standards have served The standards have served their purpose. The quality their purpose. The quality system has been system has been successfully implemented.successfully implemented.


1 2 3 4 5 6 7 8 9

But there is a considerable But there is a considerable gap between meeting the gap between meeting the standards, and creating a standards, and creating a masterpiece!masterpiece!






Measurable (particularly Measurable (particularly quantitative) assessment quantitative) assessment against pre-determined against pre-determined standards has limited standards has limited constructive effect regarding constructive effect regarding high-end performance. It high-end performance. It may even stifle progress.may even stifle progress.

A different system is A different system is required, that is more flexible required, that is more flexible and that addresses a broader and that addresses a broader range of issues (ranging from range of issues (ranging from risk management to the risk management to the pursuit of best practice).pursuit of best practice).

1 ∞Assessor’s Report CardAssessor’s Report Card

So the question is:So the question is:

What are the strategies What are the strategies that can move us beyond that can move us beyond paint-by-numbers, and help paint-by-numbers, and help us to create masterpieces?us to create masterpieces?

1 ∞Assessor’s Report CardAssessor’s Report Card

A Standards A Standards TaxonomyTaxonomy

Key Points!Key Points!

• ““Standards” influence behaviour. So we Standards” influence behaviour. So we have to be very careful about have to be very careful about whatwhat standards we use, and standards we use, and howhow we use them. we use them.

• But “standards” can mean different things But “standards” can mean different things to different people.to different people.

• There are, in fact, several different types of There are, in fact, several different types of standards. They influence different types of standards. They influence different types of change.change.

• We need to know what they are so that we We need to know what they are so that we can use them intelligently for the benefit of can use them intelligently for the benefit of higher education.higher education.

Types of StandardsTypes of Standards

1.1. Two-ended control standards.Two-ended control standards.

2.2. Low-end control standards.Low-end control standards.

3.3. Advanced control standards.Advanced control standards.

4.4. High-end target standards.High-end target standards.

5.5. Combinations.Combinations.

Standards, be they quantitative or qualitative, Standards, be they quantitative or qualitative, may be used in a number of different ways:may be used in a number of different ways:

Different strategies are required for achieving Different strategies are required for achieving each type of standard.each type of standard.

1. Standards as a two-ended control1. Standards as a two-ended control

Features:Features:• Assumes that a central norm is optimal.Assumes that a central norm is optimal.• Assumes that variances too far to either Assumes that variances too far to either

side are unacceptable.side are unacceptable.• Sometimes reinforced through negative Sometimes reinforced through negative

consequences.consequences.


Examples:Examples:• Staff:student ratios must be kept between Staff:student ratios must be kept between

1:16 and 1:24.1:16 and 1:24.• Students’ evaluation of course workloads Students’ evaluation of course workloads

must be kept reasonable (not too much; must be kept reasonable (not too much; not too little). not too little).

Assume that performance follows a normal distribution curve.

Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

Assume that performance follows a normal distribution curve.

Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

Standards set criteria that may define the tolerance thresholds for inputs, process designs or outputs.

Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

Standard

Standard


Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

Standard

Standard

LowerToleranceThreshold

UpperToleranceThreshold



Successful implementation of compliance strategies may encourage redistribution around a central norm.


Quality Measure

Qua

ntit

y M

easu

re


+

+

–

– Quality Measure

Qua

ntit

y M

easu

re


+

+

– –

Standard

Standard



Successful implementation of compliance strategies may encourage redistribution around a central norm.


Conclusions:Conclusions: Defined minima are enforced, eDefined minima are enforced, e..gg.. ensuring ensuring

students have a sufficient workload.students have a sufficient workload. Defined maxima are enforced, such as Defined maxima are enforced, such as

reducing expenditure on over-achieving.reducing expenditure on over-achieving. Innovation is suppressed.Innovation is suppressed. No all issues require both ends to be No all issues require both ends to be

controlled.controlled.


Conclusions:Conclusions:• Encourages normal behaviour.Encourages normal behaviour.• Sometimes known as “Statistical Process Sometimes known as “Statistical Process

Control” (SPC).Control” (SPC).• A “quality = conformity” model.A “quality = conformity” model.• Has some utility for HEd, especially for Has some utility for HEd, especially for

internal resource management purposes internal resource management purposes (money, workload etc.).(money, workload etc.).

2. Standards as a low-end control2. Standards as a low-end control

Features:Features:• Establishes a minimum threshhold.Establishes a minimum threshhold.• Performance above the control is acceptable Performance above the control is acceptable

and therefore ignored by any strategies and therefore ignored by any strategies associated with achieving the standard.associated with achieving the standard.

• Compliance is demanded. Performance Compliance is demanded. Performance below the control is not acceptable.below the control is not acceptable.

• Sometimes reinforced through negative Sometimes reinforced through negative consequences.consequences.


Examples:Examples:

• Academic staff must have a degree at least Academic staff must have a degree at least one level higher than the qualification level one level higher than the qualification level at which they are teaching.at which they are teaching.

• Students require >88% entrance score to Students require >88% entrance score to gain entry into a particular course.gain entry into a particular course.

• Students must demonstrate acquisition of Students must demonstrate acquisition of the specified competency to pass the unit.the specified competency to pass the unit.

Standards set criteria that may define the minimum acceptable inputs, process designs or outputs.

Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –


Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

Low-endControlStandard



Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –



Successful implementation of compliance strategies may raise the quantity of activity below the standard to an acceptable minimum.


Successful implementation of compliance strategies may raise the quantity of activity below the standard to an acceptable minimum.

Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –




Conclusions:Conclusions: Defined minima are enforced, protecting Defined minima are enforced, protecting

stakeholders from unacceptable service.stakeholders from unacceptable service. Does nothing to encourage progress at the Does nothing to encourage progress at the

high-end.high-end.


Conclusions:Conclusions:

• A “quality = risk management (guarantee)” A “quality = risk management (guarantee)” model (two-ended controls also have this).model (two-ended controls also have this).

• Has some utility for HEd, especially for Has some utility for HEd, especially for licensing purposes.licensing purposes.

• Can be applied homogeneously to a Can be applied homogeneously to a common group (e.g. universities).common group (e.g. universities).

• Precisely where to calibrate the low-end Precisely where to calibrate the low-end control is a balancing act between control is a balancing act between international benchmarks & local capability. international benchmarks & local capability.

3. Standards as an advanced control3. Standards as an advanced control

Features:Features:

• Performance above the control is Performance above the control is acceptable and therefore ignored.acceptable and therefore ignored.

• Performance below the control is acceptable Performance below the control is acceptable (provided that it is also above any low-end (provided that it is also above any low-end control standard) but not desirable.control standard) but not desirable.

• The standard is set high to reflect good The standard is set high to reflect good practice, not just required practice.practice, not just required practice.

• Sometimes reinforced through positive Sometimes reinforced through positive consequences (e.g. incentives).consequences (e.g. incentives).


Examples:Examples:• All academic staff must have a PhD.All academic staff must have a PhD.• To be eligible for promotion, academic staff To be eligible for promotion, academic staff

should have published at least five should have published at least five refereed papers in the previous three refereed papers in the previous three years.years.

• Students grades:Students grades:– >90% for an A>90% for an A– 75-89% for a B75-89% for a B– (the pass grade is a low-end control)(the pass grade is a low-end control)

Standards set criteria that define ‘good practice’ inputs, process designs or outputs.

Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –


Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

Advanced Control Standard


Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –


Successful implementation of the supporting strategiesmay result in a general shift towards the standard; with some meeting it and others not.



Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –




Successful implementation of the supporting strategiesmay result in a general shift towards the standard; with some meeting it and others not.


Conclusions:Conclusions: Promotes good practices.Promotes good practices. Does not protect from risks.Does not protect from risks. May not promote the top end.May not promote the top end.



• A “quality = excellence” model (assuming A “quality = excellence” model (assuming we can all be excellent).we can all be excellent).

• Has some utility for HEd, for example in Has some utility for HEd, for example in awarding summa cum laude or various awarding summa cum laude or various types of awards, or for seeking types of awards, or for seeking international recognition.international recognition.

• NOT a substitute for low-end controls NOT a substitute for low-end controls (they are set too high for it to be realistic (they are set too high for it to be realistic to expect sufficient compliance).to expect sufficient compliance).

4. Standards as a high-end target4. Standards as a high-end target

Features:Features:

• Does not assume that performance below Does not assume that performance below the standard is unacceptable.the standard is unacceptable.

• Does assume all performance can be Does assume all performance can be improved.improved.

• Aspirational in nature, with no particular Aspirational in nature, with no particular consequences for meeting or not meeting consequences for meeting or not meeting (therefore different from Control standards – (therefore different from Control standards – although incentives might be utilised).although incentives might be utilised).


Examples:Examples:

• Research revenue >20% total revenue Research revenue >20% total revenue within 2yrs.within 2yrs.

• Our students win more externally-fundedOur students win more externally-fundedscholarships than any other HEI.scholarships than any other HEI.

• Have a ‘world-class’ Library.Have a ‘world-class’ Library.

Target standards can be set to stretch virtually the whole organisation.

Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –


Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

TargetStandard

TargetStandard


Successful implementation of the supporting strategies may encourage wide-scale progress towards the target, because it potentially inspires everybody.

Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

TargetStandard

TargetStandard


Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

TargetStandard

TargetStandard

Successful implementation of the supporting strategies may encourage wide-scale progress towards the target, because it potentially inspires everybody.


Conclusions:Conclusions: Encourages progress at the high-end.Encourages progress at the high-end. There are significant difficulties definingThere are significant difficulties defining

target standards.target standards. Does nothing to ensure HEIs comply with Does nothing to ensure HEIs comply with

minimum standards (does not protect from minimum standards (does not protect from risks or provide guarantees).risks or provide guarantees).



• A “quality = continuous improvement” and A “quality = continuous improvement” and “quality = best practice” model.“quality = best practice” model.

• Has some utility for HEd, especially Has some utility for HEd, especially regarding strategic planning and regarding strategic planning and international benchmarking.international benchmarking.

5. Combinations5. Combinations

Multiple standards can be applied to the Multiple standards can be applied to the same issue. The two most obvious same issue. The two most obvious combinations are:combinations are:

• Low-end control plus advanced control/sLow-end control plus advanced control/s

• Low-end control plus high-end targetLow-end control plus high-end target

Quality Measure

Qua

ntit

y M

easu

re

+

+

–

– Quality Measure

Qua

ntit

y M

easu

re

+

+

– –

EXAMPLE:

All academic staff must publish one peer-reviewed article per year to maintain their contracts; staff that publish more than 6 in past four years are eligiblefor promotion. Publication in high impact journals will be encouraged and celebrated.

Low-end Control Standard Advanced Control Standard

Low-end Control Standard Advanced Control Standard

TargetStandard

TargetStandard

Who Influences Who Influences Standards?Standards?

Key PointsKey Points

• There is no one-size-fits-all set of There is no one-size-fits-all set of international standards for HEIs.international standards for HEIs.

• Many stakeholder groups try to influence Many stakeholder groups try to influence the standards-setting process.the standards-setting process.

• This gives rise to variations between This gives rise to variations between different countries/jurisdictions. Such different countries/jurisdictions. Such variations are often legitimate & essential.variations are often legitimate & essential.

• But it means that mutual recognition of But it means that mutual recognition of standards between jurisdictions is complex. standards between jurisdictions is complex. It is possible, but needs to be negotiated.It is possible, but needs to be negotiated.

Who Influences Standards?Who Influences Standards?

• Ministries?Ministries?

• HEIs?HEIs?

• Funders?Funders?

• Students?Students?

• Employers?Employers?

• Professional Bodies?Professional Bodies?

• Foreign/International EQAAs?Foreign/International EQAAs?

• Media?Media?

• OAC?OAC?

There are many legitimate stakeholders. Each has valuable

contributions to made, and limitations inherent in their different motivations.

So, consultation is essential!

Proposed WayProposed WayForward for OmanForward for Oman

Standards and LicensureStandards and Licensure

• In Oman, institutional standards are usedIn Oman, institutional standards are usedfor licensure and standards assessment for licensure and standards assessment (accreditation) purposes.(accreditation) purposes.

• Licensure constitutes an assurance thatLicensure constitutes an assurance thatlow-end control standards are met.low-end control standards are met.

• Licensure is conducted one time only,Licensure is conducted one time only,based on statements of intent.based on statements of intent.

• Thereafter, responsibility for assuring those Thereafter, responsibility for assuring those minimum standards are maintained in minimum standards are maintained in practice is with the supervising Ministries.practice is with the supervising Ministries.

Standards and AccreditationStandards and Accreditation

• Accreditation should be more than just Accreditation should be more than just confirmation that licensure remains valid.confirmation that licensure remains valid.

• Society expects that an accredited institution Society expects that an accredited institution is ‘better’ than a non-accredited institution.is ‘better’ than a non-accredited institution.

• So, standards should advance the sector; So, standards should advance the sector; not just maintain it.not just maintain it.

• Therefore, OAC accreditation will use a Therefore, OAC accreditation will use a combination of types of standards (not just combination of types of standards (not just be based on low-end control standards).be based on low-end control standards).

New Institutional StandardsNew Institutional Standards

• The Institutional Standards in ROSQA The Institutional Standards in ROSQA (2004) are being revised.(2004) are being revised.

• By January, a set of draft standards willBy January, a set of draft standards willbe assembled based on international be assembled based on international benchmarks and OAC’s expertise. benchmarks and OAC’s expertise.

• They will be structured using the same They will be structured using the same scope headings as the OAC’s scope headings as the OAC’s Quality Audit Quality Audit Manual (Institutional Accreditation: Stage 1)Manual (Institutional Accreditation: Stage 1)

• They will be subjected to extensive They will be subjected to extensive consultation early in 2009.consultation early in 2009.

New Institutional StandardsNew Institutional Standards

• They will be comprehensive.They will be comprehensive.• Different types of standards will be used. Different types of standards will be used.

In some cases, the standards must be met; In some cases, the standards must be met; in other cases, an HEI should demonstrate in other cases, an HEI should demonstrate positive progress. This distinction will be positive progress. This distinction will be made transparent.made transparent.

• Standards will differentiate between Standards will differentiate between institutional classifications.institutional classifications.

• The final standards will be published in The final standards will be published in OAC’s OAC’s Standards Assessment Manual Standards Assessment Manual (Institutional Accreditation: Stage 2).(Institutional Accreditation: Stage 2).

ProposalProposal

• Standards Assessment is still a few years Standards Assessment is still a few years away, but institutions will need a lead-in time away, but institutions will need a lead-in time (facilitated by the Quality Audit process). (facilitated by the Quality Audit process).

• One proposal to be discussed is that HEIs One proposal to be discussed is that HEIs will be Accredited if the required standards will be Accredited if the required standards are met; and Accredited with Distinction if are met; and Accredited with Distinction if good progress towards target standards is good progress towards target standards is demonstrated.demonstrated.

• OAC will use the standards to negotiate OAC will use the standards to negotiate international recognition agreements.international recognition agreements.

The OQN will be consulted on proposed new standards

In conclusion…

Standards are an essential feature of quality assurance.

Standards provide helpful guidance for improvement purposes, and the basis for

public assurances.

But standards are not a substitute for thinking.

Society expects proven leadership from thehigher education sector.

That’s the Social Covenant.

PaintMasterpie

ces

Documents

Overcoming the Paradox of Standards in Higher Education Through the Covenant between Higher Education and Society Martin Carroll Keynote Address for Oman