The Value & Economic Measures of Libraries

The Value & EconomicMeasures of Libraries

10th Northumbria InternationalLibrary Conference

Joe MatthewsJuly 25, 2013

Morning Session

Megan Oakleaf

Outline

• Performance Measures

• Value

• Value of Information

• Value of Information Services

Outline

Personal value• Direct measures• Indirect measures

Organizational value

• Academic libraries

• Public libraries• National libraries

)

Few libraries exist in a vacuum, accountable only to themselves.

There is thus always a larger context for accessing library

quality, that is, what and how well does the library contribute to

achieving the overall goals of the parent constituencies?

Sarah Pritchard

There is no systematic evidencecollected which shows the value

of academic librariesfor teaching and research staff.

Claire Creaser and Valerie Spezi

Performance Measures

Input Process

Output

Outcomes

Outcomes

Library Services

Individual

Society

Efficiency Effectiveness

Cost Effectiveness

Impact

VALUE

Cost benefit

Resources

Capability

Use Beneficial effects

Start with the end in mind:

work backwards

Refocus from the activity

to the impact

Impact

Library Control

How much? How many? How economical? How prompt?

Magnitude

% of change last year

% of overall change

Cost

Magnitude

Change

Resources used

Units processed

Cycle times

Turnaround time

Anticipatory

Library & Customers Decide

How valuable? How reliable? How accurate?

Effort expended

Cost

Benefits obtained

Dependability

Access

Accuracy

Completeness

Comprehensiveness

Currency

Customers Decide

How well? How courteous? How responsive? How satisfied?

Accuracy

Promptness

Courtesy

Expertise

Attentive

Welcoming

Anticipatory

Helpful

Empathetic

Expectations met

Materials obtained

Personal interaction

Ease of use

Environment

Comfort

Willingness to return

Live by the numbers, ….

Challenges

Lack of consensus about what should be measured and how

Lack of understanding of performance measurement and metrics

Organizational structural issues Lack of precision in measuring performance, and

alignment issuesDetermining the “bottom line” is too far awayMajority of stakeholders are too far awayLibrary staff find it difficult to see the “big”

picture

And the survey said ….

Lack of a Connection

• Budget and outputs (and outcomes) are separated

• No “bottom line” measure for libraries

• Decision-making process is bigger than the library

• Library has neither champions nor foes

• Library benefits are not widely self-evident

Orr’s Fundamental Questions

• How good is the library?

• What good does the library do?

• How well is the library managed?

We should be a bit wary of the “little library”

…For when it is good, it is very, very good

and when it is bad,

it’s a “pretty good library for a town this size.”

Eleanor Jo Rodger

Levels of Assessment …

• Individual student

• Course

• Departmental/Program

• College or University

Types of Measures

• Direct– Provide tangible, visible and

self-explanatory evidence of what students have & have not learned

• Indirect– Capture students’

perceptions of their knowledge & skills; supplement direct measures; sometimes called surrogates

Qualitative Tools

• Focus groups – open ended

• Biography• Phenomenology –

capture the “Aha!” moment

• Grounded theory• Ethnography• Case study

Qualitative Assessment

• Provides in-depth understanding of user responses and interactions

• Represents part of a long-term strategy of formative evaluative

Quantitative Tools

• Surveys

• Transaction logs

• Statistics from systems

• Observations (count)

Quantitative Assessment

• Analyses to determine library impacts on academic performance, retention rates

• Describe retention rates and GPAs in defined populations over semesters and users

• Compare users & non-users of library services while adjusting for academic preparation and background differences

• Conduct quasi-experimental designs employing multivariate analysis of covariance & hierarchical regression techniques

Useful Assessment

Be cautious about cause-and-effect relationships

The Issue

• Is it: Use library resources/services and you will get better grades.

• Or: I want to do well and so I work hard to achieve better grades (and one way I do that is to use library resources/services).

“Not surprisingly, librarians are keen to show

that the use of expensive, scholarly materials

positively correlates with higher grades,

although they cannot prove that this is so.”

Deborah Goodall & David Pattern

“There is growing pressure on all academic library managers to be more

accountable for how they use limited resources and to achieve institutional outcomes perceived as important by

college and university stakeholders….”

Elizabeth Mezick

Value of Information

• Expect value-in-use

• Library’s collection reflects a “potential value”

• Collection also reflects a “future value”

• Value of local collection is declining

Valuable is not about our professional values;

in the paradigm of the value of libraries,we are the producers,

not the consumers of services.Our sense of what is valuable

really doesn’t matter much at all unless it

matches that our our customers.

Eleanor Jo Rodger

Fundamental Changes

Libraries have changed more in the past two decades than in the prior two

centuries. Technology is the major driver . . .

We need to recognize that all this change has only begun, and that change is irreversible.

Increasingly it is important

to remember that libraries

provide few unique services.

Information is woven into our lives

Quality of InformationThis fast food approach to information consumption drives librarians crazy. “Our information is healthier and tastes better too” they shout. But nobody listens. We’re too busy Googling.”

Peter Morville

Key Characteristics of Information

Uncertainty Knowledge

Ambiguity Indeterminacy

Redundancy System dependency

Sharing Timeliness

Compression Presentation

Stability Multiple life cycles

Leakability Substitutability

Criteria for Judging ValueCustomer Criteria Value Added by the Service

Ease of use Browsing, formatting, mediation service, orientation service, ordering, physical accessibility

Noise reduction Access (item identification, subject description, subject summary), linkage, precision, selectivity

Quality Accuracy, comprehensiveness, currency, reliability, validity

Adaptability Closeness to problem, flexibility, simplicity, stimulatory

Time savings Response speed

Cost savings Cost savings

Collections are disrupted

Atoms to bits

Nature of Information is Changing

Scare, controlled

Expensive

Shaped by elites

One-way, mass consumption

Slow moving

External to our worlds

All around us

Cheap or free

Shaped by consumers

Designed for sharing,

participation & feedback

Immediate

Embedded in our worlds

Information

was ….

Information

is ….

Value of the Academic Library

If the physical proximity of

print collections had a demonstrable impact on researcher productivity,

no university would hesitate to allocate prime real estate

to library stacks.

Traditional Value Proposition

Without a great library, there can be no great university.

David Kinly,President of

The University of Illiniois1929

Universities Provide

• Private goods & services– Courses exchanged for

tuition– Research completed for

funding

• The value propositionThe value to an individual or an organization determines whether payment is made for the service

Academic Libraries Provide

• Public goods and servicesPrint and online resources are shared by all, usually without the exchange of payment

• Value propositionThe collective value of all users must be estimated to determine if a good or service should be continued

Value is determined by the user

and the

use of information

Astin’s IEO Model

Input

Output

Environment

Programs

Institutional characteristics

Library

Fellow students

Faculty

Place of residence

Student Learning Outcomes ModelIntelligence

General Reasoning

Broad Abilities

Knowledge, Understanding, and Reasoning

Abstract, ProcessOriented

Concrete,Content-Oriented

Campus Culture of Learning

• What learning experiences require use of library resources?

• What % of students engage in learning experiences in the library?

• Does the library contribute to civic learning expectations?

• Library $ spent that support teaching and learning?

• Consider using flourishing and resiliency scales to help identify the library’s impact.

• What % of librarians serve as student advisors?

Define, develop, and measure

outcomes

that contribute to

institutional effectiveness

ACRL Standards for Libraries in Higher Education

Challenge

It is not how much a library

is used that matters,

rather how does the library

impact or benefit the customer?

Perspectives on Value

Benefits

Use

Nonuse

Direct

Indirect

Option – Preservation of option for

future use by meExistence – Perceived value and significance

to the communityLegacy – Value of preservation for

future generations

Personal

Organizational

Financial

Impacts

Personal

Why Use the Library?

Reasons

Interactions

Results

For a TASK

For PERSONAL reasons

To get an OBJECT or INFORMATION

To perform an ACTIVITY

Access RESOURCES

Use of RESOURCES or SERVICES

OPERATIONS

ENVIRONMENTS

COGNITIVE results

AFFECTIVE results

ACCOMPLISHMENTS

EXPECTATIONS met

TIME aspects

MONEY estimates

Generic Learning Outcomes

Knowledge & Understanding Skills

Attitudes Enjoyment, Inspiration, Creativity

Activity, Behavior, Progression

Organizational

• Full-time students• Live on campus• Interact more with

faculty• Study more• Collaborate with their

peers

Student Learning is Affected by …

NSSE• 5 benchmarks of effective educational

practice– Level of academic challenge– Active & collaborative learning– Student-Faculty interaction– Enriching educational experiences– Supportive campus environment

• Student self-reported gains in intellectual & personal development

• No overlap between self-reported data & standardized objective tests

NSSE – Use of the Academic Library

• 50% never used the library

• Use of libraries at small, academically challenging liberal arts colleges are correlated with other purposeful activities

• Library use less intensive at larger universities

• Students who work harder use library resources

Assessing Student Achievement

• Direct measures– Capstone experience– Use of a portfolio– Standardized exam (Collegiate

Learning Assessment)

• Indirect measures

• Gains in student performance are quite low

• Individual learning is characterized by persistence

• Notable variation within and across institutions

Assessment of Higher Ed

Wabash National Study

• Different instrument – CAAP• 2,212 students

• Nearly identical results to Academically Adrift

• 44 percent no gains in the first year• 33 percent no gains in 4 years• Students only study about 15 hours

per week

“One way to demonstrate the library’s contribution is to assess whether

students’ experiences with the library

directly or indirectly contribute to desired outcomes of

college.”

George D. Kuh & Robert M. Gonyea

How to Demonstrate Impact in …

• Student enrollment• Student experiences• Student learning• Student grades (GPA) &

achievement• Student retention & graduation• Student career success• Faculty productivity• Institutional reputation• The environment

Student Enrollment

• Recruitment of prospective students

• Matriculation of admitted students

• Recommendation of current students

Student Enrollment

Student Learning

• GPA• Professional/educational test

scores• Learning assessments• Faculty judgments

Student Learning

Meta-analysis

• Entering student characteristics– SES– High school GPA– ACT/SAT scores

• Environment - Psychosocial and study skill factors– Academic goals, skills and self-

confidence– Social support & engagement

(acculturation)

Student Learning Occurs …

• in the classroom• in the laboratory• with peers• in the student union• in the dorm• in the library (for

some)• online• and

Direct Measures

• Capstone Experience

• Portfolios

• Standardized Test

• Locally-developed test

Capstone Projects

Indirect Measures

• Surveys• Retention, graduation

rates• Grades• Acceptance rates into

graduate programs• Job placement rates• Exit interviews• Alumni surveys

Student Grades

Non/Low Use

University of Wollongong

Usage of Electronic ResourcesWAM = Weighted Average Marks (Grades)

Other Studies

• Hong Kong Baptist University

• Georgia State University

Hope College

University of Minnesota

Gym Bags and Mortarboards

Use Campus Recreational Facilities

At least 25 times, first-year retention increased 1%

& 5-year graduation rates increased 2%

University of Minnesota Library

• 5,368 first-year non-transfer students

• Use of library was associated with a .23 increase in students GPA

• More use of the library, GPA also goes up

University of Minnesota Library

• Opinion surveys

• Skills testing

• Observed behaviors

Library Instruction and GPA

University of Wyoming Libraries

• Analysis of 4,489 transcripts• Slight positive relationship between

upper-level library instruction courses and GPA –

0.075 GPA difference – that’s less than

1/10th of 1 percent

• Research statement – 44%

• Evaluate Web site – objectivity – 52%

– authority – 65%

• Presentation to persuade – 12%

Library Instruction and GPA

Hong King Baptist University • 45 sample groups – N=31 to 1,223, study

majors• Pairs of data• One-fourth (11) had a positive

relationship• Results:

– 1 or 2 workshops – little impact on GPA– 3 or 4 workshops – ½ show a positive impact– 5 workshops (1 sample group) – 100% had a

higher GPA

Bibliographic Instruction• Improvement in basic library skills is

the means and not the end• Yet the means is the focus for

evaluation efforts• Evaluation efforts focus on

– Opinion surveys– Skills improvement– Pre-test & post-test knowledge

• Not the impact on student achievement

Library experiences do not seem to directly contribute to gains in information literacy, to what students gain overall in college, or to student satisfaction.

Kuh & Gonyea

How scalable is library instruction?

Student Retention & Graduation

Attrition

Retention

Persistence

Completion

Graduation Rates

• Fall-to-fall retention• Graduation rates

Student Retention

Retention Concepts

• Institutional retention– Enrolling & graduating from the same

institution

• Program retention– Enrolling & graduating with the same

major/department/school

• System retention– Students who leave one university yet

continue and complete post-secondary studies elsewhere

Measures of Retention

• Persistence (Continuation rate)– From first to second year? Entry to

graduation?

• Completion rate– From entry to graduation (Student

goals?)

• Graduation rates– Are transfers included? Time period?

• Attrition– Leaving university? Leaving higher ed?

Measures of Retention

• Stopout– Leave university with the intention (and action)

of returning later to complete a program

• Dropout– Leave university with intention (and action) of

NOT returning

• Transfer– Change institutions yet persist in higher

education– May change type of institution

• Voluntary vs. involuntary attrition?

Why Students Leave?

• Students’ decision to leave University is influenced by many personal factors– Financial reasons– Family responsibilities– Lack of academic ability– Poor fit, etc.

• Foundational Theories from Education / Psychology:– Tinto’s “Model of Student Integration” – Bean’s “Model of Student Attrition”

Tinto’s Model of Student Integration

Pre-entryAttributes

Goals /Commitments

InstitutionalExperiences

Integration

Goals /Commitments

Outcome

SES

Skills &Abilities

Quality ofEducation

Intentions

InstitutionalCommitments

Academic System

Performance

Engagement

Extra-Curricular

Peer group

Social System

AcademicIntegration

SocialIntegration

Intentions

Goal

InstitutionalMatch

Stay orLeave

Bean’s Model of Student Attrition

Grades

Courses

Educational Goals

Major & Job Certainty

Opportunity to Transfer

Family Approval

Org

aniz

ati

onal

Vari

ab

les

Pers

on

al Vari

ab

les

Envir

onm

en

tal

Vari

ab

les

Loyalty

AttitudesCertainty

PracticalValue

Intent

Dropout

Student Retention & Graduation

• Important because … rankings, revenues, educational achievement, emotional well-being

• Many reasons for drop-outs are not under the control of the university

• Engagement is the key

Indicators

• Student goal attainment

• Course retention• Subsequent course

work• Fall-to-fall

persistence• Time to degree• Degree completion• Grad school

enrollment

• Transfer rate & success

• Employer assessment

• Academic value add• Student satisfaction• Professional growth• Student involvement• Citizenship &

engagement

Australasian Survey of StudentEngagement (AUSSE)

Curtin University

University of Huddersfield

Dropp

ing

out!

“A high rate of attrition

is indicative of a failure

on the part of an institution to achieve its purpose.”

Elizabeth Mezick

Student Engagement

“many students don’t develop a personal connection with their institution. And when they don’t, they leave”

Gonzales 2010NSSE & CSEQ

Library Retention Studies

• Statistically significant relationships between library expenditures, or staffing levels and student retention

E.g. Hiscock, 1986 Hamrick, Schuh, & Shelley,

2004 Mezick, 2007

Graduation Rates & Library Expenditures

• Used IPEDS data on institutional characteristics & resource allocations

• Library expenditures was strongly correlated with graduation rates – 1.77 percent increase in graduation rates

• Greatest payoff is attributable to enhanced library expenditures (+0.92) and instruction (+0.80) while increased non-library contributions were quite modest (+0.27)

Hamrick, Schuh & Shelley


• Relationships between library use (collections) and student retention

– Student who borrowed books = more likely to persist

• E.g. Kramer & Kramer, 1968

• Impact of instruction

– Students involved in library skills programs showed lower attrition rates

• E.g. Knapp, 1966


University of Minnesota

– 77% of undergrads made use of the libraries, 85% of grad students made use of the libraries

– Students who used the library at least once were 1.54 times more likely to re-enroll


• Some library involvement in first year experience programs; specific programs for “at risk” groups

– NOT proven to have significant effect

• E.g. Hollis, 2001 Colton, et al, 2002 Aguilar & Keating, 2009 Love, 2009


• Relationship between library employment & retention

– Higher completion rate among library student workers

• E.g. Wilder, 1990Rushing & Poole, 2002

“If strong linkages between libraries and student retention can be made,

then the perceived value of the library may indeed rise.”

Steven Bell

• Some groups, some majors & seniors engage in more library-related activities

• Academic support expenditures tend to correlate with increased engagement

• Institutional academic challenge correlates with library use

Student Career SuccessGrad School Exams

Time to

First

Job

Average Salary of First

Job

Alumni Surveys

Alumni Giving

• Job placement rates• First-year job salaries• Professional/graduate school

acceptance• Internship success• Marketable skills

Student Success

• Integration of library resources and services into course syllabi, Websites, lectures, labs, reserve readings, etc.

• Faculty/librarian collaborations; cooperative curriculum, assignment, or assessment design

Faculty Teaching

Perceived Benefits for Teaching

• Savings– Of own time– Of own money– Of other resources

• Improvements– Teaching– Course-related materials– Student performance

Library Value Wheel

Impact on Faculty

• Library is the source for most journal articles (individual subscriptions are way down)

• If library subscriptions were unavailable – productivity would decrease 17%

• Library is not the source of book readings

• 42% of reading material is library provided

Time

• Academics spend a lot of time reading

• Article reading inspires new thinking, improved results, changed focus

• Award-winning academics read more

• Academics who publish more use more library resources

Ithaka Studies

• Library services not understood

• Library services not valued

• The Library is disappearing

• Number of publications, number of patents, value of technology transfer

• Tenure/promotion judgments

Faculty Research Productivity

Faculty Grants

• Number of grant proposals (funded or unfunded)

• Value of grants funded

Assessment of Research

• Payback model – form of ROI• Research impact• Research utilization ladder• Lavis decision-making impact model• Weiss logic model• HTA organization assessment

framework• Societal impact framework• Research assessment exercise• Becker medical library model

For Most Impact Models

• Indicators of research output

• Indicators of knowledge transfer

• Indicators of implementation

• Indicators of community benefit

Faculty Productivity

• Faculty recruitment• Institutional rankings• Community engagement

Institutional Reputation & Prestige

Institutional Reputation

• Changes in reputational rankings affects student & faculty recruitment

• University budget allocations to libraries have decreased

Since the library absorbs a very small percentage of a university budget, the contribution of the library is disproportionately high relative to its cost to the institution.

Sharon Weiner

• Indispensable for their research

• Maintain a high-level overview of their field

• Value for money is good

• Library not available, costs would increase 40%

• Take 31% longer to locate same information

Library Value Scorecard

• Relational Capital

• Library Capital

• Library Virtue

• Library Momentum

University & the Library Can

• Attract outstanding faculty

• Retain outstanding faculty

• Foster innovative research

• Align library activities with university goals

Value of Special Libraries

• Time saved

• Money saved

• New revenues

• Other outcomes

In addition, to ROI

• Knowledge-Value Add

• Intranet Team Forums

• Intellectual Capital Valuation

Measurements of valuewere, in fact, a

key differentiator between

successfuland unsuccessful corporate libraries

James Matarazzo

Value of the

(Rooney-Browne, 2009b).

Social Capital

• Bonding social capital

• Bridging social capital

• Linking social capital

Social Benefits

• Basic reading literacy

• Business/career

• Information literacy

• Library as place

• Summer reading

• Local history & genealogy

• Health & well-being

• Social cohesion

• General information

• Empowerment

• Social connection• Education• Employment• Health & wellness• eGovernment• Community engagement• Personal finance

Making Cities Stronger

• Building a stronger local economy

• Improving early literacy & school readiness

• Building workforce participation

• Supporting small business

• The power of place

• Literacy

• Workforce development

• Business development

• Value to homes and neighborhoods

Libraries/Building/Communities

• Developing social capital– Providing a welcoming environment– Creating a pride of place– Attracting users from all walks of life– Reaching out to the community– Appreciation of cultural differences– Building bridges to government– Encouraging collaboration across the

community


• Overcoming the digital divide– Making technology accessible– Exploiting technology to benefit the

community

• Creating informed communities– Community information– Government information– Providing a gateway to the world of

information


• Convenient and comfortable places of learning– Developing information skills– Stimulating ideas and discussion– Supporting vulnerable learners– Supporting students

Outcomes

Enhanced quality of life

Enhanced enjoyment from hobbies

Able to obtain information

Facilitates lifelong learning

Support for children’s education

Contributions

Safe and pleasant place

Supporting educational facilities

Facilitating lifelong learning

Encouraging responsible social behavior

Ensuring access to the Internet

Tracking ValueThe Engaged Library: Chicago Stories of Community Building

•Prove that public libraries build social capital

•Identify & connect the library’s assets to the community

•Assess & strengthen the library’s connections with and use of community assets

•Produce a toolkit for other libraries to adopt to

•Mapping tools to perform an inventory services, identify areas for improvement and highlight library’s contribution to the community’s wider social, educational, cultural and economic goals.

PLQIM

• Access to information• Community & personal

participation• Meeting readers’ needs• Learners’ experiences• Ethos & values• Organization & use of resources• Leadership

Valuing the Collection

Dewey SubclassNumber of Titles

2010-2011 Avg List Total Value ($)

001 - Knowledge $76.71 $0.00

002 - The book $62.45 $0.00

003 - Systems $129.77 $0.00

004 - Data processing. Computer science $89.82 $0.00

005 - Computer programming, programs, data $69.14 $0.00

006 - Special computer methods $83.60 $0.00

010 - Bibliography $73.65 $0.00

011 - Bibliographies $69.46 $0.00

012 - Bibliographies of individuals $0.00 $0.00

013 - Of works by specific classes of authors $0.00 $0.00

014 - Of anonymous and pseudonymous works $55.95 $0.00

015 - Of works from specific places $184.99 $0.00

016 - Of works on specific subjects $134.87 $0.00

017 - General subject catalogs $0.00 $0.00

018 - Catalogs arranged by author, date, etc. $0.00 $0.00

019 - Dictionary catalogs $0.00 $0.00

020 - Library and information sciences $56.06 $0.00

021 - Library relationships $62.60 $0.00

022 - Administration of the physical plant $65.50 $0.00

023 - Personnel administration $56.00 $0.00

025 - Library operations $77.02 $0.00

026 - Libraries for specific subjects $89.99 $0.00

027 - General libraries $66.25 $0.00

[email protected]

www.joematthews.org

Joe MatthewsLibrary Consultant

mailto:[email protected]

http://www.joematthews.org/

What Are the Results

Four Year CollegeFull-time

Four Year CollegePart-time

Two Year College

Stopout/Transfer

College degree – 19%

AA degree – 8%

No college degree – 60%

No college

Graduate degree – 11%