The Value & EconomicMeasures of Libraries
10th Northumbria InternationalLibrary Conference
Joe MatthewsJuly 25, 2013
Morning Session
Megan Oakleaf
Outline
• Performance Measures
• Value
• Value of Information
• Value of Information Services
Outline
Personal value• Direct measures• Indirect measures
Organizational value
• Academic libraries
• Public libraries• National libraries
)
Few libraries exist in a vacuum, accountable only to themselves.
There is thus always a larger context for accessing library
quality, that is, what and how well does the library contribute to
achieving the overall goals of the parent constituencies?
Sarah Pritchard
There is no systematic evidencecollected which shows the value
of academic librariesfor teaching and research staff.
Claire Creaser and Valerie Spezi
Performance Measures
Input Process
Output
Outcomes
Outcomes
Library Services
Individual
Society
Efficiency Effectiveness
Cost Effectiveness
Impact
VALUE
Cost benefit
Resources
Capability
Use Beneficial effects
Start with the end in mind:
work backwards
Refocus from the activity
to the impact
Impact
Library Control
How much? How many? How economical? How prompt?
Magnitude
% of change last year
% of overall change
Cost
Magnitude
Change
Resources used
Units processed
Cycle times
Turnaround time
Anticipatory
Library & Customers Decide
How valuable? How reliable? How accurate?
Effort expended
Cost
Benefits obtained
Dependability
Access
Accuracy
Completeness
Comprehensiveness
Currency
Customers Decide
How well? How courteous? How responsive? How satisfied?
Accuracy
Promptness
Courtesy
Expertise
Attentive
Welcoming
Anticipatory
Helpful
Empathetic
Expectations met
Materials obtained
Personal interaction
Ease of use
Environment
Comfort
Willingness to return
Live by the numbers, ….
Challenges
Lack of consensus about what should be measured and how
Lack of understanding of performance measurement and metrics
Organizational structural issues Lack of precision in measuring performance, and
alignment issuesDetermining the “bottom line” is too far awayMajority of stakeholders are too far awayLibrary staff find it difficult to see the “big”
picture
And the survey said ….
Lack of a Connection
• Budget and outputs (and outcomes) are separated
• No “bottom line” measure for libraries
• Decision-making process is bigger than the library
• Library has neither champions nor foes
• Library benefits are not widely self-evident
Orr’s Fundamental Questions
• How good is the library?
• What good does the library do?
• How well is the library managed?
We should be a bit wary of the “little library”
…For when it is good, it is very, very good
and when it is bad,
it’s a “pretty good library for a town this size.”
Eleanor Jo Rodger
Levels of Assessment …
• Individual student
• Course
• Departmental/Program
• College or University
Types of Measures
• Direct– Provide tangible, visible and
self-explanatory evidence of what students have & have not learned
• Indirect– Capture students’
perceptions of their knowledge & skills; supplement direct measures; sometimes called surrogates
Qualitative Tools
• Focus groups – open ended
• Biography• Phenomenology –
capture the “Aha!” moment
• Grounded theory• Ethnography• Case study
Qualitative Assessment
• Provides in-depth understanding of user responses and interactions
• Represents part of a long-term strategy of formative evaluative
Quantitative Tools
• Surveys
• Transaction logs
• Statistics from systems
• Observations (count)
Quantitative Assessment
• Analyses to determine library impacts on academic performance, retention rates
• Describe retention rates and GPAs in defined populations over semesters and users
• Compare users & non-users of library services while adjusting for academic preparation and background differences
• Conduct quasi-experimental designs employing multivariate analysis of covariance & hierarchical regression techniques
Useful Assessment
Be cautious about cause-and-effect relationships
The Issue
• Is it: Use library resources/services and you will get better grades.
• Or: I want to do well and so I work hard to achieve better grades (and one way I do that is to use library resources/services).
“Not surprisingly, librarians are keen to show
that the use of expensive, scholarly materials
positively correlates with higher grades,
although they cannot prove that this is so.”
Deborah Goodall & David Pattern
“There is growing pressure on all academic library managers to be more
accountable for how they use limited resources and to achieve institutional outcomes perceived as important by
college and university stakeholders….”
Elizabeth Mezick
Value of Information
• Expect value-in-use
• Library’s collection reflects a “potential value”
• Collection also reflects a “future value”
• Value of local collection is declining
Valuable is not about our professional values;
in the paradigm of the value of libraries,we are the producers,
not the consumers of services.Our sense of what is valuable
really doesn’t matter much at all unless it
matches that our our customers.
Eleanor Jo Rodger
Fundamental Changes
Libraries have changed more in the past two decades than in the prior two
centuries. Technology is the major driver . . .
We need to recognize that all this change has only begun, and that change is irreversible.
Increasingly it is important
to remember that libraries
provide few unique services.
Information is woven into our lives
Quality of InformationThis fast food approach to information consumption drives librarians crazy. “Our information is healthier and tastes better too” they shout. But nobody listens. We’re too busy Googling.”
Peter Morville
Key Characteristics of Information
Uncertainty Knowledge
Ambiguity Indeterminacy
Redundancy System dependency
Sharing Timeliness
Compression Presentation
Stability Multiple life cycles
Leakability Substitutability
Criteria for Judging ValueCustomer Criteria Value Added by the Service
Ease of use Browsing, formatting, mediation service, orientation service, ordering, physical accessibility
Noise reduction Access (item identification, subject description, subject summary), linkage, precision, selectivity
Quality Accuracy, comprehensiveness, currency, reliability, validity
Adaptability Closeness to problem, flexibility, simplicity, stimulatory
Time savings Response speed
Cost savings Cost savings
Collections are disrupted
Atoms to bits
Nature of Information is Changing
Scare, controlled
Expensive
Shaped by elites
One-way, mass consumption
Slow moving
External to our worlds
All around us
Cheap or free
Shaped by consumers
Designed for sharing,
participation & feedback
Immediate
Embedded in our worlds
Information
was ….
Information
is ….
Value of the Academic Library
If the physical proximity of
print collections had a demonstrable impact on researcher productivity,
no university would hesitate to allocate prime real estate
to library stacks.
Traditional Value Proposition
Without a great library, there can be no great university.
David Kinly,President of
The University of Illiniois1929
Universities Provide
• Private goods & services– Courses exchanged for
tuition– Research completed for
funding
• The value propositionThe value to an individual or an organization determines whether payment is made for the service
Academic Libraries Provide
• Public goods and servicesPrint and online resources are shared by all, usually without the exchange of payment
• Value propositionThe collective value of all users must be estimated to determine if a good or service should be continued
Value is determined by the user
and the
use of information
Astin’s IEO Model
Input
Output
Environment
Programs
Institutional characteristics
Library
Fellow students
Faculty
Place of residence
Student Learning Outcomes ModelIntelligence
General Reasoning
Broad Abilities
Knowledge, Understanding, and Reasoning
Abstract, ProcessOriented
Concrete,Content-Oriented
Campus Culture of Learning
• What learning experiences require use of library resources?
• What % of students engage in learning experiences in the library?
• Does the library contribute to civic learning expectations?
• Library $ spent that support teaching and learning?
• Consider using flourishing and resiliency scales to help identify the library’s impact.
• What % of librarians serve as student advisors?
Define, develop, and measure
outcomes
that contribute to
institutional effectiveness
ACRL Standards for Libraries in Higher Education
Challenge
It is not how much a library
is used that matters,
rather how does the library
impact or benefit the customer?
Perspectives on Value
Benefits
Use
Nonuse
Direct
Indirect
Option – Preservation of option for
future use by meExistence – Perceived value and significance
to the communityLegacy – Value of preservation for
future generations
Personal
Organizational
Financial
Impacts
Personal
Why Use the Library?
Reasons
Interactions
Results
For a TASK
For PERSONAL reasons
To get an OBJECT or INFORMATION
To perform an ACTIVITY
Access RESOURCES
Use of RESOURCES or SERVICES
OPERATIONS
ENVIRONMENTS
COGNITIVE results
AFFECTIVE results
ACCOMPLISHMENTS
EXPECTATIONS met
TIME aspects
MONEY estimates
Generic Learning Outcomes
Knowledge & Understanding Skills
Attitudes Enjoyment, Inspiration, Creativity
Activity, Behavior, Progression
Organizational
• Full-time students• Live on campus• Interact more with
faculty• Study more• Collaborate with their
peers
Student Learning is Affected by …
NSSE• 5 benchmarks of effective educational
practice– Level of academic challenge– Active & collaborative learning– Student-Faculty interaction– Enriching educational experiences– Supportive campus environment
• Student self-reported gains in intellectual & personal development
• No overlap between self-reported data & standardized objective tests
NSSE – Use of the Academic Library
• 50% never used the library
• Use of libraries at small, academically challenging liberal arts colleges are correlated with other purposeful activities
• Library use less intensive at larger universities
• Students who work harder use library resources
Assessing Student Achievement
• Direct measures– Capstone experience– Use of a portfolio– Standardized exam (Collegiate
Learning Assessment)
• Indirect measures
• Gains in student performance are quite low
• Individual learning is characterized by persistence
• Notable variation within and across institutions
Assessment of Higher Ed
Wabash National Study
• Different instrument – CAAP• 2,212 students
• Nearly identical results to Academically Adrift
• 44 percent no gains in the first year• 33 percent no gains in 4 years• Students only study about 15 hours
per week
“One way to demonstrate the library’s contribution is to assess whether
students’ experiences with the library
directly or indirectly contribute to desired outcomes of
college.”
George D. Kuh & Robert M. Gonyea
How to Demonstrate Impact in …
• Student enrollment• Student experiences• Student learning• Student grades (GPA) &
achievement• Student retention & graduation• Student career success• Faculty productivity• Institutional reputation• The environment
Student Enrollment
• Recruitment of prospective students
• Matriculation of admitted students
• Recommendation of current students
Student Enrollment
Student Learning
• GPA• Professional/educational test
scores• Learning assessments• Faculty judgments
Student Learning
Meta-analysis
• Entering student characteristics– SES– High school GPA– ACT/SAT scores
• Environment - Psychosocial and study skill factors– Academic goals, skills and self-
confidence– Social support & engagement
(acculturation)
Student Learning Occurs …
• in the classroom• in the laboratory• with peers• in the student union• in the dorm• in the library (for
some)• online• and
Direct Measures
• Capstone Experience
• Portfolios
• Standardized Test
• Locally-developed test
Capstone Projects
Indirect Measures
• Surveys• Retention, graduation
rates• Grades• Acceptance rates into
graduate programs• Job placement rates• Exit interviews• Alumni surveys
Student Grades
Non/Low Use
University of Wollongong
Usage of Electronic ResourcesWAM = Weighted Average Marks (Grades)
Other Studies
• Hong Kong Baptist University
• Georgia State University
Hope College
University of Minnesota
Gym Bags and Mortarboards
Use Campus Recreational Facilities
At least 25 times, first-year retention increased 1%
& 5-year graduation rates increased 2%
University of Minnesota Library
• 5,368 first-year non-transfer students
• Use of library was associated with a .23 increase in students GPA
• More use of the library, GPA also goes up
University of Minnesota Library
• Opinion surveys
• Skills testing
• Observed behaviors
Library Instruction and GPA
University of Wyoming Libraries
• Analysis of 4,489 transcripts• Slight positive relationship between
upper-level library instruction courses and GPA –
0.075 GPA difference – that’s less than
1/10th of 1 percent
• Research statement – 44%
• Evaluate Web site – objectivity – 52%
– authority – 65%
• Presentation to persuade – 12%
Library Instruction and GPA
Hong King Baptist University • 45 sample groups – N=31 to 1,223, study
majors• Pairs of data• One-fourth (11) had a positive
relationship• Results:
– 1 or 2 workshops – little impact on GPA– 3 or 4 workshops – ½ show a positive impact– 5 workshops (1 sample group) – 100% had a
higher GPA
Bibliographic Instruction• Improvement in basic library skills is
the means and not the end• Yet the means is the focus for
evaluation efforts• Evaluation efforts focus on
– Opinion surveys– Skills improvement– Pre-test & post-test knowledge
• Not the impact on student achievement
Library experiences do not seem to directly contribute to gains in information literacy, to what students gain overall in college, or to student satisfaction.
Kuh & Gonyea
How scalable is library instruction?
Student Retention & Graduation
Attrition
Retention
Persistence
Completion
Graduation Rates
• Fall-to-fall retention• Graduation rates
Student Retention
Retention Concepts
• Institutional retention– Enrolling & graduating from the same
institution
• Program retention– Enrolling & graduating with the same
major/department/school
• System retention– Students who leave one university yet
continue and complete post-secondary studies elsewhere
Measures of Retention
• Persistence (Continuation rate)– From first to second year? Entry to
graduation?
• Completion rate– From entry to graduation (Student
goals?)
• Graduation rates– Are transfers included? Time period?
• Attrition– Leaving university? Leaving higher ed?
Measures of Retention
• Stopout– Leave university with the intention (and action)
of returning later to complete a program
• Dropout– Leave university with intention (and action) of
NOT returning
• Transfer– Change institutions yet persist in higher
education– May change type of institution
• Voluntary vs. involuntary attrition?
Why Students Leave?
• Students’ decision to leave University is influenced by many personal factors– Financial reasons– Family responsibilities– Lack of academic ability– Poor fit, etc.
• Foundational Theories from Education / Psychology:– Tinto’s “Model of Student Integration” – Bean’s “Model of Student Attrition”
Tinto’s Model of Student Integration
Pre-entryAttributes
Goals /Commitments
InstitutionalExperiences
Integration
Goals /Commitments
Outcome
SES
Skills &Abilities
Quality ofEducation
Intentions
InstitutionalCommitments
Academic System
Performance
Engagement
Extra-Curricular
Peer group
Social System
AcademicIntegration
SocialIntegration
Intentions
Goal
InstitutionalMatch
Stay orLeave
Bean’s Model of Student Attrition
Grades
Courses
Educational Goals
Major & Job Certainty
Opportunity to Transfer
Family Approval
Org
aniz
ati
onal
Vari
ab
les
Pers
on
al Vari
ab
les
Envir
onm
en
tal
Vari
ab
les
Loyalty
AttitudesCertainty
PracticalValue
Intent
Dropout
Student Retention & Graduation
• Important because … rankings, revenues, educational achievement, emotional well-being
• Many reasons for drop-outs are not under the control of the university
• Engagement is the key
Indicators
• Student goal attainment
• Course retention• Subsequent course
work• Fall-to-fall
persistence• Time to degree• Degree completion• Grad school
enrollment
• Transfer rate & success
• Employer assessment
• Academic value add• Student satisfaction• Professional growth• Student involvement• Citizenship &
engagement
Australasian Survey of StudentEngagement (AUSSE)
Curtin University
University of Huddersfield
Dropp
ing
out!
“A high rate of attrition
is indicative of a failure
on the part of an institution to achieve its purpose.”
Elizabeth Mezick
Student Engagement
“many students don’t develop a personal connection with their institution. And when they don’t, they leave”
Gonzales 2010NSSE & CSEQ
Library Retention Studies
• Statistically significant relationships between library expenditures, or staffing levels and student retention
E.g. Hiscock, 1986 Hamrick, Schuh, & Shelley,
2004 Mezick, 2007
Graduation Rates & Library Expenditures
• Used IPEDS data on institutional characteristics & resource allocations
• Library expenditures was strongly correlated with graduation rates – 1.77 percent increase in graduation rates
• Greatest payoff is attributable to enhanced library expenditures (+0.92) and instruction (+0.80) while increased non-library contributions were quite modest (+0.27)
Hamrick, Schuh & Shelley
Library Retention Studies
• Relationships between library use (collections) and student retention
– Student who borrowed books = more likely to persist
• E.g. Kramer & Kramer, 1968
• Impact of instruction
– Students involved in library skills programs showed lower attrition rates
• E.g. Knapp, 1966
Library Retention Studies
University of Minnesota
– 77% of undergrads made use of the libraries, 85% of grad students made use of the libraries
– Students who used the library at least once were 1.54 times more likely to re-enroll
Library Retention Studies
• Some library involvement in first year experience programs; specific programs for “at risk” groups
– NOT proven to have significant effect
• E.g. Hollis, 2001 Colton, et al, 2002 Aguilar & Keating, 2009 Love, 2009
Library Retention Studies
• Relationship between library employment & retention
– Higher completion rate among library student workers
• E.g. Wilder, 1990Rushing & Poole, 2002
“If strong linkages between libraries and student retention can be made,
then the perceived value of the library may indeed rise.”
Steven Bell
• Some groups, some majors & seniors engage in more library-related activities
• Academic support expenditures tend to correlate with increased engagement
• Institutional academic challenge correlates with library use
Student Career SuccessGrad School Exams
Time to
First
Job
Average Salary of First
Job
Alumni Surveys
Alumni Giving
• Job placement rates• First-year job salaries• Professional/graduate school
acceptance• Internship success• Marketable skills
Student Success
• Integration of library resources and services into course syllabi, Websites, lectures, labs, reserve readings, etc.
• Faculty/librarian collaborations; cooperative curriculum, assignment, or assessment design
Faculty Teaching
Perceived Benefits for Teaching
• Savings– Of own time– Of own money– Of other resources
• Improvements– Teaching– Course-related materials– Student performance
Library Value Wheel
Impact on Faculty
• Library is the source for most journal articles (individual subscriptions are way down)
• If library subscriptions were unavailable – productivity would decrease 17%
• Library is not the source of book readings
• 42% of reading material is library provided
Time
• Academics spend a lot of time reading
• Article reading inspires new thinking, improved results, changed focus
• Award-winning academics read more
• Academics who publish more use more library resources
Ithaka Studies
• Library services not understood
• Library services not valued
• The Library is disappearing
• Number of publications, number of patents, value of technology transfer
• Tenure/promotion judgments
Faculty Research Productivity
Faculty Grants
• Number of grant proposals (funded or unfunded)
• Value of grants funded
Assessment of Research
• Payback model – form of ROI• Research impact• Research utilization ladder• Lavis decision-making impact model• Weiss logic model• HTA organization assessment
framework• Societal impact framework• Research assessment exercise• Becker medical library model
For Most Impact Models
• Indicators of research output
• Indicators of knowledge transfer
• Indicators of implementation
• Indicators of community benefit
Faculty Productivity
• Faculty recruitment• Institutional rankings• Community engagement
Institutional Reputation & Prestige
Institutional Reputation
• Changes in reputational rankings affects student & faculty recruitment
• University budget allocations to libraries have decreased
Since the library absorbs a very small percentage of a university budget, the contribution of the library is disproportionately high relative to its cost to the institution.
Sharon Weiner
• Indispensable for their research
• Maintain a high-level overview of their field
• Value for money is good
• Library not available, costs would increase 40%
• Take 31% longer to locate same information
Library Value Scorecard
• Relational Capital
• Library Capital
• Library Virtue
• Library Momentum
University & the Library Can
• Attract outstanding faculty
• Retain outstanding faculty
• Foster innovative research
• Align library activities with university goals
Value of Special Libraries
• Time saved
• Money saved
• New revenues
• Other outcomes
In addition, to ROI
• Knowledge-Value Add
• Intranet Team Forums
• Intellectual Capital Valuation
Measurements of valuewere, in fact, a
key differentiator between
successfuland unsuccessful corporate libraries
James Matarazzo
Value of the
(Rooney-Browne, 2009b).
Social Capital
• Bonding social capital
• Bridging social capital
• Linking social capital
Social Benefits
• Basic reading literacy
• Business/career
• Information literacy
• Library as place
• Summer reading
• Local history & genealogy
• Health & well-being
• Social cohesion
• General information
• Empowerment
• Social connection• Education• Employment• Health & wellness• eGovernment• Community engagement• Personal finance
Making Cities Stronger
• Building a stronger local economy
• Improving early literacy & school readiness
• Building workforce participation
• Supporting small business
• The power of place
• Literacy
• Workforce development
• Business development
• Value to homes and neighborhoods
Libraries/Building/Communities
• Developing social capital– Providing a welcoming environment– Creating a pride of place– Attracting users from all walks of life– Reaching out to the community– Appreciation of cultural differences– Building bridges to government– Encouraging collaboration across the
community
Libraries/Building/Communities
• Overcoming the digital divide– Making technology accessible– Exploiting technology to benefit the
community
• Creating informed communities– Community information– Government information– Providing a gateway to the world of
information
Libraries/Building/Communities
• Convenient and comfortable places of learning– Developing information skills– Stimulating ideas and discussion– Supporting vulnerable learners– Supporting students
Outcomes
Enhanced quality of life
Enhanced enjoyment from hobbies
Able to obtain information
Facilitates lifelong learning
Support for children’s education
Contributions
Safe and pleasant place
Supporting educational facilities
Facilitating lifelong learning
Encouraging responsible social behavior
Ensuring access to the Internet
Tracking ValueThe Engaged Library: Chicago Stories of Community Building
•Prove that public libraries build social capital
•Identify & connect the library’s assets to the community
•Assess & strengthen the library’s connections with and use of community assets
•Produce a toolkit for other libraries to adopt to
•Mapping tools to perform an inventory services, identify areas for improvement and highlight library’s contribution to the community’s wider social, educational, cultural and economic goals.
PLQIM
• Access to information• Community & personal
participation• Meeting readers’ needs• Learners’ experiences• Ethos & values• Organization & use of resources• Leadership
Valuing the Collection
Dewey SubclassNumber of Titles
2010-2011 Avg List Total Value ($)
001 - Knowledge $76.71 $0.00
002 - The book $62.45 $0.00
003 - Systems $129.77 $0.00
004 - Data processing. Computer science $89.82 $0.00
005 - Computer programming, programs, data $69.14 $0.00
006 - Special computer methods $83.60 $0.00
010 - Bibliography $73.65 $0.00
011 - Bibliographies $69.46 $0.00
012 - Bibliographies of individuals $0.00 $0.00
013 - Of works by specific classes of authors $0.00 $0.00
014 - Of anonymous and pseudonymous works $55.95 $0.00
015 - Of works from specific places $184.99 $0.00
016 - Of works on specific subjects $134.87 $0.00
017 - General subject catalogs $0.00 $0.00
018 - Catalogs arranged by author, date, etc. $0.00 $0.00
019 - Dictionary catalogs $0.00 $0.00
020 - Library and information sciences $56.06 $0.00
021 - Library relationships $62.60 $0.00
022 - Administration of the physical plant $65.50 $0.00
023 - Personnel administration $56.00 $0.00
025 - Library operations $77.02 $0.00
026 - Libraries for specific subjects $89.99 $0.00
027 - General libraries $66.25 $0.00
www.joematthews.org
Joe MatthewsLibrary Consultant
What Are the Results
Four Year CollegeFull-time
Four Year CollegePart-time
Two Year College
Stopout/Transfer
College degree – 19%
AA degree – 8%
No college degree – 60%
No college
Graduate degree – 11%