Measuring Scholarly and Public Impact Letrsquos Talk Metrics

RACHEL BORCHARDT MONDAY MAY 16TH 2017

BORCHARDAMERICANEDU CONFERENCE FOR HIGH-IMPACT RESEARCH

Why metrics

Quantifiable measures of research impact

Differentiate articles journals authors institutions

Imperfect calculations of a fuzzy concept

How are metrics used

Evaluate researchers

Hiring

Tenurepromotion

Merit awards

Grant funding

Benchmark programs and institutions

Image from AU Faculty Manual httpwwwamericaneduloadercfmcsModule=securitygetfileamppageid=4304081

What are metrics

Traditional Bibliometrics

Based on citation counts

Emerging Altmetrics

Based on online tools

Image from Plum Analytics

Categories of bibliometrics

Article-level

Journal-level

Author-level

Institutional-level

Traditional bibliometrics for journals

Impact Factor and Citation Counts created to measure

Journals and journal articles

Scholarly (journal) impact

Initially created for librarians then largely adopted by STEM

Image from Journal Citation Reports (library database)

Some Impact Factor limitations

Retrospective measurement

Narrow measurement

Confined by what is indexed as a citation

Canrsquot be compared between disciplines

Other known issues language co-authorship disproportionate article citation impact

Image from Essential Science Indicators (library database)

One way to correct ndash disciplinary context

Image from Journal Citation Reports (library database)

Contextual limitations ndash ldquoPolitical Sciencerdquo category

List from httpwwwauspsaorgaupublications

Other major bibliometrics Metrics from Web of Science Eigenfactor Immediacy Index Cited Half-Life

Metrics from ElsevierScopus SJR SNIP IPP CiteScore

Metrics from Google Scholar H5-index H5-median

Image from httpwwwjournalmetricscom

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

How are metrics used

Evaluate researchers

Hiring

Tenurepromotion

Merit awards

Grant funding

Benchmark programs and institutions

Image from AU Faculty Manual httpwwwamericaneduloadercfmcsModule=securitygetfileamppageid=4304081

What are metrics

Traditional Bibliometrics

Based on citation counts

Emerging Altmetrics

Based on online tools

Image from Plum Analytics

Categories of bibliometrics

Article-level

Journal-level

Author-level

Institutional-level

Traditional bibliometrics for journals

Impact Factor and Citation Counts created to measure

Journals and journal articles

Scholarly (journal) impact

Initially created for librarians then largely adopted by STEM

Image from Journal Citation Reports (library database)

Some Impact Factor limitations

Retrospective measurement

Narrow measurement

Confined by what is indexed as a citation

Canrsquot be compared between disciplines

Other known issues language co-authorship disproportionate article citation impact

Image from Essential Science Indicators (library database)

One way to correct ndash disciplinary context

Image from Journal Citation Reports (library database)

Contextual limitations ndash ldquoPolitical Sciencerdquo category

List from httpwwwauspsaorgaupublications

Other major bibliometrics Metrics from Web of Science Eigenfactor Immediacy Index Cited Half-Life

Metrics from ElsevierScopus SJR SNIP IPP CiteScore

Metrics from Google Scholar H5-index H5-median

Image from httpwwwjournalmetricscom

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

What are metrics

Traditional Bibliometrics

Based on citation counts

Emerging Altmetrics

Based on online tools

Image from Plum Analytics

Categories of bibliometrics

Article-level

Journal-level

Author-level

Institutional-level

Traditional bibliometrics for journals

Impact Factor and Citation Counts created to measure

Journals and journal articles

Scholarly (journal) impact

Initially created for librarians then largely adopted by STEM

Image from Journal Citation Reports (library database)

Some Impact Factor limitations

Retrospective measurement

Narrow measurement

Confined by what is indexed as a citation

Canrsquot be compared between disciplines

Other known issues language co-authorship disproportionate article citation impact

Image from Essential Science Indicators (library database)

One way to correct ndash disciplinary context

Image from Journal Citation Reports (library database)

Contextual limitations ndash ldquoPolitical Sciencerdquo category

List from httpwwwauspsaorgaupublications

Other major bibliometrics Metrics from Web of Science Eigenfactor Immediacy Index Cited Half-Life

Metrics from ElsevierScopus SJR SNIP IPP CiteScore

Metrics from Google Scholar H5-index H5-median

Image from httpwwwjournalmetricscom

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

Categories of bibliometrics

Article-level

Journal-level

Author-level

Institutional-level

Traditional bibliometrics for journals

Impact Factor and Citation Counts created to measure

Journals and journal articles

Scholarly (journal) impact

Initially created for librarians then largely adopted by STEM

Image from Journal Citation Reports (library database)

Some Impact Factor limitations

Retrospective measurement

Narrow measurement

Confined by what is indexed as a citation

Canrsquot be compared between disciplines

Other known issues language co-authorship disproportionate article citation impact

Image from Essential Science Indicators (library database)

One way to correct ndash disciplinary context

Image from Journal Citation Reports (library database)

Contextual limitations ndash ldquoPolitical Sciencerdquo category

List from httpwwwauspsaorgaupublications

Other major bibliometrics Metrics from Web of Science Eigenfactor Immediacy Index Cited Half-Life

Metrics from ElsevierScopus SJR SNIP IPP CiteScore

Metrics from Google Scholar H5-index H5-median

Image from httpwwwjournalmetricscom

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

Traditional bibliometrics for journals

Impact Factor and Citation Counts created to measure

Journals and journal articles

Scholarly (journal) impact

Initially created for librarians then largely adopted by STEM

Image from Journal Citation Reports (library database)

Some Impact Factor limitations

Retrospective measurement

Narrow measurement

Confined by what is indexed as a citation

Canrsquot be compared between disciplines

Other known issues language co-authorship disproportionate article citation impact

Image from Essential Science Indicators (library database)

One way to correct ndash disciplinary context

Image from Journal Citation Reports (library database)

Contextual limitations ndash ldquoPolitical Sciencerdquo category

List from httpwwwauspsaorgaupublications

Other major bibliometrics Metrics from Web of Science Eigenfactor Immediacy Index Cited Half-Life

Metrics from ElsevierScopus SJR SNIP IPP CiteScore

Metrics from Google Scholar H5-index H5-median

Image from httpwwwjournalmetricscom

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

Some Impact Factor limitations

Retrospective measurement

Narrow measurement

Confined by what is indexed as a citation

Canrsquot be compared between disciplines

Other known issues language co-authorship disproportionate article citation impact

Image from Essential Science Indicators (library database)

One way to correct ndash disciplinary context

Image from Journal Citation Reports (library database)

Contextual limitations ndash ldquoPolitical Sciencerdquo category

List from httpwwwauspsaorgaupublications

Other major bibliometrics Metrics from Web of Science Eigenfactor Immediacy Index Cited Half-Life

Metrics from ElsevierScopus SJR SNIP IPP CiteScore

Metrics from Google Scholar H5-index H5-median

Image from httpwwwjournalmetricscom

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

One way to correct ndash disciplinary context

Image from Journal Citation Reports (library database)

Contextual limitations ndash ldquoPolitical Sciencerdquo category

List from httpwwwauspsaorgaupublications

Other major bibliometrics Metrics from Web of Science Eigenfactor Immediacy Index Cited Half-Life

Metrics from ElsevierScopus SJR SNIP IPP CiteScore

Metrics from Google Scholar H5-index H5-median

Image from httpwwwjournalmetricscom

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

Contextual limitations ndash ldquoPolitical Sciencerdquo category

List from httpwwwauspsaorgaupublications

Other major bibliometrics Metrics from Web of Science Eigenfactor Immediacy Index Cited Half-Life

Metrics from ElsevierScopus SJR SNIP IPP CiteScore

Metrics from Google Scholar H5-index H5-median

Image from httpwwwjournalmetricscom

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

Other major bibliometrics Metrics from Web of Science Eigenfactor Immediacy Index Cited Half-Life

Metrics from ElsevierScopus SJR SNIP IPP CiteScore

Metrics from Google Scholar H5-index H5-median

Image from httpwwwjournalmetricscom

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

SJR comparison ndash ldquoPolitical Science and International Relationsrdquo

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

H5-index comparison ndash Political Science

List from Google Scholar Metrics

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

Another alternative ndash citation distributions

httpwwwnaturecomnewsbeat-it-impact-factor-publishing-elite-turns-against-controversial-metric-120224

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

Why altmetrics

Measures broader scholarly outputs

Books presentations data sets software code and more

More measures of scholarship

Views downloads shares mentions in blogs policy papers and more

More immediate metrics

Measures beyond scholarly impact

Policy public and more

httpsbeckerwustleduimpact-assessment

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

What are altmetrics

Definition The creation and study of new metrics based on the social web for analyzing and informing scholarship

Altmetricsorg

My definition The umbrella classification of non-citation based metrics

httpaltmetricsorgmanifesto httptwittercomjasonpriem

Categories of altmetrics

Article-level [individual work]

Journal-level

Author-level

Institutional-level

Where do altmetrics come from

How do we collect altmetrics

Directly from the individual tools

From publishers (views download data)

From (some) library databases

From scholarly networks

Through aggregating tools

Slideshare views

PLOS article metrics Web of Science usage ResearchGate metrics Altmetric metrics

ldquoImpactrdquo and metrics

Inherently fuzzy and subjective

Attention

Engagement

Some correlate to bibliometrics some donrsquot

Deciphering impact - ImpactStory

ImpactStory contextual elements

Deciphering impact from tools - Altmetric

Altmetric contextual elements

httpswwwaltmetriccomdetails585373

Impact correlations

httpsarxivorgabs12034745 httponlinelibrarywileycomdoi101002asi23309full

Where do altmetrics come from

How do we collect altmetrics

Directly from the individual tools

From publishers (views download data)

From (some) library databases

From scholarly networks

Through aggregating tools

Slideshare views

PLOS article metrics Web of Science usage ResearchGate metrics Altmetric metrics

ldquoImpactrdquo and metrics

Inherently fuzzy and subjective

Attention

Engagement

Some correlate to bibliometrics some donrsquot

Deciphering impact - ImpactStory

ImpactStory contextual elements

Deciphering impact from tools - Altmetric

Altmetric contextual elements

httpswwwaltmetriccomdetails585373

Impact correlations

httpsarxivorgabs12034745 httponlinelibrarywileycomdoi101002asi23309full

How do we collect altmetrics

Directly from the individual tools

From publishers (views download data)

From (some) library databases

From scholarly networks

Through aggregating tools

Slideshare views

PLOS article metrics Web of Science usage ResearchGate metrics Altmetric metrics

ldquoImpactrdquo and metrics

Inherently fuzzy and subjective

Attention

Engagement

Some correlate to bibliometrics some donrsquot

Deciphering impact - ImpactStory

ImpactStory contextual elements

Deciphering impact from tools - Altmetric

Altmetric contextual elements

httpswwwaltmetriccomdetails585373

Impact correlations

httpsarxivorgabs12034745 httponlinelibrarywileycomdoi101002asi23309full

ldquoImpactrdquo and metrics

Inherently fuzzy and subjective

Attention

Engagement

Some correlate to bibliometrics some donrsquot

Deciphering impact - ImpactStory

ImpactStory contextual elements

Deciphering impact from tools - Altmetric

Altmetric contextual elements

httpswwwaltmetriccomdetails585373

Impact correlations

httpsarxivorgabs12034745 httponlinelibrarywileycomdoi101002asi23309full

Deciphering impact - ImpactStory

ImpactStory contextual elements

Deciphering impact from tools - Altmetric

Altmetric contextual elements

httpswwwaltmetriccomdetails585373

Impact correlations

httpsarxivorgabs12034745 httponlinelibrarywileycomdoi101002asi23309full

ImpactStory contextual elements

Deciphering impact from tools - Altmetric

Altmetric contextual elements

httpswwwaltmetriccomdetails585373

Impact correlations

httpsarxivorgabs12034745 httponlinelibrarywileycomdoi101002asi23309full

Deciphering impact from tools - Altmetric

Altmetric contextual elements

httpswwwaltmetriccomdetails585373

Impact correlations

httpsarxivorgabs12034745 httponlinelibrarywileycomdoi101002asi23309full

Altmetric contextual elements

httpswwwaltmetriccomdetails585373

Impact correlations

httpsarxivorgabs12034745 httponlinelibrarywileycomdoi101002asi23309full

Impact correlations

httpsarxivorgabs12034745 httponlinelibrarywileycomdoi101002asi23309full

Discipline-specific impact

httpswwwresearchtrendscomissue-30-september-2012the-use-of-big-datasets-in-bibliometric-research

Limitations of impact example

httpswwwaltmetriccomdetails2313862policy-documents

httpswwwhealthypeoplegov2020topics-objectivestopicAdolescent-Health

httpswwwgooglecomsearchq=22Trends+in+adolescent+and+young+adult+health+in+the+United+States22+site3Agovampoq=22Trends+in+adolescent+and+young+adult+health+in+the+United+States22+site3Agovampaqs=chrome69i573775j0j8ampsourceid=chromeampie=UTF-8

Limitations of impact example

httpswwwaltmetriccomdetails2313862policy-documents

httpswwwhealthypeoplegov2020topics-objectivestopicAdolescent-Health

httpswwwgooglecomsearchq=22Trends+in+adolescent+and+young+adult+health+in+the+United+States22+site3Agovampoq=22Trends+in+adolescent+and+young+adult+health+in+the+United+States22+site3Agovampaqs=chrome69i573775j0j8ampsourceid=chromeampie=UTF-8

Current uses of altmetrics

External

Applying for funding promotiontenure

Showcasing achievements

Internal

Better understand engagement with research

Who why where

More on grant funders

httpswwwnaturecomnaturejobssciencearticles101038nj7471-397a

Grant funders NSF

Broader Impacts

httpswwwnsfgovodoiaspecialbroaderimpacts

More on grant funders

httpswwwnaturecomnaturejobssciencearticles101038nj7471-397a

Grant funders NSF

Broader Impacts

httpswwwnsfgovodoiaspecialbroaderimpacts

Grant funders NSF

Broader Impacts

httpswwwnsfgovodoiaspecialbroaderimpacts

Grant funders Wellcome Trust

httpwwwhefceacukpubsrereportsyear2015metrictide

Putting it all together - suggestions

Develop personal impact goals

Discipline-specific research-specific university-specific etc

Demonstrate achievement of those goals

Tell your story

Bibliometrics altmetrics and qualitative measures

Complementary roles

Track early and often

Getting help

Rachel Borchardt

borchardamericanedu

202-885-3657

(I work year-round)

Metrics guide httptinyurlcomAUmetrics

Borrow a copy of my book

Putting it all together - suggestions

Develop personal impact goals

Discipline-specific research-specific university-specific etc

Demonstrate achievement of those goals

Tell your story

Bibliometrics altmetrics and qualitative measures

Complementary roles

Track early and often

Getting help

Rachel Borchardt

borchardamericanedu

202-885-3657

(I work year-round)

Metrics guide httptinyurlcomAUmetrics

Borrow a copy of my book

Getting help

Rachel Borchardt

borchardamericanedu

202-885-3657

(I work year-round)

Metrics guide httptinyurlcomAUmetrics

Borrow a copy of my book

Questions