S519: Evaluation of Information Systems

Analyzing data:

Rank

Ch9-p171

Last week

What is synthesis methodology?, why do we need that?

What is synthesis for grading? What is the rubric for dimensions What is the rubric for components How to merge all the conclusions to get the final

grade?

Synthesizing for „ranking“

What are „ranking“ evaluations? Examples? Difference comparing with „grading“ evaluation?

Qualitative and quantitative

Qualitative Qualitative weight and sum (QWS)

Quantitative Numerical weight and sum (NWS)

Numerical Weight and Sum (NWS)

It is a quantitative synthesis method for summing evaluand performance across multiple criteria.

It includes Assign numerical importance weight and a numerical

performance score to each criteria (dimention) Mutliply weights by performance scores Sum these products The summing result represents the overall merit of the

evaluand

Numerical Weight and Sum (NWS)

It fits for There are only a small number of criteria (why) There is some other mechanism for taking bars into

account (why) There is defensible needs-based strategy for ascribing

weights.

Training program evaluation

A comparative evaluation on three different interventions for training managers A mountain retreat featuring interactive sessions with multiple

world-class management gurus An in-house training and mentoring program run by human

resources, A set of videos and latest book on management from

managment guru Peter Drucker

Training program evaluation

Needs assessment for this evaluation Bear in mind that this is a comparison evaluation How do you want to compare these programs, what are the key

features of the programs

Identify the dimension of merit (Process, Outcomes and Cost)

Decide the importance of the merit (giving weights to merits, based on needs?)

See Table 9.8

Training program evaluation

Next steps Data collection (what are your experiences for your

project data collection?) Data analysis

Rate their performance based on pre-defined ratings: excellent, very good, good, fair, or poor) (see Table 9.9 for this example)

Convert weights into numbers (see Table 9.10) Convert ratings into numbers (see Table 9.10)

Synthesis step (how? See Table 9.11)

How to interpret Table 9.11

Exercise

Do it by your own hand: Converting Table9.9 to Table9.10 (defining your own numeric

value for importance and grading scales) and try to find out which program is the best comparing with others.

If suddently, the cost criterias become extremely important, will this change the final result? Work on your own Form the pair and discussion

Pros and cons for NWS?

Qualitative Weight and Sum (QWS)

It is non-numerical synthesis methodology for summing the performances of an evaluand on multiple criteria to determine overall merit.

It is a ranking method for determining the relative merit of two or more evaluands

It is not suitable for grading It fits for

Personnel selection, products/service/proposal selection

QWS

Step1: Determine importance in terms of maximum possible value How (see Chapter 7, six strategies) Table 9.12 (compare with Table 9.8)

Step2: Set bars Bar is the cut point between acceptable and

unacceptable criteria. Such as: Too expensive to afford Too long away from their work

QWS

Step3: Create value determination rubrics Rubrics are level-based (see Chapter 8)

Description on each level, how to deal with bar? Unacceptableno noticeable valuemarginally

valuablevaluableextremely valuable Such as what performance would look like at each level

Each dimension can have its own rubrics or each group of dimensions can have their own rubrics

Each group of questions can have their own rubrics Synthesis step can have its own rubrics

Example: Rubric for rating finanical cost of training (see table 9.14) How to set up similar rubric for Table 9.12 on extremely valuable

criteria (see Table 9.15)

QWS

Step4: Check equivalence of value levels across dimensions The validity of the QWS method is highly

dependent on ensuring the rough equivalence on the value levels defined for each dimension For example, whether table 9.14 and table 9.15 have

the roughly equivalent value levels How to do that? Put them into a matrix.

See table 9.16

QWS

Step5: rate value of actual performance on each dimension Rating table 9.9 according to rubric (table9.16) See Table 9.17

Step6: tally the number of ratings at each level and look for a clear winner For each program, how many symbols they got? Throw out programs with unacceptable ratings, see

whether there is a clear winner?

QWS

Step7: refocus Delete the rows with similar score (see table9.18) Count how many symbols each of them got? Can we find the clear winner?

Yes or no? Why? How should we go further?

Exercise Form a group to work on this Library wants to subscribe to one of the three magazines

and ask you to conduct an evaluation and propose the best solution: Journal of the American Society for Information Science and

Technology Journal of Information Science Scientometrics (or choose some magezines you are familiar with)

List the criteria you think are important Following the steps of NWS and QWS and tell me your

findings and show me the justification of your findings.