Psy 407 Validity

8/13/2019 Psy 407 Validity

1/43


2/43

Determining Validity

There are several ways to measure validity. The mostcommonly addressed include:

- Face Validity

- Construct & Content Validity

- Convergent & Divergent Validity

- Predictive Validity

- Discriminant Validity


3/43

Validity

Refers to measuring what we intend to measure.

If math and vocabulary truly represent intelligence then

a math and vocabulary test might be said to have high

validity when used as a measure of intelligence.


4/43

Face Validity Is the extent to which it is self-evident that a scale is measuring

what it is suppose to measure.For Example - you might look at a measure of math ability, read through

the questions, and decide that yep, it seems like this is a good measure of

math ability

It would clearly be weak evidence because it is essentially a

subjective judgment call. Just because it is weak evidence doesn't

mean that it is wrong. We need to rely on our subjective judgment

throughout the research process.

For example- suppose you were taking an instrument reportedly

measuring your attractiveness, but the questions were asking you to

identify the correctly spelled word in each list. Not much of a link

between the claim of what it is supposed to do and what it actuallydoes.


5/43


6/43

Face Validity (cont.)

In all cases, face validity is not based on empirical research

evidence.

When used alone, face validity provides very weak support for the

overall validity of a scale.


7/43

Face Validity (cont.)

Possible Advantage of face validity... If the respondent trusts the test and believes it is

measuring what it should be measuring, they may

provide more useful, thoughtful, and accurate answers. E.g., if a nursing test seems to measure constructs not

associated with nursing, the respondent may become

frustrated and answer poorly or without much thought


8/43

Possible Disadvantage of face validity...

If the respondent knows what information weare looking for, they might try to bend &

shape their answers to what they think wewant.

i.e., fake good or fake bad


9/43

Content Validity

Does the test contain items from the desired content

domain?

Based on assessment by experts in that content domain.

Is especially important when a test is designed to have

low face validity.

Is generally simpler for tests of ability than for

psychological constructs to demonstrate content

validity

.


10/43

For Example- Easier for math experts to agreeon an item for an algebra test than it is forpsych experts to agree whether or not an item

should be placed in a personality measure


11/43

Content Validity (cont.)

Basic Procedure for Assessing Content Validity:

1. Describe the content domain

2. Determine the areas of the content domain that aremeasured by each test item

For Example - In developing a nursing licensure exam, experts on

the field of nursing would identify the information and issues

required to be an effective nurse and then choose (or rate) items

that represent those areas of information and skills.


12/43

Content Validity (cont.)

Lawshe (1975) proposed that each rater should respond to

the following question for each item in content validity:

Is the skill or knowledge measured by this item

1. Essential

2. Useful but not essential

3. Not necessary

We need to have a high proportion of essential

responses from the raters


13/43

Construct Validity

Construct Validity basically refers to the general validity

of the measurement tool.

Does the instrument measure the construct that it is

intended to measure?

There is no statistical test that will provide an absolute

measure for construct validity. Therefore, construct

validity is never proven, it can only be supported.


14/43

A) Assess how well the test predicts the relevantconstruct

Assume that we have developed theJohnson

Inventory of Aggression (JAS) A person who scores high on the JAS should

be more likely to engage in aggressive behaviorin any situation relative to those who score lowon the JAS


15/43

B) Compare the new measure to an existing, valid measure (i .e, the

JAS should correlate highly with an existing valid measure of

aggression) .

Although a valid measure will exist but a new scale is being

created that will have some advantage over the older measure.

Advantages of new measure A) more consistent with current theory B) shorter and more accurate

-


16/43

Sometimes existing valid measures dontexist. That is often why the new scale isbeing created in the first place.


17/43


18/43

Tyson (1997) contends that there is a clearracial divide between Blacks and Whites intheir perceptions of the prevalence of

discrimination in America Crocker (1999) found that when compared to

White Americans, Black Americans (across allsocioeconomic levels) were much more likely

to believe in conspiracy theories that suggestthat the government engaged in organizedefforts to harm Blacks


19/43

Monteith & Spicer (2000)showed thatBlack negative attitudes towards

Whites generally reflected reactionsto perceived racism (eg., Theyalways have negative thoughts about

Blacks)


20/43

A) Media and experiences


21/43

83% of Blacks have experienced racism in their day-to-daylife and 68% experiencing racism during healthcare.(Peters, 2006)

70% of Blacks reported being treated unfairly by strangers,34% reported unfair treatment by helping professions, and31% were called derogatory names connected to race.(Pieterse & Carter, 2007).

67% of Black participants felt that their life would havebeen different over the past year if they had notexperienced these racist events


22/43

However, despite popular opinions,Blacks are not a monolithic group!

There is variation among Blacks regarding their

racism expectations involving Whites


23/43

In his book Country of Strangers,Shiplersuggests that Blacks in America range fromthose who tend to see racism in everyencounter with a white to those who trymightily not to see it all (p. 448).

Yet there was no direct measureof variation in Black beliefs inWhite racism

To address the limitations in the racism studies


24/43

To address the limitations in the racism studiesliterature,Johnson & Lecci (2003) developedthe Johnson-Lecci Scale of Black antiWhite bias

This scale was based on responses and lifeexperiences of approximately 450 Black collegestudents

The scale had 20 items and four subscales

A major subscale was the ingroupstigmatization and discrimination expectation(I.e., the expectation that the typical White personwill discriminate against Blacks)


25/43

***All questions were answered on a four point scale (1-strongly disagree, 4-strongly agree)

I believe that most whites really do support the ideas andthoughts of racist political groups

I believe that most whites really believe that blacks aregenetically inferior.

I believe that most whites would discriminate against blacks, ifthey could get away with it

I believe that most of the negative actions of whites towards

blacks are due to racist feelings I believe that most whites would harm blacks if they could get

away with it.

I believe that most whites think that they are superior to blacks.


26/43

However, scales are useless if they dontpredict!


27/43

The JLS has been shown to predict:

A) the probability of perceiving racism in anambiguously racist scenario (Johnson & Lecci, 2003) Restaurant example, airline choice

B) the number of a Black persons White friends

(Johnson & Lecci, 2003) C) the probability that a Black person will confront a

racist person (Johnson et al., 2006)

D) racial preferences regarding a mental health

counselor (Ferguson et al., 2008) E) the probability that of prosocial responses

towards a White person-in-need (Johnson et al.,2008)

F) bias against lighter skinned Blacks who were inneed ohnson et al., under review


28/43

What would be a comparable measure inPacific Island/Fiji culture?

What might be some of the items on the scale?


29/43

Discriminant Validity The statistical assessment of Construct Validity

Does the instrument show the right pattern of interrelationships

with other instruments.

Discriminant Validity has two parts:

Convergent Validity

Divergent Validity


30/43

Convergent & Divergent Validity

Convergent Validity: the extent to which the scale

correlates with measures of the same or related concepts.

e.g.,

A new scale to measure Assertiveness should

correlate with existing measures of Assertiveness, andwith existing measures of related concepts like

Independence.


31/43

Divergent Validity: the extent to which does notcorrelate with measures of unrelated or distinctconcepts.

e.g., An assertiveness scale shouldnot correlate with measures of aggressiveness.


32/43

This type of validity measures the relationship between thepredictor and the criterion, and the accuracy with which the

predictor is able to predict performance on the criterion.

What is the actual association between a test to predictjob

performance and actualjob performance

If the test has high criterion validity, then it should be close

Criterion-Related Validity (cont.)


33/43

HOW THIS TYPE IS ESTABLISHED:Criterion-related validity can be either concurrent or

predictive. The distinguishing factor is the time when criterion

and predictor data are collected.

Concurrent- criterion data are collected before or at the same

time that the predictor is administered.

Predictive- criterion data are collected after the predictor is

administered.

Criterion-Related Validity (cont.)


34/43

Concurrent Validity This type of validity indicates the correlation between the

predictor and criterion when data on both were collected ataround the same time.

Is used to determine a persons current status.

For ExampleIf a person seems depressed, they should score fairly

high on a depression inventory given the same day


35/43

Predictive Validity This type of validity also indicates the correlation between

the predictor (X) and the criterion (Y). However, criteriondata are collected after predictor data are obtained. In other

words, this method determines the degree, that X can

accurately predict Y

For Example - giving high school juniors the ACT test for

admission to a university.

The test is the predictor and first semester grades in college are

the criterion. If the correlation is large, this means the ACT

is useful for predicting future grades.


36/43

Predictive Validity (cont.)

The extent to which scores on the scale are related to,and predictive of, some future outcome that is of

practical utility.

e.g., If higher scores on the SAT are positively correlated with

higher G.P.A.s and visa versa, then the SAT is said to

have predictive validity.

The Predictive Validity of the SAT is mildly supported by

the relation of that scale with performance in graduate

school.


37/43

Predictive Validity

A predictive validity study consists of two basic steps:

1.Obtain test scores from a group of respondents, but do not

use the test in making a decision.

2.At some later time, obtain a performance measure for those

respondents, and correlate these measures with test scores to

obtain predictive validity.

A Test Must Be Valid -


38/43

When evaluating test to real-life predictions, even very

modest correlations of r= .02 or .03 can be of

considerable importance. For example, the impact of

chemotherapy on breast cancer survival is r= .03.

In selection, hiring, and counseling contexts, current

interpretations suggest that correlations as low as r= .02

or .03 are meaningful, with many psychological (andmedical test) assessments and real life criteria falling in

the r = .10 to .30 level, and a few rising beyond that level.

A Test Must Be Valid -

Predictive Validity


39/43

Resilience-----an ability to recover from oradjust easily to misfortuneor change

Sh ld i t lli b di ti f th
http://www.merriam-webster.com/dictionary/misfortunehttp://www.merriam-webster.com/dictionary/misfortune


40/43

Should intelligence be predictive of theprobability of joining a cult?

More importance should a measure of

resilience be predictive of (i.e., correlate with)joining a cult?


41/43

FACTORS THAT INFLUENCE VALIDITY

Inadequate sample

Items that do not function as intended

Improper arrangement/unclear directionsToo few items for interpretation

Improper test administration

Scoring that is subjective


42/43

RELATIONSHIP BETWEEN

RELIABILITY AND VALIDITY

Reliability means

nothing when theproblem is Validity.

Reliability caps

validity


43/43

Reliability and validity are two different standards used to gage the

usefulness of a test. Though different, they work together.

It would not be beneficial to design a test with good reliability that

did not measure what it was intended to measure. The inverse,

accurately measuring what you desire to measure with a test that isso flawed that results are not reproducible, is impossible.

Reliability is a necessary requirement for validity. This means that

you have to have good reliability in order to have validity. Reliability

actually puts a cap or limit on validity, and if a test is not reliable, it

can not be valid.

- from the MSCEIT Manual

Relationship Between Reliability & Validity

Documents

Psy 407 Validity