Upload
pemea2008
View
220
Download
0
Embed Size (px)
Citation preview
8/9/2019 Pedrajita, J.
1/48
Item Bias Detection ModelsItem Bias Detection Models
for Test Reliabilityfor Test Reliabilityand Validityand Validity
JOSE Q. PEDRAJITA, PhDJOSE Q. PEDRAJITA, PhDCollege of EducationCollege of Education
University of the PhilippinesUniversity of the Philippines
8/9/2019 Pedrajita, J.
2/48
TABLE OF CONTENTSTABLE OF CONTENTS
Background of the StudyBackground of the Study ObjectivesObjectives
Methodology and HypothesesMethodology and Hypotheses
ResultsResults ConclusionsConclusions
RecommendationsRecommendations
AcknowledgementAcknowledgement
8/9/2019 Pedrajita, J.
3/48
BACKGROUND OF THE STUDYBACKGROUND OF THE STUDY
One way to investigate potential bias at the item level is throughOne way to investigate potential bias at the item level is through differentialdifferentialitem functioningitem functioning (DIF) analysis.(DIF) analysis.
What is Differential Item Functioning (DIF) Analysis?What is Differential Item Functioning (DIF) Analysis?
-- is a means of statistically identifying unexpected differences inis a means of statistically identifying unexpected differences in
performance across matched groups of examinees. Itperformance across matched groups of examinees. Itcompares the performance of matched majority (orcompares the performance of matched majority (orreference) and minority (or focal) group examinees.reference) and minority (or focal) group examinees.
What is Differential Item Functioning (DIF)?What is Differential Item Functioning (DIF)?
-- Refers to the differing probabilities of success on an item ofRefers to the differing probabilities of success on an item ofexaminees of the same ability but belonging to differentexaminees of the same ability but belonging to differentgroups; that is, when examinees from different groups have agroups; that is, when examinees from different groups have adifferent probability or likelihood of answering an itemdifferent probability or likelihood of answering an itemcorrectly after controlling for overall test performance.correctly after controlling for overall test performance.
8/9/2019 Pedrajita, J.
4/48
BACKGROUND OF THE STUDYBACKGROUND OF THE STUDY
DIF is said to be present in a test:DIF is said to be present in a test: When despite controls for overall test performance,When despite controls for overall test performance,
examinees from different groups have a differentexaminees from different groups have a differentprobability of answering an item correctly or whenprobability of answering an item correctly or whenexaminees from two subpopulations with the same traitexaminees from two subpopulations with the same trait
level have different expected scores on the same itemlevel have different expected scores on the same item((CamilliCamilli andand ShepardShepard, 1994;, 1994; KamataKamata and Vaughn, 2004).and Vaughn, 2004).
When individuals having the same ability, but fromWhen individuals having the same ability, but from
different groups, do not have the same probability ofdifferent groups, do not have the same probability ofgetting the item right (getting the item right (HambletonHambleton,, SwaminathanSwaminathan, and, andRogers, 1991).Rogers, 1991).
8/9/2019 Pedrajita, J.
5/48
BACKGROUND OF THE STUDYBACKGROUND OF THE STUDY
An item that exhibits DIF may or may notbe biased foror againstAn item thatexhibitsDIF mayormaynotbe biased fororagainstanygroup(anygroup(KanjeeKanjee, 2007), 2007)
DIF maybe attributed toitem biasbutmayalsoreflectperformanceDIF maybe attributed toitembiasbutmayalsoreflectperformancedifferencesthatthe testisdesigned tomeasure (differencesthatthe testisdesigned tomeasure (CamilliCamilli andandShepardShepard, 1994)., 1994).
WhatisItemBias?WhatisItemBias?
refers to invalidity or systematic error in how a test itemrefers to invalidity or systematic error in how a test itemmeasures a construct for the members of a particular groupmeasures a construct for the members of a particular group((CamilliCamilli andand ShepardShepard, 1994). When a test item unfairly favors, 1994). When a test item unfairly favors
one group of examinees over another, the item is biased.one group of examinees over another, the item is biased.
8/9/2019 Pedrajita, J.
6/48
OBJECTIVESOBJECTIVES
This study isThis study is
designed to detect potentially biased test items anddesigned to detect potentially biased test items and
tests the effect of potentially biased itemstests the effect of potentially biased itemselimination on theelimination on the
contentcontent validityvalidity,,
concurrentconcurrent validityvalidity,, andand
internal consistencyinternal consistency reliabilityreliability
of an Achievement Test in Chemistry.of an Achievement Test in Chemistry.
8/9/2019 Pedrajita, J.
7/48
METHODOLOGYMETHODOLOGY
DIF ModelsDIF Models
ChiChi--Square (XSquare (X22)) This approach to the identification of test item bias examinesThis approach to the identification of test item bias examines
the likelihood or probability of test takers from differentthe likelihood or probability of test takers from differentgroups with the same ability levels correctly responding to angroups with the same ability levels correctly responding to anitem. An item is considered unbiased when all persons at aitem. An item is considered unbiased when all persons at a
given ability level have an equal probability of correctlygiven ability level have an equal probability of correctlyanswering an item regardless of their group membership.answering an item regardless of their group membership.With this strategy, the proportion of responses within abilityWith this strategy, the proportion of responses within abilitycategories for two groups diverse in some criterion iscategories for two groups diverse in some criterion isexamined.examined.
Hypothesis:Hypothesis:
There is no significant difference in proportions attaining aThere is no significant difference in proportions attaining acorrect response across total score categories on the testcorrect response across total score categories on the testitems between the public and private, the male and female,items between the public and private, the male and female,
and the low and high ability examineesand the low and high ability examinees ..
8/9/2019 Pedrajita, J.
8/48
METHODOLOGYMETHODOLOGYDIF ModelsDIF Models
DistractorDistractor Response Analysis (DRA)Response Analysis (DRA) The essential strategy of this technique is to examine theThe essential strategy of this technique is to examine the
incorrect alternatives to a test item for differences in patternsincorrect alternatives to a test item for differences in patternsof response among different subgroups of a population. Theof response among different subgroups of a population. Thefunction of distracter is to determine the significance of thefunction of distracter is to determine the significance of the
differences among two or more groups response frequenciesdifferences among two or more groups response frequenciesin the discrete categories of question distracters.in the discrete categories of question distracters.
Hypothesis:Hypothesis:
There is no significant difference in proportions selectingThere is no significant difference in proportions selecting
distracters on the test items between the public and private,distracters on the test items between the public and private,the male and female, and the low and high abilitythe male and female, and the low and high abilityexamineesexaminees..
8/9/2019 Pedrajita, J.
9/48
METHODOLOGYMETHODOLOGY
DIF ModelsDIF ModelsLogistic Regression (LR)Logistic Regression (LR)
A kind of regression analysis often used when the dependentA kind of regression analysis often used when the dependentvariable is dichotomous and scored 0 or 1. It can also bevariable is dichotomous and scored 0 or 1. It can also beused when the dependent variable has more than twoused when the dependent variable has more than two
categories. It is usually used for predicting whethercategories. It is usually used for predicting whethersomething will happen or notsomething will happen or not -- anything that can beanything that can beexpressed as event/nonexpressed as event/non--event. Independent variables mayevent. Independent variables maybe categorical or continuous.be categorical or continuous.
Hypothesis:Hypothesis: The population value is zero for either the differenceThe population value is zero for either the difference
between the proportions correct or the logs odds ratio on thebetween the proportions correct or the logs odds ratio on thetest items between the public and private, the male andtest items between the public and private, the male andfemale, and the low and high ability examineesfemale, and the low and high ability examinees ..
8/9/2019 Pedrajita, J.
10/48
METHODOLOGYMETHODOLOGY
DIF ModelsDIF Models
MantelMantel--HaenszelHaenszel Statistic (MH)Statistic (MH) The MH is a nonThe MH is a non--parametric contingency table procedureparametric contingency table procedure
commonly used to perform statistical test for uniform DIF. It iscommonly used to perform statistical test for uniform DIF. It is
also used to estimate a log odds ratio (also used to estimate a log odds ratio (MHMH) that yields a measure) that yields a measureof effect size for evaluating the amount of DIF that is present. Thisof effect size for evaluating the amount of DIF that is present. Thisratio value is rescaled asratio value is rescaled as DD == -- 2.352.35MHMH to produce the deltato produce the delta--MHMH(D(D--MH). A positive DMH). A positive D--MH indicates DIF in favor of the focalMH indicates DIF in favor of the focalgroups, and a negative value signifies DIF in favor of thegroups, and a negative value signifies DIF in favor of thereference groups. The degrees of DIF in test items are labeled A,reference groups. The degrees of DIF in test items are labeled A,B, and C (ETS item category) to indicateB, and C (ETS item category) to indicate negligiblenegligible,, moderatemoderate,,
andand largelarge amounts of DIF (amounts of DIF (GierlGierl, 1999)., 1999).
Hypothesis:Hypothesis:
There isnosignificantrelationship betweengroupmembershipandtestThere isnosignificantrelationship betweengroupmembershipandtestperformance onthe testitemsbetweenthe publicandprivate,the maleperformance onthe testitemsbetweenthe publicandprivate,the male
andfemale,andthe lowandhighabilityexamineesandfemale,andthe lowandhighabilityexaminees ..
8/9/2019 Pedrajita, J.
11/48
METHODOLOGYMETHODOLOGY
Content Validity AnalysisContent Validity Analysis
The content validity of the original test and its versionThe content validity of the original test and its version
was estimated based on the remaining items afterwas estimated based on the remaining items aftereliminating the biased items identified by each of the DIFeliminating the biased items identified by each of the DIFmethods.methods.
The degree to which the items comprise an adequateThe degree to which the items comprise an adequate
sample were based on a sixsample were based on a six--point scale, namely : Adequatepoint scale, namely : Adequate(86(86--100%), Moderately Adequate (71100%), Moderately Adequate (71--85%), Slightly85%), Slightly
Adequate (56Adequate (56--70%), Slightly Inadequate (4170%), Slightly Inadequate (41--55%),55%),Moderately Inadequate (26Moderately Inadequate (26--40%), and Inadequate (25%40%), and Inadequate (25%and below).and below).
8/9/2019 Pedrajita, J.
12/48
METHODOLOGYMETHODOLOGY
Concurrent Validity AnalysisConcurrent Validity Analysis
Concurrent validity evidence was secured by examining theConcurrent validity evidence was secured by examining the
relationship between the examinees test score and their grade pointrelationship between the examinees test score and their grade pointaverage in Science III.average in Science III.
The Pearson r was used to assessed the concurrent validity of theThe Pearson r was used to assessed the concurrent validity of theinstrument (original and test version) by correlating the examinees scoresinstrument (original and test version) by correlating the examinees scores
to their GPA in Science III.to their GPA in Science III.
Hypothesis:Hypothesis:
There isnosignificantrelationshipbetweenthe examineestestThere isnosignificantrelationshipbetweenthe examineestest
scoresandtheirgradesinScience III.scoresandtheirgradesinScience III.
InternalConsistency Reliability AnalysisInternalConsistency Reliability Analysis
The internal consistency reliability was determined byThe internal consistency reliability was determined bycalculating the KRcalculating the KR--20 Formula coefficient for the20 Formula coefficient for the
original and the test version.original and the test version.
8/9/2019 Pedrajita, J.
13/48
METHODOLOGYMETHODOLOGYStatistical Criteria for Identifying Potentially Biased Test ItemsStatistical Criteria for Identifying Potentially Biased Test Items
--------------------------------------------------------------------------------------------------------------------------------------------------------
DIF Methods Focusof AnalysisDIF Methods Focusof Analysis Measure of BiasMeasure of Bias
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ChiSquareChiSquare Difference inproportionattaininga Significance ofchisquareDifference inproportionattaininga Significance ofchisquare
correctresponse acrosstotalscorecorrectresponse acrosstotalscore
categoriescategories
Distracter Response Difference inproportionsselecting Significance ofchisquareDistracter Response Difference inproportionsselecting Significance ofchisquare
Analysis distractersAnalysis distracters
Logistic Regression Odds orlikelihoodofgettingan Significance ofchisquareLogistic Regression Odds orlikelihoodofgettingan Significance ofchisquare
itemrightitemright
MantelMantel--HaenszelHaenszel Performstatisticaltestfor Significance ofchisquarePerformstatisticaltestfor Significance ofchisquare
Statistic evaluatingthe amountofDIF andlarge DIFeffectStatistic evaluatingthe amountofDIF andlarge DIFeffect
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
8/9/2019 Pedrajita, J.
14/48
METHODOLOGYMETHODOLOGY
Test VersionsTest Versions
Chi SquareChi SquareDistracter Response AnalysisDistracter Response AnalysisLogistic RegressionLogistic RegressionMantelMantel--HaenszelHaenszel StatisticStatistic
ValidityValidity
ContentContentConcurrentConcurrent
DIF ModelsDIF ModelsChi SquareChi SquareDistracter Response AnalysisDistracter Response AnalysisLogistic RegressionLogistic RegressionMantelMantel--HaenszelHaenszel StatisticStatistic
Matched GroupingMatched Grouping Test ReliabilityTest ReliabilityClass TypeClass Type
publicpublicprivateprivate
GenderGendermalemale Chemistry AchievementChemistry Achievementfemalefemale TestTest
English AbilityEnglish Abilitylowlow
highhigh METHODOLOGICAL FLOWCHART OF THE STUDYMETHODOLOGICAL FLOWCHART OF THE STUDY
8/9/2019 Pedrajita, J.
15/48
METHODOLOGYMETHODOLOGY
The SampleThe SamplePublic School ExamineesPublic School Examinees
MaleMale FemaleFemale TotalTotal
AmorsoloAmorsolo LowLow 22 00 22
HighHigh 1818 19 19 3737
TotalTotal 2020 1919 39 39
KasilagKasilag LowLow 15 15 1414 2929
HighHigh 44 33 77
TotalTotal 1919 1717 3636
MalangMalang LowLow 77 1010 1717
HighHigh 44 44 88TotalTotal 1111 1414 2525
Aggregate TotalAggregate Total 50 50 10050 50 100
the public junior high school examinees was taken from the top, middle, andthe public junior high school examinees was taken from the top, middle, andlower class sections.lower class sections.
8/9/2019 Pedrajita, J.
16/48
METHODOLOGYMETHODOLOGY
The SampleThe Sample
Private School ExamineesPrivate School Examinees
MaleMale FemaleFemale TotalTotalPeachPeach LowLow 00 00 00
HighHigh 1919 2020 3939TotalTotal 1919 2020 39 39
PurplePurple LowLow 1818 1010 2828HighHigh 11 77 88
TotalTotal 1919 1717 3636
FuschiaFuschia LowLow 1010 99 1919HighHigh 22 44 66
TotalTotal 1212 1313 2525
Aggregate TotalAggregate Total 50 50 10050 50 100
the private junior high school examinees was taken the from top, middle, andthe private junior high school examinees was taken the from top, middle, andlower class sectionslower class sections
8/9/2019 Pedrajita, J.
17/48
METHODOLOGYMETHODOLOGY
Research InstrumentResearch InstrumentThe preparation of the researcherThe preparation of the researcher--constructed Chemistryconstructed Chemistry
Achievement Test involved the following steps:Achievement Test involved the following steps:
1. Development of a Table of Specifications1. Development of a Table of Specifications
2. Consultation with adviser/experts2. Consultation with adviser/experts
3. Generation of an item pool3. Generation of an item pool
4. Review of the initial item pool by experts4. Review of the initial item pool by experts5. Field5. Field--testingtesting
6. Item Analysis and test revision6. Item Analysis and test revision
Data Collection ProcedureData Collection Procedure1) Administration of the test to the original intact classes1) Administration of the test to the original intact classes2) Checking and scoring the test2) Checking and scoring the test
8/9/2019 Pedrajita, J.
18/48
METHODOLOGYMETHODOLOGY
Data Analysis ProcedureData Analysis Procedure
The analysis of data involved:The analysis of data involved:(a) assignment of examinees to the three comparison groups(a) assignment of examinees to the three comparison groups
matched by section and total score;matched by section and total score;
(b) organizing data for every item into a three(b) organizing data for every item into a three--way contingencyway contingencytable;table;
(c) encoding data in the Statistical Analysis System (SAS)(c) encoding data in the Statistical Analysis System (SAS)computer program;computer program;
(d) analysis for detecting and testing for differential item(d) analysis for detecting and testing for differential itemfunctioning/item bias for each comparison group,functioning/item bias for each comparison group,
(e) eliminating the correct responses on the biased items(e) eliminating the correct responses on the biased itemsidentified in the public/private matched group by each of theidentified in the public/private matched group by each of the
DIF methods, respectively;DIF methods, respectively;(f) retaining the unbiased items, herein referred to as the test(f) retaining the unbiased items, herein referred to as the test
version for each of the DIF methods, respectively;version for each of the DIF methods, respectively;
(g) assessment of the test versions content validity, concurrent(g) assessment of the test versions content validity, concurrentvalidity, and internal consistency reliability.validity, and internal consistency reliability.
8/9/2019 Pedrajita, J.
19/48
RESULTSRESULTSClass Type BiasClass Type Bias
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Chi SquareChi Square Dis tracter LogisticDistracter Logistic MantelMantel--HaenszelHaenszel
Response Analysis RegressionResponse Analysis Regression StatisticStatistic
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Potentially Biased Items 1 1 1Potentially Biased Items 1 1 1 11
Against the Private School 3 3 3 Against the Private School 3 3 3 33
ExamineesExaminees 5 5 55 5 5
9 9 9 99 9 9 9
10 1010 1019 19 1919 19 19 1919
2121 21 2121 21
26 2626 26
3030 30 30 3030 30 30
3131
33 33 3333 33 33 3333
3535
434347 47 4747 47 47 4747
5050
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Sub TotalSub Total 9 129 12 1111 1010
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
8/9/2019 Pedrajita, J.
20/48
RESULTSRESULTS
Class Type BiasClass Type Bias
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Chi SquareChi Square Dis tracter LogisticDistracter Logistic MantelMantel--HaenszelHaenszel
Response Analysis RegressionResponse Analysis Regression StatisticStatistic
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Potentially Biased ItemsPotentially Biased Items 2 22 2
Against the Public School Against the Public School 8 8 88 8 8
ExamineesExaminees 13 13 13 1313 13 13 13
1414 141416 16 1616 16 16 1616
22 22 2222 22 22 2222
3232 3232
3636 3636 3636
37 37 3737 37 37
4040 4040
4141 4141 4141
4646
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SubtotalSubtotal 4 6 11 124 6 11 12
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TotalItems IdentifiedTotalItems Identified 13 1813 18 2222 2222
8/9/2019 Pedrajita, J.
21/48
Sample Test ItemsSample Test Items
Item3Item3
Whentwoatomsinamolecule are heldtogether bythe transferof an electron fromone atomtotheWhentwoatomsinamolecule are heldtogether bythe transferof an electron fromone atomtotheother,the chemical bond betweenthemis knownasother,the chemical bond betweenthemis knownas
a.atomic bond.a.atomic bond. c.ionic bond.c.ionic bond.
b.covalent bond.b.covalent bond. d.metallic bond.d.metallic bond.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ChiChi--Square AnalysisSquare Analysis
Item Ability Level Computed XItem Ability Level Computed X22
Tabular XTabular X22
Sig. Prob.Sig. Prob. Biased Against Biased Against
33 1515 1919 0.65610.6561 6.25 0.01256.25 0.0125 0.41790.4179
2020 2424 2.68672.6867 0.10120.1012
2525 2929 0.00000.0000 1.00001.0000
3030 4141 6.32346.3234 0.0119* Private0.0119* Private
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Distracter Response AnalysisDistracter Response Analysis
ItemItem Distracters Computed XDistracters Computed X22 ProbabilityProbability Biased Against Biased Against
33 AA 0.16030.1603 0.68890.6889
BB 13.905113.9051 0.0002*0.0002* PrivatePrivate
DD 1.39691.3969 0.23730.2373
8/9/2019 Pedrajita, J.
22/48
Item3Item3
Logistic Regression AnalysisLogistic Regression Analysis
ItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
33 SI 3SI 3 6.67 6.67 0.08320.0832
CTCT 11 8.768.76 0.0031* Private0.0031* Private
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Distracter Response AnalysisDistracter Response Analysis
ItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased AgainstRatioRatio CategoryCategory
33 8.5611 0.0034* 2.4140 0.888.5611 0.0034* 2.4140 0.88 -- 2.072.07 C*C* PrivatePrivate
8/9/2019 Pedrajita, J.
23/48
Item 19Item 19
Whatisthe molarmassof FeWhatisthe molarmassof Fe22OO33 ? (Fe = 56 O = 16)? (Fe = 56 O = 16)
a. 48a. 48 c. 112c. 112b.72b.72 d. 160d. 160
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ChiChi--Square AnalysisSquare Analysis
Item Ability Level Computed XItem Ability Level Computed X22 Tabular XTabular X22 Sig. Prob.Sig. Prob. Biased Against Biased Against
1919 1515 2828 6.42066.4206 5.026 0.025 0.0113* Private5.026 0.025 0.0113* Private
2929 4141 0.05030.0503 0.82250.8225
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
8/9/2019 Pedrajita, J.
24/48
Item 19Item 19
Logistic Regression AnalysisLogistic Regression Analysis
ItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
1919 SISI 11 24.0524.05 < 0.0001< 0.0001
CTCT 11 5.345.34 0.0208*0.0208* PrivatePrivate
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MantelMantel--HaenszelHaenszel AnalysisAnalysis
ItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased Against
RatioRatio CategoryCategory
1919 5.3047 5.3047 0.0213* 2.1095 0.750.0213* 2.1095 0.75 -- 1.751.75 C*C* PrivatePrivate
8/9/2019 Pedrajita, J.
25/48
Item22Item22
A medicine dropperiscalibrated bycounting the numberof dropsthatdeliver 1 ml.IfA medicine dropperiscalibrated bycounting the numberof dropsthatdeliver 1 ml.If20drops equal 1 ml,the volume of adropis20drops equal 1 ml,the volume of adropis
a.0.005mla.0.005ml c.0.25mlc.0.25ml
b.0.05mlb.0.05ml d.0.5mld.0.5ml
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ChiChi--Square AnalysisSquare Analysis
Item Ability Level Computed XItem Ability Level Computed X22 Tabular XTabular X22 Sig. Prob.Sig. Prob. Biased AgainstBiased Against
2222 1515 2323 8.08878.0887 5.73 0.0166 0.0045* Public5.73 0.0166 0.0045* Public
2424 3030 12.747412.7474 0.0004* Public0.0004* Public
3131 4141 1.12161.1216 0.28960.2896
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Distracter Response AnalysisDistracter Response Analysis
Item Distracters Computed XItem Distracters Computed X22 ProbabilityProbability Biased Against Biased Against
2222 A 7.0913 A 7.0913 0.0077*0.0077* PublicPublic
CC 6.96166.9616 0.0083*0.0083* PublicPublic
DD 15.885615.8856 < 0.0001*< 0.0001* PublicPublic
8/9/2019 Pedrajita, J.
26/48
Item22Item22
Logistic Regression AnalysisLogistic Regression Analysis
ItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
2222 SISI 22 12.73 0.0017 12.73 0.0017
CTCT 11 24.84 < 0.0001* Public24.84 < 0.0001* Public
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------MantelMantel--HaenszelHaenszel AnalysisAnalysis
ItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased Against
RatioRatio CategoryCategory
2222 23.388223.3882
8/9/2019 Pedrajita, J.
27/48
Item 13Item 13
A certain gashasa volume of 1.25A certain gashasa volume of 1.25 lili whenpressure is.8 atm. Whatisthe newwhenpressure is.8 atm. Whatisthe new
pressure whenthe gasis compressedtoa volume of 1pressure whenthe gasis compressedtoa volume of 1 lili ??a.0.001 atma.0.001 atm c. 1.0atmc. 1.0atm
b.0.1 atmb.0.1 atm d. 10.0atmd. 10.0atm
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ChiChi--Square AnalysisSquare Analysis
Item Ability Level Computed XItem Ability Level Computed X22 Tabular XTabular X22 Sig. Prob. Biased AgainstSig. Prob. Biased Against
1313 1515 22 1.893022 1.8930 5.735.73 0.0166 0.16890.0166 0.1689
2323 30 6.814630 6.8146 0.0090* Public0.0090* Public
3131 41 2.332641 2.3326 0.1267 0.1267
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Distracter Response AnalysisDistracter Response Analysis
ItemItem Distracters Computed XDistracters Computed X
22
ProbabilityProbability Biased Against Biased Against
1313 AA 2.03952.0395 0.15330.1533
BB 6.90096.9009 0.0086*0.0086* PublicPublic
DD 15.363815.3638 < 0.0001*< 0.0001* PublicPublic
8/9/2019 Pedrajita, J.
28/48
Item 13Item 13
Logistic Regression AnalysisLogistic Regression Analysis
ItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
1313 SISI 22 0.930.93 0.62920.6292
CTCT 11 14.0514.05 0.0002*0.0002* PublicPublic
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MantelMantel--HaenszelHaenszel AnalysisAnalysis
ItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased AgainstRatioRatio CategoryCategory
1313 13.561513.5615 0.0002* 0.30630.0002* 0.3063 -- 1.181.18 2.782.78 C*C* PublicPublic
8/9/2019 Pedrajita, J.
29/48
RESULTSRESULTS
Gender BiasGender Bias------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Chi SquareChi Square Dis tracter LogisticDistracter Logistic MantelMantel--HaenszelHaenszel
Response Analysis RegressionResponse Analysis Regression StatisticStatistic
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Biased Items Against 17Biased Items Against 17 17 17 17 17
the Female Examineesthe Female Examinees 2727 27 27 27 27
34 3434 34
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Biased Items AgainstBiased Items Against 11 11
the Male Examineesthe Male Examinees 33 33
4242 4242 4242
4747 47 47
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Total Number ofTotal Number of
Items IdentifiedItems Identified 11 22 7 77 7
8/9/2019 Pedrajita, J.
30/48
Sample Test ItemsSample Test ItemsItem 17Item 17
Choose the correct electronicconfigurationof the elementChoose the correct electronicconfigurationof the element4747Na.Na.
a. 1sa. 1s22 2s2s22 2p2p66 3s3s22 3p3p66 4s4s22 3d3d1010 4p4p66 5s5s22 4d4d99
b. 1sb. 1s22 2s2s22 2p2p66 3s3s22 3p3p66 4s4s22 3d3d1010 4p4p66 5s5s33 4d4d99
c. 1sc. 1s22 2s2s22 2p2p66 3s3s22 3p3p66 4s4s22 3d3d1010 4p4p77 5s5s22 4d4d88
d. 1sd. 1s22 2s2s22 2p2p66 3s3s22 3p3p66 4s4s22 3d3d99 4p4p55 5s5s33 4d4d1010
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ChiChi--Square AnalysisSquare Analysis
Item Ability Level Computed XItem Ability Level Computed X22 Tabular XTabular X22 Sig. Prob. Biased AgainstSig. Prob. Biased Against
1717 1515 4141 5.34925.3492 3.84 .05 0.0207* Female3.84 .05 0.0207* Female
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Logistic Regression AnalysisLogistic Regression Analysis
ItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
1717 SISI 00 0.000.00
SexSex 11 6.586.58 0.0103*0.0103* FemaleFemale
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MantelMantel--HaenszelHaenszel AnalysisAnalysis
ItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased Against
Ratio CategoryRatio Category
1717 6.3342 0.0118* 3.11646.3342 0.0118* 3.1164 1.141.14 -- 2.67 C*2.67 C* FemaleFemale
8/9/2019 Pedrajita, J.
31/48
Item27Item27
Whichof the following illustratesthe compressibilityof gases?Whichof the following illustratesthe compressibilityof gases?
a. filling uparubbertire withaira. filling uparubbertire withair
b.increase in volume of doughwhen bakedb.increase in volume of doughwhen bakedc.rising ofc.rising of casserolescasseroles coverwhenwater beginsto boilcoverwhenwater beginsto boil
d.smelling the aromaof brewedcoffeed.smelling the aromaof brewedcoffee
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Distracter Response AnalysisDistracter Response AnalysisItemItem Distracters Computed X Distracters Computed X22 ProbabilityProbability Biased Against Biased Against
2727 BB 2.93062.9306 0.08690.0869
CC 6.20036.2003 0.0128*0.0128* FemaleFemaleDD 3.79893.7989 0.05130.0513
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Logistic Regression AnalysisLogistic Regression AnalysisItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
2727 SI 3SI 3 10.1410.14 0.01740.0174
SexSex 11 9.57 9.57 0.0020*0.0020* FemaleFemale
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MantelMantel--HaenszelHaenszel AnalysisAnalysisItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased Against
Ratio CategoryRatio Category
2727 9.25159.2515 0.0024*0.0024* 2.53062.5306 0.930.93 -- 2.18 C*2.18 C* FemaleFemale
8/9/2019 Pedrajita, J.
32/48
Item 1Item 1
A neighborchoose to burnhis garbage afterthe failure of garbage truckstocollectA neighborchoose to burnhis garbage afterthe failure of garbage truckstocollectthem. Aftera fewminutes,smoke fromthe burning garbage startedto enterthethem. Aftera fewminutes,smoke fromthe burning garbage startedto enterthehouse. Whatpropertyof gasesisillustrated by thissituation?house. Whatpropertyof gasesisillustrated by thissituation?
a.compressibilitya.compressibility c. elasticityc. elasticity
b.b.diffusibilitydiffusibility d. expansibilityd. expansibility
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Logistic Regression AnalysisLogistic Regression Analysis
ItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
11 SISI 33 24.1924.19 < 0.0001< 0.0001
SexSex 11 4.924.92 0.0265*0.0265* MaleMale
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MantelMantel--HaenszelHaenszel AnalysisAnalysisItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased Against
Ratio CategoryRatio Category
11 4.79224.7922 0.0286* 0.50660.0286* 0.5066 -- 0.68 1.6 C*0.68 1.6 C* MaleMale
8/9/2019 Pedrajita, J.
33/48
Item3Item3
Whentwoatomsinamolecule are heldtogether bythe transferof an electron fromWhentwoatomsinamolecule are heldtogether bythe transferof an electron fromone atomtothe other,the chemical bond betweenthemis knownasone atomtothe other,the chemical bond betweenthemis knownas
a.atomic bond.a.atomic bond. c.ionic bond.c.ionic bond.
b.covalent bond.b.covalent bond. d.d.metalliicmetalliic bond.bond.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Logistic Regression AnalysisLogistic Regression Analysis
ItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
33 SI 3SI 3 9.589.58 0.02250.0225
Sex 1Sex 1 4.594.59 0.0321*0.0321* MaleMale
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MantelMantel--HaenszelHaenszel AnalysisAnalysis
ItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased Against
Ratio CategoryRatio Category
33 4.48254.4825 0.0342* 0.52400.0342* 0.5240 -- 0.650.65 1.521.52 C*C* MaleMale
8/9/2019 Pedrajita, J.
34/48
RESULTSRESULTS
Ability BiasAbility BiasChi SquareChi Square Dist racter LogisticDistracter Logistic MantelMantel--HaenszelHaenszel
Response Analysis RegressionResponse Analysis Regression StatisticStatistic
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Biased Items AgainstBiased Items Against 2929 2929
the High Abilitythe High Ability 3838 3838
Examinees 45Examinees 45 4545
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Biased Items AgainstBiased Items Against 22 22 22
the Low Abilitythe Low Ability 33 33 33 33ExamineesExaminees 66 66 66 66
77
88 88 88 88
1313 1313 1313 1313
1515
17171919 1919 1919 1919
2222 2222 22223030 3030
3636 3636
4848 4848 4848
5050 5050 5050
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total Number ofTotal Number of
Items IdentifiedItems Identified 77 1111 13 1413 14
8/9/2019 Pedrajita, J.
35/48
Sample Test ItemsSample Test ItemsItem29Item29
Howmany valence electronsdoesHowmany valence electronsdoes ClCl atomhave ?atomhave ?
a.2a.2 c. 17c. 17
b.7b.7 d. 18d. 18
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Logistic Regression AnalysisLogistic Regression Analysis
ItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
2929 SISI 22 8.238.23 0.01630.0163
AbilityAbility 11 4.984.98 0.0256*0.0256* High AbilityHigh Ability
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MantelMantel--HaenszelHaenszel AnalysisAnalysis
ItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased Against
RatioRatio CategoryCategory
2929 4.8128 0.0282* 2.0515 0.724.8128 0.0282* 2.0515 0.72 -- 1.691.69 C*C* High AbilityHigh Ability
8/9/2019 Pedrajita, J.
36/48
Item36Item36
Waterisaddedtoashproduced fromthe burning of wood. The solutionchangesredWaterisaddedtoashproduced fromthe burning of wood. The solutionchangesredlitmuspaperto blue. Howwouldyouclassifythe solution?litmuspaperto blue. Howwouldyouclassifythe solution?
a.acida.acid c.metalc.metal
b. baseb. base d.nond.non--metalmetal
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Logistic Regression AnalysisLogistic Regression Analysis
ItemItem LR StatisticsLR Statistics Biased Against Biased Against
SourceSource dfdf Chi SquareChi Square ProbProb
3636 SI 3SI 3 0.47 0.47 0.92640.9264
AbilityAbility 11 4.154.15 0.0416* Low Ability0.0416* Low Ability
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MantelMantel--HaenszelHaenszel AnalysisAnalysis
ItemItem MHX MHX22 ProbProb OddsOdds MHMH DD--MH ETS Item Biased AgainstMH ETS Item Biased Against
RatioRatio CategoryCategory
3636 4.05234.0523 0.0441* 0.53010.0441* 0.5301 -- 0.630.63 1.491.49 C*C* Low AbilityLow Ability
8/9/2019 Pedrajita, J.
37/48
RESULTSRESULTS
Content Validityof the Test VersionContent Validityof the Test Version
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TestTest VersionVersion NoNo.. ofof ItemsItems Percent Percent DescriptionDescription
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OriginalTestOriginalTest 5050 100100 AdequateAdequate
Chi SquareChi Square 37 37 7474 ModeratelyadequateModeratelyadequate
DistracterDistracter 3232 6464 SlightlyadequateSlightlyadequate
Logistic Regression 28Logistic Regression 28 5656 SlightlyadequateSlightlyadequate
MantelMantel--HaenszelHaenszel 2828 5656 SlightlySlightly adequateadequate
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------The degree to which the items comprise an adequate sample were based on a sixThe degree to which the items comprise an adequate sample were based on a six--point scale, namely :point scale, namely :Adequate (86Adequate (86--100%), ModeratelyAdequate (71100%), ModeratelyAdequate (71--85%), SlightlyAdequate (5685%), SlightlyAdequate (56--70%), Slightly Inadequate70%), Slightly Inadequate
(41(41--55%), Moderately Inadequate (2655%), Moderately Inadequate (26--40%), and Inadequate (25% andbelow).40%), and Inadequate (25% andbelow).
8/9/2019 Pedrajita, J.
38/48
RESULTSRESULTS
Concurrent Validityof the Test VersionsConcurrent Validityof the Test Versions
------------------------------------------------------------------------------------------------------------------------------------------------------------------------Test VersionTest Version Validity CoefficientValidity Coefficient DescriptionDescription
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
OriginalOriginalTestTest 00..585585**** moderatemoderate relationshiprelationship
MantelMantel--HaenszelHaenszel 00..507507**** moderatemoderate relationshiprelationship
ChiChi SquareSquare 00..504504**** moderatemoderate relationshiprelationship
LogisticLogistic RegressionRegression 0 0..499499**** moderatemoderate relationshiprelationship
DistracterDistracter 00..462462**** moderatemoderate relationshiprelationship
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------****pp
8/9/2019 Pedrajita, J.
39/48
RESULTSRESULTSInternal Consistency Reliabilityof the Test VersionInternal Consistency Reliabilityof the Test Version
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Test Version No.of ItemsTest Version No.of Items Reliability CoefficientReliability Coefficient
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Original TestOriginal Test 5050 0.710.71
Chi SquareChi Square 3737 0.640.64
DistracterDistracter 3232 0.620.62
Logistic RegressionLogistic Regression 2828 0.570.57
MantelMantel--HaenszelHaenszel 2 828 00..5757
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The internal consistency reliability was determined by calculating the KRThe internal consistency reliability was determined by calculating the KR--20 Formula20 Formulacoefficient for the original and the test version.coefficient for the original and the test version.
8/9/2019 Pedrajita, J.
40/48
The presence of potential class type bias, gender bias,The presence of potential class type bias, gender bias,
and ability bias in the Chemistry Achievement Test canand ability bias in the Chemistry Achievement Test can
be attributed to:be attributed to:
1.1. Discrepancies in the curriculum of the public andDiscrepancies in the curriculum of the public andprivate school investigated;private school investigated;
2.2. Unfamiliarity with the content of the items whichUnfamiliarity with the content of the items whichcaused the examinees to be attracted to the incorrectcaused the examinees to be attracted to the incorrect
options;options;3.3. Disparities of the matched examinees exposure to theDisparities of the matched examinees exposure to theinformation, concepts, vocabularies, or skills reflectedinformation, concepts, vocabularies, or skills reflectedin the content of the biased items;in the content of the biased items;
4.4. Items which may reflect information and/or skills thatItems which may reflect information and/or skills thatwas not experienced by the matched examinees;was not experienced by the matched examinees;
5.5. Ambiguities in the item stem, keyed response, orAmbiguities in the item stem, keyed response, ordistracters; anddistracters; and
6.6. Overly difficult reading level or inability of the matchedOverly difficult reading level or inability of the matchedexaminees to comprehend or understand the conceptsexaminees to comprehend or understand the conceptsreflected on the biased items.reflected on the biased items.
8/9/2019 Pedrajita, J.
41/48
CONCLUSIONSCONCLUSIONS
The results of the differential item functioning analysis showedThe results of the differential item functioning analysis showed
that there were potentially biased test items between the publicthat there were potentially biased test items between the publicand private, the male and female, and the low and high abilityand private, the male and female, and the low and high abilityexaminees. Hence, potentialexaminees. Hence, potential class type biasclass type bias,, gender biasgender bias, and, andEnglish ability biasEnglish ability bias were present in the Chemistry Achievementwere present in the Chemistry AchievementTest. Overall, it appears that students from public schoolsTest. Overall, it appears that students from public schoolsperformed better than those from private schools; male andperformed better than those from private schools; male and
female examinees performed fairly; and low ability examineesfemale examinees performed fairly; and low ability examineesperformed miserably than the high English ability examinees.performed miserably than the high English ability examinees.
There were agreement and disagreement among the DIFThere were agreement and disagreement among the DIFmethods in the identity and number of potentially biased itemsmethods in the identity and number of potentially biased itemsidentified. There were items which were identified identically (a)identified. There were items which were identified identically (a)by the four models, b) by any three of the four models, c) byby the four models, b) by any three of the four models, c) byany two of the four models, and d) by a single model.any two of the four models, and d) by a single model.
8/9/2019 Pedrajita, J.
42/48
CONCLUSIONSCONCLUSIONSThe Logistic Regression and the MantelThe Logistic Regression and the Mantel--HaenszelHaenszel StatisticStatisticyielded very similar results with respect to uniform DIF. Theyielded very similar results with respect to uniform DIF. Thetwo procedures result in similar number and identity oftwo procedures result in similar number and identity ofitems identified. Hence, there is a high degree ofitems identified. Hence, there is a high degree ofcorrespondence between these two procedures.correspondence between these two procedures.
Elimination of biased items in a test tend to decrease itsElimination of biased items in a test tend to decrease its
content validity, concurrent validity, and internalcontent validity, concurrent validity, and internalconsistency reliability, as it diminishes the length or numberconsistency reliability, as it diminishes the length or numberof items of the test.of items of the test.
The use of statistical methods in identifying biased testThe use of statistical methods in identifying biased test
items is a relatively better kind of item analysis because byitems is a relatively better kind of item analysis because bysubjecting test items to DIF approaches, test items whichsubjecting test items to DIF approaches, test items whichwere unfairly difficult and widely discriminating for awere unfairly difficult and widely discriminating for aparticular group of examinees are determined. Byparticular group of examinees are determined. Byeliminating, replacing, or revising these biased items aeliminating, replacing, or revising these biased items avalid, reliable, and fairer test would be made.valid, reliable, and fairer test would be made.
8/9/2019 Pedrajita, J.
43/48
RECOMMENDATIONSRECOMMENDATIONS
1. Test experts and developers should use contingency table1. Test experts and developers should use contingency table(CT) methods, particularly the LR and MH methods, in item bias(CT) methods, particularly the LR and MH methods, in item biasdetection. The two procedures result in similar number of itemsdetection. The two procedures result in similar number of items(and similar items) being identified. The two methods are viable in(and similar items) being identified. The two methods are viable inthe detection of DIF and are widely implemented in both testthe detection of DIF and are widely implemented in both testconstruction and research settings.construction and research settings.
2. Educational evaluation practitioners should engage in item2. Educational evaluation practitioners should engage in itembias detection and use Logistic Regression or Mantelbias detection and use Logistic Regression or Mantel--HaenszelHaenszelStatistic for bias correction, which means that identified biasedStatistic for bias correction, which means that identified biaseditems should be revised or replaced. Then, reitems should be revised or replaced. Then, re--administer the testadminister the testand subject it anew to item bias detection in order to further refineand subject it anew to item bias detection in order to further refine
and purify the required item content of a test. This process couldand purify the required item content of a test. This process couldmake differentially functioning items between groups of interest bemake differentially functioning items between groups of interest bemore valid, reliable, and fair. Bias correction may maintain ormore valid, reliable, and fair. Bias correction may maintain orimprove the measurement qualities of a test such as its contentimprove the measurement qualities of a test such as its contentvalidity, concurrent validity, and internal consistency reliability.validity, concurrent validity, and internal consistency reliability.
8/9/2019 Pedrajita, J.
44/48
RECOMMENDATIONSRECOMMENDATIONS
3.3. PotentiallyPotentially biased items should either be revised or replacedbiased items should either be revised or replacedsince its elimination and nonsince its elimination and non--replacement lessen the number of items inreplacement lessen the number of items ina test. The lesser the number of items, the smaller was the contenta test. The lesser the number of items, the smaller was the contentvalidity, concurrent validity, and internal consistency.validity, concurrent validity, and internal consistency.
4. In this study, matching was done by conditioning simultaneously4. In this study, matching was done by conditioning simultaneouslyon test score, and a categorical variable, namely,on test score, and a categorical variable, namely, total scoretotal score andand classclasstypetype for the public/private comparison group,for the public/private comparison group, total scoretotal score andand sexsex forforthe male/female comparison group, andthe male/female comparison group, and total scoretotal score andand English abilityEnglish abilityfor the low/high ability comparison group. In connection with the abovefor the low/high ability comparison group. In connection with the above--mentioned conditioning, it is also recommended that a study bementioned conditioning, it is also recommended that a study be
conducted using Logistic Regression or Mantelconducted using Logistic Regression or Mantel--HaenszelHaenszel Statistic byStatistic byincorporating more than two or multiple ability estimate into a DIF/itemincorporating more than two or multiple ability estimate into a DIF/itembias analysis. That is, matching should be conditioned simultaneously onbias analysis. That is, matching should be conditioned simultaneously ontotal score, a categorical variable, and additional educationaltotal score, a categorical variable, and additional educationalbackground variables like age, verbal ability, mathematical ability, socialbackground variables like age, verbal ability, mathematical ability, social
class, educational attainment, type of community, and the like..class, educational attainment, type of community, and the like..
8/9/2019 Pedrajita, J.
45/48
RECOMMENDATIONSRECOMMENDATIONS
5.5. Future studies shouldFuture studies should focus onfocus on other psychometric issuesother psychometric issues
not addressed in this study. These include matters related tonot addressed in this study. These include matters related tocomparative study ofcomparative study ofItem Response TheoryItem Response Theory (IRT) and(IRT) andContingency TableContingency Table (CT) methods on any relevant psychometric(CT) methods on any relevant psychometricissue, such as of test equating, and item banking.issue, such as of test equating, and item banking.
6. Educational institutions, educational evaluators, and test6. Educational institutions, educational evaluators, and testexperts and developers should give increasingattention toexperts and developers should give increasingattention toequity of test scores for various subpopulations of examinees,equity of test scores for various subpopulations of examinees,be it regular or students with learningdisabilities. Test equitybe it regular or students with learningdisabilities. Test equitycan be achieved by ensuring that a test measures onlycan be achieved by ensuring that a test measures only
constructconstruct--relevant differences between subpopulations ofrelevant differences between subpopulations ofexaminees. To achieve test equity amongsubpopulations ofexaminees. To achieve test equity amongsubpopulations ofexaminees, bias testingmust be conducted especially for veryexaminees, bias testingmust be conducted especially for veryimportant tests like entrance examination and professionalimportant tests like entrance examination and professionallicensure examination.licensure examination.
8/9/2019 Pedrajita, J.
46/48
RECOMMENDATIONSRECOMMENDATIONS
7.7. One of the objectives of this paper is on detectingOne of the objectives of this paper is on detecting
DIF/biased items. However, it is also recommended that furtherDIF/biased items. However, it is also recommended that furtherstudies be conducted to go beyond detecting biased items andstudies be conducted to go beyond detecting biased items andobtain additional information about DIF/biased items. Someobtain additional information about DIF/biased items. Someitems may show larger magnitude of DIF, while some othersitems may show larger magnitude of DIF, while some othersshow relatively small magnitude of DIF. In such a situation, it isshow relatively small magnitude of DIF. In such a situation, it is
of interest to investigate sources of such variation.of interest to investigate sources of such variation.
8/9/2019 Pedrajita, J.
47/48
REFERENCESREFERENCESCamilliCamilli, G. and, G. and ShepardShepard, L. (1994)., L. (1994). Methodsfor Identifying Biased TestItemsMethodsfor Identifying Biased TestItems. Volume 4, Sage. Volume 4, Sage
Publications, Inc., California.Publications, Inc., California.Ferguson, G. A. andFerguson, G. A. and TakaneTakane, Y. (1989)., Y. (1989). Statistical Analysis in Psychology and EducationStatistical Analysis in Psychology and Education. Sixth. SixthEdition, New York : McGrawEdition, New York : McGraw--Hill, Inc.Hill, Inc.FraenkelFraenkel, J.R. and, J.R. and WallenWallen, N.E. (1993)., N.E. (1993). How ToDesign and Evaluate Research in EducationHow ToDesign and Evaluate Research in Education..Second Edition,Second Edition, MsGrawMsGraw--Hill Book, Inc.Hill Book, Inc.GierlGierl, M. J. (1999)., M. J. (1999). Differential Item Functioningon the Alberta Education Social Studies 30Differential Item Functioningon the Alberta Education Social Studies 30Diploma ExaminationDiploma Examination. Canadian Social Studies, Vol. 33, No. 2. Canadian Social Studies, Vol. 33, No. 2..KamataKamata, A. and Vaughn B. (2004)., A. and Vaughn B. (2004). An Introduction toDifferential Item Functioning Analysis.An Introduction toDifferential Item Functioning Analysis.LearningDisabilities: A Contemporary Journal 2(2), 49LearningDisabilities: A Contemporary Journal 2(2), 49 69.69.KanjeeKanjee, A. (2007)., A. (2007). Using Logistic Regression toDetect BiasWhen Multiple Groups Are Tested.Using Logistic Regression toDetect BiasWhen Multiple Groups Are Tested.South African Journal ofPsychology, 37, 47South African Journal ofPsychology, 37, 47 61.61.MazorMazor, K. E.,, K. E., KanjeeKanjee, A., and, A., and ClauserClauser, B. E. (1995)., B. E. (1995). Using Logistic Regression and the MantelUsing Logistic Regression and the Mantel--HaenszelHaenszel With Multiple AbilityEstimatestoDetectDifferential Item Functioning.With Multiple AbilityEstimatestoDetectDifferential Item Functioning. Journal ofJournal ofEducational Measurement, 32, 131Educational Measurement, 32, 131 144.144.OsterlindOsterlind, Steven J. (1983)., Steven J. (1983). TestItem BiasTestItem Bias. Sage Publications, Inc., California.. Sage Publications, Inc., California.Reynolds, C. R., Livingston, R. B., andReynolds, C. R., Livingston, R. B., and WillsonWillson, V. (2006). Measurement and Assessment in, V. (2006). Measurement and Assessment inEducation. Pearson Education, Inc.Education. Pearson Education, Inc.
Rogers, H. J. &Rogers, H. J. & SwaminathanSwaminathan, H. (1993)., H. (1993). A Comparison ofLogistic Regression and MantelA Comparison ofLogistic Regression and Mantel--HaenszelHaenszel Proceduresfor DetectingDifferential Item FunctioningProceduresfor DetectingDifferential Item Functioning. Applied Psychological. Applied PsychologicalMeasurement, 17, 105Measurement, 17, 105 116.116.SwaminathanSwaminathan, H. and Rogers, H. J. (1990)., H. and Rogers, H. J. (1990). DetectingDifferential Item Functioning Using LogisticDetectingDifferential Item Functioning Using LogisticRegression Procedures.Regression Procedures. Journal ofEducational Measurement, 27, 361Journal ofEducational Measurement, 27, 361 370.370.
8/9/2019 Pedrajita, J.
48/48
ACKNOWLEDGEMENTACKNOWLEDGEMENTThe presenter expresses his sincere appreciationThe presenter expresses his sincere appreciation
and gratitude to :and gratitude to :
Conference OrganizersConference Organizers
Dr. CarloDr. Carlo MagnoMagno Chair, NCEME Scientific CommitteeChair, NCEME Scientific Committee
Dr.Dr. LinaLina MiclatMiclat Secretary, PEMEASecretary, PEMEA
Thank you!Thank you!