55:"It is the characteristic of a set of test scores that relates to the amount of random error from the measurement process that might be embedded in the scores. Scores that are highly reliable are precise, reproducible, and consistent from one testing occasion to another. That is, if the testing process were repeated with a group of test takers, essentially the same results would be obtained. Various kinds of reliability coefficients, with values ranging between 0.00 (much error) and 1.00 (no error), are usually used to indicate the amount of error in the scores."
741:. Reliability estimates from one sample might differ from those of a second sample (beyond what might be expected due to sampling variations) if the second sample is drawn from a different population because the true variability is different in this second population. (This is true of measures of all types—yardsticks might measure houses well yet have poor reliability when used to measure the lengths of insects.)
138:, reliability does place a limit on the overall validity of a test. A test that is not perfectly reliable cannot be perfectly valid, either as a means of measuring attributes of a person or as a means of predicting scores on a criterion. While a reliable test may provide useful valid information, a test that is not reliable cannot possibly be valid.
145:
consistently measured the weight of an object as 500 grams over the true weight, then the scale would be very reliable, but it would not be valid (as the returned weight is not the true weight). For the scale to be valid, it should return the true weight of an object. This example demonstrates that a
753:
indices, the latter index involving computation of correlations between the items and sum of the item scores of the entire test. If items that are too difficult, too easy, and/or have near-zero or negative discrimination are replaced with better items, the reliability of the measure will increase.
712:
In splitting a test, the two halves would need to be as similar as possible, both in terms of their content and in terms of the probable state of the respondent. The simplest method is to adopt an odd-even split, in which the odd-numbered items form one half of the test and the even-numbered items
626:
With the parallel test model it is possible to develop two forms of a test that are equivalent in the sense that a person's true score on form A would be identical to their true score on form B. If both forms of the test were administered to a number of people, differences between scores on form A
708:
There are several ways of splitting a test to estimate reliability. For example, a 40-item vocabulary test could be split into two subtests, the first one made up of items 1 through 20 and the second made up of items 21 through 40. However, the responses from the first half may be systematically
658:
is less of a problem. Reactivity effects are also partially controlled; although taking the first test may change responses to the second test. However, it is reasonable to assume that the effect will not be as strong with alternate forms of the test as with two administrations of the same test.
744:
Reliability may be improved by clarity of expression (for written assessments), lengthening the measure, and other informal means. However, formal psychometric analysis, called item analysis, is considered the most effective way to increase reliability. This analysis consists of computation of
255:
The central assumption of reliability theory is that measurement errors are essentially random. This does not mean that errors arise from random processes. For any individual, an error in measurement is not a completely random event. However, across a large number of individuals, the causes of
168:
In practice, testing measures are never perfectly consistent. Theories of test reliability have been developed to estimate the effects of inconsistency on the accuracy of measurement. The basic starting point for almost all theories of test reliability is the idea that test scores reflect the
622:
The key to this method is the development of alternate test forms that are equivalent in terms of content, response processes and statistical characteristics. For example, alternate forms exist for several tests of general intelligence, and these tests are generally seen equivalent.
131:. That is, a reliable measure that is measuring something consistently is not necessarily measuring what you want to be measured. For example, while there are many reliable tests of specific abilities, not all of them would be valid for predicting, say, job performance.
529:
557:
It was well known to classical test theorists that measurement precision is not uniform across the scale of measurement. Tests tend to distinguish better for test-takers with moderate trait levels and worse among high- and low-scoring test-takers.
259:
If errors have the essential characteristics of random variables, then it is reasonable to assume that errors are equally likely to be positive or negative, and that they are not correlated with true scores or with errors on other tests.
84:
assesses the degree to which test scores are consistent from one test administration to the next. Measurements are gathered from a single rater who uses the same methods or instruments and the same testing conditions. This includes
185:
Temporary and specific characteristics of individual: comprehension of the specific test task, specific tricks or techniques of dealing with the particular test materials, fluctuations of memory, attention or
352:
1206:
Ritter, N. (2010). Understanding a widely misunderstood statistic: Cronbach's alpha. Paper presented at
Southwestern Educational Research Association (SERA) Conference 2010, New Orleans, LA (ED526237).
95:
assesses the degree to which test scores are consistent when there is a variation in the methods or instruments used. This allows inter-rater reliability to be ruled out. When dealing with
419:
701:
The correlation between these two split halves is used in estimating the reliability of the test. This halves reliability estimate is then stepped up to the full test length using the
865:
726:, which is usually interpreted as the mean of all possible split-half coefficients. Cronbach's alpha is a generalization of an earlier form of estimating internal consistency,
574:
The goal of estimating reliability is to determine how much of the variability in test scores is due to errors in measurement and how much is due to variability in true scores.
984:
399:
217:
is the replicable feature of the concept being measured. It is the part of the observed score that would recur across different measurement occasions in the absence of error.
155:
76:
assesses the degree of agreement between two or more raters in their appraisals. For example, a person gets a stomach ache and different doctors all give the same diagnosis.
808:
401:
provides an index of the relative influence of true and error scores on attained test scores. In its general form, the reliability coefficient is defined as the ratio of
885:
175:
2. Inconsistency factors: features of the individual or the situation that can affect test scores but have nothing to do with the attribute being measured.
713:
form the other. This arrangement guarantees that each half will contain an equal number of items from the beginning, middle, and end of the original test.
733:
These measures of reliability differ in their sensitivity to different sources of error and so need not be equal. Also, reliability is a property of the
534:
Unfortunately, there is no way to directly observe or calculate the true score, so a variety of methods are used to estimate the reliability of a test.
1150:
51:
is the overall consistency of a measure. A measure is said to have a high reliability if it produces similar results under consistent conditions:
605:
985:
http://www.ncme.org/ncme/NCME/Resource_Center/Glossary/NCME/Resource_Center/Glossary1.aspx?hkey=4bb87415-44dc-4088-9ed9-e8515326a061#anchorR
1289:
1279:
252:
The goal of reliability theory is to estimate errors in measurement and to suggest ways of improving tests so that errors are minimized.
17:
604:
The correlation between scores on the first test and the scores on the retest is used to estimate the reliability of the test using the
289:
1028:
1294:
1173:
1035:
1009:
702:
680:
This method treats the two halves of a measure as alternate forms. It provides a simple solution to the problem that the
566:. The IRT information function is the inverse of the conditional observed score standard error at any given test score.
1309:
1061:
189:
Aspects of the testing situation: freedom from distractions, clarity of instructions, interaction of personality, etc.
1134:
1314:
524:{\displaystyle \rho _{xx'}={\frac {\sigma _{T}^{2}}{\sigma _{X}^{2}}}=1-{\frac {\sigma _{E}^{2}}{\sigma _{X}^{2}}}}
722:: assesses the consistency of results across items within a test. The most common internal consistency measure is
727:
172:
1. Consistency factors: stable characteristics of the individual or the attribute that one is trying to measure.
669:
It may also be difficult if not impossible to guarantee that two alternate forms of a test are parallel measures
587:: directly assesses the degree to which test scores are consistent from one test administration to the next.
405:
variance to the total variance of test scores. Or, equivalently, one minus the ratio of the variation of the
814:
577:
Four practical strategies have been developed that provide workable methods of estimating test reliability.
146:
perfectly reliable measure is not necessarily valid, but that a valid measure necessarily must be reliable.
644:
The correlation between scores on the two alternate forms is used to estimate the reliability of the test.
31:
549:. Each method comes at the problem of figuring out the source of error in the test somewhat differently.
122:
196:
The goal of estimating reliability is to determine how much of the variability in test scores is due to
1329:
228:. It represents the discrepancies between scores obtained on tests and the corresponding true scores.
182:
Temporary but general characteristics of the individual: health, fatigue, motivation, emotional strain
1217:
1319:
912:
907:
897:
275:
Reliability theory shows that the variance of obtained scores is simply the sum of the variance of
760:
942:
917:
649:
582:
538:
369:
86:
80:
72:
637:
At some later time, administering an alternate form of the same test to the same group of people
1324:
932:
902:
709:
different from responses in the second half due to an increase in item difficulty and fatigue.
609:
1290:
The relationships between correlational and internal consistency concepts of test reliability
1193:
Cortina, J.M., (1993). What Is
Coefficient Alpha? An Examination of Theory and Applications.
952:
927:
730:. Although the most commonly used, there are some misconceptions regarding Cronbach's alpha.
247:
135:
128:
922:
718:
559:
542:
256:
measurement error are assumed to be so varied that measure errors act as random variables.
107:
870:
8:
1285:
Uncertainty models, uncertainty quantification, and uncertainty processing in engineering
723:
1258:
1144:
1087:
998:
972:
937:
192:
Chance factors: luck in selection of answers by sheer guessing, momentary distractions
59:
For example, measurements of people's height and weight are often extremely reliable.
1262:
1250:
1169:
1130:
1067:
1057:
1005:
197:
697:
Correlating scores on one half of the test with scores on the other half of the test
1240:
1232:
655:
225:
1032:
1039:
947:
142:
96:
647:
This method provides a partial solution to many of the problems inherent in the
562:
extends the concept of reliability from a single index to a function called the
1236:
1303:
1071:
44:
1284:
1254:
1218:"The reliability of a two-item scale: Pearson, Cronbach or Spearman-Brown?"
357:
This equation suggests that test scores vary as the result of two factors:
231:
This conceptual breakdown is typically represented by the simple equation:
221:
1245:
206:
40:
666:
It may be very difficult to create several alternate forms of a test
597:
Re-administering the same test to the same group at some later time
1090:
The
Research Methods Knowledge Base. Last Revised: 20 October 2006
1129:(6th ed.). Upper Saddle River, N.J.: Pearson/Prentice Hall.
112:, assesses the consistency of results across items within a test.
27:
Overall consistency of a measure in statistics and psychometrics
347:{\displaystyle \sigma _{X}^{2}=\sigma _{T}^{2}+\sigma _{E}^{2}}
654:. For example, since the two forms of the test are different,
537:
Some examples of the methods to estimate reliability include
634:
Administering one form of the test to a group of individuals
67:
There are several general classes of reliability estimates:
1215:
1280:
Internal and external reliability and validity explained.
1033:
Common
Language: Marketing Activities and Metrics Project
1127:
Psychological testing : principles and applications
236:
Observed test score = true score + errors of measurement
1083:
1081:
1031:(MASB) endorses this definition as part of its ongoing
737:
rather than the measure itself and are thus said to be
873:
817:
763:
684:
faces: the difficulty in developing alternate forms.
627:
and form B may be due to errors in measurement only.
422:
372:
292:
1078:
1216:Eisinga, R.; Te Grotenhuis, M.; Pelzer, B. (2012).
600:
Correlating the first set of scores with the second
997:
879:
859:
802:
640:Correlating scores on form A with scores on form B
523:
393:
346:
149:
1125:Davidshofer, Kevin R. Murphy, Charles O. (2005).
272:3. Errors on different measures are uncorrelated
1301:
1051:
662:However, this technique has its disadvantages:
156:Reproducibility (statistics) § Reliability
691:Administering a test to a group of individuals
606:Pearson product-moment correlation coefficient
594:Administering a test to a group of individuals
1023:
1021:
983:National Council on Measurement in Education
363:2. Variability due to errors of measurement.
1149:: CS1 maint: multiple names: authors list (
1124:
1004:(4th Canadian ed.). Toronto: Pearson.
269:2. True scores and errors are uncorrelated
220:Errors of measurement are composed of both
116:
1168:. Hillsdale, N.J.: L. Erlbaum Associates.
1018:
1000:Psychology : the science of behaviour
1244:
1189:
1187:
1185:
1163:
1120:
1118:
1116:
1029:Marketing Accountability Standards Board
995:
241:
123:Validity (statistics) § Reliability
1114:
1112:
1110:
1108:
1106:
1104:
1102:
1100:
1098:
1096:
552:
14:
1302:
1225:International Journal of Public Health
1182:
996:al.], Neil R. Carlson ... [et (2009).
860:{\displaystyle R(t)=\exp(-\lambda t),}
200:and how much is due to variability in
1295:The problem of negative reliabilities
1093:
169:influence of two sorts of factors:
24:
25:
1341:
1273:
1195:Journal of Applied Psychology, 78
1054:Essentials of abnormal psychology
703:Spearman–Brown prediction formula
266:1. Mean error of measurement = 0
134:While reliability does not imply
163:
150:Difference from reproducibility
1209:
1200:
1157:
1045:
989:
977:
965:
851:
839:
827:
821:
794:
788:
773:
767:
360:1. Variability in true scores
13:
1:
958:
569:
803:{\displaystyle R(t)=1-F(t).}
366:The reliability coefficient
7:
891:
728:Kuder–Richardson Formula 20
394:{\displaystyle \rho _{xx'}}
127:Reliability does not imply
18:Reliability (psychometrics)
10:
1346:
1164:Gulliksen, Harold (1987).
694:Splitting the test in half
245:
153:
120:
101:parallel-forms reliability
29:
1310:Comparison of assessments
1237:10.1007/s00038-012-0416-3
1052:Durand, V. Mark. (2015).
547:parallel-test reliability
409:and the variation of the
141:For example, if a set of
117:Difference from validity
1038:12 February 2013 at the
913:Homogeneity (statistics)
908:Consistency (statistics)
898:Coefficient of variation
93:Inter-method reliability
62:
1315:Statistical reliability
943:Reliability engineering
918:Test-retest reliability
650:test-retest reliability
583:Test-retest reliability
539:test-retest reliability
178:These factors include:
87:intra-rater reliability
81:Test-retest reliability
73:Inter-rater reliability
1166:Theory of mental tests
1056:. : Cengage Learning.
971:William M.K. Trochim,
933:Accuracy and precision
903:Congeneric reliability
881:
861:
804:
610:item-total correlation
525:
395:
348:
57:
953:Validity (statistics)
928:Levels of measurement
882:
880:{\textstyle \lambda }
862:
805:
682:parallel-forms method
617:Parallel-forms method
526:
396:
349:
281:errors of measurement
279:plus the variance of
248:Classical test theory
242:Classical test theory
53:
1088:Types of Reliability
923:Internal consistency
887:is the failure rate.
871:
815:
761:
719:Internal consistency
564:information function
560:Item response theory
553:Item response theory
543:internal consistency
420:
370:
290:
263:It is assumed that:
108:Internal consistency
30:For other uses, see
751:item discrimination
735:scores of a measure
518:
503:
475:
460:
343:
325:
307:
99:, it may be termed
938:Reliability theory
877:
857:
800:
521:
504:
489:
461:
446:
391:
344:
329:
311:
293:
198:measurement errors
1330:Survival analysis
1175:978-0-8058-0024-1
1011:978-0-205-64524-4
747:item difficulties
675:Split-half method
545:reliability, and
519:
476:
16:(Redirected from
1337:
1267:
1266:
1248:
1222:
1213:
1207:
1204:
1198:
1191:
1180:
1179:
1161:
1155:
1154:
1148:
1140:
1122:
1091:
1085:
1076:
1075:
1049:
1043:
1025:
1016:
1015:
1003:
993:
987:
981:
975:
969:
886:
884:
883:
878:
866:
864:
863:
858:
809:
807:
806:
801:
739:sample dependent
724:Cronbach's alpha
656:carryover effect
530:
528:
527:
522:
520:
517:
512:
502:
497:
488:
477:
474:
469:
459:
454:
445:
440:
439:
438:
400:
398:
397:
392:
390:
389:
388:
353:
351:
350:
345:
342:
337:
324:
319:
306:
301:
226:systematic error
21:
1345:
1344:
1340:
1339:
1338:
1336:
1335:
1334:
1320:Market research
1300:
1299:
1276:
1271:
1270:
1220:
1214:
1210:
1205:
1201:
1192:
1183:
1176:
1162:
1158:
1142:
1141:
1137:
1123:
1094:
1086:
1079:
1064:
1050:
1046:
1040:Wayback Machine
1026:
1019:
1012:
994:
990:
982:
978:
970:
966:
961:
948:Reproducibility
894:
872:
869:
868:
816:
813:
812:
762:
759:
758:
572:
555:
513:
508:
498:
493:
487:
470:
465:
455:
450:
444:
431:
427:
423:
421:
418:
417:
381:
377:
373:
371:
368:
367:
338:
333:
320:
315:
302:
297:
291:
288:
287:
250:
244:
237:
166:
160:
158:
152:
143:weighing scales
125:
119:
65:
35:
28:
23:
22:
15:
12:
11:
5:
1343:
1333:
1332:
1327:
1322:
1317:
1312:
1298:
1297:
1292:
1287:
1282:
1275:
1274:External links
1272:
1269:
1268:
1231:(4): 637–642.
1208:
1199:
1181:
1174:
1156:
1135:
1092:
1077:
1063:978-1305633681
1062:
1044:
1017:
1010:
988:
976:
963:
962:
960:
957:
956:
955:
950:
945:
940:
935:
930:
925:
920:
915:
910:
905:
900:
893:
890:
889:
888:
876:
856:
853:
850:
847:
844:
841:
838:
835:
832:
829:
826:
823:
820:
810:
799:
796:
793:
790:
787:
784:
781:
778:
775:
772:
769:
766:
699:
698:
695:
692:
671:
670:
667:
642:
641:
638:
635:
602:
601:
598:
595:
571:
568:
554:
551:
532:
531:
516:
511:
507:
501:
496:
492:
486:
483:
480:
473:
468:
464:
458:
453:
449:
443:
437:
434:
430:
426:
411:observed score
387:
384:
380:
376:
355:
354:
341:
336:
332:
328:
323:
318:
314:
310:
305:
300:
296:
246:Main article:
243:
240:
239:
238:
235:
194:
193:
190:
187:
183:
165:
162:
151:
148:
118:
115:
114:
113:
104:
90:
77:
64:
61:
26:
9:
6:
4:
3:
2:
1342:
1331:
1328:
1326:
1325:Psychometrics
1323:
1321:
1318:
1316:
1313:
1311:
1308:
1307:
1305:
1296:
1293:
1291:
1288:
1286:
1283:
1281:
1278:
1277:
1264:
1260:
1256:
1252:
1247:
1242:
1238:
1234:
1230:
1226:
1219:
1212:
1203:
1196:
1190:
1188:
1186:
1177:
1171:
1167:
1160:
1152:
1146:
1138:
1136:0-13-189172-3
1132:
1128:
1121:
1119:
1117:
1115:
1113:
1111:
1109:
1107:
1105:
1103:
1101:
1099:
1097:
1089:
1084:
1082:
1073:
1069:
1065:
1059:
1055:
1048:
1041:
1037:
1034:
1030:
1024:
1022:
1013:
1007:
1002:
1001:
992:
986:
980:
974:
968:
964:
954:
951:
949:
946:
944:
941:
939:
936:
934:
931:
929:
926:
924:
921:
919:
916:
914:
911:
909:
906:
904:
901:
899:
896:
895:
874:
854:
848:
845:
842:
836:
833:
830:
824:
818:
811:
797:
791:
785:
782:
779:
776:
770:
764:
757:
756:
755:
752:
748:
742:
740:
736:
731:
729:
725:
721:
720:
714:
710:
706:
704:
696:
693:
690:
689:
688:
687:It involves:
685:
683:
678:
676:
668:
665:
664:
663:
660:
657:
653:
651:
645:
639:
636:
633:
632:
631:
630:It involves:
628:
624:
620:
618:
613:
611:
607:
599:
596:
593:
592:
591:
590:It involves:
588:
586:
584:
578:
575:
567:
565:
561:
550:
548:
544:
540:
535:
514:
509:
505:
499:
494:
490:
484:
481:
478:
471:
466:
462:
456:
451:
447:
441:
435:
432:
428:
424:
416:
415:
414:
412:
408:
404:
385:
382:
378:
374:
364:
361:
358:
339:
334:
330:
326:
321:
316:
312:
308:
303:
298:
294:
286:
285:
284:
282:
278:
273:
270:
267:
264:
261:
257:
253:
249:
234:
233:
232:
229:
227:
223:
218:
216:
211:
209:
208:
203:
199:
191:
188:
184:
181:
180:
179:
176:
173:
170:
164:General model
161:
157:
147:
144:
139:
137:
132:
130:
124:
111:
109:
105:
102:
98:
94:
91:
88:
83:
82:
78:
75:
74:
70:
69:
68:
60:
56:
52:
50:
46:
45:psychometrics
42:
37:
33:
19:
1228:
1224:
1211:
1202:
1197:(1), 98–104.
1194:
1165:
1159:
1126:
1053:
1047:
999:
991:
979:
967:
750:
746:
743:
738:
734:
732:
717:
715:
711:
707:
700:
686:
681:
679:
674:
672:
661:
648:
646:
643:
629:
625:
621:
616:
614:
603:
589:
581:
579:
576:
573:
563:
556:
546:
536:
533:
410:
406:
402:
365:
362:
359:
356:
280:
276:
274:
271:
268:
265:
262:
258:
254:
251:
230:
222:random error
219:
214:
212:
205:
201:
195:
177:
174:
171:
167:
159:
140:
133:
126:
106:
100:
92:
79:
71:
66:
58:
54:
48:
38:
36:
1246:2066/116735
973:Reliability
608:: see also
407:error score
277:true scores
202:true scores
110:reliability
49:reliability
32:Reliability
1304:Categories
959:References
570:Estimation
403:true score
215:true score
207:true value
154:See also:
121:See also:
41:statistics
1263:215730043
1145:cite book
1072:884617637
875:λ
846:λ
843:−
837:
783:−
506:σ
491:σ
485:−
463:σ
448:σ
425:ρ
375:ρ
331:σ
313:σ
295:σ
1255:23089674
1036:Archived
892:See also
436:′
386:′
186:accuracy
136:validity
129:validity
1261:
1253:
1172:
1133:
1070:
1060:
1008:
867:where
652:method
585:method
1259:S2CID
1221:(PDF)
97:forms
63:Types
1251:PMID
1170:ISBN
1151:link
1131:ISBN
1068:OCLC
1058:ISBN
1027:The
1006:ISBN
749:and
224:and
43:and
1241:hdl
1233:doi
834:exp
716:4.
673:3.
615:2.
580:1.
210:).
39:In
1306::
1257:.
1249:.
1239:.
1229:58
1227:.
1223:.
1184:^
1147:}}
1143:{{
1095:^
1080:^
1066:.
1020:^
705:.
677::
619::
612:.
541:,
413::
283:.
213:A
47:,
1265:.
1243::
1235::
1178:.
1153:)
1139:.
1074:.
1042:.
1014:.
855:,
852:)
849:t
840:(
831:=
828:)
825:t
822:(
819:R
798:.
795:)
792:t
789:(
786:F
780:1
777:=
774:)
771:t
768:(
765:R
515:2
510:X
500:2
495:E
482:1
479:=
472:2
467:X
457:2
452:T
442:=
433:x
429:x
383:x
379:x
340:2
335:E
327:+
322:2
317:T
309:=
304:2
299:X
204:(
103:.
89:.
34:.
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.