Classical test theory and item response theory validation of a building design and modeling fundamentals test in vocational education

Abstract

Classical test theory (CTT) and item response theory (IRT) rest on different measurement assumptions: CTT describes items relative to the tested group, whereas IRT places items and persons on a common latent ability scale. Using both frameworks therefore offers complementary validity evidence for classroom tests in vocational education, where such evidence is rarely reported. This research and development study developed a test for the Fundamentals of Building Design, Modeling, and Information subject (Dasar-dasar Desain Pemodelan dan Informasi Bangunan, Basic DPIB) and examined its items with CTT and the two-parameter logistic (2PL) IRT model. Forty multiple-choice items were written from a competency-based blueprint and judged by five experts (Aiken's V = 0.80–0.95). After a tryout with 30 students, 32 items were administered to 103 DPIB students selected by proportional stratified random sampling. CTT showed high split-half reliability (Spearman–Brown = 0.95), good-to-very-good discrimination for 30 items, and 13 items with at least one poorly functioning distractor; no item was classified as difficult. The 2PL model produced positive discrimination (a = 0.848 to 4.833) and mostly negative difficulty (b = −2.597 to 0.034), and test information peaked at θ ≈ −0.98. Both frameworks identified the same easiest, hardest, and most discriminating items, but they contributed different evidence: CTT pinpointed distractors to revise, whereas IRT showed that measurement precision is concentrated below average ability. The test is therefore better suited to identifying students who have not yet mastered basic competencies than to distinguishing high achievers; adding difficult items and recalibrating with a larger sample are required before high-stakes use.