Development of a Toolbox for Psychological Test Development

Psychological tests and questionnaires are commonly used for assessing various traits of persons. Examples for the use of such tests are found in clinical psychology (e.g., dementia research), educational research (e.g. the PISA study), traffic psychology, or personal selection and development. The results of tests that are developed in a professional setting must have a valid interpretation and are therefore subject to strict quality controls. This quality control typically entails empirical studies with the tests that are to be to assess their psychometric characteristics, such as their reliability and aspects of their fairness. The data analysis of these empirical studies uses statistical methods from the field of psychometrics, which is located at the intersection of psychology and statistics.

This project concerns the development of new statistical methods and software for planning and the post hoc assessment of such evaluation studies. The resulting toolbox can be used in the development and analysis of psychological tests. The theoretical framework of this project are models of item response theory. These models provide a mathematical description of the interaction of test takers and the tasks of psychological tests. If such a model provides a sufficiently accurate description of this interaction, test evaluators can make assessments about the fairness, reliability, and also the interpretation of psychological tests. On the other hand, specific violations of fairness and also insufficient reliability can also be described by models from the item response theory framework. From a statistical perspective, the evaluation of psychological tests with methods from this framework leads to the question of whether a model that indicates desirable properties of the evaluated tests provides a better description than a model that does not. It is of high practical relevance that such models can be discerned with minimal effort in, for instance, the collection of data.

The goal of the project is the development of software and new statistical methods that support the planning of such studies. Central aspects where this toolbox should support in test development is the selection of suitable statistical methods and the sample collection, like the size and the composition of the sample. This toolbox also allows the evaluation of ongoing or already concluded empirical studies on the evaluation of psychological tests.