Developing a performance assessment task in the Finnish higher education context: Conceptual and empirical insights.
critical thinking
higher education
moral judgement
performance assessment
Journal
The British journal of educational psychology
ISSN: 2044-8279
Titre abrégé: Br J Educ Psychol
Pays: England
ID NLM: 0370636
Informations de publication
Date de publication:
Sep 2019
Sep 2019
Historique:
received:
30
10
2018
revised:
09
03
2019
pubmed:
19
5
2019
medline:
6
2
2020
entrez:
19
5
2019
Statut:
ppublish
Résumé
Performance assessment development is a three-phase process of (1) defining the construct of what is to be measured, (2) constructing the test items, task, and scoring criteria, and (3) collecting empirical evidence on extent to which they tap into the intended construct. In the light of the three-phase process, this pilot study presents the development of performance assessment for research on critical thinking and moral judgement to be utilized in the Finnish higher education context. In particular, it explores the cognitive processes and moral judgement that occur while completing the assessment and how both task and respondent characteristics are associated with these processes. The study was conducted with five participants drawn from purposeful sampling. We collected the data from cognitive laboratories with think-alouds and interviews. The data were analysed using a qualitative abductive approach. The instructions together with sources given for completing the assessment triggered participants' critical and moral judgement. The results also indicate that the performance assessment being developed has relevance to everyday life. The theme of assessment was experienced meaningful, and it maintained the participants' interest throughout the completion of the task. The assessment was open-ended and allowed multiple lines of thinking. The participants perceived the task as being challenging. The prior knowledge of the participants, their integrity, and capability to monitor thinking and performance was related to the ways of performing the task. Based on preliminary evidence drawn from the pilot data, the study concludes by outlining the strengths of the task as well as modifications that are needed in order to develop this specific performance assessment further.
Sections du résumé
BACKGROUND
BACKGROUND
Performance assessment development is a three-phase process of (1) defining the construct of what is to be measured, (2) constructing the test items, task, and scoring criteria, and (3) collecting empirical evidence on extent to which they tap into the intended construct.
AIMS
OBJECTIVE
In the light of the three-phase process, this pilot study presents the development of performance assessment for research on critical thinking and moral judgement to be utilized in the Finnish higher education context. In particular, it explores the cognitive processes and moral judgement that occur while completing the assessment and how both task and respondent characteristics are associated with these processes.
SAMPLE
METHODS
The study was conducted with five participants drawn from purposeful sampling.
METHODS
METHODS
We collected the data from cognitive laboratories with think-alouds and interviews. The data were analysed using a qualitative abductive approach.
RESULTS
RESULTS
The instructions together with sources given for completing the assessment triggered participants' critical and moral judgement. The results also indicate that the performance assessment being developed has relevance to everyday life. The theme of assessment was experienced meaningful, and it maintained the participants' interest throughout the completion of the task. The assessment was open-ended and allowed multiple lines of thinking. The participants perceived the task as being challenging. The prior knowledge of the participants, their integrity, and capability to monitor thinking and performance was related to the ways of performing the task.
CONCLUSIONS
CONCLUSIONS
Based on preliminary evidence drawn from the pilot data, the study concludes by outlining the strengths of the task as well as modifications that are needed in order to develop this specific performance assessment further.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
551-563Informations de copyright
© 2019 The British Psychological Society.
Références
American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME) (2014). Standards for educational and psychological testing. Washington, DC: Author.
American Philosophical Association (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction, the Delphi report: Research findings and recommendations prepared for the committee on pre-college philosophy. P. Facione, (Project Director). ERIC Doc. No. ED 315-423.
Arum, R., & Roksa, J. (2011). Academically adrift. Limited learning on college campuses. Chicago, IL: The University of Chicago Press.
Biddle, B. J., & Thomas, E. J. (Eds.) (1966). Role theory. Concepts and research. New York, NY: John Wiley & Sons.
Brückner, S., & Pellegrino, J. W. (2016). Integrating the analysis of mental operations into multilevel models to validate an assessment of higher education students’ competency in business and economics. Journal of Educational Measurement, 53, 293-312. https://doi.org/10.1111/jedm.12113
Buchmann, M. (1986). Role over person: Morality and authenticity in teaching. Teachers College Record, 87, 529-543.
Denzin, N. K. (2012). Triangulation 2.0. Journal of Mixed Methods Research, 6, 80-88. https://doi.org/10.1177/1558689812437186
Dewey, J. (1910). How we think. Boston, MA: D.C. Heath & Co. https://doi.org/10.1037/10903-000
Dewey, J. (1927). The public and its problems. Reprinted in J. A. Boydston (Ed.), John Dewey: The later works: 1925-1953 (LW) (Vol. 2, pp. 253-372). Carbondale, IL: Southern Illinois University Press.
Ercikan, K., & Pellegrino, J. W. (2017). Validation of score meaning using examinee response processes for the next generation of assessments. New York, NY: Routledge. https://doi.org/10.4324/9781315708591
Greene, J. (2013). Moral tribes: Emotion, reason, and the gap between us and them. New York, NY: Penguin Press.
Haidt, J. (2012). The righteous mind: Why good people are divided by politics and religion. New York, NY: Vintage.
Halpern, D. F. (2014). Thought and knowledge (5th ed.). New York, NY: Psychology Press.
Hyytinen, H., Nissinen, K., Ursin, J., Toom, A., & Lindblom-Ylänne, S. (2015). Problematising the equivalence of the test results of performance-based critical thinking tests for undergraduate students. Studies in Educational Evaluation, 44, 1-8. https://doi.org/10.1016/j.stueduc.2014.11.001.
Hyytinen, H., Toom, A., & Postareff, L. (2018). Unraveling the complex relationship in critical thinking, approaches to learning and self-efficacy beliefs among first-year educational science students. Learning and Individual Differences, 67, 132-142. https://doi.org/10.1016/j.lindif.2018.08.004.
Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar, Straus and Giroux.
Kane, M. (2013). The argument-based approach to validation. School Psychology Review, 42, 448-457.
Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., Blazevski, J., Bonney, C. R., … Kelly, K. L. (2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42(3), 139-151. https://doi.org/10.1080/00461520701416231
Leighton, J. (2017). Collecting and analyzing verbal response process data in the service of interpretive and validity arguments. In K. Ercikan & J. W. Pellegrino (Eds.), Validation of score meaning using examinee response processes for the next generation of assessments (pp. 25-37). New York, NY: Routledge. https://doi.org/10.4324/9781315708591
MacIntyre, A. (1985). After virtue: A study in moral theory. Second postscript. London, UK: Duckworth.
McClelland, D. C. (1973). Testing for competence rather than testing for “intelligence”. American Psychologist, 28(1), 1-14. https://doi.org/10.1037/h0034092
Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25, 6-20. https://doi.org/10.1080/03075079.2015.1004241
Pellegrino, J. W., N. Chudowsky, & R. Glaser (Eds.) (2001). Knowing What Students Know: The Science and Design of Educational Assessment. Washington, DC: The National Academies Press. https://doi.org/10.17226/10019.
Shavelson, R. J. (2010). On the measurement of competency. Empirical Research in Vocational Education and Training, 2(1), 41-63.
Shavelson, R. J., Zlatkin-Troitschanskaia, O., & Mariño, J. (2018). International performance assessment of learning in higher education (iPAL): Research and development. In O. Zlatkin-Troitschanskaia, M. Toepper, H. A. Pant, C. Lautenbach & C. Kuhn (Eds.), Assessment of learning outcomes in higher education - Cross-national comparisons and perspectives (pp. 193-214). Wiesbaden, Germany: Springer.
Sockett, H. (2009). Dispositions as virtues: The complexity of the construct. Journal of Teacher Education, 60, 291-303. https://doi.org/10.1177/0022487109335189
Timmermans, S., & Tavory, I. (2012). Theory construction in qualitative research: From grounded theory to abductive analysis. Sociological Theory, 30, 167-186. https://doi.org/10.1177/0735275112457914
Toom, A., & Husu, J. (2018). Teachers’ work in changing educational contexts: Balancing between the role and the person. In H. Niemi, A. Toom, A. Kallioniemi & J. Lavonen (Eds.), The teacher's role in the changing globalizing world (pp. 1-9). Leiden, The Netherlands: Brill.
Toom, A., Husu, J., & Tirri, K. (2015). Cultivating student teachers’ moral competencies in teaching during teacher education. In C. J. Craig & L. Orland-Barak (Eds.), International teacher education: Promising pedagogies (Part C): Advances in research on teaching (Vol. 22C, pp. 13-31). Bingley, UK: Emerald Publishing.
Zlatkin-Troitschanskaia, O., Shavelson, R. J., & Kuhn, C. (2015). The international state of research on measurement of competency in higher education. Studies in Higher Education, 40, 393-411. https://doi.org/10.1080/03075079.2015.1004241