ABSTRACT

Background

Problem-based learning (PBL) is a small group education in which students direct their own learning processes. Our aim in our study is to evaluate the program of the PBL module that we conducted with first-year students.

Materials and Methods

A total of 176 first-year students and 16 instructors attended the training. The PBL module was completed in 3 days. A pre-test with 16 questions and a post-test consisting of the same pre-test questions were administered to the students at the end of the training. Data analysis was performed using SPSS 25. Compliance of the data with normal distribution was determined using the Shapiro-Wilk test. The changes between pre- and post-training scores were examined using the Wilcoxon test. In this study, the type I error rate was 0.05. Approximately 4 weeks after the training, a focus group was held, and the interview was recorded. The qualitative data were evaluated by two researchers. The transcribed text was divided into themes and subthemes. The Kirkpatrick and logic model for program evaluation were used as the program evaluation model.

Results

According to Kirkpatrick level 1 evaluation, satisfaction with the training was high in both surveys and focus group studies. According to Kirkpatrick level 2, a significant increase in success was detected between the pretest and posttest of the students (pre-test average: 7, post-test average: 12, p<0.05). In addition, the average success rate of students was 35 out of 39 points. With the logic model evaluation, the inputs (resources), activities, results, short, medium and long-term goals of education were revealed, and it was determined that the short- and medium-term goals were achieved.

Conclusion

As a result of the program evaluation, it was determined that the satisfaction of first-year students and instructors with the PBL education was high, and the PBL success of the students was at the desired level.

Introduction

Program evaluation is the process of systematically collecting information about the structure and quality of an educational program. Program evaluation measures whether educational programs can achieve learning objectives. It is decided whether the training program will continue or not through program evaluation strategies (1). The most frequently used program evaluation models in medical education are the Kirkpatrick program evaluation model, Context, Input, Process, Product model, and logic model.

Problem-based learning (PBL) is a student-centered and structured learning method. This method was created to reduce the intensive information load in classical medical education and ensure that theoretical knowledge becomes more permanent. The most important feature of this method is that it allows vertical integration between basic and clinical medical sciences (2).

A clinical problem defined in PBL is transformed into a scenario. This scenario was discussed with a group of 7-8 students and a training facilitator. Students try to solve this clinical problem and diagnose the patient using their knowledge. They used brainstorming to discuss possible hypotheses. When their current knowledge could not explain the hypothesis, they reached their knowledge limit and set new learning goals.

They shared the new learning objectives that they had researched with the group in the next session (3, 4). PBL originates from constructivist theory and has been associated with multiple intelligence theory (5, 6).

Although research has shown that PBL is appreciated by students, no studies have examined the benefits of PBL using systematic program evaluation methods (1, 3, 5, 6, 7).

In this research, our aim is to evaluate the PBL training received by Hamidiye Medical Faculty first term students. For this purpose, we aim to make a more systematic evaluation using program evaluation models. Although program evaluation models are diverse, their use in educational research is not very common. Although some researchers have evaluated their PBL training, these evaluations were not based on a program evaluation method (1, 7, 8, 9, 10). In our study, we aimed to carry out the program evaluation of PBL education by using the

Kirkpatrick program evaluation model and the Logic evaluation model together. In this context, we believe that our study will contribute to the literature.

Material and Methods

Our PBL training, which we evaluated, was conducted by 176 first-term students at the Hamidiye Medical Faculty of Medicine and 16 faculty members. Before the training, 2 meetings were held with 16 trainers regarding the implementation of problem-based training. The students were divided into groups of 11 people. PBL groups consisting of an instructor and 11 students were created. Before the PBL scenario training, students were given a pretest designed to cover all scenario learning objectives (16 questions). After the students completed the PBL training, which consisted of approximately 2 hours and 3 separate sessions, they were given a posttest with the same questions as the pretest. In addition, at the end of the training, both the instructors and students were asked to evaluate by conducting a survey.

We used Kirkpatrick’s 4-stage model and the logic model together in the program evaluation of our study. We evaluated whether students and instructors were satisfied with the training at the first level of Kirkpatrick through surveys and a focus group qualitative study we conducted with the students at the end of the training.We evaluated educational success at the Kirkpatrick 2^nd level using the difference between the pre- and post-tests and the model form in which the instructors evaluated the students. Using the logical model, we evaluated the inputs/resources of education, the activities carried out during education, the outputs of education and the short, medium and long-term outcomes of these outputs.

The focus group method was used for the qualitative part of the study. Ten first-year students and 2 qualitative researchers participated in the focus group study. Verbal permission was obtained from the students for the focus group meeting, and brief information was provided about the focus group study. The answers given to the semi-structured questions prepared by the trainers with the support of the literature were recorded. The interviews lasted about 45-50 minutes. At the end of the focus group discussion, the recordings were transcribed on the same day. The themes and subthemes related to the content were determined by two medical evaluators. Ethics Committee approval was obtained from the University of Health Sciences Türkiye, Hamidiye Scientific Research Ethics Committee (approval number: 24/29, date: 19.01.2024).

Statistical Analysis

Data analysis was performed using IBM SPSS 25. The suitability of the data for normal distribution was examined using the Shapiro-Wilk test. For quantitative variables, median, minimum, and maximum values are presented. The changes in test scores before and after training were examined using the Wilcoxon test. In this study, the type I error rate was 0.05 (p<0.05). Exploratory factor analysis was assessed using Kaiser Meyer Olkin (KMO) and Bartlett’s test, confirmatory factor analysis using Root Mean Square Error of Approximation (RMSEA), and reliability using Cronbach’s alpha.

Thematic analysis was used to analyze qualitative data. The interview records transcribed after the focus group meeting were arranged and tabulated by removing themes and subthemes.

G Power (3.1.9.7) indicated that the power to detect small effects for the most complicated analyses (using effect size w=0.3, and desired power=0.95, alpha=0.05, two tailed; c² tests -Goodness-of-fit tests: Contingency tables) would total n=172.

Results

According to student and instructor surveys in the Kirkpatrick level 1 evaluation, satisfaction with PBL was found to be at the desired level. The results of the 5-point Likert-type student PBL evaluation survey are presented in Table 1. The survey was completed by 119 of the 176 students who participated in the training. Fourty-seven of the students participating in the survey were female, and 72 were male. The average age of the students is 18.6±1.10 (15-24; median:18). The satisfaction survey of the PBL instructors is presented in Table 2. Eeleven out of the 16 instructors completed the survey. The verbal comments of the students and instructors are included at the end of both tables.

The answers received as a result of the focus group interview were shaped around 2 main themes. These are the positive aspects of the applied PBL and the aspects that require improvement. The main themes and subthemes are presented in Table 3. In the focus group interview, sentence samples from students were coded as participants 1,2,3... (P1,2,3...).

The scale was found to be highly reliable (Cronbach’s alpha: 0.944) and valid (exploratory factor analysis, KMO: 0.926; and Bartlett’s test: 973.602; confirmatory factor analysis, RMSE: 0.733).

According to the Kirkpatrick level 2 evaluation results, term 1 students received an average of 35 points out of 39 based on performance evaluations made in 3 separate PBL sessions. The overall success score was calculated as the median. Minimum and maximum scores are 5 and maximum score is 39. It was determined that there was a significant difference between the success scores of the students before and after the training and that the post-test scores were higher than the pre-test scores (p<0.001) (Table 4).

In the evaluation conducted using the logic model, the resources used for training, activities performed, program outputs, and results were examined (Table 5).

Discussion

Kirkpatrick level 1 evaluation measures satisfaction with PBL training. This evaluation was conducted through student and educator surveys and focus group studies. According to the survey results, the students stated that a positive educational atmosphere was created, and the modules were well organized. Similar to our study, AlHaqwi (11) emphasized the importance of educators in PBL at their two medical faculties. In their studies, the authors stated the importance of the trainer in training motivation and creating a friendly training environment (11). In a study where Akdogan et al. (12) expressed the opinions of students in PBL, they stated that learning goals were easily achieved with a suitable scenario and the support of instructors. Similarly, in our study, the students stated that they actively participated in the discussions and easily achieved their learning goals. Trainers similarly expressed their satisfaction with the PBL training. However, instructors state that some students arrive to the modules without being sufficiently prepared and therefore cannot achieve their desired performances. Krasne et al. (13) also showed in their study that first-year medical students performed lower than third-year medical students, especially in terms of access to resources and effective educational preparation. Students emphasized important details in their verbal responses to the surveys and focus group discussions. Students say that they like PBL more than lecture halls, that their motivation increases, and that they learn to research. They also stated that with this training, they felt like a doctor for the first time and that they had taken on their own learning responsibilities. There are similar studies in the literature that support our findings. Nandi et al. (14), in their study comparing traditional medical education with PBL, showed that PBL supports self-learning and positively affects students’ social skills and motivation. Lee et al. (15) showed that PBL prepared students more professionally for medicine, while Koh et al. (16) showed that it supported social and cognitive skills much better. In the focus group discussion, students mentioned some negative experiences. These are problems with the scenario and problems with the organization. Some students stated that directing students to resources was insufficient. After training, various arrangements were made regarding these problems. When the literature review reveals that there are similar organizational or scenario-related problems with PBL. In their longitudinal study, Okubo et al. (17) mentioned the positive features of PBL as well as organizational problems that can be observed during training. Musal et al. (1) obtained interesting results regarding PBL training that they evaluated through a focus group study. Musal et al. (1) expressed the negative aspects of PBL as problems related to the training hall and training materials.

Kirkpatrick’s level 2 assessment evaluates student achievement. In our study, there was a significant increase in the pre- and post-tests, which consisted of the same questions for the students. Additionally, students were evaluated through a method that monitored their class participation, appropriate information gathering, and professional attitudes. In this evaluation, students received an average of 35 of 39 points. Qualitative and quantitative studies in the literature indicate that student success in PBL is high (7).

Evaluation using the logic model is important because it offers people a more systematic evaluation opportunity. In this study, we created a wide-ranging scale for the logic model evaluation. This scale consisted of resources, activities, outputs, and outcomes. Thanks to this table, potential deficiencies in the program can be understood more easily. In the literature, we did not find a program evaluation study that used the logic model in PBL. However, there are many logical model studies on medical education in the literature. Armstrong et al. (18) demonstrated that they achieved the expected outcomes in a logic model study in which they evaluated the success of their faculty development program they created. In this study, we detected positive behavioral improvements in the students at the end of the training. We can say that this is the most important output of the PBL. In surveys and focus group studies, we revealed that students feel that they are learning more effectively than in classical education. In addition, PBL is used by educators and students to develop social skills, improve motivation, and support self-learning. At the end of the evaluation, we made revisions to our training organization and PBL training. Rajashekaraet et al. (19) used the logic model in their studies showing whether 8 basic learning goals were achieved at the end of the training.

Conclusion

In our study, we believe that PBL is a useful training for both instructors and students in general and in many aspects, and we should increase the number of PBL trainings. It can be concluded that in the long term, physicians trained with this PBL training will be much more successful in communication, social accountability, and a positive approach to patient care.