Evaluation of Project-Based Learning at a Polytechnic in Malaysia: an Input Aspect Evaluation Research

The purpose of the study was to evaluate the effectiveness of Project-Based Learning in the Mechanical Engineering Department at the Polytechnic in Malaysia. This study was designed with Context, Input, Process and Product (CIPP) model. Project-Based Learning is a different approach than the traditional learning in the sense that it is a student-centered learning approach based on the principles of constructivism. However, this present paper focused on the input aspect with comparing students’ and supervisor’ perspective. The input dimension assesses the module and supervisor’s readiness. This study involved a random sample of 118 students and 43 supervisors in the Mechanical Engineering Department in Polytechnic Kota Bharu Malaysia. The instrument used in this study were a set of questionnaire. The study found that both the students and supervisors agreed that Project-Based Learning approach is appropriate for the final project course J5012. In addition, both groups of respondents believed that the supervisors possesed adequate technical knowledge and implemented supervisory duties effectively. Nevertheless, students perceived the module content was hard to understand. Kata kunci: Project-Based Learning, Input Aspect.


Introduction
Higher education sector is at the forefront in driving the nation's progress and development. This sector plays a key role in shifting Malaysia's position to become a high income economy by 2020. Economic competitiveness of a country is based on the skills of its workforce. The skills and competencies of the workforce are dependent on the quality of the country's education and training system. Technical and Vocational Education (TVE) is one of the various disciplines of education that can generate economic growth of a country [1]. TVE was designed to provide opportunities for students who have the tendency toward vocational fields and technology to fulfill the technical workforce. It is perceived as one of the crucial elements in enhancing economics of productivity [2,3]. Project-Based Learning was introduced in the Malaysian polytechnics curriculum in order to produce creative and innovative graduates. It is believed that students using Project-Based Learning are actively involved in authentic inquiry, knowledge construction, autonomous learning, scaffolding, and proposing creative solutions [4].
Project-Based Learning grew out of the architecture field that began in Italy during the late 16th century [5]. This method is becoming even more meaningful in today's society as educators are having diverse learners in their classrooms; students with different learning styles and abilities. Furthermore, Project-Based Learning builds on students' individual strengths and allows them to be creative and innovative in solving the problem. However, Project-Based Learning is full of challenges. Many teachers are not formally trained to handle Project-Based Learning. Especially for teachers who have never experienced Project-Based Learning before, Figure 1 The Framework of Concept Study Adaptation from CIPP Model by Stufflebeam (1971)

b. Research Objectives
The purpose of the study was to evaluate the Project-Based Learning (Final Project course J5012) at a Malaysian polytechnic, in the mechanical engineering department. Specifically, the objectives of this study was to evaluate the input dimension of the Project-Based Learning from the students and supervisors perspectives. This study was designed to answer these research questions. What are the perceptions of the students and supervisors regarding the input dimension of the Project-Based Learning?

c. Limitations of the Study
This study only involved the final-year students from the Department of Mechanical Engineering who took the final project course J5012 at one of the polytechnic in Malaysia IMPROVEMENT OF PROJECT-BASED LEARNING MODEL respondents consisted of supervisors and students from that polytechnic only. This study focused and evaluated the effectiveness of Project-Based Learning which was focused on one aspect of evaluation (input). Thus, the findings of this study may have a limited generalization.

Literature Review
Project-Based Learning has a long history. "Project" previously known as a method of institutionalized instruction. Project-Based Learning is rooted in constructivism and cooperative/ collaborative learning [12]. Constructivism has led to some learning models such as Project-Based Learning, Problem-Based Learning, Work-Based learning, or others models which are basically student-centered. Basically, Project-Based Learning is about solving real-world problem. Project-Based Learning is an instructional approach that emphasizes student-centered learning by assigning project [12,13]. Thus, in appearance, Project-Based Learning tends to be the appropriate approach to nurture higher order thinking. Project-Based Learning is a preferred method for many courses which the tasks are fit with this method [14]. In the nutshell, Project-Based Learning was introduced because of the ineffectiveness of traditional learning. It provides students with real-world learning, engages students' in investigation task, enhances students' inquiry process and produces a tangible artifact.
Students' participation of Project-Based Learning were what they think, do and interact [15]. By doing a project, students are trained to work in team, to plan, to organize, to negotiate and to execute the project at hand. However, students may have difficulty in Project Based Learning, such as, difficulty to define a research project, to find resources, to manage time, to collaborate with others and to revise the project [13]. In Project-Based Learning, supervisors are responsible to monitor the adequacy of resources, information, learning contexts, project time, and tasks [16]. The Buck Institute for Education believes that the important aspects of Project-Based Learning are: critical thinking, problem-solving, collaboration, and communication [17]. However, Project-Based Learning is a popular instructional strategy and it is believed to be a better method compared to other instructional methods. Finally, Project-Based Learning is assumed to be effective for teaching the students' complex processes like planning, communicating, problem-solving, and decision-making.

Methodology
Descriptive research involves providing careful descriptions of a phenomenon [18]. Thus, there are three main types of descriptive methods: observational methods, case-study methods, and survey methods [19]. Case study is widely used in organizational studies and across the social sciences [20]. The research design was based on the research objectives and the conceptual framework. The demographic variables in this study were ethnic, gender, residential area, and program areas. Additional supervisors variables included academic qualification, and years of teaching experience. This study employed CIPP (Context, Input, Process, and Product) evaluation model to assess the effectiveness of Project-Based Learning. However, this article focused on one of CIPP aspect, that was Input Aspect. The input dimension focuses on the module and supervisors' readiness.
There were two target populations in this study. The first population was final-project course (J5012) supervisors in the Mechanical Engineering Department in the selected polytechnic. The second population was final-year students who took the final-project course (J5012). The directory of supervisors and students was obtained from the Mechanical Engineering Department. The Mechanical Engineering Department was divided into three  [22]. The random sample of 70 students was selected from DPT, 42 students were selected from DAD, and 6 students were selected from DEM. The sample consisted of 118 students was perceived to be adequate based on the Krejcie and Morgan Sample size Table [22]. As mentioned earlier, a population of 76 supervisors was identified. However, only 43 supervisors were willing to participate in the study.

Result
The purpose of this case study was to assess the effectiveness of Project-Based Learning at the polytechnic by using the CIPP evaluation model. The focus of the Input dimension is the module and supervisors' readiness. The respondents of the study were divided into two groups, the students (n=118) and the supervisors (n=43). The demographic information of the students was illustrated in Table 1. The supervisors' profile was presented in Table 3. As illustrated in Table 1, the demographic information for the students enrolled in the course J5012 (final project course). They consisted of 72.9% male and 27.1% female students. The majority of the respondents were Malays (95.8%) and most of them (65.3%) lived in rural areas. More than half of the students (53.4%) were in the mechanical agriculture program, whereas 39.8% of the students were in the mechanical automotive program and only 5.9% were in the general mechanical program.  Table 2 shows the basic knowledge of Project-Based Learning from the students' perspective. Out of 118 respondents, more than half (55.1%) believe that they knew about Project-Based Learning. About one-fourth (27.1%) of the students were uncertain and the rest (17.8%) did not know about Project-Based Learning. However, majority of the students (74.6%) agreed that they were comfortable working with Project-Based Learning through medium e-SOLMS. Nevertheless, almost half (50.8%) of the respondents were rarely active (once in a week) in e-SOLMS to gain information regarding the final project.  Table 3. Illustrates the demographic profile of the supervisors. The respondents consisted of 79.1% males and 20.9% females. All of the supervisors were Malays and almost half of them (48.8%) lived in the city and the other half (48.8%) lived in rural areas. Majority of the supervisors (53.5%) were in the mechanical automotive engineering program. The supervisors from the mechanical agricultural engineering and the general mechanical engineering programs made up 25.6% and 20.9% respectively. In term of academic qualifications, 27.9% and 65.1% of the supervisors possessed Master degree and Bachelors degree, respectively and only 7% were Diploma graduates. Most (41.9%) of the supervisors had 0 to 5 years of teaching experience whereas 39.5% of the supervisors had more than 10 years experiences and only 18.6% had 6 -10 years of experience.  Table 4 presents basic knowledge of Project-Based Learning from the supervisors' perspective. Out of 43 supervisors, the majority of them (81.4%) claimed that they knew about Project-Based Learning, and only one supervisor (2.3%) did not know about Project-Based Learning. All the supervisors claimed that they were comfortable working with Project-Based Learning through e-SOLMS, although most (67.4%) of them only use e-SOLMS once a week. The second section of the questionnaire contained Likert items. Similar items were posed to the students and the supervisors. The findings were organized around the study's research questions. The following abbreviations were used: M for means, SD for standard deviations and ME for margins of error.

a. Research Question: What are the perceptions of the students and supervisors regarding the input dimension of the Project-Based Learning?
Regarding the input dimension of Project-Based Learning, this dimension focuses on project modules and the supervisors' readiness. Table 5 shows the means, margins of error, and standard deviations for items L1 to L7 that were posed to the students to answer Research Question 1 on the perceived effectiveness of the Project-Based Learning module. Regarding item L1, the students agreed (M=3.79; SD=0.82) that Project-Based Learning approach is suitable for project course J5012. Majority of the students claimed (M=3.73; SD=0.83) that they were confident to start their project by using Project-Based Learning approach (L2). On item L3, the students believed (M=3.70; SD=0.85) that the project management module was designed based on Project-Based Learning approach. With respect to student activity in the module (item L4), the students agreed (M=3.80; SD=0.85) that student's project activity was based on the module. However, the students were only slightly agreed (M=3.59; SD=0.72) that module content were easy to understand (item L5). In terms of the weekly goal setting, the students believed (M=3.76; SD=0.83) that the goal setting has helped them to focus on what to be accomplished (item L6). Finally, the students agreed (M=3.89; SD=0.92) that though the project was challenging at first but the students gradually succeeded to complete the project (item L7). However, the standard deviations for item L7 is relatively high which suggested a lack of consensus among the students regarding this item.  Table 6) were formulated to identify the students' perception regarding their supervisors' readiness for Project-Based Learning. The supervisor's readiness was an Input Dimension in the present study. Regarding the supervisors' knowledge (item L8), the students believed (M=4.07; SD=0.74) that supervisor' knowledge was appropriate with student's project title. For item L9, the students agreed (M=4.10; SD=0.87) that the supervisors have done effective supervisory duties. Finally, for item L10, majority of the students strongly agreed (M=4.25; SD=0.70) that the students and their supervisors determined the objectives which to be achieved in the project. At 95% confidence level, the margins of error for items L8 through L10 ranged from 0.06 to 0.08. 14 .77 Table 7 shows the means, margins of error, and standard deviations for items S1 to S7 that posed to answer Research Question 1 based on the supervisors' perception. Regarding item S1, the supervisors strongly agreed (M=4.58; SD=0.54) that Project-Based Learning approach is appropriate for project course J5012. The supervisor also strongly agreed (M=4.49; SD=0.50) that they could supervise the project effectively with Project-Based Learning approach (item S2). Regarding to item S3, the supervisor believed (M=4.23; SD=0.64) that the project management module was appropriate for the students. In relation to students project activity, the supervisor strongly believed (item S4, M=4.33; SD=0.52) that student's project activity was based on the module and they also claimed that module content was easy to understand (item S5, M=4.42; SD=0.54). In terms of item S6, the supervisors strongly agreed (M=4.42; SD=0.58) that the weekly goal setting in the module has helped their students to focus on what to be accomplished in the project. Finally the respondents were strongly believed (M=4.35; SD=0.57) that the module was designed based on Project-Based Learning approach (item S7). Total (Items S1 to S7) 4.40 .55 Note: In parenthesis is the margin of error at 95 % confidence level.
Items S8 through S10 (see Table 8) were formulated to identify the supervisors' readiness for Project-Based Learning. Regarding the supervisors' technical knowledge (item S8), the supervisors strongly agreed (M=4.21; SD=0.63) that their knowledge was appropriate with the students' project. For item S9, the supervisors strongly believed (M=4.28; SD=0.59) that they implemented supervisory duties effectively. Finally, the supervisors strongly agreed (M=4.44; SD=0.54) that the students determine their project goal with their supervisor. The relatively small standard deviation (items S8-S10) for the supervisors indicated a high consensus among the respondents. For item L1 through L10 (see Table 5 and Table 6), the margin of error for the students' data were lower than the margins error of the supervisors' data at the 95 % confidence level (see Table 7 and Table 8). One possible explanation is that the student sample size was larger (n=118) than the supervisor sample size (n=43). The margin of error of a confidence interval tends to increase when the sample size decreases and the standard deviation increases. The total means for the input dimension in terms of Project-Based Learning module, the supervisors' mean (M=4.40; SD=0.55) is higher than the students' mean (M=3.75; SD=0.83). This could implicate that the supervisors have more favourable attitude towards the module than their students. Regarding to supervisors' readiness, total mean for the supervisors (M=4.31; SD=0.58) is higher than the students (M=4.14; SD=0.77). This could conclude the supervisors believed that they had higher level of readiness than it was perceived by their students.
In comparing the students and the supervisors' data, the null hypothesis one (Ho1) was formulated. The independent samples t-test was carried out to determine whether there was a significant difference between the two mean scores. The result show that t (159) = -5.526, p = 0.00, the p-value was smaller than 0.05. Thus the Ho1 was rejected (see Table 9). The mean score of Input Dimension for the students was 3.86 and for the supervisors was 4.37. The difference between these two groups was significant at the 0.05 -level. It concluded that supervisors' mean is higher than students'. Thus, it indicated that the supervisors perceived better than the students regarding the items in the Input Dimension.

Discussion
In general, both students and supervisors stated that Project-Based Learning approach was appropriate for the Project Course J5012. This implies that the polytechnic should continue the implementation of Project-Based Learning. It is crucial to implement Project-Based Learning in order to enhance higher-order thinking [23]. In addition, Project-Based Learning could be implemented into other technical institutions and vocational colleges in Malaysia. In term of project module and supervisors' readiness as input dimensions of Project-Based Learning, few students did not know the module content. However, they knew that they had to follow the weekly activities in the project module. This clearly suggests that it is necessary to provide exposure about the module content to the students and their supervisors. In long term, the module needs to be reviewed and improvements be made. In addition, the project coordinator and the supervisors need to monitor closely the students' progress on their project work. This was supported by Zhang et al. 2011) who stated that the task force member (supervisors and project coordinator) should hold weekly meeting to provide updates on the students' implementation [24]. They also shared experiences and discussed any potential problems that needed to be addressed.
Essentially, the finding of this study may benefit the Ministry of Education in realizing Malaysian Goverment Transformation Program (GTP) and Economic Transformation Program (ETP). This study found that Project-Based Learning has enhanced the students' knowledge and skills in several critical aspects. Therefore, based on the results and limitations of this study, several recommendations for practice and future research are offered: