3. The sample
Beyond the general incompetence of the Hungarian university students, students of computer sciences perform even worse. In our project, based on the official Hungarian student management system (NEPTUN), we analysed the progress of the Software Engineering (SOE) BSc students of the Faculty of Informatics starting their studies in the 2011/2012 academic year. According to the sample curricula of the SOE BSc, this course is planned to be covered in six semesters.
Data gained from the NEPTUN are compared to the result of the TAaAS tests, administered on the first week of the students’ tertiary studies, and to the students’ results from secondary education, used in the acceptance process. The comparison of the TAaAS tests and the acceptance points plays crucial role in our analyses, since the TAaAS tests focus on the students’ knowledge in computer science, on their level in digital literacy and computational thinking, while the acceptance points rather show the students’ general intelligence and their social background. The question was which measurements would predict a successful progress in tertiary computer science education.
For the present studies the NEPTUN was downloaded October 18, 2014, after finalizing the 2013/2014 academic year.
3.1. The calculation of the acceptance level
In the Hungarian education system to enter tertiary education there are no entering exams.
Students finish their secondary education, and following it, they pass their graduation exams, which can be done on two levels – intermediate and advanced. There are two ways to calculate the acceptance point to tertiary education. (1) The points arrive from the students’ marks from secondary education in five subjects – Hungarian, Mathematics, History, a foreign language, and one selected subject –, 200 points, and from their results in the graduation exams, where the compulsory subjects are the same, only the selected subject can be different, another 200 points. (2) If the students think their marks in secondary education are not satisfactory they can double their results from the graduation exams, consequently 400 points. Beyond this semi-disciplinary calculation extra 80 points can be gathered from passing the advanced level graduation exams, foreign language exams at least on level B2, finishing in the top ten in the national disciplinary competitions, being pregnant and/or having children, being disabled, coming from a disadvantageous family and social background, or arriving from outside of Hungary with a Hungarian nationality. Institutes in tertiary education – referred to personality rights –do not have information which calculation is applied to the students, how they gain their acceptance points.
Altogether, 480 points can be collected from previous studies and social background. In addition, the calculating algorithm is under a continuous change, so students in consecutive years enter tertiary studies in different conditions. In the analysed year 200 points were the acceptance level, which means that only 47.1% of the available points were enough to start tertiary education.
We have to emphasize here that starting tertiary education in computer studies neither requires formal studies in informatics nor passing a graduation exam in the subject.
3.2. The TAaAS tests
The TAaAS tests are administered on the first week of tertiary studies in computer sciences and informatics (Csernoch & Biró, 2013). At the Faculty of Informatics we have three BSc courses. In this study we focus on the results of the SOE students entered tertiary education in September 2011, whose sample curricula is planned for six semesters. Consequently, they are the first students who took the TAaAS test and finish tertiary studies officially. The other two courses are planned for seven semesters, and their first results will arrive in the next semester. Considering the timing, from now on in each semester we will be provided with the progresses and results of the students of the following years, which can be compared to the results of the TAaAS tests.
The TAaAS tests consist of tasks considering the students’ level of computational thinking in general, their terminology use, and their algorithmic skills. For testing the students’ algorithmic skills we have selected two tasks from the national programming contests planned for 5–8th graders. The tasks contain four programs in pseudo code. One is different from the other three in its nature. In this task the output is limited to four possible choices, while the other three tasks are “What do the programs do?”
kind, where the students had to explain the programs in natural language sentences.
3.3. The characteristics of the different measures
In Table 1 we list the major characteristics of the retrospective (prerequisites), the discipline specific in-progress and the closing requirements. The retrospective results are calculated from the students’ secondary studies, from their graduation exams, and from their social background (Secondary education, Graduation exam, Social background, SGS). The in-progress results and requirements of
graduation are recorded in the university e-progress book (NEPTUN), entitled Students’ Result Assessment (SRA). The third group contains the discipline specific results gained from the TAaAS project (Csernoch & Biró, 2013).
Table 1. Comparison of the different premises during tertiary computer science studies prerequisites
SGS SRA TAaAS
– results from secondary studies – results from graduation exams – extracurricular results
– social background
– graduation results in Informatics and Mathematics – TAaAS results in traditional programming
– TAaAS results in non- traditional programming in-progress results
– the examination of study-paths – length of study
– results in exams
– results in labs and seminars – discipline specific, crucial subjects and exams
– comparisons of the results of the subjects
– post- and delayed post-tests in non-traditional programming environments
closing tertiary education – the output of studies – successful, semi-successful results: absolutorium, closing exam, diploma
– time frame – results
4. Research questions
– How the acceptance points are related to the in-progress and to the closing results in tertiary education?
– How the acceptance points are related to the discipline-specific knowledge?
– How much tertiary computer education would rely on knowledge built up in secondary education?
– How the graduation exams in informatics affect studies in tertiary education?
5. Results
In the 2011/2012 academic year 120 SOE students started their studies in tertiary computer science education, and 115 students completed the TAaAS test. The section of the two sets consists of 113 students. The NEPTUN downloaded in 2014 October would provide information on the students’
current statuses at the university (Table 2). Based on these data we have defined three groups of the students:
– finished studies in time (finished),
– left the university earlier, previous to the predicted time (deleted), – still at the university (active).
Table 2. Students’ classification based on their statuses after seven semesters
status frequency percent
active 47 41.6
deleted 53 46.9
finished 13 11.5
It is clear from Table 2 that only very few students (11.5%) were able to finish their studies in time, almost half of them left their studies earlier, and more than 41% of them is still in the education system.
In the following analyses we compare the students’ results from secondary education (Table 3) and from the TAaAS project (Table 4). The acceptance points (AP) are retrieved from NEPTUN, the graduation exam results in Informatics and Mathematics at intermediate and advanced levels (GEII, GEMI, GEIA, GEMA, respectively) from the TAaAS project. The acceptance points show significant differences between the groups (Table 3, AP) (Kruskal-Wallis-test, p = 0.005). However, the paired comparison reveals that there is only difference between the active and the deleted groups. This means that the acceptance points are not able to distinguish between the finished and the other two groups. This finding is supported by the analyses of the students’ results in the TAaAS tests, where their knowledge in programming was tested (Table 4).
Considering the results of the graduation exams both in informatics (Table 3, GEII and GEIA, at intermediate and advanced level, respectively) and mathematics (Table 3, GEMI and GEMA, at intermediate and advanced level, respectively) no significance difference was found at either levels. This means that tertiary education accepts students whose knowledge in the two most important subjects is not clear, and institutes are not prepared for the students’ lack of knowledge.
Table 3. Students’ results from secondary education
AP GEII GEIA GEMI GEMA
max 480 100 100 100 100
active 376.62 85.67 77.14 81.32 87
deleted 350.71 84.95 68.87 74.85 –
finished 383.85 83.50 80.00 83.78 –
Among the traditional programming tasks of the TAaAS project a 4-output task (Table 4, 4O) and three “What do the programs do?” type of tasks (Table 4, WA, WB, and WC) were included. In the 4- output task no significant different was found between the three groups of students (Kruskal-Wallis-test, p = 0.3694). However, all the WA, WB, and WC tasks show significant differences (Kruskal-Wallis-test, p = 0.0035, p = 0.0044, and p = 0.0024, respectively). The paired comparison revealed that these differences are clearly detectable between the finished and the deleted groups in all the three tasks.
Table 4. Students’ results in traditional programming environments in the TAaAS project
4O WA AB WC
active 68.32 68.62 52.13 46.28
deleted 66.67 47.64 39.15 30.19
finished 86.32 86.54 82.69 71.15
The active students seem being those who can learn anything, hopefully, also programming, but it takes longer. The finished students were no better in the general subjects than the others, but arrived with higher level of programming knowledge. It can also be predicted that our education system in computer science is not prepared for students who are well educated in any other subject except programming. This finding suggests that we can succeed only in teaching those students who arrive ready-made, and in any other cases both students and tertiary education struggle. We have to be faced with the fact that secondary education does not develop the students’ computational thinking and algorithmic skills to the level which is required in computer science education. It is the responsibility of tertiary education to apply proven effective methods (Mayer, 1981; Kirschner, 2006; Hattie, 2012) to prepare students for high level programming.
6. Conclusion
In the TAaAS+ project we compare the different sources of students’ measurement in tertiary computer science education. The reason of our doing is to find explanation for the high attrition rate of computer science students. In the 2014/2015 academic year we have reached the first possible finishing students participating in the TAaAS project, launched in September 2011.
The results of the analyses clearly present that those students who finished their studies according to the prescheduled curricula, arrived into tertiary computer science education with firm knowledge in programming. It has also been found that this result is neither predictable from the students’
acceptance grades nor from their results in the graduation exams in Informatics and Mathematics. Being aware of these findings we can conclude that both students and tertiary education are unprepared for students with high level intelligence but low level programming knowledge, and this shortage definitely would explain the extremely high attrition rate in computer science education.
Acknowledgement
The research was supported partly by the TÁMOP-4.2.2.C-11/1/KONV-2012-0001 and TÁMOP- 4.1.2.B.2-13/1-2013-0009, SZAKTÁRNET projects. The projects have been supported by the European Union, co-financed by the European Social Fund. The research was supported partly by the Hungarian Scientific Research Fund under Grant No. OTKA K-105262.
References
Biró, P. & Csernoch, M. (2013). Deep and surface structural metacognitive abilities of the first year students of Informatics. In: 4th IEEE International Conference on Cognitive Infocommunications.
2013 Proceedings. IEEE, Budapest, pp. 521-526.
Hattie, J. (2012) Visible Learning for Teachers. Routledge.
Kirschner, P.A., Sweller, J., Clark, R.E. (2006) Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching. Educational Psychologist, 41(2), pp. 75–86.
Mayer, R. E. (1981). The Psychology of How Novices Learn Computer Programming. ACM Computing Surveys, vol. 13 (1), pp. 121–141.
Palkovics, L. (2014) Nem a bolognai rendszer rossz, hanem a magyarországi megvalósítása.
http://www.hirado.hu/2014/10/18/palkovics-nem-a-bolognai-rendszer-rossz-hanem-a- magyarorszagi-megvalositasa/, retrieved February 2, 2015.
Palkovics, L. (2015) Nem egyetemistából, hanem diplomásból kellene több az államtitkár szerint.
http://eduline.hu/felsooktatas/2015/1/13/Palkovics_nem_hallgatobol_kell_tobb_hanem_d_L5D20 M, retrieved February 2, 2015.