In my thesis, I will present LabEX, a system for instructors to understand student progress and learning experience during programming lab sessions. LabEX uses statistical techniques that help distinguish struggling students and understand their degree of struggle. LabEX was evaluated in an entry-level programming course taken by more than two hundred students at UNIST, proving that it enhances the quality of programming lab sessions.
Introduction
An instructor can easily get a wrong impression of how students are doing based on his/her limited, biased interaction with students - students who ask very basic questions or those who leave the session early after completing all the tasks. More importantly, it is even more difficult for lab instructors to qualitatively assess and help improve student solutions during the lab session, as feedback is usually provided only to those participants who ask for help. This dissertation presents LabEX, a lab session monitoring system that helps instructors better understand how students are learning in real time by providing them with a comprehensive overview of the lab sessions.
LabEX uses more detailed information about student interactions and time spent on task to rank struggling students. A code clustering technique is used by LabEX to group similar solutions and display them to lab assistants to aid them in qualitative review by students. Grouping similar solutions helps assistants easily identify creative solutions and share them with participants.
We believe that such in-situ feedback significantly improves participants' learning experience in the laboratory. The research shows that LabEX significantly improves the lab session experiences for both laboratory assistants and students.
Background and Related Works
Design Goals
Jupyter's interface provides students with greater ease in writing, running, and submitting their solutions, unlike the UNIX command-line environment commonly used in typical programming lab sessions. Ideally, teachers should allocate their help to students who are struggling more than to students who ask a lot of questions. Therefore, it is necessary to enable assistants to easily identify students who are struggling and need help.
The quality of code matters in programming and it would be ideal for lab staff to qualitatively review all students' codes. Since it is not feasible to read all lines of code, LabEX provides an efficient tool to help assistants review more code with less effort.
Understanding Struggling Students
Estimating Distribution of Problem Solving Times
Estimating Parameters with Order Statistics
Although it is not possible to obtain the distribution parameters in advance, we can estimate the parameters incrementally as we acquire data points during a laboratory session by exploiting the properties of order statistics [2]. The k data points are not sampled uniformly, but they are the smallest k samples and thus correlated with each other. To account for the sampling bias, we use order statistics to estimate the distribution parameters.
Given the probability distribution of X and n random samples from X, the kth order statistic of n samples is the distribution of the kth smallest value in the n samples. Instead of maximizing the upper likelihood to get an accurate estimate, we calculate the MLE for individual data points to get multiple possible estimates; then we choose the one that fits best. Since there are two unknowns, two data points are required; we randomly select two data points ({xi,oi},{xj,oj}) to calculate the value.
Because some outliers may exist in the first few data points, we exclude them in the estimation if the parameters calculated with these data points are significantly different from other estimated values. After collecting a sufficient number of candidate estimates of µ and σ, we verify how well they describe the k data points.
Modeling Students Struggle
To obtain candidate estimation values, we precompute MLE for the order statistics for the standard log-normal distribution with Monte-Carlo simulation. We then calculate µ and σ with the completion time xi and oi using the equation ln(oi) = (ln(xi)- µ)/ σ. The problem lead time distributions we estimate can be used for multiple purposes.
In LabEX we mainly use them in two ways; to infer the difficulty of problems and identify struggling students. After estimating the distribution of task completion time, we select a threshold time that represents the lowest X% (default 30%) students in the distribution; we consider only the bottom 30% as potentially struggling students. To do this, we define the degree of struggle using the three measures: the time spent on the task, the frequency of executions, and the amount of code change between executions.
The values of the three measures are first normalized to standard normal/log-normal distributions; We calculate grades for all tasks for struggling students and sort them by grades so lab staff can identify who needs the most help.
Qualitative Code Review
Clustering Students’ Solution
The values of the three measures are first normalized to standard normal/log-normal distributions; .. then we calculate the distance of how far the three values are from the mean of the distributions. Although the code gives correct answers for all positive integer inputs, the code is generally considered poor due to its style. His solution reveals that 1) he is not aware of the modulo operator in Python, 2) he does not fully understand how if statement works (unnecessary checks in line 6), 3) he is confused about loop checks ( unnecessary continue statement in line 9), and 4) he doesn't know that making special cases is not good style (line 2).
Our algorithm is based on hierarchical clustering [16] and can thus easily control the granularity of the clustering. Then the distances (default edit distance [17], but tree edit distance [18] can also be used) of all signatures are calculated. This is repeated while the retrieved distance is less than a certain threshold, or all distances are retrieved.
Although the algorithm is not as accurate or robust as other AST fingerprinting algorithms, it is lightweight and effective in our context. The first two steps of the algorithm – creating signatures and calculating distances – can be performed incrementally as a new solution is submitted.
Comparing Code Clustering Algorithms
Reviewing with Clusters
LabEX User Interface
Students Interface
In LabEX, students can easily retrieve their previous experiments, which is useful for both students and lab staff when assisting them.
Lab Staff Interface
Lab staff can click on the individual box representing a specific student and assignment to see the student's solution code for the assignment. The list is sorted by difficulty level and contains the student's name, their desk number, the assignment ID, and the time spent on the assignment. You can click on an entry in the list to view the student's current and previous solution codes.
As previously described, similar solutions are grouped together so that an instructor can effectively review solution code. The site provides a slider for adjusting the granularity of groupings, so an instructor can group solutions more (or less) aggressively as needed. The code review interface can be used during lab sessions to identify poor solutions and provide real-time feedback to students.
Moreover, the interface helps in finding creative/interesting solutions so that they can be shared with the rest of the students during or after the lab sessions. If an instructor decides to include the student's code in the list of verified solutions, they simply click a button next to the student's code.
Mobile View
On the right side we list the struggling students that LabEX identifies based on the statistical model.
Evaluation
- Case Study
- Real-World Experiment
- Methodology
- Qualitative Observations
- Post-Study Interview
- Discussions and Limitations
In collaboration with the lab instructor and TAs, we adopted LabEX in the lab sessions for a total of six weeks. The laboratory sessions are conducted in a naturalistic manner; there were no specific tasks for laboratory staff or students; and we mostly observed. In the interviews, we asked them to discuss some of their interactions that we observed during the laboratory sessions.
After activating LabEX monitoring interface in the third week, we proposed to try the scheme again and limit the number of questions in the lab sessions. The lab instructor was hesitant at first, but when the scheme was implemented, it was very successful. With the progress summary in LabEX, the lab staff were fully aware of the overall progress and could identify struggling students to help them; the instructor and TAs seemed comfortable running the lab sessions.
Perhaps this is because they took the same lab class just a few semesters ago; they may have felt frustration and embarrassment before in lab sessions and may want to help students in similar situations. All the lab staff enjoyed using the system and helping students who needed more help.
Conclusion
ChangJoo Ah for his help through the live testing of LabEX in programming lab sessions. Last but not least, I would like to thank my family: my mother Safieh Aghamirkarimi and my wife Marcela, for their enormous support throughout my life. Guo, “Codeopticon: Real-Time, One-to-Many Human Tutoring for Computer Programming,” in Proceedings of the 28th Annual ACM Symposium on User Interface Software.
Kennedy, “Visualizing patterns of student engagement and performance in MOOCs,” Proceedings of the Fourth International Conference on Learning Analytics And Knowledge, vol. Schneider, “Deconstructing disengagement: analyzing subpopulations of learners in massive open online courses,” in Proceedings of the Third International Conference on Learning Analytics and Knowledge, 2013. Zhao, “DropoutSeer: visualizing learning patterns in massive open online courses for reasoning and predicting failure,” in Visual Analytics, Science and Technology (VAST), 2016.
Guibas, "Syntactic and functional variability of a million code submissions in a machine learning mooc," i AIED Workshops Proceedings Volume, 2013. Roussel, "Syntax tree fingerprinting for source code similarity detection," i IEEE 17th International Conference on Program Comprehension, 2009 .
