Title:  Improving Student Learning Through Hierarchical Reinforcement Learning Induced Pedagogical Policies

Location:  Remote Exam, Zoom Link:
https://ncsu.zoom.us/j/234252650

Examination Committee:
Dr. Min Chi (Chair)
Dr. Roger Azevedo (Graduate School Representative)
Dr. Tiffany Barnes
Dr. James Lester

All members of the university community are invited via the Zoom link provided above.

* * *
Abstract:  In interactive e-learning environments such as Intelligent Tutoring Systems, there are pedagogical decisions to make at two main levels of granularity: whole problems and single steps. Here, we focus on making the problem-level decisions of worked example (WE) vs. problem solving (PS) and the step-level decisions of elicit vs. tell. More specifically, we first investigate the impact of decision granularity on student learning and then explore taking granularity into account in data-driven pedagogical policy induction. 
In a series of classroom studies, we explored the impact of three types of granularity: problem-level only (Prob-Only), step-level only (Step-Only) and both problem and step levels (Both) on student learning. Results showed that Prob-Only can be effective for Low incoming competence students, Step-Only can be effective for High ones, and Both can be effective for both Low and High students. This suggests that granularity indeed can have an impact on student learning. However, overall there was no significant difference among the three granularity conditions. One possible reason is that the pedagogical decisions there were randomly made rather than adaptively.

Prior research has shown that effective pedagogical decision-making can significantly improve student learning. In recent years, there has been growing interest in applying data-driven approaches to induce pedagogical policies directly from student-system interaction logs. However, most of the prior works treated all system decisions equally, or independently without considering the long-term impact of higher-level actions or the interaction of decisions made at different levels. Here, we apply reinforcement learning (RL) to induce pedagogical policies that make decisions at different granularity levels and evaluate their effectiveness in empirical classroom studies.

We first applied RL to induce a problem-level and a step-level policy and evaluated their effectiveness in a classroom study by comparing them with two random yet reasonable policies, one at the problem-level and one at the step-level. Results showed that there was no significant difference between the two RL conditions and none of them was significantly more effective than the two random baseline conditions. The results suggest that RL induced policies that make decisions at a single granularity level may not always be effective.

On the other hand, results from the granularity study showed that Both-level decisions can benefit more students than either the problem-level or step-level. It is possible that considering both levels of decisions in RL policy induction can lead to effective policies. Therefore, we propose and apply an offline, off-policy Gaussian Processes based Hierarchical Reinforcement Learning (HRL) framework to induce a hierarchical pedagogical policy that makes decisions at both the problem and step levels. In an empirical classroom study, the HRL policy was compared with a Deep Q-Network (DQN) induced step-level policy and a random yet reasonable step-level baseline policy. Results showed that the HRL policy was significantly more effective than the DQN induced policy and the random baseline policy. The results suggest that by taking decision granularity into account, HRL indeed can induce effective policies that can significantly improve student learning.