Understanding cognitive states of human under different difficulty levels is useful in improving human-human and human-machine interactions. For example, a crucial factor in designing games is maintaining the engagement of players. An ideal scenario would be decoding the cognitive states of players to dynamically adjust the difficulty level of the game. The same idea can also be applied in learning, where an easy level does not provide a sense of achievement while a too difficult level frustrates students. Therefore, the main idea of improving the interactions is reaching and maintaining at an optimal difficulty level.
The optimal difficulty level is, however, not simple to define and is also a function of skills which varies with time. An automatic approach is to build the interaction loop by the employment of a cognitive state decoder which helps dynamically adjusting the difficulty level. Ideally, the level will converge to the optimal level after a certain time. Even if the skills are improved afterward, the cognitive states reflecting the same objective difficulty level will be different as the person is more skilled and more confident.

Based on this idea, two online experiments were conducted in this thesis for demonstrating the feasibility. The behavioral outcomes using cognitive state decoders were compared with those based on behaviors or decisions of subjects, i.e., ground-truth-like baselines. The outcomes are promising; several subjects had similar results between the two conditions. In some rare cases, decoding cognitive states even outperformed the ground-truth-like condition. These pieces of evidence support the idea of decoding cognitive state to improve the interactions. The backbone of this approach is decoding the cognitive states of interest from physiological signals. Electroencephalography (EEG) is the selected physiological signal for its non-invasiveness and quick response. Based on the defined protocols, this thesis proposed a two-stage processing method and compared it with previously highlighted engagement and attention indices as well as some state-of-the-arts classifiers. The open-loop validation supports the proposed decoding framework, which is further validated by one experiment with both open-loop and closed-loop settings, as well as another open-loop experiment involving a second task.

In order to improve the interaction, the temporal dynamics of cognitive states should be well captured. Previous literature of cognitive state computing mostly focuses on setups where the cognitive states should remain constant over a certain period. This thesis further probes the dynamics of cognitive states around the onset where the cognitive states are supposed to change. The analysis suggests that the latencies introduced by either the decoder or subjects are on a scale of seconds for the designed protocol. This is not a negligible scale if the targeted application requires a quick reaction or a precise timing from the decoder.

In sum, the closed-loop experiments support the main idea of improving interactions through decoding cognitive states from EEG signals. The proposed decoder has been shown effective in decoding difficulty-related cognitive states, for which one should also be careful about the latencies.