Decoding the subjective perception of task difficulty may help improve operator performance, i.e., automatically optimize the task difficulty level. Here, we aim to decode a compound of cognitive states that covaries with the task difficulty level. We designed a protocol composed of two different subtasks, flying and visual recognition, to induce different difficulty levels. We first showed that electroencephalography (EEG) signals can be a reliable source for discriminating different compound states. To gain insight into the underlying components in the compound states, we examined the attentional index and engagement index as in our previous study. We showed that, first, attention and engagement are essential components but fail to provide the best accuracy, and, second, our model is consistent with our previous study, which means that lateralized modulations in the α bands are representative of the flying task. We also analyzed a practical issue in the design of adaptive human–machine interaction (HMI) systems, namely, the latency of changes in the user’s compound state. We hypothesized that the EEG correlates of the task difficulty level do not instantaneously reflect the changes in the task difficulty. We validated the hypothesis by measuring the time required for our decoders to provide stable accuracy after the task changed. This amount of time, or latency, could be as high as ten seconds. The results suggest that the latency of changes in the user’s compound state between different tasks is a factor that should be taken into account when building adaptive HMI systems.