In this work, periodically captured thermal images from infrared (IR) thermography were utilized as health indicators and integrated into a hybrid deep learning model to non-destructively predict the fatigue life of fiber–polymer composites under continuous-fatigue, interrupted-fatigue, and creep-fatigue loading patterns for structural health monitoring (SHM). The hybrid model was based on a convolutional recurrent neural network (CRNN). This model used a ResNet50 architecture for feature extraction from thermal images, followed by dimension reduction using principal component analysis (PCA), and finally a multi-layer recurrent neural network (MLRNN) to address the necessity of considering load history effects for accurate fatigue life prediction. The performance of the trained model was evaluated using the coefficient of determination (R2), calculated by regressing the predicted fatigue life values against the experimental ones at various input data points, corresponding to different percentages of thermal images used per experiment. It was observed that by using thermal images obtained from the first 20% of each experiment, the model reached R2 score of more than 0.90 on the test dataset. Accuracy gradually increased when up to 40% of the initial fatigue life was incorporated into the model, after which it remained stable. In general, this trend was consistent across all loading patterns. To evaluate the robustness of the MLRNN model, it was trained using different sequence lengths, and tested under various subsampling scenarios. It was observed that the best performance was achieved with a sequence length of 5, which enabled the model to well predict the fatigue life even in a very sparse subsampling scenario. The proposed approach demonstrated the potential of combining thermal analysis and machine learning methods to accurately predict the fatigue life of composites in the early stages of their service life, even with limited available data.