This work proposes a transfer learning-based encoder-decoder framework to predict the relationship between loading conditions and residual stiffness in carbon fiber reinforced composites and adhesives. The encoder, built from a Convolutional Neural Network (CNN) and Bidirectional Long Short-term Memory (Bi-LSTM), extracts time-series loading signals into latent variables and captures their dependencies. The decoder employs a Multilayer Perceptron (MLP) to map these latent features to residual stiffness. Transfer learning strategy is used to account for individual variability and further improve accuracy. The model's effectiveness and robustness are validated through random and constant loading fatigue experiments from two different material systems. Under random fatigue data, the model demonstrates strong learning capabilities. Under random fatigue data, the model demonstrates strong learning capabilities. Compared to classical models like Support Vector Machine (SVM) and Random Forest, or simpler deep learning architectures like individual CNN and Bi-LSTM networks, the proposed architecture shows enhanced prediction accuracy and regression results, achieving a Root Mean Square Error (RMSE) of 0.154 and a Coefficient of Determination (R2) of 0.931. In constant amplitude fatigue datasets, the model accurately identifies different materials and exhibits satisfactory robustness when reasonable training dataset size is used.