Shape Memory Alloys (SMAs) have gained significant attention due to their high work density in order to fabricate compact and lightweight actuators. However, the major drawback of thermally activated SMAs is the time that takes for these materials to cool down. This study aims to explain an effective passive way to decrease the cooling time of Shape Memory Alloys (SMAs) based actuators. The idea is to use highly conductive elastomer layers to improve the heat conduction and convection in SMA sheet. Different types of elastomer in terms of thermal conductivity and thickness were tested in order to improve cooling time. The Finite Element Method is performed for the SMA sheet sandwiched between elastomer layers with and without silicone in between as well as for SMA without elastomer. The results show approximately three times improvement for the actuator with elastomer layers but without silicone in between compared to SMA itself.