Facial gestures and movements play a critical role in communicating, eating, and expressing emotions, making the assessment of oro-facial functions vital in clinical practice. Neurological conditions such as stroke and facial palsy significantly impact these movements, necessitating accurate differentiation for appropriate diagnosis. This paper proposes an automated approach to classifying facial asymmetry in stroke and peripheral facial paralysis, leveraging computer vision and machine learning algorithms. Using public datasets like the Toronto NeuroFace and Massachusetts Eye and Ear Infirmary (MEEI), we employed facial landmark localization, comprehensive feature extraction, and classification techniques to discern subtle variations in facial movements and expressions across different conditions. Our approach achieved promising results, with accuracy up to 88% to distinguish facial impairments due to stroke and facial palsy. Tasks such as lip spreading, blinking, and eyebrow-raising demonstrated high accuracy, aligning with previous findings. Our study lays the groundwork for improving the diagnosis and treatment of oro-facial impairments using machine learning, with potential applications in clinical and emergency settings to enhance patient care and diagnostic accuracy.