The evolution of edge computing has advanced the accessibility of E-health recommendation services, encompassing areas such as medical consultations, prescription guidance, and diagnostic assessments. Traditional methodologies predominantly utilize centralized recommendations, relying on servers to store client data and dispatch advice to users. However, these conventional approaches raise significant concerns regarding data privacy and often result in computational inefficiencies. E-health recommendation services, distinct from other recommendation domains, demand not only precise and swift analyses but also a stringent adherence to privacy safeguards, given the users’ reluctance to disclose their identities or health information. In response to these challenges, we explore a new paradigm called on-device recommendation tailored to E-health diagnostics, where diagnostic support (such as biomedical image diagnostics), is computed at the client level. We leverage the advances of federated learning to deploy deep learning models capable of delivering expert-level diagnostic suggestions on clients. However, existing federated learning frameworks often deploy a singular model across all edge devices, overlooking their heterogeneous computational capabilities. In this work, we propose an adaptive federated learning framework utilizing BlockNets, a modular design rooted in the layers of deep neural networks, for diagnostic recommendation across heterogeneous devices. Our framework offers the flexibility for users to adjust local model configurations according to their device’s computational power. To further handle the capacity skewness of edge devices, we develop a data-free knowledge distillation mechanism to ensure synchronized parameters of local models with the global model, enhancing the overall accuracy. Through comprehensive experiments across five real-world datasets, against six baseline models, within six experimental setups, and various data distribution scenarios, our architecture demonstrates unparalleled performance and robustness in terms of both accuracy and efficiency.