Large Language Models (LLMs) have gained significant attention in recent years due to their potential to revolutionize various industries and sectors. However, scaling LLMs further requires access to substantial linguistic resources that are being rapidly depleted. Moreover, the available text sources such as emails, social media interactions, or internal documents may contain private information, making them susceptible to misuse. On-premises Federated Learning (FL) with privacy-preserving model updates is an alternative avenue for LLMs’ development that ensures data sovereignty and enables peers to collaborate while ensuring that the sensitive parts of their private data cannot be reconstructed. However, in the case of large-scale FL, there is also a risk of malicious users attempting to poison LLMs for their benefit. The problem of protecting the learning procedure against such users is known as Byzantine-robustness, and it is crucial to develop models that perform accurately despite faulty machines and poisonous data. Designing FL methods that are simultaneously privacy-preserving and Byzantine-robust is challenging. However, ongoing research suggests ways to incorporate the differentially-private Gaussian mechanism for privacy preservation and spectral robust-averaging for robustness. However, whether this approach applies to LLMs or whether a major player in the domain would emerge and capture all private information sources through network effects remains to be seen.