Go with the Flow
Churn-Tolerant Decentralized Training of Large Language Models
Nikolay Blagoev (University of Neuchâtel, Worker Thread)
Bart Cox (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Jérémie Decouchant (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Lydia Y. Chen (University of Neuchâtel)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Motivated by the emergence of large language models (LLMs) and the importance of democratizing their training, we propose GWTF, the first fully decentralized churn-tolerant practical training framework for LLMs. Differently from existing distributed and federated training frameworks, GWTF enables the efficient collaborative training of a LLM on heterogeneous clients that volunteer their resources. In addition, GWTF addresses node churn, i.e., clients joining or leaving the system at any time, and network instabilities, i.e., network links becoming unstable or unreliable. The core of GWTF is a novel decentralized flow algorithm that finds the most effective routing that maximizes the number of microbatches trained with the lowest possible delay. We extensively evaluate GWTF on GPT-like and LLaMalike models and compare it against the prior art. Our results indicate that GWTF reduces the training time by up to 45 % in realistic and challenging scenarios that involve heterogeneous client nodes distributed over 10 different geographic locations with a high node churn rate.