NB
N. Blagoev
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
2 records found
1
Go With The Flow: Fault-Tolerant Decentralized Training of Large Language Models
Decentralised Training of Large Language Models
Motivated by the emergence of Large Language Models (LLMs) and the importance of democratizing their training, we propose Go With The Flow, the first practical decentralized training framework for LLMs. Differently from existing distributed and federated training frameworks, Go With The Flow enables the collaborative training of an LLM on a set of heterogeneous client nodes that dedicate different resources for an undefined amount of time. Our work addresses node churn, i.e., clients joining or leaving the system, and network instabilities, i.e., network links becoming unstable or unreliable. The core of Go With The Flow is a decentralized flow algorithm that finds the most effective routing to train a maximum number of microbatches with a minimum delay. We extensively evaluate our work on LLama-like and GPT-like models, compare it against the prior art and achieve up to 45\% training time reduction in realistic and challenging scenarios of heterogeneous client nodes distributed at 10 different geographic locations with a high node churn rate. We further demonstrate resilient training in such challenging environments, without sacrificing convergence.
...
Motivated by the emergence of Large Language Models (LLMs) and the importance of democratizing their training, we propose Go With The Flow, the first practical decentralized training framework for LLMs. Differently from existing distributed and federated training frameworks, Go With The Flow enables the collaborative training of an LLM on a set of heterogeneous client nodes that dedicate different resources for an undefined amount of time. Our work addresses node churn, i.e., clients joining or leaving the system, and network instabilities, i.e., network links becoming unstable or unreliable. The core of Go With The Flow is a decentralized flow algorithm that finds the most effective routing to train a maximum number of microbatches with a minimum delay. We extensively evaluate our work on LLama-like and GPT-like models, compare it against the prior art and achieve up to 45\% training time reduction in realistic and challenging scenarios of heterogeneous client nodes distributed at 10 different geographic locations with a high node churn rate. We further demonstrate resilient training in such challenging environments, without sacrificing convergence.
As automated negotiating agents become more and more part of our daily life, additional care needs to be taken that the agents can negotiate fairly. Humans each have their own intrinsic view on fairness, which affects the negotiation processes and the degree to which the outcome is viewed as satisfactory. However, most current agents are built around a specific notion of fairness, which could be potentially unwanted in certain scenarios or by some individuals. This paper presents a generic framework through which varying fairness notions can be implemented. The performance of the framework is tested through a selection of proposed agents with different fairness views. The results show that the agents are able to reach an agreement on the pareto frontier that is fair given their notion. Thus they do not sacrifice optimality for fairness.
...
As automated negotiating agents become more and more part of our daily life, additional care needs to be taken that the agents can negotiate fairly. Humans each have their own intrinsic view on fairness, which affects the negotiation processes and the degree to which the outcome is viewed as satisfactory. However, most current agents are built around a specific notion of fairness, which could be potentially unwanted in certain scenarios or by some individuals. This paper presents a generic framework through which varying fairness notions can be implemented. The performance of the framework is tested through a selection of proposed agents with different fairness views. The results show that the agents are able to reach an agreement on the pareto frontier that is fair given their notion. Thus they do not sacrifice optimality for fairness.