This repository contains the code for the paper "Cooperation and Fairness in Multi-Agent Reinforcement Learning", which introduces a method to incorporate fairness for multi-agent navigation tasks. The method builds on the InforMARL framework and extends it to ensure fair cooperation in scenarios like MPE's simple spread (coverage) and formation.
Paper Link: π π https://arxiv.org/abs/2410.14916
Paper Website: π» π Website
- Introduction
- Features
- Environment
- Installation
- Usage
- Code Structure
- Results
- Citation
- Acknowledgements
The Fair-MARL method addresses fairness in cooperative multi-agent reinforcement learning (MARL), where agents must not only achieve task success but also do so in a manner that promotes fairness in navigation for all agents. This is particularly relevant in tasks involving navigation, such as:
- Coverage Navigation: Agents must spread out to cover target locations.
- Formation: Agents must arrange themselves in specific formations.
Our approach extends the InforMARL framework to include fairness in the goal assignment and rewards, enabling agents to learn policies that are both efficient and fair.
- Fair Goal Assignment: Incorporates fairness principles in the goal assignment process.
- Fairness Reward: Includes a fairness reward that is based on agents's distance traveled.
The code is implemented for use with the Multi-Agent Particle Environment (MPE), specifically for tasks like simple_spread
. The environment simulates continuous spaces where agents must collaborate to achieve a common goal.
You can find the MPE environment here: Multi-Agent Particle Environment (MPE)
To get started with the Fair-MARL method, clone this repository and install the required dependencies. Ensure you have pip version pip==23.1.2. Installing torch beforehand ensures the correct installation of other components.:
git clone https://github.com/yourusername/fair-marl.git
cd fair-marl
pip install torch==2.0.1
pip install -r requirements.txt
NOTE: Using a conda environment is preferred. Please use the following command to create a conda environment with the correct python version.
conda create -n fairmarl python=3.11
- Python 3.11+
- PyTorch
- OpenAI Gym
- Multi-Agent Particle Environment (MPE)
Training scripts are located in the folder train_scripts
. To train the Fair-MARL agents on the coverage tasks the command is alongg the following lines:
python -u onpolicy/scripts/train_mpe.py \
--project_name "test" \
--env_name "GraphMPE" \
--algorithm_name "rmappo" \
--seed 2 \
--experiment_name "test123" \
--scenario_name "navigation_graph"
This will train agents using the Fair-MARL method on the chosen task (navigation_graph
in this case). Additional parameters for training, such as the number of agents, can be modified in the configuration file or passed as command-line arguments.
NOTE: Please note that for training we have enabled wandb logging by default. Please inspect your logging mechanishm or use the flag
--use_wandb
to prevent wandb longging.
After training, you can evaluate the trained agents by running:
python onpolicy/scripts/eval_mpe.py \
--model_dir='model_weights/FA_FR' \
--render_episodes=2 \
--world_size=3 \
--num_agents=3 \
--num_obstacles=0 \
--seed=0 \
--num_landmarks=3 \
--episode_length=50 \
--use_dones=False \
--collaborative=False --model_name='FA_FR' \
--scenario_name='nav_base_formation_graph_nogoal' \
--goal_rew=30 \
--fair_rew=1 \
--save_gifs \
--use_render \
--num_walls=0 \
--zeroshift=5 \
--min_obs_dist 0.5
This will load the trained model and evaluate its performance in the specified environment. Additional parameters for evaluation, such as the number of agents, can be modified in the configuration file or passed as command-line arguments.
.
βββ README.md # Project Overview
βββ license # Project license file
βββ requirements.txt # Dependencies
βββ train_scripts # Training Script
βββ eval_scripts # Sample Evaluation Script using Trained models
βββ model_weights/ # Directory for saving trained models
βββ utils/ # Configuration files for different environments and algorithms
βββ multi-agent/ # Fair-MARL specific code
β βββ custom-scenarios # Core Fair-MARL Algorithm
β βββ navigation_environment.py # Fairness-based goal assignment logic
β βββ agent.py # Multi-agent definitions
β βββ utils.py # Utility functions
βββ onpolicy/ # MPE environment files (if necessary)
-
multiagent/custom_scenarios/navigation_graph.py
: Implements the Fair-MARL reinforcement learning algorithm. -
marl_fair_assign.py
: Contains the logic for fair goal assignment. -
We have created adocument detailing the network architecture here:
-
We have created a document for easy understandng of our codebase here
Here we summarize the results from the experiments. The Fair-MARL method achieves fairer goal assignment and better cooperation compared to baseline methods. For example:
- Coverage navigation: Fair-MARL agents spread out more equitably to different target locations.
- Formation: Agents arrange themselves in stable formations while ensuring fairness in positional assignments.
For detailed results and analysis, please refer to our paper.
If you find this repository helpful in your research, please cite the corresponding paper:
@article{aloor2024cooperation,
title={Cooperation and Fairness in Multi-Agent Reinforcement Learning},
author={Aloor, Jasmine and Nayak, Siddharth Nagar and Dolan, Sydney and Balakrishnan, Hamsa},
journal={Journal on Autonomous Transportation Systems},
year={2024},
publisher={ACM New York, NY},
doi={10.1145/3702012}
}
- Known issues with pytorch geometric and torch-scatter packege installation. Please refer to the requirements.txt to note the versions being used in the code. Correct order of installation of Pytorch Geometric packages if encountering any errors:
pip install --verbose git+https://github.com/pyg-team/pyg-lib.git
pip install --verbose torch_scatter
pip install --verbose torch_sparse
pip install --verbose torch_cluster
pip install --verbose torch_spline_conv
- Rendering issues with Linux users: Follow the instructions to access display for the visualization of evaluation tests.
Please file an issue if you have any questions or requests about the code or the paper. If you prefer your question to be private, you can alternatively email me at jjaloor@mit.edu
- InforMARL: https://nsidn98.github.io/InforMARL/ Paper: Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation
- Satellite Navigation and Coordination with Limited Information Sharing
We would be happy to accept PRs that help extend or improve this work.