Scalable Reinforcement Learning Framework for Traffic Signal Control under Communication Delays

Info

This study proposes a two-stage framework to address observation delay in TSC. In the first stage, a scene prediction module and a scene context encoder are utilized to process historical and current traffic data to generate preliminary traffic signal actions. In the second stage, an action refinement module, informed by human-defined traffic rules and real-time traffic metrics, adjusts the preliminary actions to account for the latency in observations. This modular design allows device deployment with varying computational resources while facilitating system customization, ensuring both adaptability and scalability, particularly in edge-computing environments.

Schematic of the two-stage RL-based TSC framework with delay mitigation.

Illustration of the RL training process within the action generation stage (Stage 1).

The Predict-LSTM training process within the RL framework.

Getting Start

Install TransSimHub

The simulation environment we used is TransSimHub, which is based on SUMO and can be used for TSC, V2X and UAM simulation. More information is available via docs.

You can install TransSimHub by cloning the GitHub repository. Follow these steps:

git clone https://github.com/Traffic-Alpha/TransSimHub.git
cd TransSimHub
pip install -e .

After the installation is complete, you can use the following Python command to check if TransSimHub is installed and view its version:

import tshub
print(tshub.__version__)

Run model locally

• train.py，train model；
• • Example：

python train.py --stack=6 --delay=0 --model_name=scnn --net_env=train_four_345 --net_name=4phases.net.xml

• • model zoo： scnn ernn eattention ecnn inference predict
• • env：‘train_four_3 3phases.net.xml‘ ’train_four_345 4phases.net.xml‘ ’train_four_345 6phases.

• test.py, Test the trained model, specify the type of model, and the name of the environment to be tested, the test environment needs to have junction type and phase selection The model reads the model trained at delay=0 by default when reading, you can change it by yourself.

• • Example：

python test.py --stack=6 --delay=0 --model_name=ernn --net_name=train_four_3 

predict model

• FlowData_create.py,Traffic flow data generation for predictive model training.
• RNN_predict.py, Used to train the traffic prediction model, here used is LSTM.
• train_predict.py, For LSTM-based prediction and RL model training.

License and Citation

All assets and code in this repository are under the Apache 2.0 license unless specified otherwise. The language data is under CC BY-NC-SA 4.0. Other datasets (including nuScenes) inherit their own distribution licenses. Please consider citing our paper and project if they help your research.

@article{pang2024scalable,
  title={Scalable Reinforcement Learning Framework for Traffic Signal Control under Communication Delays},
  author={Pang, Aoyu and Wang, Maonan and Chen, Yirong and Pun, Man-On and Lepech, Michael},
  journal={IEEE Open Journal of Vehicular Technology},
  year={2024},
  publisher={IEEE}
}

Scripts

The script files used for training can be found in scripts.

Acknowledgment

感谢滕羽菲对画图工作进行的帮助，她非常善良可爱，乐于助人