louis_lifu
/
AReaL
mirror of https://github.com/inclusionAI/AReaL
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
AReaL/README.md

11 KiB
Raw Blame History

AReaL: Ant Reasoning Reinforcement Learning for LLMs

| Paper | Documentation | Ask DeepWiki | 🤗 Models & Data | WeChat Group |

ReaL

AReaL (Ant Reasoning RL) is an open-source fully asynchronous reinforcement learning training system for large reasoning models developed at the RL Lab, Ant Research. Built upon the open-source project RealHF, we are fully committed to open-source by providing training details, data, and infrastructure required to reproduce results along with the model itself. AReaL aims to help everyone build their own AI agents easily and affordably. Our team loves milk tea because it's delicious, customizable, and affordable. We hope you enjoy our project just like how you enjoy real-world milk tea (cheers).

AReaL Highlights

  • ⚡ [NEW] AReaLite: Our new release AReaLite is a light-weight and AI-centric codebase that prioritizes better development experiences for AI researchers. As a result, AReaLite delivers most AReaL functionalities while maintains its high performance with much fewer lines of code. This allows users to build their own agentic and RLVR training workflows with minimal effort.
  • 🔥 Asynchronous RL: With algorithm-system co-design, AReaL supports fully asynchronous RL for the fastest training speed! Experimental support for multi-turn agentic RL is also provided.
  • 🛠️ Open & Reproducible: We continuously release all code, datasets, and training recipes for RL training of LLMs.
  • 🚀 Scalability: AReaL can seamlessly adapt to different computational resource settings, ranging from a single node to 1K GPUs.
  • 🔪 Cutting-Edge Performance: AReaL can produce models with cutting-edge reasoning capabilities in math and coding. We are also actively working on agentic tasks.

News

[2025/07/31] (AReaLite) We introduce AReaLite, a light-weight version of AReaL designed specifically for AI researchers and rapid prototyping. AReaLite features an AI-centric API design that prioritizes ease of use and algorithm development, while inherently supporting fully asynchronous agentic RL. With 80% fewer lines of code, AReaLite maintains 90% of AReaL's high performance and core functionality. Check out our AReaLite design doc and the quickstart guide to begin your journey with AReaLite!

[2025/06/03] (v0.3, boba²) We release boba² (double-boba) for fully asynchronous RL training, which achieves a 2.77x speedup while obtaining on-par or even better training performance compared to synchronous systems. Moreover, asynchronous RL makes it extremely easy to set up multi-turn agentic RL training! Check out our v0.3 overview blog and the research paper.

[2025/03/31] (v0.2, boba) Here comes our next milestone release - boba! Please call it A-ReaL-boba! This release includes much faster training with SGLang support and SOTA 7B and 32B models on math reasoning. Check our v0.2 technical blog.

[2025/02/24] (v0.1) Our initial release includes reproducible results for 1.5B and 7B LRMs. Check our v0.1 technical blog.

AReaLite Release Highlights

New highlights in AReaLite:

  • Instead of the system-centric architecture in old AReaL, AReaLite follows an AI-centric API design that aims to provide the following key features:

    • Light-weight & easy-to-write algorithm and training workflow customization.
    • Easy to scale up without knowing system and infrastructure details.
    • Adaptable and plugable: Smooth to integrate with other modern AI applications.

    These features make AReaLite easy for AI researchers to adopt, understand, and develop effectively and efficiently. To learn more about the design principles of AReaL, please read the AReaLite design doc!

  • A much more light-weight codebase compared to old AReaL codebase with only 20% # lines of code, with a detailed code walkthrough on an GRPO-on-GSM8K example. Save your time & efforts for code reading!

  • Smoother customization for your own algorithms and agentic & RLVR rollout RL within a single file! Check here for agent & RLVR customization and here for algorithm customization.

Good old stuff from AReaL:

  • High performance and scalability with fully asynchronous RL training. Check our boba² (v0.3) blog for details.

  • A single command line to launch an experiment, no matter on a single node or a large-scale distributed cluster.

Now, let us run an example experiment with AReaLite following the quickstart guide below!

Getting Started with AReaLite

Our training scripts will automatically download the dataset (openai/gsm8k) and model (Qwen/Qwen2-1.5B-Instruct). On a single node, runs:

python3 -m arealite.launcher.local examples/arealite/gsm8k_grpo.py --config examples/arealite/configs/gsm8k_grpo.yaml

On a Ray cluster with 2 nodes & 8 GPUs each node, runs (Remember to change paths in the YAML file to your own shared storage):

python3 -m arealite.launcher.ray examples/arealite/gsm8k_grpo.py --config examples/arealite/configs/gsm8k_grpo.yaml \
  cluster.n_nodes=2 \
  cluster.n_gpus_per_node=8

For more detailed guide on how to run experiments in AReaLite, please check out our quickstart guide!

Switching from legacy AReaL to AReaLite

We also provide a convenient script to convert your AReaL YAML config into AReaLite config in one command line. First you need to locate your AReaL config either modified from files from examples folder, or generated when you run your experiments in <fileroot>/<expr_name>/<trial_name> folder. Runs:

python3 examples/arealite/convert_config.py -f <config_path> -o <output_path>

Then you should be able to run experiments with your old settings on AReaLite!

AReaLite vs legacy AReaL

AReaLite is an initiative to fully refactor AReaL, addressing historical issues such as redundant code and unnecessary system-level abstractions. Currently, AReaLite provides a lightweight codebase that enables fast prototyping for new RL training workflows and algorithms on a relatively small scale. For large-scale experiments (1K+ GPUs), we recommend using the battle-tested legacy AReaL to ensure stability. In the future, we will continue developing AReaLite by expanding its APIs, migrating legacy features, introducing new functionality, and validating the system through large-scale experiments.

Resources

Quickstart

Code Walkthrough

Customization

AReaL Legacy

For old AReaL documentation, check the legacy sections in our Documentation. To reproduce AReaL boba & boba² results, check our reproduction guide with legacy AReaL.

Future Plan

AReaL is under active development. We plan to have minor releases weekly and major releases monthly. Community engagement and contributions are extremely welcome. We are also hiring interns and full-time employees with open positions in both the US and China.

For the research and development plan already in place, please see the following list:

System Development

  • Support for SGLang
  • RL training with coding problems
  • Asynchronous generation and RL training
  • Optimizations for distributed training: expert parallel for MOE and zero-bubble pipelining
  • RL for vision-language models (VLM)
  • Multi-turn agentic RL
  • Function calling and tool use

Algorithm Development

  • RL training recipes for 1.5B and 7B models
  • A complete RL training recipe for 32B models
  • Sample-efficient multi-task RL algorithms
  • Agentic capabilities with end-to-end RL
  • Stable RL training for larger MOE models

Acknowledgement

We would like to note that major contributors are from the RL Lab at Ant Research and the Institute for Interdisciplinary Information Sciences, Tsinghua University.

Our team has also received invaluable assistance from the Data Intelligence Lab at Ant Research for data support and from the Super Computing Technology (SCT) team at Ant Group, particularly in the realm of large-scale cluster operations and maintenance.

We also appreciate all the pioneering works from the community, particularly the ReaLHF project from OpenPsi Inc. and other projects, including but not limited to DeepScaleR, Open-Reasoner-Zero, OpenRLHF, VeRL, SGLang, QwQ, Light-R1 and DAPO.

Citation

@inproceedings{mei2025real,
  author       = {Mei, Zhiyu and Fu, Wei and Li, Kaiwei and Wang, Guangju and Zhang, Huanchen and Wu, Yi},
  title        = {ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation},
  booktitle    = {Proceedings of the Eighth Conference on Machine Learning and Systems,
                  MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025},
  publisher    = {mlsys.org},
  year         = {2025},
}

@misc{fu2025areal,
      title={AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning},
      author={Wei Fu and Jiaxuan Gao and Xujie Shen and Chen Zhu and Zhiyu Mei and Chuyi He and Shusheng Xu and Guo Wei and Jun Mei and Jiashu Wang and Tongkai Yang and Binhang Yuan and Yi Wu},
      year={2025},
      eprint={2505.24298},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.24298},
}