Wei Fu
|
c60d128b14
|
Support asynchronous RL training, Qwen3, and the latest SGLang (#47)
* feat: one buffer for each task
* feat: support "one buffer for each task" for async
* make kv_cache_dtype configurable
Signed-off-by: Tiwei Bie <tiwei.btw@antgroup.com>
* style: use plural form
fix: use _seed_from_key to set different seeds for data loaders
fix: call load_data for one buffer each time
* PullRequest: 125 Support running async experiments in the 2407 image.
Merge branch fw/async2407 of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/125
Signed-off-by: 晓雷 <meizhiyu.mzy@antgroup.com>
* .
* fix: handle multiple datasets in recover indices
fix: `isinstance(self.__datasets, PullerStreamDataset)`
feat: use the "spec" request to obtain the number of datasets
fix: revert rollout worker
* fix: revert async_rl_exp.py
* fix flag for list (cuda_graph_bs)
* format
* [FIX] fix async task reward [sglang bf16-> fp16]
* fix: define `self.__datasets` in advance
* PullRequest: 130 [Refactor] Remove deprecated search related code
Merge branch mzy/remove-search of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/130
Signed-off-by: 博惟 <bowei.fw@antgroup.com>
* remove search related
* PullRequest: 131 [Refactor] Change terminology "model parallel" into "tensor parallel" to align with megatron.
Merge branch mzy/mp-to-tp of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/131?tab=comment
Signed-off-by: 博惟 <bowei.fw@antgroup.com>
* change mp to tp
* .
* .
* PullRequest: 142 Fix an error for megatron backend destroy
Merge branch fw/fix-meagatron-destroy of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/142
Reviewed-by: 晓雷 <meizhiyu.mzy@antgroup.com>
* .
* PullRequest: 143 Fix the port conflict issue of generation servers
Merge branch fw/fix-gen-port of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/143?tab=comment
Reviewed-by: 晓雷 <meizhiyu.mzy@antgroup.com>
* somehow fix the port issue
* add clearance period
* .
* .
* PullRequest: 145 Add code environment
Merge branch fw/code-env of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/145?tab=comment
Reviewed-by: 晓雷 <meizhiyu.mzy@antgroup.com>
* add code env
* somehow fix the port issue
* fix
* PullRequest: 144 Add decoupled PPO loss
Merge branch fw/decoupled-ppo-loss of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/144?tab=comment
Reviewed-by: 晓雷 <meizhiyu.mzy@antgroup.com>
* fix ppo step logging, nan in stats tracker, and add decoupled loss
* .
* somehow fix the port issue
* fix typo
* PullRequest: 146 Merge SLURM logs and save experiment configs in yaml format.
Merge branch fw/better-logging of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/146
Reviewed-by: 晓雷 <meizhiyu.mzy@antgroup.com>
* merge all slurm logs into one
* write config to yaml
* PullRequest: 141 Merge changes during NeurIPS submission
Merge branch fw/async-dev of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/141
Reviewed-by: 晓雷 <meizhiyu.mzy@antgroup.com>
* .
* .
* .
* .
* .
* .
* .
* .
* .
* update script
* .
* .
* .
* .
* [ADD] add least req scheduling
* fix test genreq
* .
* .
* fix stats tracker nan
* .
* .
* .
* .
* .
* .
* .
* uppper clip decoupled objective
* add throughput exp script
* .
* remove behav upper clip param
* .
* .
* .
* plot curve
* update thpt script
* .
* master worker raise error when exiting
* update script
* add gen throughput logging
* .
* .
* add decoupled wandb data
* .
* fix port issue and add no training option
* .
* enlarge ttl
* remove gserver manager await staled
* update weights in groups
* .
* .
* .
* add port clearance period
* .
* .
* .
* add plot script
* add sft throughput eval
* .
* log tokens in null interface
* 消融实验和interruptible generation
* 画图脚本/运行脚本/数据结果
* .
* remove scripts
* add port test
* remove force_sync_reward
* revert some changes
* .
* revert
* revert fix
* fix
* revert
* fix typo
* support qwen3 training
* PullRequest: 147 Support interruption in SGLang and fix a KeyError in gather-scatter communication
Merge branch fw/sglang046-with-abort-request of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/147?tab=diff
Reviewed-by: 晓雷 <meizhiyu.mzy@antgroup.com>
* fix ppo step logging, nan in stats tracker, and add decoupled loss
* .
* somehow fix the port issue
* initial commit
* add interupt request
* fix data transfer issue
* max concurrent rollouts defaults to train batch size
* merge main
* add patch
* fix patch typp
* revert sglang
* fix typo
* fix minor typo
* .
* pip show editable sglang path
* PullRequest: 149 fix: code faas max_retries
Merge branch xss/fix_code_verifier of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/149
Reviewed-by: 博惟 <bowei.fw@antgroup.com>
* fix: code faas max_retries
* PullRequest: 150 [Bug Fix] Fix key errors in `_run_scatter` in data transfer
Merge branch mzy/fix-scatter-groups of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/150
Reviewed-by: 博惟 <bowei.fw@antgroup.com>
* fix scatter groups key error
* fix test
* .
* PullRequest: 151 Fix Qwen3 import error when using transformers with a lower version
Merge branch fw/fix-qwen3 of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/151
Reviewed-by: 温差 <xushusheng.xss@antgroup.com>
* merge all slurm logs into one
* write config to yaml
* .
* PullRequest: 152 Support sglang0.4.6 and fix master_worker import error
Merge branch adopt_sglang046 of git@code.alipay.com:inclusionAI/AReaL.git into main
https://code.alipay.com/inclusionAI/AReaL/pull_requests/152
Reviewed-by: 博惟 <bowei.fw@antgroup.com>
* Support sglang0.4.6 and fix master_worker import error
* remove disable_mla option
---------
Signed-off-by: Tiwei Bie <tiwei.btw@antgroup.com>
Co-authored-by: wanghuaijie.whj <wanghuaijie.whj@antgroup.com>
Co-authored-by: Tiwei Bie <tiwei.btw@antgroup.com>
Co-authored-by: kira.gw <kira.gw@antgroup.com>
Co-authored-by: shenxujie.sxj <shenxujie.sxj@antgroup.com>
Co-authored-by: 晓雷 <meizhiyu.mzy@antgroup.com>
Co-authored-by: sam.gjx <sam.gjx@antgroup.com>
Co-authored-by: 温差 <xushusheng.xss@antgroup.com>
Co-authored-by: 履渊 <yuhong.gyh@antgroup.com>
|
2025-05-26 09:45:13 +08:00 |