# raylight **Repository Path**: yangecool/raylight ## Basic Information - **Project Name**: raylight - **Description**: No description available - **Primary Language**: Unknown - **License**: Apache-2.0 - **Default Branch**: main - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2026-06-11 - **Last Updated**: 2026-06-11 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # Raylight Raylight. Using Ray Worker to manage multi GPU sampler setup. With XDiT-XFuser and FSDP to implement parallelism. *"Why buy 5090 when you can buy 2x5070s"-Komikndr* ## UPDATE
Click to expand changelog - [BETA] Add LTXV Custom nodes expansion with Raylight compatibility - DP Latent Noise List, FSDP, and DP improvement, added by avtc - Add WanT2V Causal - Experimental : ControlNet support, by avtc - Experimental : Unified Parallel sampler, now all parallel type can be use at the same time, USP x DP x CFG x FSDP - ERNIE and Anima support - GPU Selector: add `GPU_SELECT` input to `RayInitializerAdvanced` to restrict Ray workers to specific GPUs (for example `0,1,2`, or leave empty for all GPUs), credit to [avtc](https://github.com/avtc) - Multi-Prompt DP: add `DPConditioningList` node for per-GPU conditioning in Data Parallel mode, credit to avtc - `mmap` option for reduce RAM, with GGUF variant implementation by avtc - Docs: add developer docs for FSDP, USP, CFG, and expansion modules - Now TensorCoreFP8 and NVFP4 works in FSDP2, check on [CK-Distributed](https://github.com/komikndr/comfy-kitchen-distributed) - LTX-2/ LTX2.3 USP - Kandinsky5 model - Fix FSDP error cause by Ray cannot pickle None type return by `comfy.supported_models_base.BASE.__getattr__` - TeaCache and EasyCache added thanks to [rmatif](https://github.com/rmatif/raylight/tree/easycache) - Flux2, Hunyuan 1.5 USP, FSDP - Fix broken tqdm progress bar - AMD ROCm Aiter attention - Z Image, Lumina, Model USP FSDP - SDXL and SD 1.5 supported through CFG - New parallelism, CFG. Check models note below about Flux or Hunyuan - Qwen Image fix for square dim - Hunyuan Video support for FSDP and USP - Chroma/Radiance support for FSDP and USP - GGUF added thanks to [City96](https://github.com/city96/ComfyUI-GGUF), only in USP mode, not in FSDP - Reworked the entire FSDP loader. Model loading should now be more stable and faster, as Raylight no longer kills active workers to reset the model state. Previously, this was necessary because Comfy could not remove FSDP models from VRAM, which caused memory leaks. - No need to install FlashAttn. - SageAttn is now supported. - Full FSDP support for Wan, Qwen, Flux, and Hunyuan Video. - Full LoRA support. - FSDP CPU offload, analogous to block swap/DisTorch.
## Table of Contents - [Raylight](#raylight) - [UPDATE](#update) - [Documentation](#documentation) - [What exactly is Raylight](#what-exactly-is-raylight) - [Raylight vs MultiGPU vs ComfyUI Worksplit vs ComfyUI-Distributed](#raylight-vs-multigpu-vs-comfyui-worksplit-branch-vs-comfyui-distributed) - [RTM and Known Issues](#rtm-and-known-issues) - [Operation](#operation) - [Tested GPU](#gpu-architectures) - [Supported Models](#supported-models) - [Scaled vs Non-Scaled Models](#scaled-vs-non-scaled-models) - [Attention](#attention) - [Example Wan](#wan-t2v-13b) - [Benchmark](#5090-vs-rtx-2000-ada) - [Installation](#installation) - [Support Me](#support) ## Documentation For developers and advanced users, see the [docs](docs/) folder for detailed guides on USP, CFG, FSDP, and expansion modules. - Top most head is `dev`, pr merge into `dev` or any custom branch name e.g `komikndr:raylight/feat-something` <- `john_doe:raylight/feat-something`, DONT MERGE INTO MAIN - If commit message contain "Partial: " it meaant the code is finished, tested on single worker, but not yet being tested in parallel setup. ## What exactly is Raylight Raylight is a parallelism node for ComfyUI, where the tensor of an image or video sequence is split among GPU ranks. Raylight, as its partial namesake, uses [Ray](https://docs.ray.io/en/latest/ray-core/walkthrough.html) to manage its GPU workers. [Introduction to Raylight on Youtube](https://youtu.be/KQxrkJAV4eI?si=JHjZAKZ3RGBCtmFx) image So how does it split among the ranks? It uses Unified Sequence Parallelism (USP), embedded inside [XDiT](https://github.com/xdit-project/xDiT), a core library of Raylight that splits and allgathers tensors among GPU ranks. Unfortunately, although it splits across GPUs, each GPU must still load the full model weight. And let's be honest, most of us do not have a 4090 or 5090. In my opinion, buying a second 4070 is monetarily less painful than buying a 5090. This is where [FSDP](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html) comes in. Its job is to split the model weights among GPUs. **TLDR:** Raylight is multi-GPU nodes for Comfy, USP for splitting the work, and FSDP for splitting the model weights. ### Raylight vs MultiGPU vs ComfyUI Worksplit branch vs ComfyUI-Distributed - [MultiGPU](https://github.com/pollockjj/ComfyUI-MultiGPU) Loads models selectively on specified GPUs without sharing workload. Includes CPU RAM offloading, which also benefits single-GPU users. - [ComfyUI Worksplit branch](https://github.com/comfyanonymous/ComfyUI/pull/7063) Splits workload at the CFG level, not at the tensor level. Since most workflows use `CFG=1.0` like Wan with Lora, this approach provides limited use cases. - [ComfyUI-Distributed](https://github.com/robertvoy/ComfyUI-Distributed) Distribute jobs among workers. Run your workflow on multiple GPUs simultaneously with varied seeds. Easily connect to local/remote/cloud worker like RunPod. - **Raylight** Provides both tensor split in sequence parallelism (USP), CFG parallelism and model weight sharding (FSDP). Your GPUs will 100% being used at the same time. In technical sense it _combine your VRAM_. This enables efficient multi-GPU utilization and scales beyond single high-memory GPUs (e.g., RTX 4090/5090). ## RTM and Known Issues - Scroll further down for the installation guide. - Model without new ComfyUI quant usually can be sharded using FSDP. - If there is an error about NCCL installation, just install `pip install nvidia-nccl-cu12==2.28.9`. Raylight use this NCCL lib instead of torch baked in NCCL. - If NCCL communication fails before running (e.g., watchdog timeout), set the following environment variables: ```bash export NCCL_P2P_DISABLE=1 export NCCL_SHM_DISABLE=1 ``` But this will hurt performance, it is like a sanity check if the Raylight can work, but there is so much performance left on the table. - Example WF just open from your comfyui menu and browse templates - **GPU Topology** is very important, not all PCIe in your motherboard is equal. - VRAM leakage, when using [Ring > 1 instead of Ulysses](https://github.com/feifeibear/long-context-attention/issues/112). Solution : just increase Ulysses degree for now. - The PyTorch **NCCL** version will be replaced to `2.28.9` to fix issues with FP8 communication. - The PyTorch version will be `2.8.1` due to relaxed `dtype` constraints when using FSDP. You can still use `2.7.1` or earlier. However, FSDP will not be function correctly in those versions. ## Operation ### Mode **Sequence Parallel** This mode splits the sequence among GPUs, the full model will be loaded into each GPU. Use the XFuser KSampler to increase the Ulysses degree according to the number of your GPUs, while keeping the Ring degree at 1 for small systems. ![USP](docs/images/USP.PNG) --- **Data Parallel** The full sequence will be processed independently on each GPU. Use the Data Parallel KSampler. There are two options, enable FSDP, or disable all options in `Ray Init Actor`. By disabling them, it will run in DP mode. Both FSDP and DP modes must have the Ulysses and Ring degrees set to 0. FSDP will shard the weights, but each GPU will still work independently, as the name suggests, Fully Sharded (Weight) Data Parallel. Data parallel also can use different prompt per generation using Conditional List ![DP](docs/images/DP.PNG) ![DP_COND_FSDP](docs/images/DP_DPCOND_FSDP.PNG) --- **Sequence + FSDP** Activate FSDP, and set the Ulysses degree to the number of GPUs. Use the XFuser KSampler. ![USP_FSDP](docs/images/USP_FSDP.PNG) --- **Unified Parallel** This is experimental mode where all type of parallel group can work at a sime time, USP, FSDP, DP, CFG. ![UNIFIED](docs/images/UNIFIED.PNG) ![UNIFIED_DP_COND](docs/images/UNIFIED_DP_COND.PNG) --- ### Side Notes - **Rule of thumb**, if you have enough VRAM, just use USP, if not, enable the FSDP, and if that is still not enough, enable also the FSDP CPU Offload. - FSDP CPU Offload is intended for systems with very low VRAM, though it will come with a performance hit work akin to DisTorch from MultiGPU. ## GPU Architectures ### NVidia 1. **Volta** : Use latest Yunchang libs, check notes below. 2. **Turing**: Same as Volta 3. **Ampere**: Tested 4. **Ada Lovelace**: Tested 5. **Blackwell**: Tested ### AMD 1. **MI3XX** : User confirmed working on 8xMI300X using ROCm compiled PyTorch and Flash Attention 2. 2. **MI210** : Personally tested and working on MI210 using ROCm compiled PyTorch and builtin `torch.nn.Functional.SDPA` ### Intel 1. **Arc Pro B60** : Using [LLM Scaler](https://github.com/intel/llm-scaler/blob/main/omni/README.md/#wan22). ## Supported Models **Wan** | Model | USP | FSDP | CFG | |-------------------|-----|------|-----| | Wan2.1 14B T2V | ✅ | ✅ | ✅ | | Wan2.1 14B I2V | ✅ | ✅ | ✅ | | Wan2.2 14B I2V | ✅ | ✅ | ✅ | | Wan2.2 14B I2V | ✅ | ✅ | ✅ | | Wan2.1 1.3B T2V | ✅ | ✅ | ✅ | | Wan2.1 1.3B AR | ✅ | ❌ | ✅ | | Wan2.2 5B TI2V | ✅ | ✅ | ✅ | | Wan2.1 Vace | ✅ | ❌ | ✅ | **Flux** | Model | USP | FSDP | CFG | |-------------------|-----|------|-----| | Flux Dev | ✅ | ✅ | ❌ | | Flux Kontext | ✅ | ✅ | ❌ | | Flux Krea | ✅ | ✅ | ❌ | | Flux 2 | ✅ | ✅ | ❌ | | Flux ControlNet | ✅ | ✅ | ❌ | **Chroma** | Model | USP | FSDP | CFG | |-------------------|-----|------|-----| | Chroma | ✅ | ✅ | ✅ | | Chroma Radiance | ✅ | ✅ | ✅ | | Chroma ControlNet | ✅† | ✅† | ✅ | **Qwen** | Model | USP | FSDP | CFG | |-------------------|-----|------|-----| | Qwen Image/Edit | ✅ | ✅ | ✅ | | ControlNet | ✅ | ✅ | ✅ | **Z Image, Lumina 2** | Model | USP | FSDP | CFG | |-------------------|-----|------|-----| | Z Image | ✅ | ✅ | ✅ | | Lumina 2 | ✅ | ✅ | ✅ | **Hunyuan Video** | Model | USP | FSDP | CFG | |-------------------|-----|------|-----| | Hunyuan Video | ✅ | ✅ | ❌ | | Hunyuan 1.5 | ✅ | ✅ | ❌ | | ControlNet | ✅† | ✅† | ❌ | **Kandinsky5** | Model | USP | FSDP | CFG | |-------------------|-----|------|-----| | Kandinsky5 I2V | ✅ | ❓ | ❌ | | Kandinsky5 T2V | ✅ | ❓ | ❌ | **LTX-2** | Model | USP | FSDP | CFG | |----------|-----|------|-----| | LTX-2 | ✅ | ❓ | ❌ | | LTX-2.3 | ✅ | ❓ | ❌ | **ERNIE-Image** | Model | USP | FSDP | CFG | |--------------------|-----|------|-----| | Ernie Image | ✅ | ✅ | ✅ | | Ernie Image Turbo | ✅ | ✅ | ✅ | **Anima** | Model | USP | FSDP | CFG | |--------------------|-----|------|-----| | Anima Preview3 | ✅ | ✅ | ❌ | **UNet** | Model | USP | FSDP | CFG | |--------|-----|------|-----| | SD1.5 | ❌ | ❌ | ✅ | | SDXL | ❌ | ❌ | ✅ | **Legend:** - ✅ = Supported - ❌ = Not currently supported. - ✅† = Code-ready, awaiting ControlNet model weights. - ❓ = Logically supported, awaiting for testing. - T = Text - I = Image - A = Audio - V = Video **Notes:** - Non standard Wan variant (Phantom, S2V, etc...) is not tested - CFG parallel for Flux, Hunyuan, is technically supported by Raylight, but since these models do not support conditional batches (CFG = 1), enabling it has no effect. - Chroma and Hunyuan Video ControlNet support is implemented in the USP/FSDP code paths but no ControlNet model weights are currently available in ComfyUI to test with. - Only one ControlNet model per workflow is supported. To apply the same model with different images or strengths, use multiple Apply ControlNet (Ray) nodes connected to a single Load ControlNet (Ray) node. ## Attention | Attention Variant | Time (s) | |----------------------|----------| | sage_fp8 | 10.75 | | sage_fp16_cuda | 11.00 | | sage_fp16_triton | 11.17 | | flash | 11.24 | | torch | 11.36 | **Notes:** - Tested on Wan 2.1 T2V 14B 832x480 33 frame 2 RTX 2000 ADA ## Wan T2V 1.3B image https://github.com/user-attachments/assets/40deddd2-1a87-44de-98a5-d5fc3defbecd ## Wan T2V 14B on RTX 2000 ADA ≈ RTX 4060 TI 16GB Screenshot 2025-08-20 125936 ## Qwen Image 20B on RTX 2000 ADA ≈ RTX 4060 TI 16GB , 4x Playback speed up https://github.com/user-attachments/assets/d5e262c7-16d5-4260-b847-27be2d809920 ## 5090 vs RTX 2000 ADA | Model | Model dtype | Parallelism (when applicable) | CFG | Steps | Resolution (W x H x Frame) | 1× RTX 2000 ADA (s/it) | 2× RTX 2000 ADA (s/it) | 5090 (s/it) | |--------|-------------|-----------------------------|-----|--------|-----------------------------|-------------------------|-------------------------|--------------| | SD 1.5 | FP32 | CFG Parallel = 2 | 7 | 20 | 512 × 512 | 0.11 | 0.05 | 0.003 | | SDXL | FP16 | CFG Parallel = 2 | 8 | 20 | 720 × 1024 | 0.33 | 0.19 | 0.004 | | Wan 2.1 1.3B T2V LX2V | BF16 | Ulysses = 2 | 1 | 4 | 480 × 832 × 33 | 2.70 | 1.65 | 0.46 | | Wan 2.1 14B T2V LX2V | FP8 E4M3 | Ulysses = 2 | 1 | 4 | 480 × 480 × 33 | 9.23 | 5.18 | 3.05 | | Wan 2.1 14B T2V LX2V (FSDP) | FP8 E4M3 | Ulysses = 2 | 1 | 4 | 480 × 640 × 81 | OOM | 22.51 | 3.05 | | Flux | FP8 E4M3 | Ulysses = 2 | 1 | 20 | 1024 × 1024 | 2.22 | 1.26 | 0.29 | | Chroma | FP8 E4M3 Scaled | Ulysses = 2 | 3.5 | 25 | 1024 × 1024 | 5.14 | 3.24 | 0.35 | | Chroma Radiance | FP8 E4M3 Scaled | Ulysses = 2 | 3.5 | 25 | 1024 × 1024 | 8.11 | 4.32 | 0.51 | | Hunyuan Video T2V | GGUF Q4 | Ulysses = 2 | 1 | 20 | 480 × 832 × 33 | 13.21 | 7.69 | 2.06 | | Qwen Image (FSDP) | FP8 E4M3 | Ulysses = 2 | 2.5 | 20 | 1024 × 1024 | OOM | 5.68 | 0.98 | **Notes:** - **RTX 2000 ADA ≈ RTX 4060 Ti** in performance. - All benchmarks were executed using **ComfyUI native workflows**, with **no Kijai wrapper** involved. - **Wan 2.1 14B T2V (FSDP)** is only applicable to **dual RTX 2000 ADA**: - **Single RTX 2000 ADA:** OOM - **5090:** does not require FSDP - Results represent the **average of 5 runs**, after warm-up. - **All speeds are normalized to seconds per iteration (s/it).** For models reporting **iterations per second**, we compute `1 / (it/s)` to convert. - **RTX 2000 ADA topology:** P2P is supported but runs over **SYS** path (NUMA cross-socket), meaning **no NVLink is present** and peer bandwidth is reduced. ## Installation **Manual** 1. Clone this repository under `ComfyUI/custom_nodes`. 2. `cd raylight` 3. Install dependencies: your_python_env - pip install -r requirements.txt 4. Install `pip install nvidia-nccl-cu12==2.28.9`, this is only needed if you are using FSDP with fp8 models with Nvidia GPUs 5. Install FlashAttention 2 (optional): - Option A (NOT recommended due to long build time): pip install flash-attn --no-build-isolation - Option B (recommended, use prebuilt wheel): For Torch 2.8: ```bash wget https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.3.14/flash_attn-2.8.2+cu128torch2.7-cp311-cp311-linux_x86_64.whl -O flash_attn-2.8.2+cu128torch2.7-cp311-cp311-linux_x86_64.whl ``` ```bash pip install flash_attn-2.8.2+cu128torch2.7-cp311-cp311-linux_x86_64.whl` ``` For other versions, check: https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/ 6. Attention backend can be install like any other libs e.g: SageAttn, AITER, FA3 (optional) 7. Restart ComfyUI. **ComfyUI Manager** 1. Find raylight in the manager and install it. **Windows** 1. After numerous testing, it still does not work on out of the box PyTorch, however if you want to try: 2. First, build [NCCL](https://github.com/MyCaffe/NCCL) for Windows. 3. Recommended steps: - Manually clone the **Raylight** repo - Switch to the `dev` branch for now - Inside the top-most Raylight folder (where `pyproject.toml` exists), run: ```bash ..\..\..\python_embeded\python.exe -m pip install -r .\requirements.txt ..\..\..\python_embeded\python.exe -m pip install -e . ``` 4. Advice, just run in WSL, and symlink your ComfyUI model dir from windows to WSL. ## Support [PayPal](https://paypal.me/Komikndr) Thanks for the support :) (I want to buy 2nd GPU (5060Ti) so i dont have to rent cloud GPU) [RunPod](https://runpod.io?ref=yruu07gh)