Mike0021 commited on
Commit
00836cf
·
verified ·
1 Parent(s): e444c38

Initial Edit Anything LTX-2.3 Space

Browse files
Files changed (5) hide show
  1. GOAL.md +123 -0
  2. README.md +24 -8
  3. app.py +435 -0
  4. requirements.txt +13 -0
  5. rollout.jsonl +0 -0
GOAL.md ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Goal: Build a HuggingFace Space for "Edit Anything" — LTX-2.3 Video Editing LoRAs
2
+
3
+ ## Context
4
+
5
+ The model repo is at https://huggingface.co/Alissonerdx/EditAnything — it hosts experimental LTX-2.3 (22B) video editing LoRAs by Alissonerdx. There are THREE training tracks:
6
+
7
+ ### 1. Edit Anything v0.1 (motion transfer) — Standard LoRA
8
+ - `edit_anything_30k_v0.1_motion_transfer_r128.safetensors` (1.31 GB, rank 128)
9
+ - `edit_anything_30k_v0.1_motion_transfer_r256.safetensors` (2.62 GB, rank 256)
10
+ - Two-stage training: image-only pretraining (~30k pairs) then video fine-tune with first_frame_conditioning > 0
11
+ - Motion transfer: replace first frame with edited still, model copies motion from guide video
12
+ - Load through regular ComfyUI LoraLoader before LoopingSampler
13
+ - On sampler: editanything_module disconnected, ref_image = edited first frame, guide_frames = guide video, all enable_* flags OFF
14
+
15
+ ### 2. Edit Anything v1.1 (no-reference multitask) — Standard LoRA
16
+ - `edit_anything_v1.1_r256.safetensors` (rank 256)
17
+ - Prompt-only multitask editing: Add, Remove, Replace, Style
18
+ - No reference image, no first-frame conditioning
19
+ - Standalone — load as regular LoRA on vanilla LTX-2.3
20
+ - Prompt patterns:
21
+ - Add: 15-30+ words, "Add <detailed subject>, <position>, <context>"
22
+ - Remove: 4-10 words, "Remove the <object>"
23
+ - Replace: 20-35 words, "Replace <original+location> with <new subject>"
24
+ - Style: "Convert the video into a <STYLE> style" (300+ styles in training)
25
+ - NO compositional prompts (no "Add X and remove Y")
26
+ - NO "change background" as a standalone task
27
+ - NO global color grade / lighting change (only Style template)
28
+
29
+ ### 3. Reference V2V — Experimental IC-LoRA + sidecar modules (TWO builds)
30
+ - Build 1 (2-extras): `edit_anything_reference_v0.1_r128_ref_adaln_proj-role_embedding.{standard,module}.safetensors`
31
+ - Build 2 (4-extras): `...-ref_attn-ref_visual_proj.{standard,module}.safetensors`
32
+ - *.standard.safetensors = LoRA on attn1/attn2/ff → standard ComfyUI LoRA loader
33
+ - *.module.safetensors = role_embedding, ref_adaln_proj, ref_visual_proj, ref_attn → needs LTXVEditAnythingModuleLoader (BFSnodes custom nodes)
34
+ - Both files of a pair must be loaded together
35
+ - Ref V2V uses reference image for identity transfer (Add/Replace)
36
+ - Trained on ~1600 Add/Replace video pairs (very small, often fails)
37
+ - enable_adaln: on, enable_visual_crossattn: on for 4-extras build, enable_role_embedding: off for 4-extras
38
+
39
+ ## CRITICAL: Research Existing LTX-2.3 LoRA Spaces
40
+
41
+ Before writing any code, you MUST research the existing official LTX-2.3 LoRA spaces in the ltx-community org on HuggingFace. These are the reference implementations for how to run LTX-2.3 LoRAs on ZeroGPU HuggingFace Spaces.
42
+
43
+ ### Spaces to study (fetch their app.py source code):
44
+
45
+ 1. **LTX-2.3 Video Inpainting** — https://huggingface.co/spaces/ltx-community/ltx-2.3-inpaint
46
+ - Raw app.py: https://huggingface.co/spaces/ltx-community/ltx-2.3-inpaint/raw/main/app.py
47
+ - Uses `diffusers.LTX2InContextPipeline` with `LTX2ReferenceCondition` and `conditioning_attention_mask`
48
+ - IC-LoRA from `Lightricks/LTX-2.3-22b-IC-LoRA-In-Outpainting`
49
+ - Two-stage inference: Stage 1 (IC-LoRA + attention mask) → Stage 2a (spatial x2 latent upsample) → Stage 2b (refine on bare distilled model)
50
+ - Uses SAM3 for video mask generation
51
+
52
+ 2. **LTX-2.3 Video Outpaint** — https://huggingface.co/spaces/ltx-community/ltx-2.3-outpaint
53
+ - Raw app.py: https://huggingface.co/spaces/ltx-community/ltx-2.3-outpaint/raw/main/app.py
54
+ - Same diffusers pipeline as inpaint but with margin masking for outpainting
55
+
56
+ 3. **LTX-2.3 Day to Night** — https://huggingface.co/spaces/ltx-community/ltx-2.3-day-to-night
57
+ - Raw app.py: https://huggingface.co/spaces/ltx-community/ltx-2.3-day-to-night/raw/main/app.py
58
+ - Uses NATIVE LTX-2 codebase (`ltx_core` + `ltx_pipelines`) cloned from github.com/Lightricks/LTX-2
59
+ - Uses `ICLoraPipeline` from `ltx_pipelines.ic_lora`
60
+ - Has ZeroGPU patches for safetensors loader and attention backend
61
+ - Uses AOTI compiled transformer for acceleration
62
+
63
+ 4. **LTX-2.3 LoRA Trainer** — https://huggingface.co/spaces/ltx-community/ltx2-lora-trainer
64
+
65
+ ### Official Lightricks IC-LoRAs (for reference on how IC-LoRAs work with diffusers):
66
+ - `Lightricks/LTX-2.3-22b-IC-LoRA-In-Outpainting` — used by inpaint/outpaint spaces
67
+ - `Lightricks/LTX-2.3-22b-IC-LoRA-Day-To-Night` — used by day-to-night space
68
+ - `Lightricks/LTX-2.3-22b-IC-LoRA-Colorization`
69
+ - `Lightricks/LTX-2.3-22b-IC-LoRA-Deblur`
70
+ - `Lightricks/LTX-2.3-22b-IC-LoRA-Decompression`
71
+ - `Lightricks/LTX-2.3-22b-IC-LoRA-Water-Simulation`
72
+ - `Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients`
73
+ - `Lightricks/LTX-2.3-22b-IC-LoRA-HDR`
74
+ - `Lightricks/LTX-2.3-22b-IC-LoRA-Motion-Track-Control`
75
+
76
+ ### Base model:
77
+ - diffusers: `diffusers/LTX-2.3-Distilled-Diffusers`
78
+ - native: `Lightricks/LTX-2.3`
79
+
80
+ ## Key Technical Decisions
81
+
82
+ The EditAnything LoRAs are NOT IC-LoRAs (except the Ref V2V build). They are standard LoRAs that load through `pipe.load_lora_weights()`. The v0.1 and v1.1 LoRAs should work with the diffusers `LTX2InContextPipeline`:
83
+
84
+ ```python
85
+ from diffusers import LTX2InContextPipeline
86
+ pipe = LTX2InContextPipeline.from_pretrained("diffusers/LTX-2.3-Distilled-Diffusers", torch_dtype=torch.bfloat16)
87
+ pipe.load_lora_weights("Alissonerdx/EditAnything", weight_name="edit_anything_v1.1_r256.safetensors")
88
+ ```
89
+
90
+ For the v0.1 motion transfer, you need first_frame_conditioning — look at how the existing spaces handle reference conditions (`LTX2ReferenceCondition` with frames).
91
+
92
+ For Ref V2V, the sidecar modules are NOT standard LoRA adapters — they need custom loading. This is the hardest part. You may need to study the BFSnodes ComfyUI code to understand how the modules are loaded, then adapt that logic for a diffusers-based Space. Consider whether the Ref V2V track is feasible in a diffusers-only Space, and if not, focus on v0.1 + v1.1 first.
93
+
94
+ ## What to Build
95
+
96
+ A HuggingFace Space (Gradio app) that provides a UI for the EditAnything LoRAs. The Space should:
97
+
98
+ 1. Let users upload a video and choose an edit mode:
99
+ - **Motion Transfer** (v0.1): upload guide video + edited first frame → model copies motion
100
+ - **Prompt Edit** (v1.1): upload video + choose edit type (Add/Remove/Replace/Style) + prompt → model edits
101
+ - **Ref V2V** (if feasible): upload video + reference image + prompt → model adds/replaces using reference
102
+
103
+ 2. Load the appropriate LoRA based on the selected mode
104
+
105
+ 3. Use the diffusers `LTX2InContextPipeline` with two-stage inference (matching the inpaint/outpaint spaces)
106
+
107
+ 4. Run on ZeroGPU with proper `@spaces.GPU(duration=...)` decorators
108
+
109
+ 5. Follow all HF Spaces best practices (README.md with SDK: gradio, proper requirements.txt, etc.)
110
+
111
+ ## Implementation Notes
112
+
113
+ - Use `diffusers/LTX-2.3-Distilled-Diffusers` as the base model
114
+ - Use `LTX2InContextPipeline` and `LTX2LatentUpsamplePipeline` from diffusers
115
+ - Use `DISTILLED_SIGMA_VALUES` and `STAGE_2_DISTILLED_SIGMA_VALUES` from `diffusers.pipelines.ltx2.utils`
116
+ - Download LoRA weights from `Alissonerdx/EditAnything` repo via `hf_hub_download`
117
+ - The v0.1 LoRA uses first_frame_conditioning (pass edited first frame as reference condition)
118
+ - The v1.1 LoRA is standalone, no reference needed — just prompt + guide video
119
+ - Handle both landscape and portrait videos
120
+ - Include prompt templates/guidance for each edit type
121
+ - Resolution presets: Fast (768×448), Quality (960×544)
122
+ - Frame choices: 49, 73, 97, 121
123
+ - FPS: 24
README.md CHANGED
@@ -1,13 +1,29 @@
1
  ---
2
- title: Edit Anything Ltx23
3
- emoji: 🏆
4
- colorFrom: pink
5
- colorTo: gray
6
  sdk: gradio
7
- sdk_version: 6.19.0
8
- python_version: '3.12'
9
  app_file: app.py
10
- pinned: false
 
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: Edit Anything LTX-2.3
3
+ emoji: 🎬
4
+ colorFrom: blue
5
+ colorTo: indigo
6
  sdk: gradio
7
+ sdk_version: 6.10.0
 
8
  app_file: app.py
9
+ short_description: Edit Anything LTX-2.3 video LoRAs
10
+ startup_duration_timeout: 1h
11
  ---
12
 
13
+ # Edit Anything LTX-2.3
14
+
15
+ Gradio Space for the experimental Edit Anything LTX-2.3 video editing LoRAs by
16
+ Alisson Pereira dos Anjos.
17
+
18
+ Implemented:
19
+
20
+ - Motion Transfer v0.1 using `edit_anything_30k_v0.1_motion_transfer_r128.safetensors`
21
+ - Prompt Edit v1.1 using `edit_anything_v1.1_r256.safetensors`
22
+ - Two-stage LTX-2.3 diffusers inference, following the ltx-community inpaint/outpaint pattern
23
+
24
+ Ref V2V is listed but disabled. Its `.module.safetensors` sidecars contain
25
+ non-standard AdaLN, role embedding, visual projection, and reference-attention
26
+ branches consumed by BFSnodes. Those branches are not standard diffusers LoRA
27
+ adapters.
28
+
29
+ Model source: https://huggingface.co/Alissonerdx/EditAnything
app.py ADDED
@@ -0,0 +1,435 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+
3
+ os.environ.setdefault("HF_HOME", "/tmp/.cache/huggingface")
4
+ os.environ.setdefault("HF_MODULES_CACHE", "/tmp/hf_modules")
5
+ os.environ.setdefault("MPLCONFIGDIR", "/tmp/matplotlib")
6
+ os.environ.setdefault("GRADIO_SSR_MODE", "false")
7
+ os.environ.setdefault("TORCH_COMPILE_DISABLE", "1")
8
+ os.environ.setdefault("TORCHDYNAMO_DISABLE", "1")
9
+
10
+ for _path in (
11
+ os.environ["HF_HOME"],
12
+ os.environ["HF_MODULES_CACHE"],
13
+ os.environ["MPLCONFIGDIR"],
14
+ ):
15
+ os.makedirs(_path, exist_ok=True)
16
+
17
+ import random
18
+ import tempfile
19
+ import time
20
+
21
+ import gradio as gr
22
+ import imageio.v3 as iio
23
+ import numpy as np
24
+ import spaces
25
+ import torch
26
+ from huggingface_hub import hf_hub_download
27
+ from PIL import Image, ImageOps
28
+ from safetensors.torch import load_file
29
+
30
+ from diffusers import LTX2InContextPipeline, LTX2LatentUpsamplePipeline
31
+ from diffusers.pipelines.ltx2.latent_upsampler import LTX2LatentUpsamplerModel
32
+ from diffusers.pipelines.ltx2.pipeline_ltx2_condition import LTX2VideoCondition
33
+ from diffusers.pipelines.ltx2.pipeline_ltx2_ic_lora import LTX2ReferenceCondition
34
+ from diffusers.pipelines.ltx2.utils import DISTILLED_SIGMA_VALUES, STAGE_2_DISTILLED_SIGMA_VALUES
35
+ from diffusers.utils import encode_video, load_video
36
+
37
+
38
+ BASE_MODEL = "diffusers/LTX-2.3-Distilled-Diffusers"
39
+ EDIT_REPO = "Alissonerdx/EditAnything"
40
+ MOTION_LORA = "edit_anything_30k_v0.1_motion_transfer_r128.safetensors"
41
+ PROMPT_LORA = "edit_anything_v1.1_r256.safetensors"
42
+ UPSAMPLER_REPO = "dg845/LTX-2.3-Spatial-Upsampler-Diffusers"
43
+
44
+ FPS = 24
45
+ NUM_STEPS = len(DISTILLED_SIGMA_VALUES)
46
+ MAX_SEED = np.iinfo(np.int32).max
47
+ HF_TOKEN = os.environ.get("HF_TOKEN")
48
+
49
+ MODE_MOTION = "Motion Transfer (v0.1)"
50
+ MODE_PROMPT = "Prompt Edit (v1.1)"
51
+ MODE_REF = "Ref V2V (experimental, not enabled)"
52
+
53
+ ADAPTERS = {
54
+ MODE_MOTION: ("motion_v01_r128", MOTION_LORA),
55
+ MODE_PROMPT: ("prompt_v11_r256", PROMPT_LORA),
56
+ }
57
+
58
+ RES_PRESETS = {
59
+ "Fast (768x448)": (768, 448),
60
+ "Quality (960x544)": (960, 544),
61
+ }
62
+ FRAME_CHOICES = [49, 73, 97, 121]
63
+
64
+
65
+ @spaces.GPU(duration=1)
66
+ def _zerogpu_probe():
67
+ return "ready"
68
+
69
+
70
+ print("Loading LTX-2.3 distilled diffusers pipeline...", flush=True)
71
+ pipe = LTX2InContextPipeline.from_pretrained(BASE_MODEL, torch_dtype=torch.bfloat16)
72
+ pipe.to("cuda")
73
+ pipe.vae.enable_tiling()
74
+
75
+ print("Loading Edit Anything standard LoRAs...", flush=True)
76
+ for adapter_name, filename in ADAPTERS.values():
77
+ lora_path = hf_hub_download(EDIT_REPO, filename, token=HF_TOKEN)
78
+ pipe.load_lora_weights(load_file(lora_path), adapter_name=adapter_name)
79
+ pipe.set_adapters(ADAPTERS[MODE_PROMPT][0], 1.0)
80
+
81
+ print("Loading stage-2 spatial latent upsampler...", flush=True)
82
+ _upsampler = LTX2LatentUpsamplerModel.from_pretrained(
83
+ UPSAMPLER_REPO,
84
+ subfolder="latent_upsampler",
85
+ torch_dtype=torch.bfloat16,
86
+ )
87
+ _upsampler.to("cuda")
88
+ upsample_pipe = LTX2LatentUpsamplePipeline(vae=pipe.vae, latent_upsampler=_upsampler)
89
+ print("Pipeline ready.", flush=True)
90
+
91
+
92
+ def _src_fps(path, default=FPS):
93
+ try:
94
+ return float(iio.immeta(path, plugin="pyav").get("fps", default)) or default
95
+ except Exception:
96
+ return default
97
+
98
+
99
+ def _probe_video(path):
100
+ frames = load_video(path)
101
+ if not frames:
102
+ raise gr.Error("Could not read frames from the uploaded video.")
103
+ return frames
104
+
105
+
106
+ def _pick_resolution(first_frame, preset):
107
+ width, height = RES_PRESETS[preset]
108
+ if first_frame.height > first_frame.width:
109
+ width, height = height, width
110
+ return width, height
111
+
112
+
113
+ def _load_frames(path, num_frames, width, height):
114
+ frames = _probe_video(path)
115
+ source_fps = _src_fps(path)
116
+ out = []
117
+ for i in range(num_frames):
118
+ idx = min(int(round(i / FPS * source_fps)), len(frames) - 1)
119
+ frame = frames[idx].convert("RGB")
120
+ out.append(ImageOps.fit(frame, (width, height), Image.LANCZOS))
121
+ return out
122
+
123
+
124
+ def _prepare_first_frame(image, width, height):
125
+ if image is None:
126
+ raise gr.Error("Motion Transfer needs an externally edited first frame.")
127
+ if not isinstance(image, Image.Image):
128
+ image = Image.fromarray(np.asarray(image))
129
+ return ImageOps.fit(image.convert("RGB"), (width, height), Image.LANCZOS)
130
+
131
+
132
+ def _compose_prompt(mode, edit_type, prompt, style_name):
133
+ prompt = (prompt or "").strip()
134
+ style_name = (style_name or "").strip()
135
+
136
+ if mode == MODE_REF:
137
+ raise gr.Error(
138
+ "Ref V2V is not enabled in this diffusers Space. Its .module.safetensors "
139
+ "sidecar installs custom AdaLN, role embedding, and ref-attention branches "
140
+ "through BFSnodes; those branches are not standard diffusers LoRA adapters."
141
+ )
142
+
143
+ if mode == MODE_PROMPT and edit_type == "Style":
144
+ style = style_name or prompt
145
+ if not style:
146
+ raise gr.Error("Style mode needs a style name, for example 'Watercolor Painting'.")
147
+ if style.lower().startswith("convert the video into"):
148
+ return style
149
+ return f"Convert the video into a {style} style."
150
+
151
+ if not prompt:
152
+ raise gr.Error("Enter an edit prompt.")
153
+ return prompt
154
+
155
+
156
+ def _duration(*args, **kwargs):
157
+ preset = next((a for a in args if isinstance(a, str) and a in RES_PRESETS), "Fast (768x448)")
158
+ num_frames = next((a for a in args if isinstance(a, int) and a in FRAME_CHOICES), 73)
159
+ per_frame = 1.75 if "Quality" in str(preset) else 1.35
160
+ return int(120 + int(num_frames) * per_frame)
161
+
162
+
163
+ def _export(video_np, audio, path):
164
+ kwargs = {}
165
+ if audio is not None:
166
+ kwargs = {
167
+ "audio": audio[0].float().cpu(),
168
+ "audio_sample_rate": pipe.vocoder.config.output_sampling_rate,
169
+ }
170
+ encode_video(video_np, fps=FPS, output_path=path, **kwargs)
171
+
172
+
173
+ def _set_adapter(mode, scale):
174
+ adapter_name = ADAPTERS[mode][0]
175
+ pipe.set_adapters(adapter_name, float(scale))
176
+ return adapter_name
177
+
178
+
179
+ def _run_two_stage(
180
+ prompt,
181
+ reference_conditions,
182
+ conditions,
183
+ width,
184
+ height,
185
+ num_frames,
186
+ seed,
187
+ adapter_name,
188
+ lora_scale,
189
+ conditioning_attention_strength,
190
+ ):
191
+ pipe.set_adapters(adapter_name, float(lora_scale))
192
+ generator = torch.Generator(device="cuda").manual_seed(int(seed))
193
+
194
+ video_latent, audio_latent = pipe(
195
+ prompt=prompt,
196
+ negative_prompt="",
197
+ reference_conditions=reference_conditions,
198
+ conditions=conditions,
199
+ reference_downscale_factor=1,
200
+ conditioning_attention_strength=float(conditioning_attention_strength),
201
+ width=width,
202
+ height=height,
203
+ num_frames=num_frames,
204
+ frame_rate=FPS,
205
+ num_inference_steps=NUM_STEPS,
206
+ sigmas=DISTILLED_SIGMA_VALUES,
207
+ guidance_scale=1.0,
208
+ stg_scale=0.0,
209
+ audio_guidance_scale=1.0,
210
+ audio_stg_scale=0.0,
211
+ generator=generator,
212
+ output_type="latent",
213
+ return_dict=False,
214
+ )
215
+
216
+ up_latent = upsample_pipe(latents=video_latent, output_type="latent", return_dict=False)[0]
217
+
218
+ pipe.disable_lora()
219
+ try:
220
+ video_out, audio_out = pipe(
221
+ prompt=prompt,
222
+ negative_prompt="",
223
+ latents=up_latent,
224
+ audio_latents=audio_latent,
225
+ width=width * 2,
226
+ height=height * 2,
227
+ num_frames=num_frames,
228
+ frame_rate=FPS,
229
+ num_inference_steps=len(STAGE_2_DISTILLED_SIGMA_VALUES),
230
+ sigmas=STAGE_2_DISTILLED_SIGMA_VALUES,
231
+ noise_scale=STAGE_2_DISTILLED_SIGMA_VALUES[0],
232
+ guidance_scale=1.0,
233
+ stg_scale=0.0,
234
+ audio_guidance_scale=1.0,
235
+ audio_stg_scale=0.0,
236
+ generator=generator,
237
+ output_type="np",
238
+ return_dict=False,
239
+ )
240
+ finally:
241
+ pipe.set_adapters(adapter_name, float(lora_scale))
242
+
243
+ return video_out, audio_out
244
+
245
+
246
+ @spaces.GPU(duration=_duration, size="xlarge")
247
+ @torch.inference_mode()
248
+ def edit_anything(
249
+ mode,
250
+ video,
251
+ edited_first_frame,
252
+ edit_type,
253
+ prompt,
254
+ style_name,
255
+ preset,
256
+ num_frames,
257
+ seed,
258
+ randomize_seed,
259
+ lora_scale,
260
+ guide_strength,
261
+ source_attention,
262
+ progress=gr.Progress(track_tqdm=True),
263
+ ):
264
+ if video is None:
265
+ raise gr.Error("Upload a source video.")
266
+ if mode not in ADAPTERS and mode != MODE_REF:
267
+ raise gr.Error("Choose a supported edit mode.")
268
+
269
+ final_prompt = _compose_prompt(mode, edit_type, prompt, style_name)
270
+ if randomize_seed:
271
+ seed = random.randint(0, MAX_SEED)
272
+ seed = int(seed)
273
+ num_frames = int(num_frames)
274
+
275
+ progress(0.03, desc="Preparing source frames")
276
+ first = _probe_video(video)[0].convert("RGB")
277
+ width, height = _pick_resolution(first, preset)
278
+ guide_frames = _load_frames(video, num_frames, width, height)
279
+
280
+ reference_conditions = [
281
+ LTX2ReferenceCondition(frames=guide_frames, strength=float(guide_strength))
282
+ ]
283
+ conditions = None
284
+ edited_anchor = None
285
+
286
+ if mode == MODE_MOTION:
287
+ edited_anchor = _prepare_first_frame(edited_first_frame, width, height)
288
+ conditions = [LTX2VideoCondition(frames=edited_anchor, index=0, strength=1.0)]
289
+
290
+ adapter_name = _set_adapter(mode, lora_scale)
291
+ started = time.perf_counter()
292
+
293
+ progress(0.12, desc="Running LTX-2.3 stage 1")
294
+ video_out, audio_out = _run_two_stage(
295
+ prompt=final_prompt,
296
+ reference_conditions=reference_conditions,
297
+ conditions=conditions,
298
+ width=width,
299
+ height=height,
300
+ num_frames=num_frames,
301
+ seed=seed,
302
+ adapter_name=adapter_name,
303
+ lora_scale=lora_scale,
304
+ conditioning_attention_strength=source_attention,
305
+ )
306
+
307
+ progress(0.92, desc="Encoding output video")
308
+ result = (np.clip(video_out[0], 0, 1) * 255).astype(np.uint8)
309
+ if edited_anchor is not None and len(result) > 0:
310
+ result[0] = np.array(edited_anchor.resize((width * 2, height * 2), Image.LANCZOS))
311
+
312
+ out_path = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False).name
313
+ _export(result, audio_out, out_path)
314
+
315
+ elapsed = time.perf_counter() - started
316
+ print(
317
+ f"[METRIC] mode={mode!r} frames={num_frames} preset={preset!r} "
318
+ f"seed={seed} elapsed_s={elapsed:.2f}",
319
+ flush=True,
320
+ )
321
+ details = (
322
+ f"Seed: {seed}\n"
323
+ f"Prompt: {final_prompt}\n"
324
+ f"Mode: {mode}\n"
325
+ f"Elapsed seconds: {elapsed:.2f}"
326
+ )
327
+ return out_path, seed, details
328
+
329
+
330
+ def _mode_hint(mode):
331
+ if mode == MODE_MOTION:
332
+ return (
333
+ "Upload a guide video and an externally edited first frame. "
334
+ "The first frame anchors appearance; the video supplies motion."
335
+ )
336
+ if mode == MODE_PROMPT:
337
+ return (
338
+ "Upload a source video and use a single Add, Remove, Replace, or Style prompt. "
339
+ "Avoid multi-action prompts."
340
+ )
341
+ return (
342
+ "Ref V2V is shown for completeness but disabled in this diffusers build; "
343
+ "it requires BFSnodes sidecar module injection."
344
+ )
345
+
346
+
347
+ def _edit_type_hint(edit_type):
348
+ if edit_type == "Add":
349
+ return "Pattern: Add <detailed subject>, <position>, <context>."
350
+ if edit_type == "Remove":
351
+ return "Pattern: Remove the <object>. Keep it short."
352
+ if edit_type == "Replace":
353
+ return "Pattern: Replace <original subject and location> with <new subject>."
354
+ return "Pattern: Convert the video into a <STYLE> style."
355
+
356
+
357
+ with gr.Blocks(title="Edit Anything LTX-2.3") as demo:
358
+ gr.Markdown(
359
+ "# Edit Anything LTX-2.3\n"
360
+ "Experimental video editing with the Edit Anything standard LoRAs on LTX-2.3 Distilled."
361
+ )
362
+ mode_hint = gr.Markdown(_mode_hint(MODE_PROMPT))
363
+
364
+ with gr.Row():
365
+ with gr.Column():
366
+ mode = gr.Dropdown(
367
+ [MODE_PROMPT, MODE_MOTION, MODE_REF],
368
+ value=MODE_PROMPT,
369
+ label="Edit mode",
370
+ )
371
+ video_in = gr.Video(label="Source / guide video")
372
+ edited_frame = gr.Image(
373
+ label="Edited first frame for Motion Transfer",
374
+ type="pil",
375
+ image_mode="RGB",
376
+ )
377
+ edit_type = gr.Radio(
378
+ ["Add", "Remove", "Replace", "Style"],
379
+ value="Replace",
380
+ label="Prompt Edit task",
381
+ )
382
+ edit_hint = gr.Markdown(_edit_type_hint("Replace"))
383
+ prompt = gr.Textbox(
384
+ label="Edit prompt",
385
+ lines=4,
386
+ placeholder="Replace the bronze statue on the left with a tall man wearing a navy raincoat and brown boots.",
387
+ )
388
+ style_name = gr.Textbox(
389
+ label="Style name",
390
+ placeholder="Watercolor Painting",
391
+ )
392
+ with gr.Accordion("Settings", open=False):
393
+ preset = gr.Dropdown(list(RES_PRESETS), value="Fast (768x448)", label="Resolution")
394
+ num_frames = gr.Dropdown(FRAME_CHOICES, value=73, label="Frames at 24 fps")
395
+ randomize_seed = gr.Checkbox(True, label="Randomize seed")
396
+ seed = gr.Slider(0, MAX_SEED, value=42, step=1, label="Seed")
397
+ lora_scale = gr.Slider(0.2, 1.4, value=1.0, step=0.05, label="LoRA scale")
398
+ guide_strength = gr.Slider(0.2, 1.0, value=1.0, step=0.05, label="Guide video strength")
399
+ source_attention = gr.Slider(
400
+ 0.2,
401
+ 1.0,
402
+ value=1.0,
403
+ step=0.05,
404
+ label="Source/reference attention",
405
+ )
406
+ run = gr.Button("Generate", variant="primary")
407
+ with gr.Column():
408
+ video_out = gr.Video(label="Edited result")
409
+ details = gr.Textbox(label="Run details", lines=6)
410
+
411
+ mode.change(_mode_hint, inputs=mode, outputs=mode_hint)
412
+ edit_type.change(_edit_type_hint, inputs=edit_type, outputs=edit_hint)
413
+ run.click(
414
+ edit_anything,
415
+ inputs=[
416
+ mode,
417
+ video_in,
418
+ edited_frame,
419
+ edit_type,
420
+ prompt,
421
+ style_name,
422
+ preset,
423
+ num_frames,
424
+ seed,
425
+ randomize_seed,
426
+ lora_scale,
427
+ guide_strength,
428
+ source_attention,
429
+ ],
430
+ outputs=[video_out, seed, details],
431
+ api_name="generate",
432
+ )
433
+
434
+ if __name__ == "__main__":
435
+ demo.launch(show_error=True)
requirements.txt ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ git+https://github.com/huggingface/diffusers@ea802951f5fb235b6af8fe9247f56187d49748b2
2
+ gradio==6.10.0
3
+ spaces==0.41.1
4
+ transformers==4.57.6
5
+ accelerate==1.12.0
6
+ peft==0.18.1
7
+ safetensors==0.7.0
8
+ sentencepiece==0.2.1
9
+ torchvision
10
+ imageio[ffmpeg]==2.37.3
11
+ imageio-ffmpeg==0.6.0
12
+ av==16.0.1
13
+ pillow==12.0.0
rollout.jsonl ADDED
The diff for this file is too large to render. See raw diff