MOSS-Image-SFT Segmentation Control LoRA

MOSS-Image-SFT Segmentation Control LoRA

Restyles a flat-color region map into a new picture with MOSS-Image-SFT, keeping the region shapes while your prompt decides what each region becomes. For ComfyUI image artists.

Made by thedeoxen on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is a control LoRA for the image model MOSS-Image-SFT. You give it a map of flat color regions, and it keeps the layout of those regions while your text prompt decides what each region turns into. It works as an edit adapter that takes one control image, and it is not the official MOSS color or depth adapter. The chosen checkpoint comes from training step 4500.

Who it suits: Image creators who plan a picture as colored shapes first and want the prompt to supply the subject, materials and style.

Industry | Who uses it | How they use it | Example
Illustration | Concept artists | Block a scene into color regions, then render each mass from a prompt | A rough map of a figure becomes a costumed character
Film and animation | Storyboard artists | Fix a color-script layout and vary the rendering | The same shapes in several moods
Game art | Environment artists | Hold the placement of characters, furniture and backdrop fixed while swapping the look | A room layout restyled in a new setting
Marketing | Visual designers | Reuse one composition across different looks | One layout, three product scenes

You can skip it if you need a depth or color control, or if you do not use ComfyUI, since the card describes the workflow only there.

Where to use it: It runs in ComfyUI, loaded as a LoRA on top of the MOSS-Image-SFT model.

• ComfyUI with the Text Encode MOSS-Image Edit node
• A LoRA loader on the UNet
• The UniFormer semantic segmentor, used by the included workflow to build the control map

Use case | Who | What they get back
Scene blocking | Concept artists | A finished picture that follows the region layout
Layout reuse | Designers | New renders that keep people, furniture and background in place
Color-script planning | Storyboard artists | Pictures where only the shapes were decided up front

How to set it up: You need ComfyUI, the base MOSS-Image-SFT parts, and the LoRA and workflow files from this repository.

• ComfyUI with the MOSS-Image nodes
• MOSS-Image-SFT: its UNet, its text encoder and its VAE, under the file names the card lists
• The LoRA weights file and the ComfyUI workflow file from this repository

1. Download this repository's files, or fetch them with the command below.
2. Load the MOSS-Image-SFT UNet, text encoder with type moss_image, and VAE.
3. Load this LoRA on the UNet.
4. Open the included workflow in ComfyUI.

pip install -U huggingface_hub
hf download thedeoxen/MOSS-Image-segmentation-controlnet

How to use it: Load the LoRA at strength 1.0, then build a flat-color control map and feed it as the only reference image to the Text Encode MOSS-Image Edit node. Describe the picture you want in the prompt.

1. In both text encoders, positive and negative, pick ai-toolkit as the edit format to match the training.
2. Sample with 30 steps, CFG 4, and the euler sampler with the simple scheduler.
3. You need no trigger word. The control image fixes the structure, and the prompt supplies who or what appears, what it is made of, what it wears and where it is.

Tool | What it does
LoRA loader | Applies this LoRA to the UNet at strength 1.0
Text Encode MOSS-Image Edit | Takes the control map as the single reference image
UniFormer segmentor | Builds the region map from a photo in the included workflow
ComfyUI workflow file | Loads a photo, builds the map, encodes it and samples

Example: Feed in a flat-color map of a standing figure and ask for a full-length photo of a ringmaster in a red coat and top hat in front of a painted circus tent. The pose and the outline of the figure stay, and the text fills in clothing, colors and setting.

For newcomers and experts: A segmentation map is a picture where each area has one flat color to mark a region, such as a person, a chair or a wall. This LoRA treats that map as a layout and lets your words paint the details. A first small try is to run the included ComfyUI workflow on a photo and write a short prompt.

For experts, linear layers carry a rank of 64 and convolutions a rank of 16, trained so the control image plays the part of the reference in an edit. The region colors act as ids, not target colors. The card says UniFormer maps fit the weights better than segmentors that leave the background black, and it trained with kv cache on, which is why the edit format must be ai-toolkit.

Start from a photo or a drawn region map
UniFormer builds a flat-color control map
The map goes to Text Encode MOSS-Image Edit
The LoRA and your prompt guide the sampler
A new picture follows the region layout

Good to know: These tips come from the card, and the example previews show the control image next to three prompt results.

• Plain fills: Smooth shading or texture inside the map blurs where one region ends and the next begins.
• Color ids: A region color will not become that color in the result, so name the material in the prompt.
• Strength: If the result sticks too tightly to the map, try about 0.7 instead of the 1.0 used in training.

• License: Apache 2.0.
• Cost to run: none named, the source lists no API key, account or paid service.
• Data it reaches: the photos and control maps you load in ComfyUI, plus the model files you download.
• Not for: use as the official MOSS color or depth adapter, which the card says this is not.

On Hugging Face, 13 November 2026

• Task: image to image
• Library: diffusers
• Downloads in the last 30 days: 415
• Likes: 4
• License: apache-2.0
• Languages: en
• Based on: OpenMOSS-Team/MOSS-Image-SFT
• Tags: diffusers, moss-image, controlnet, lora, comfyui, segmentation, image-to-image

Links

• Model: https://huggingface.co/thedeoxen/MOSS-Image-segmentation-controlnet
• Files: https://huggingface.co/thedeoxen/MOSS-Image-segmentation-controlnet/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download thedeoxen/MOSS-Image-segmentation-controlnet

Then follow the usage in its model card. Nothing in this kit runs by itself.