ConCor-1: weights, remote code, processor and model card

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+example.png filter=lfs diff=lfs merge=lfs -text
+example_2.png filter=lfs diff=lfs merge=lfs -text
+model.jpg filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,206 @@
+---
+library_name: transformers
+license: apache-2.0
+language:
+  - en
+pipeline_tag: image-segmentation
+base_model:
+  - Qwen/Qwen3.5-0.8B
+tags:
+  - vision-language
+  - multimodal
+  - vision-language-grounding
+  - concept-correspondence
+  - phrase-grounding
+  - referring-expression-segmentation
+  - open-vocabulary-segmentation
+---
+
+# ConCor-1
+
+## Vision-Language Grounding as Bidirectional Concept Correspondence
+
+[Jieyu Zhang](https://jieyuz2.github.io)<sup>1</sup>\*,
+[Ziqi Gao](https://uwgzq.github.io)<sup>1,2</sup>\*,
+[Luke Zettlemoyer](https://homes.cs.washington.edu/~lsz/)<sup>1,3</sup>,
+[Ranjay Krishna](https://www.ranjaykrishna.com/)<sup>1</sup>
+
+<sup>1</sup>University of Washington · <sup>2</sup>Allen Institute for AI · <sup>3</sup>FAIR at Meta  
+\*Equal contribution
+
+**Quick links**
+
+📄 [Paper](https://arxiv.org) ·
+🌐 [Project page](https://uwgzq.github.io/papers/ConCor-1/) ·
+💻 [GitHub](https://github.com/uwGZQ/ConCor-1) ·
+🤗 [Data](https://huggingface.co/datasets/UWGZQ/ConCor-1-Data) ·
+🎨 [Demo](https://huggingface.co/spaces/UWGZQ/ConCor-1-demo)
+
+## Model Summary
+
+ConCor-1 is a vision-language grounding model designed to explicitly predict **bidirectional concept correspondences** between language and visual content. Built on a pretrained Qwen3.5-0.8B vision-language backbone, it introduces a set of learnable bridge tokens, each representing a candidate text–image correspondence. By jointly attending to visual and textual tokens, these bridge tokens aggregate multimodal context to represent potential alignments between image regions and text spans. For each bridge token, ConCor-1 simultaneously predicts a text mask identifying the grounded text segment, an image mask localizing the corresponding visual instance, and a presence score indicating whether the proposed pairing is a valid correspondence.
+
+![ConCor-1 architecture: vision tokens from the ViT and language tokens from the text enter the pretrained VLM alongside a set of learnable bridge tokens; each bridge token's output state feeds a vision segmentation head, a text segmentation head and a presence head, producing one image mask, one text mask and one presence score per candidate correspondence.](model.jpg)
+
+
+## Usage
+
+### Environment
+
+The released checkpoint is supported with the following core inference stack:
+
+| Dependency | Supported version |
+|---|---|
+| Python | >=3.11 |
+| PyTorch | 2.8.0 |
+| torchvision | 0.23.0 |
+| Transformers | 5.3.0 |
+
+Install a PyTorch/torchvision build compatible with your CUDA environment first. Then install the remaining inference dependencies:
+
+```bash
+pip install "transformers==5.3.0" "safetensors>=0.4.0" \
+  "huggingface_hub>=0.30.0" "numpy>=1.24" "pillow>=10.0" \
+  "flash-linear-attention>=0.4.1"
+```
+
+The example below uses PyTorch SDPA for the backbone's full-attention layers. To use FlashAttention-2, install `flash-attn>=2.8.0` and set `attn_implementation="flash_attention_2"`. `causal-conv1d>=1.4.0` is an optional acceleration dependency for the linear-attention layers.
+
+### Image + caption
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModel, AutoProcessor
+
+model_id = "UWGZQ/ConCor-1"  # or a local path to this repository
+
+processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModel.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    dtype=torch.bfloat16,
+    attn_implementation="sdpa",  # "flash_attention_2" also works
+).to("cuda").eval()
+
+image = Image.open("example.png").convert("RGB")
+text = (
+    "This image depicts a close-up of a brown bear in a natural outdoor setting. "
+    "The background consists of lush green grass. In the foreground, a large brown bear "
+    "is positioned centrally."
+)
+# A referring expression works the same way:
+# text = "the large brown bear in the foreground"
+
+inputs = processor(images=image, text=text, return_tensors="pt").to("cuda")
+with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
+    outputs = model(**inputs)
+
+correspondences = processor.post_process_correspondences(
+    outputs, text=text, target_sizes=[(image.height, image.width)]
+)[0]
+
+for correspondence in correspondences:
+    print(
+        f"{correspondence['presence_score']:.3f}",
+        correspondence["text_phrases"],   # phrases of this correspondence's text mask
+        correspondence["text_spans"],     # character spans into `text`
+        correspondence["mask"].shape,     # bool array, (height, width)
+    )
+```
+
+
+`post_process_correspondences` also takes `presence_threshold`, `text_threshold`, `image_threshold` and `nms_iou_threshold` (defaults `0.1`, `0.45`, `0.45`, `0.5`).
+
+`example_inference.py` wraps this up as a script, including a mask-overlay renderer:
+
+```bash
+python example_inference.py \
+    --image example.png \
+    --text "This image depicts a close-up of a brown bear in a natural outdoor setting. The background consists of lush green grass. In the foreground, a large brown bear is positioned centrally." \
+    --output overlay.png
+```
+
+
+### Image + category list
+
+```python
+image = Image.open("example_2.png").convert("RGB")
+text = "baseball player . baseball glove . grass . fence . dog"
+
+inputs = processor(images=image, text=text, return_tensors="pt").to("cuda")
+with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
+    outputs = model(**inputs)
+
+for correspondence in processor.post_process_correspondences(
+    outputs, text=text, target_sizes=[(image.height, image.width)]
+)[0]:
+    print(f"{correspondence['presence_score']:.3f}", correspondence["text_phrases"])
+```
+
+
+### Batched image–text pairs
+
+```python
+images = [Image.open("example.png").convert("RGB"), Image.open("example_2.png").convert("RGB")]
+texts = [
+    "This image depicts a close-up of a brown bear in a natural outdoor setting. "
+    "The background consists of lush green grass. In the foreground, a large brown bear "
+    "is positioned centrally.",
+    "baseball player . baseball glove . grass . fence . dog",
+]
+
+inputs = processor(images=images, text=texts, return_tensors="pt").to("cuda")
+with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
+    outputs = model(**inputs)
+
+results = processor.post_process_correspondences(
+    outputs,
+    text=texts,                                                        # the same list of texts
+    target_sizes=[(image.height, image.width) for image in images],    # one (height, width) per sample
+)
+
+for index, correspondences in enumerate(results):
+    print(f"sample {index}: {len(correspondences)} correspondence(s)")
+    for correspondence in correspondences:
+        print(f"  {correspondence['presence_score']:.3f}", correspondence["text_phrases"])
+```
+
+
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `configuration_concor1.py` | `ConCor1Config` |
+| `modeling_concor1.py` | `ConCor1ForConceptCorrespondence` and its heads |
+| `processing_concor1.py` | `ConCor1Processor`: sequence construction + correspondence post-processing |
+| `example_inference.py` | image + text → correspondences, with mask overlay |
+| `example.png`, `example_2.png` | the images used in the examples above |
+| `model.jpg` | the architecture figure above |
+| `model.safetensors` | weights: bf16 backbone, fp32 prediction heads |
+| `requirements.txt` | the inference dependencies listed above |
+
+
+
+## Demo
+
+An interactive ZeroGPU demo lives at
+[`UWGZQ/ConCor-1-demo`](https://huggingface.co/spaces/UWGZQ/ConCor-1-demo).
+
+
+## License and Use
+
+The weights and the code in this repository are released under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license. The backbone is [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), also Apache 2.0.
+
+
+
+## Citation
+
+```bibtex
+@article{zhang2026concor,
+  title   = {Vision-Language Grounding as Bidirectional Concept Correspondence},
+  author  = {Zhang, Jieyu and Gao, Ziqi and Zettlemoyer, Luke and Krishna, Ranjay},
+  year    = {2026}
+}
+```

config.json

@@ -0,0 +1,169 @@
+{
+  "architectures": [
+    "ConCor1ForConceptCorrespondence"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_concor1.ConCor1Config",
+    "AutoModel": "modeling_concor1.ConCor1ForConceptCorrespondence",
+    "AutoProcessor": "processing_concor1.ConCor1Processor"
+  },
+  "backbone_config": {
+    "image_token_id": 248056,
+    "model_type": "qwen3_5",
+    "text_config": {
+      "_name_or_path": "",
+      "architectures": null,
+      "attention_bias": false,
+      "attention_dropout": 0.0,
+      "attn_output_gate": true,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "dtype": "bfloat16",
+      "eos_token_id": 248044,
+      "full_attention_interval": 4,
+      "head_dim": 256,
+      "hidden_act": "silu",
+      "hidden_size": 1024,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "initializer_range": 0.02,
+      "intermediate_size": 3584,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "layer_types": [
+        "linear_attention",
+        "linear_attention",
+        "linear_attention",
+        "full_attention",
+        "linear_attention",
+        "linear_attention",
+        "linear_attention",
+        "full_attention",
+        "linear_attention",
+        "linear_attention",
+        "linear_attention",
+        "full_attention",
+        "linear_attention",
+        "linear_attention",
+        "linear_attention",
+        "full_attention",
+        "linear_attention",
+        "linear_attention",
+        "linear_attention",
+        "full_attention",
+        "linear_attention",
+        "linear_attention",
+        "linear_attention",
+        "full_attention"
+      ],
+      "linear_conv_kernel_dim": 4,
+      "linear_key_head_dim": 128,
+      "linear_num_key_heads": 16,
+      "linear_num_value_heads": 16,
+      "linear_value_head_dim": 128,
+      "mamba_ssm_dtype": "float32",
+      "max_position_embeddings": 262144,
+      "mlp_only_layers": [],
+      "model_type": "qwen3_5_text",
+      "mtp_num_hidden_layers": 1,
+      "mtp_use_dedicated_embeddings": false,
+      "num_attention_heads": 8,
+      "num_hidden_layers": 24,
+      "num_key_value_heads": 2,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "pad_token_id": null,
+      "partial_rotary_factor": 0.25,
+      "problem_type": null,
+      "return_dict": true,
+      "rms_norm_eps": 1e-06,
+      "rope_parameters": {
+        "mrope_interleaved": true,
+        "mrope_section": [
+          11,
+          11,
+          10
+        ],
+        "partial_rotary_factor": 0.25,
+        "rope_theta": 10000000,
+        "rope_type": "default"
+      },
+      "tie_word_embeddings": true,
+      "use_cache": true,
+      "vocab_size": 248462
+    },
+    "tie_word_embeddings": true,
+    "video_token_id": 248057,
+    "vision_config": {
+      "_name_or_path": "",
+      "architectures": null,
+      "chunk_size_feed_forward": 0,
+      "deepstack_visual_indexes": [],
+      "depth": 12,
+      "dtype": null,
+      "hidden_act": "gelu_pytorch_tanh",
+      "hidden_size": 768,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "in_channels": 3,
+      "initializer_range": 0.02,
+      "intermediate_size": 3072,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "model_type": "qwen3_5",
+      "num_heads": 12,
+      "num_position_embeddings": 2304,
+      "out_hidden_size": 1024,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "patch_size": 16,
+      "problem_type": null,
+      "return_dict": true,
+      "spatial_merge_size": 2,
+      "temporal_patch_size": 2
+    },
+    "vision_end_token_id": 248054,
+    "vision_start_token_id": 248053,
+    "vocab_size": 248462
+  },
+  "bidirectional_full_attention": true,
+  "bridge_grid_levels": [
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10
+  ],
+  "bridge_token_id_start": 248077,
+  "correspondence_dim": 256,
+  "dtype": "bfloat16",
+  "fuse_vision_encoder_features": true,
+  "image_max_pixels": 1003520,
+  "image_min_pixels": 1003520,
+  "image_threshold": 0.45,
+  "merge_size": 2,
+  "model_type": "concor1",
+  "nms_iou_threshold": 0.5,
+  "num_bridge_tokens": 385,
+  "num_mask_upsample_blocks": 2,
+  "patch_size": 16,
+  "presence_hidden_dim": 256,
+  "presence_threshold": 0.1,
+  "text_threshold": 0.45,
+  "transformers_version": "5.3.0"
+}

configuration_concor1.py

@@ -0,0 +1,202 @@
+# coding=utf-8
+# Copyright 2026 The ConCor-1 authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""ConCor-1 configuration.
+
+ConCor-1 is the concept-correspondence model of *Vision-Language Grounding as
+Bidirectional Concept Correspondence*. It wraps a pretrained Qwen3.5-0.8B
+vision-language backbone, appends ``num_bridge_tokens`` learnable **bridge
+tokens** to the multimodal sequence, and predicts, for every bridge token, an
+image mask, a text mask, and a correspondence presence score.
+"""
+
+from __future__ import annotations
+
+from typing import List, Optional, Sequence, Union
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.models.qwen3_5.configuration_qwen3_5 import Qwen3_5Config
+
+
+class ConCor1Config(PretrainedConfig):
+    r"""Configuration class for [`ConCor1ForConceptCorrespondence`].
+
+    Args:
+        backbone_config (`Union[Qwen3_5Config, dict]`, *optional*):
+            Configuration of the Qwen3.5 vision-language backbone used as a
+            contextual image-text encoder. Defaults to the Qwen3.5-0.8B
+            architecture.
+        num_bridge_tokens (`int`, *optional*, defaults to 385):
+            Number of bridge tokens `Q` appended after the image and text
+            tokens. Each bridge token stands for one candidate image-text
+            correspondence. With the default multi-scale grid levels,
+            `Q = sum(s**2 for s in 1..10) = 385`.
+        bridge_grid_levels (`Sequence[int]`, *optional*, defaults to `(1, ..., 10)`):
+            Multi-scale spatial grid levels the bridge tokens are organised
+            into. Level `s` contributes `s**2` bridge tokens, ordered in raster
+            order over an `s x s` grid. Only used for the training-time
+            Hungarian assignment and for interpreting a bridge index; the
+            forward pass itself does not depend on it.
+        bridge_token_id_start (`int`, *optional*, defaults to 248077):
+            First token id used for bridge tokens. Qwen3.5's tokenizer defines
+            248077 tokens while its embedding table has 248320 rows, so ids
+            from 248077 on are unused vocabulary slots; the embedding table is
+            grown when `num_bridge_tokens` needs more than the reserved slots.
+        correspondence_dim (`int`, *optional*, defaults to 256):
+            Dimension of the shared correspondence space that bridge, text and
+            visual features are projected into before the bilinear scorers.
+        presence_hidden_dim (`int`, *optional*, defaults to 256):
+            Hidden dimension of the presence head's SwiGLU MLP.
+        patch_size (`int`, *optional*, defaults to 16):
+            Vision-encoder patch size, in pixels.
+        merge_size (`int`, *optional*, defaults to 2):
+            Spatial merge factor of the backbone's patch merger. One merged
+            visual token therefore covers `patch_size * merge_size` pixels per
+            side (32 px for the released model).
+        num_mask_upsample_blocks (`int`, *optional*, defaults to 2):
+            Number of transposed-convolution blocks in the vision segmentation
+            head's convolutional decoder. Each block upsamples by 2, so image
+            masks are predicted on a grid of
+            `patch_size / 2**num_mask_upsample_blocks` pixel cells (4 px).
+        fuse_vision_encoder_features (`bool`, *optional*, defaults to `True`):
+            Fuse pre-merger ViT patch features into the expanded patch-level
+            features to restore local spatial detail.
+        bidirectional_full_attention (`bool`, *optional*, defaults to `True`):
+            Run the backbone's full-attention layers with a bidirectional mask
+            so bridge tokens can access the complete multimodal context. The
+            linear-attention layers keep their original behaviour. The released
+            checkpoint was trained this way; setting this to `False` reproduces
+            stock causal attention and will degrade predictions badly.
+        image_min_pixels (`int`, *optional*, defaults to 1003520):
+            `min_pixels` passed to the image processor; the released checkpoint
+            was trained and evaluated with `min_pixels == max_pixels`, i.e. a
+            fixed ~1.0 M pixel budget (1024 visual tokens).
+        image_max_pixels (`int`, *optional*, defaults to 1003520):
+            `max_pixels` passed to the image processor.
+        presence_threshold (`float`, *optional*, defaults to 0.1):
+            Default presence-score threshold used by the processor's
+            post-processing (paper: 0.1).
+        text_threshold (`float`, *optional*, defaults to 0.45):
+            Default text-mask probability threshold (paper: 0.45).
+        image_threshold (`float`, *optional*, defaults to 0.45):
+            Default image-mask probability threshold (paper: 0.45).
+        nms_iou_threshold (`float`, *optional*, defaults to 0.5):
+            Default IoU threshold for the correspondence NMS that removes
+            duplicate bridge predictions (paper: 0.5).
+
+    Example:
+
+    ```python
+    >>> from transformers import AutoConfig, AutoModel
+    >>> config = AutoConfig.from_pretrained("UWGZQ/ConCor-1", trust_remote_code=True)
+    >>> config.num_bridge_tokens
+    385
+    ```
+    """
+
+    model_type = "concor1"
+    sub_configs = {"backbone_config": Qwen3_5Config}
+
+    def __init__(
+        self,
+        backbone_config: Optional[Union[Qwen3_5Config, dict]] = None,
+        num_bridge_tokens: int = 385,
+        bridge_grid_levels: Sequence[int] = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10),
+        bridge_token_id_start: int = 248077,
+        correspondence_dim: int = 256,
+        presence_hidden_dim: int = 256,
+        patch_size: int = 16,
+        merge_size: int = 2,
+        num_mask_upsample_blocks: int = 2,
+        fuse_vision_encoder_features: bool = True,
+        bidirectional_full_attention: bool = True,
+        image_min_pixels: int = 1003520,
+        image_max_pixels: int = 1003520,
+        presence_threshold: float = 0.1,
+        text_threshold: float = 0.45,
+        image_threshold: float = 0.45,
+        nms_iou_threshold: float = 0.5,
+        **kwargs,
+    ):
+        if backbone_config is None:
+            backbone_config = Qwen3_5Config()
+        elif isinstance(backbone_config, dict):
+            backbone_config = Qwen3_5Config(**backbone_config)
+        self.backbone_config = backbone_config
+
+        self.num_bridge_tokens = num_bridge_tokens
+        self.bridge_grid_levels = list(bridge_grid_levels)
+        self.bridge_token_id_start = bridge_token_id_start
+        self.correspondence_dim = correspondence_dim
+        self.presence_hidden_dim = presence_hidden_dim
+        self.patch_size = patch_size
+        self.merge_size = merge_size
+        self.num_mask_upsample_blocks = num_mask_upsample_blocks
+        self.fuse_vision_encoder_features = fuse_vision_encoder_features
+        self.bidirectional_full_attention = bidirectional_full_attention
+        self.image_min_pixels = image_min_pixels
+        self.image_max_pixels = image_max_pixels
+        self.presence_threshold = presence_threshold
+        self.text_threshold = text_threshold
+        self.image_threshold = image_threshold
+        self.nms_iou_threshold = nms_iou_threshold
+
+        super().__init__(**kwargs)
+
+        if self.bridge_grid_levels:
+            expected = sum(s * s for s in self.bridge_grid_levels)
+            if expected != self.num_bridge_tokens:
+                raise ValueError(
+                    f"num_bridge_tokens={self.num_bridge_tokens} does not match "
+                    f"bridge_grid_levels={self.bridge_grid_levels} (sum of s^2 = {expected})."
+                )
+
+    @property
+    def hidden_size(self) -> int:
+        """Backbone hidden size (bridge / text token feature dim)."""
+        return self.backbone_config.text_config.hidden_size
+
+    @property
+    def vision_hidden_size(self) -> int:
+        """Pre-merger ViT hidden size (patch-level feature dim)."""
+        return self.backbone_config.vision_config.hidden_size
+
+    @property
+    def bridge_token_ids(self) -> List[int]:
+        """Token ids of the `num_bridge_tokens` bridge slots, in order."""
+        start = self.bridge_token_id_start
+        return list(range(start, start + self.num_bridge_tokens))
+
+    @property
+    def merged_token_size(self) -> int:
+        """Pixels per side covered by one merged visual token (32 px)."""
+        return self.patch_size * self.merge_size
+
+    @property
+    def mask_cell_size(self) -> int:
+        """Pixels per side of one image-mask cell (4 px for the released model)."""
+        return self.patch_size // (2 ** self.num_mask_upsample_blocks)
+
+    @property
+    def bridge_cells(self) -> List[tuple]:
+        """`(level, row, col)` of every bridge token, in bridge order."""
+        cells = []
+        for level in self.bridge_grid_levels:
+            for row in range(level):
+                for col in range(level):
+                    cells.append((level, row, col))
+        return cells
+
+
+__all__ = ["ConCor1Config"]

example_inference.py

@@ -0,0 +1,129 @@
+#!/usr/bin/env python3
+# coding=utf-8
+"""ConCor-1 inference example: an image + its text in, correspondences out.
+
+Given an image and a paired text, ConCor-1 predicts the full set of
+correspondences between visually referential text spans and instance-level
+image segments — the text spans are *not* given as queries; the model decides
+which parts of the text are grounded.
+
+Usage (the bundled `example.png` with its COCONut-PanCap caption):
+    python example_inference.py --image example.png \
+        --text "This image depicts a close-up of a brown bear in a natural outdoor setting. \
+The background consists of lush green grass. In the foreground, a large brown bear is \
+positioned centrally."
+
+    # a category list works just as well as a caption
+    python example_inference.py --image example.png --text "bear . grass . tree . person"
+
+    # write a mask overlay next to the printed correspondences
+    python example_inference.py --image example.png --text "..." --output overlay.png
+"""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+from typing import List
+
+import numpy as np
+import torch
+from PIL import Image, ImageDraw
+from transformers import AutoModel, AutoProcessor
+
+# Visually distinct overlay colours.
+COLORS = [
+    (239, 64, 64), (64, 204, 64), (64, 115, 255), (255, 204, 0), (255, 115, 0),
+    (217, 51, 217), (0, 204, 217), (153, 89, 13), (102, 255, 102), (140, 26, 255),
+    (255, 153, 204), (0, 140, 0), (179, 179, 0), (0, 26, 153), (204, 140, 51),
+    (128, 128, 128), (255, 0, 128), (0, 255, 140), (140, 0, 0), (0, 140, 140),
+]
+
+
+def overlay_masks(image: Image.Image, correspondences: List[dict], alpha: float = 0.5) -> Image.Image:
+    """Blend each correspondence's mask over the image and label it with its phrases."""
+    canvas = np.array(image.convert("RGB"), dtype=np.float32)
+    for index, correspondence in enumerate(correspondences):
+        mask = correspondence.get("mask")
+        if mask is None or not mask.any():
+            continue
+        color = np.array(COLORS[index % len(COLORS)], dtype=np.float32)
+        canvas[mask] = (1.0 - alpha) * canvas[mask] + alpha * color
+
+    overlaid = Image.fromarray(canvas.astype(np.uint8))
+    draw = ImageDraw.Draw(overlaid)
+    for index, correspondence in enumerate(correspondences):
+        mask = correspondence.get("mask")
+        if mask is None or not mask.any():
+            continue
+        rows, columns = np.nonzero(mask)
+        label = " / ".join(correspondence["text_phrases"]) or "(no text span)"
+        label = f"{label}  {correspondence['presence_score']:.2f}"
+        anchor = (int(columns.min()), max(int(rows.min()) - 12, 0))
+        draw.text(anchor, label, fill=COLORS[index % len(COLORS)])
+    return overlaid
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--model", default=str(Path(__file__).parent), help="model repo id or local path")
+    parser.add_argument("--image", required=True, type=Path)
+    parser.add_argument("--text", required=True, help="the text to ground (caption, category list, referring expression)")
+    parser.add_argument("--output", type=Path, default=None, help="write a mask overlay here (PNG)")
+    parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
+    parser.add_argument(
+        "--attn_implementation",
+        default="sdpa",
+        choices=["flash_attention_2", "sdpa", "eager"],
+        help="sdpa needs no extra dependency; flash_attention_2 reproduces the paper's numbers exactly",
+    )
+    parser.add_argument("--presence_threshold", type=float, default=None, help="default 0.1")
+    parser.add_argument("--text_threshold", type=float, default=None, help="default 0.45")
+    parser.add_argument("--image_threshold", type=float, default=None, help="default 0.45")
+    parser.add_argument("--nms_iou_threshold", type=float, default=None, help="default 0.5")
+    args = parser.parse_args()
+
+    processor = AutoProcessor.from_pretrained(args.model, trust_remote_code=True)
+    model = AutoModel.from_pretrained(
+        args.model,
+        trust_remote_code=True,
+        dtype=torch.bfloat16,
+        attn_implementation=args.attn_implementation,
+    ).to(args.device).eval()
+
+    image = Image.open(args.image).convert("RGB")
+    inputs = processor(images=image, text=args.text, return_tensors="pt").to(args.device)
+
+    with torch.inference_mode(), torch.autocast(args.device, dtype=torch.bfloat16):
+        outputs = model(**inputs)
+
+    correspondences = processor.post_process_correspondences(
+        outputs,
+        text=args.text,
+        target_sizes=[(image.height, image.width)],
+        presence_threshold=args.presence_threshold,
+        text_threshold=args.text_threshold,
+        image_threshold=args.image_threshold,
+        nms_iou_threshold=args.nms_iou_threshold,
+    )[0]
+
+    print(f"\nimage: {args.image}  ({image.width}x{image.height})")
+    print(f"text:  {args.text}")
+    print(f"\n{len(correspondences)} correspondence(s):")
+    for correspondence in correspondences:
+        mask = correspondence["mask"]
+        phrases = " / ".join(correspondence["text_phrases"]) or "(no text span)"
+        area = 100.0 * mask.mean() if mask is not None else 0.0
+        print(
+            f"  presence={correspondence['presence_score']:.3f}  "
+            f"bridge={correspondence['bridge_index']:3d}  "
+            f"mask={area:5.2f}% of image  spans={correspondence['text_spans']}  |  {phrases}"
+        )
+
+    if args.output is not None:
+        overlay_masks(image, correspondences).save(args.output)
+        print(f"\nwrote {args.output}")
+
+
+if __name__ == "__main__":
+    main()

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bdd5c7f935c7288241191066d0d3860d82ffe0d371071dd3f010e5317d9aab4
+size 1760946056

modeling_concor1.py

@@ -0,0 +1,804 @@
+# coding=utf-8
+# Copyright 2026 The ConCor-1 authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch ConCor-1 model — vision-language grounding as bidirectional concept
+correspondence.
+
+ConCor-1 uses a pretrained Qwen3.5-0.8B vision-language model as a *contextual
+image-text encoder* (not as a generator) and appends `Q` learnable **bridge
+tokens** to the multimodal sequence:
+
+    [<vision_start>, <image_pad> x N_v, <vision_end>, text tokens x N_t, bridge tokens x Q]
+
+For every bridge token, three lightweight heads predict one candidate
+image-text correspondence:
+
+  * `presence_head`              — a scalar presence score: is this a valid correspondence?
+  * `text_segmentation_head`     — a binary mask over the input text tokens
+  * `vision_segmentation_head`   — a binary mask over the image, on a 4-pixel cell grid
+
+The backbone's full-attention layers are run with a **bidirectional** mask so
+that bridge tokens see the complete multimodal context and can differentiate
+from one another; the linear-attention (gated delta-net) layers keep their
+original behaviour. This model is therefore not autoregressive: it does not
+support `use_cache`, `past_key_values` or `generate()`.
+"""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass
+from typing import List, Optional, Tuple
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from transformers.masking_utils import create_bidirectional_mask
+from transformers.modeling_outputs import ModelOutput
+from transformers.modeling_utils import PreTrainedModel
+from transformers.models.qwen3_5 import modeling_qwen3_5 as qwen3_5
+from transformers.models.qwen3_5.modeling_qwen3_5 import (
+    Qwen3_5Model,
+    Qwen3_5ModelOutputWithPast,
+    Qwen3_5TextModel,
+)
+from transformers.utils import logging
+
+from .configuration_concor1 import ConCor1Config
+
+logger = logging.get_logger(__name__)
+
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Backbone: Qwen3.5 with bidirectional full-attention layers
+# ══════════════════════════════════════════════════════════════════════════════
+
+
+class ConCor1BidirectionalTextModel(Qwen3_5TextModel):
+    """Qwen3.5 text model whose *full-attention* layers are bidirectional.
+
+    Identical to [`Qwen3_5TextModel`] except that the mask handed to the
+    full-attention layers is built with `create_bidirectional_mask` instead of
+    `create_causal_mask`, and `is_causal=False` is forced so the attention
+    backend cannot silently fall back to causal masking. The linear-attention
+    layers are untouched, preserving Qwen3.5's pretrained hybrid-attention
+    structure.
+
+    Bidirectional attention is only defined for full-sequence forward passes,
+    so KV caching / incremental decoding is rejected.
+    """
+
+    @qwen3_5.merge_with_config_defaults
+    @qwen3_5.capture_outputs
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values=None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        **kwargs,
+    ) -> Qwen3_5ModelOutputWithPast:
+        bidirectional = kwargs.pop("bidirectional_full_attention", True)
+
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if bidirectional:
+            if past_key_values is not None:
+                raise ValueError(
+                    "ConCor-1's bidirectional attention does not support `past_key_values`; "
+                    "use a full-sequence forward pass."
+                )
+            if use_cache:
+                raise ValueError(
+                    "ConCor-1's bidirectional attention does not support `use_cache=True` or "
+                    "autoregressive `generate()`. Pass `use_cache=False`."
+                )
+            use_cache = False
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids)
+
+        if use_cache and past_key_values is None:
+            past_key_values = qwen3_5.Qwen3_5DynamicCache(config=self.config)
+
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position = torch.arange(
+                past_seen_tokens,
+                past_seen_tokens + inputs_embeds.shape[1],
+                device=inputs_embeds.device,
+            )
+
+        if position_ids is None:
+            position_ids = cache_position.view(1, 1, -1).expand(4, inputs_embeds.shape[0], -1)
+        elif position_ids.ndim == 2:
+            position_ids = position_ids[None, ...].expand(4, position_ids.shape[0], -1)
+
+        if position_ids.ndim == 3 and position_ids.shape[0] == 4:
+            text_position_ids = position_ids[0]
+            position_ids = position_ids[1:]
+        else:
+            text_position_ids = None
+
+        # Linear-attention layers keep the original mask path.
+        linear_attn_mask = self._update_linear_attn_mask(attention_mask, cache_position)
+
+        if bidirectional:
+            full_attn_mask = create_bidirectional_mask(
+                config=self.config,
+                inputs_embeds=inputs_embeds,
+                attention_mask=attention_mask,
+            )
+        else:
+            full_attn_mask = qwen3_5.create_causal_mask(
+                config=self.config,
+                inputs_embeds=inputs_embeds,
+                attention_mask=attention_mask,
+                cache_position=cache_position,
+                past_key_values=past_key_values,
+                position_ids=text_position_ids,
+            )
+
+        hidden_states = inputs_embeds
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        for decoder_layer in self.layers[: self.config.num_hidden_layers]:
+            if decoder_layer.layer_type == "linear_attention":
+                layer_mask = linear_attn_mask
+                layer_kwargs = kwargs
+            else:
+                layer_mask = full_attn_mask
+                layer_kwargs = dict(kwargs)
+                if bidirectional:
+                    layer_kwargs["is_causal"] = False
+
+            hidden_states = decoder_layer(
+                hidden_states,
+                position_embeddings=position_embeddings,
+                attention_mask=layer_mask,
+                position_ids=text_position_ids,
+                past_key_values=past_key_values,
+                use_cache=use_cache,
+                cache_position=cache_position,
+                **layer_kwargs,
+            )
+
+        hidden_states = self.norm(hidden_states)
+
+        return Qwen3_5ModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+        )
+
+
+class ConCor1VisionLanguageBackbone(Qwen3_5Model):
+    """Qwen3.5 vision-language backbone with a bidirectional text model."""
+
+    def __init__(self, config):
+        super().__init__(config)
+        # Same weights and layout as Qwen3_5TextModel; only the attention mask of
+        # the full-attention layers differs (see ConCor1BidirectionalTextModel).
+        self.language_model.__class__ = ConCor1BidirectionalTextModel
+
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Building blocks
+# ══════════════════════════════════════════════════════════════════════════════
+
+
+class ConCor1SwiGLUProjection(nn.Module):
+    """SwiGLU projection `norm(silu(W_g x) * W_u x)`, following the backbone's FFN style."""
+
+    def __init__(self, in_features: int, out_features: int):
+        super().__init__()
+        self.gate_proj = nn.Linear(in_features, out_features, bias=False)
+        self.up_proj = nn.Linear(in_features, out_features, bias=False)
+        self.norm = nn.RMSNorm(out_features)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        return self.norm(F.silu(self.gate_proj(hidden_states)) * self.up_proj(hidden_states))
+
+
+class ConCor1LayerNorm2d(nn.LayerNorm):
+    """LayerNorm over the channel dimension of `(B, C, H, W)` tensors."""
+
+    def __init__(self, num_channels: int, eps: float = 1e-6):
+        super().__init__(num_channels, eps=eps, elementwise_affine=True)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = hidden_states.permute(0, 2, 3, 1)
+        hidden_states = F.layer_norm(
+            hidden_states, self.normalized_shape, self.weight, self.bias, self.eps
+        )
+        return hidden_states.permute(0, 3, 1, 2)
+
+
+class ConCor1PresenceHead(nn.Module):
+    """Presence head: SwiGLU MLP over a bridge token, projected to a scalar logit.
+
+    `z_pres[j] = W_o · norm(silu(W_g b_j) * W_u b_j)`
+    """
+
+    def __init__(self, hidden_size: int, intermediate_size: int):
+        super().__init__()
+        self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
+        self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
+        self.norm = nn.RMSNorm(intermediate_size)
+        self.out_proj = nn.Linear(intermediate_size, 1, bias=False)
+
+    def forward(self, bridge_features: torch.Tensor) -> torch.Tensor:
+        bridge_features = bridge_features.to(self.gate_proj.weight.dtype)
+        hidden = self.norm(
+            F.silu(self.gate_proj(bridge_features)) * self.up_proj(bridge_features)
+        )
+        return self.out_proj(hidden)
+
+
+class ConCor1BilinearCorrespondenceScorer(nn.Module):
+    """Bilinear scorer between bridge tokens and a sequence of features.
+
+    Both sides are projected into a shared `correspondence_dim` space with
+    independent SwiGLU MLPs, then scored with a learnable bilinear form:
+
+        z[j, n] = phi_f(f_n)^T W phi_b(b_j)
+
+    Used both as the text segmentation head (features = text tokens) and as the
+    mask predictor of the vision segmentation head (features = decoded spatial
+    cells).
+    """
+
+    def __init__(self, feature_dim: int, bridge_dim: int, correspondence_dim: int = 256):
+        super().__init__()
+        self.correspondence_dim = correspondence_dim
+        self.feature_proj = ConCor1SwiGLUProjection(feature_dim, correspondence_dim)
+        self.bridge_proj = ConCor1SwiGLUProjection(bridge_dim, correspondence_dim)
+        self.bilinear = nn.Bilinear(correspondence_dim, correspondence_dim, 1, bias=False)
+
+    def forward(self, features: torch.Tensor, bridge_features: torch.Tensor) -> torch.Tensor:
+        """
+        Args:
+            features: `(B, N, feature_dim)`
+            bridge_features: `(B, Q, bridge_dim)`
+
+        Returns:
+            `(B, Q, N)` correspondence logits.
+        """
+        parameter_dtype = self.bilinear.weight.dtype
+        features = features.to(parameter_dtype)
+        bridge_features = bridge_features.to(parameter_dtype)
+        projected_features = self.feature_proj(features)          # (B, N, C)
+        projected_bridges = self.bridge_proj(bridge_features)     # (B, Q, C)
+        weight = self.bilinear.weight.squeeze(0)                  # (C, C)
+        logits = torch.einsum("bni, ij, bqj -> bqn", projected_features, weight, projected_bridges)
+        return logits.to(projected_features.dtype)
+
+
+class ConCor1PatchExpander(nn.Module):
+    """Expand each merged visual token back into its `merge_size**2` patch features.
+
+    The backbone merges 2x2 patch neighbourhoods into one visual token, so mask
+    prediction over merged tokens would be spatially coarse. Four independent
+    SwiGLU MLPs map one merged token (`hidden_size`) to the top-left, top-right,
+    bottom-left and bottom-right pre-merger patch features (`vision_hidden_size`).
+
+    Output order is interleaved: the four children of a merged token are
+    contiguous, matching the pre-merger ViT patch order.
+    """
+
+    def __init__(self, hidden_size: int, patch_dim: int, num_children: int = 4):
+        super().__init__()
+        self.patch_dim = patch_dim
+        self.num_children = num_children
+        self.branches = nn.ModuleList(
+            [ConCor1SwiGLUProjection(hidden_size, patch_dim) for _ in range(num_children)]
+        )
+
+    def forward(
+        self,
+        visual_features: torch.Tensor,
+        patch_features: Optional[torch.Tensor] = None,
+    ) -> torch.Tensor:
+        """
+        Args:
+            visual_features: `(B, N_vis, hidden_size)` merged visual tokens.
+            patch_features: `(B, N_vis * num_children, patch_dim)`, optional
+                pre-merger ViT patch features fused into the expanded features.
+
+        Returns:
+            `(B, N_vis * num_children, patch_dim)`, interleaved.
+        """
+        visual_features = visual_features.to(self.branches[0].gate_proj.weight.dtype)
+        batch_size, num_visual, _ = visual_features.shape
+
+        if patch_features is not None:
+            expected = num_visual * self.num_children
+            if patch_features.shape[1] != expected:
+                raise ValueError(
+                    f"patch_features.shape[1]={patch_features.shape[1]} != "
+                    f"N_vis * num_children = {expected}"
+                )
+            if patch_features.shape[2] != self.patch_dim:
+                raise ValueError(
+                    f"patch_features.shape[2]={patch_features.shape[2]} != patch_dim={self.patch_dim}"
+                )
+            patch_features = patch_features.view(
+                batch_size, num_visual, self.num_children, self.patch_dim
+            )
+
+        children = []
+        for index, branch in enumerate(self.branches):
+            child = branch(visual_features)
+            if patch_features is not None:
+                child = child + patch_features[:, :, index, :].to(
+                    device=child.device, dtype=child.dtype
+                )
+            children.append(child)
+
+        stacked = torch.stack(children, dim=2)  # (B, N_vis, num_children, patch_dim)
+        return stacked.reshape(batch_size, num_visual * self.num_children, self.patch_dim)
+
+
+class ConCor1UpsampleBlock(nn.Module):
+    """2x upsampling block: transposed conv → SiLU → depthwise 3x3 refine → LayerNorm2d."""
+
+    def __init__(self, dim: int):
+        super().__init__()
+        self.up = nn.ConvTranspose2d(dim, dim, kernel_size=2, stride=2)
+        self.act = nn.SiLU()
+        self.refine = nn.Conv2d(dim, dim, kernel_size=3, padding=1, groups=dim, bias=False)
+        self.norm = ConCor1LayerNorm2d(dim)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.up(hidden_states)
+        hidden_states = self.act(hidden_states)
+        hidden_states = self.refine(hidden_states)
+        return self.norm(hidden_states)
+
+
+class ConCor1ConvolutionalDecoder(nn.Module):
+    """Lightweight convolutional decoder: `num_blocks` successive 2x upsamplings."""
+
+    def __init__(self, dim: int, num_blocks: int = 2):
+        super().__init__()
+        self.blocks = nn.ModuleList([ConCor1UpsampleBlock(dim) for _ in range(num_blocks)])
+
+    @property
+    def scale_factor(self) -> int:
+        return 2 ** len(self.blocks)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        for block in self.blocks:
+            hidden_states = block(hidden_states)
+        return hidden_states
+
+
+class ConCor1VisionSegmentationHead(nn.Module):
+    """Predict one image mask per bridge token.
+
+    Two components, as described in the paper:
+
+      1. a *feature decoder* that reconstructs dense visual features from the
+         spatially compressed backbone visual tokens — `patch_expander`
+         (+ optional fusion of pre-merger ViT features) followed by
+         `convolutional_decoder`;
+      2. a *mask predictor* that scores every decoded spatial cell against every
+         bridge token with a bilinear form — `mask_predictor`.
+    """
+
+    def __init__(self, config: ConCor1Config):
+        super().__init__()
+        self.merge_size = config.merge_size
+        self.patch_expander = ConCor1PatchExpander(
+            hidden_size=config.hidden_size,
+            patch_dim=config.vision_hidden_size,
+            num_children=config.merge_size ** 2,
+        )
+        self.convolutional_decoder = ConCor1ConvolutionalDecoder(
+            dim=config.vision_hidden_size,
+            num_blocks=config.num_mask_upsample_blocks,
+        )
+        self.mask_predictor = ConCor1BilinearCorrespondenceScorer(
+            feature_dim=config.vision_hidden_size,
+            bridge_dim=config.hidden_size,
+            correspondence_dim=config.correspondence_dim,
+        )
+
+    def decode_features(
+        self,
+        visual_features: torch.Tensor,
+        patch_features: Optional[torch.Tensor],
+        image_grid_thw: torch.Tensor,
+        visual_token_mask: torch.Tensor,
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        """Expand, fuse and upsample the visual tokens into a dense feature map.
+
+        Returns:
+            features: `(B, N_cells_max, patch_dim)` in row-major order, zero-padded.
+            grid_hw: `(B, 2)` — the `(height, width)` of each sample's cell grid.
+        """
+        expanded = self.patch_expander(visual_features, patch_features=patch_features)
+
+        batch_size = expanded.shape[0]
+        dim = expanded.shape[-1]
+        merge_size = self.merge_size
+        scale = self.convolutional_decoder.scale_factor
+
+        per_sample: List[torch.Tensor] = []
+        grid_hw = expanded.new_zeros((batch_size, 2), dtype=torch.long)
+
+        for index in range(batch_size):
+            num_visual = int(visual_token_mask[index].sum().item())
+            if num_visual == 0:  # text-only sample
+                per_sample.append(expanded.new_zeros(1, dim))
+                continue
+
+            patch_h = int(image_grid_thw[index, 1].item())
+            patch_w = int(image_grid_thw[index, 2].item())
+            merged_h, merged_w = patch_h // merge_size, patch_w // merge_size
+
+            # Interleaved children → row-major 2D patch grid.
+            features = expanded[index, : num_visual * merge_size ** 2]
+            features = features.reshape(merged_h, merged_w, merge_size, merge_size, dim)
+            features = features.permute(0, 2, 1, 3, 4).reshape(patch_h, patch_w, dim)
+
+            # (1, D, patch_h, patch_w) → conv decoder → (1, D, patch_h * s, patch_w * s)
+            feature_map = features.permute(2, 0, 1).unsqueeze(0)
+            feature_map = self.convolutional_decoder(feature_map)
+
+            grid_hw[index, 0] = patch_h * scale
+            grid_hw[index, 1] = patch_w * scale
+            per_sample.append(feature_map.squeeze(0).permute(1, 2, 0).reshape(-1, dim))
+
+        num_cells = max(max(f.shape[0] for f in per_sample), 1)
+        padded = expanded.new_zeros(batch_size, num_cells, dim)
+        for index, features in enumerate(per_sample):
+            padded[index, : features.shape[0]] = features
+        return padded, grid_hw
+
+    def forward(
+        self,
+        visual_features: torch.Tensor,
+        bridge_features: torch.Tensor,
+        patch_features: Optional[torch.Tensor],
+        image_grid_thw: torch.Tensor,
+        visual_token_mask: torch.Tensor,
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        features, grid_hw = self.decode_features(
+            visual_features, patch_features, image_grid_thw, visual_token_mask
+        )
+        return self.mask_predictor(features, bridge_features), grid_hw
+
+
+def extract_tokens_by_mask(hidden_states: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
+    """Gather the hidden states where `mask` is True, keeping left-to-right order.
+
+    Rows with fewer selected positions than the batch maximum are zero-padded.
+
+    Args:
+        hidden_states: `(B, S, D)`
+        mask: `(B, S)` boolean
+
+    Returns:
+        `(B, max_selected, D)`
+    """
+    _, _, dim = hidden_states.shape
+    is_valid = mask.bool()
+
+    max_count = max(int(is_valid.sum(dim=1).max().item()), 1)
+
+    sorted_indices = torch.argsort(is_valid.int(), dim=1, descending=True, stable=True)
+    selected = sorted_indices[:, :max_count]
+
+    extracted = torch.gather(hidden_states, 1, selected.unsqueeze(-1).expand(-1, -1, dim))
+    valid = torch.gather(is_valid, 1, selected)
+    return extracted * valid.unsqueeze(-1).to(extracted.dtype)
+
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Model
+# ══════════════════════════════════════════════════════════════════════════════
+
+
+@dataclass
+class ConCor1Output(ModelOutput):
+    """Correspondence predictions of [`ConCor1ForConceptCorrespondence`].
+
+    Args:
+        presence_logits: `(B, Q)` — logit that bridge `q` holds a valid
+            image-text correspondence.
+        text_mask_logits: `(B, Q, N_text)` — per-bridge logits over the text
+            tokens selected by `text_token_mask` (the bridge's text mask).
+        image_mask_logits: `(B, Q, N_cells)` — per-bridge logits over the decoded
+            image cells, row-major, zero-padded across the batch.
+        image_mask_grid_hw: `(B, 2)` — `(height, width)` of every sample's cell
+            grid, so `image_mask_logits[b, q, : h * w].reshape(h, w)` is the mask
+            map. One cell covers `config.mask_cell_size` pixels per side of the
+            (resized) image.
+        last_hidden_state: `(B, S, D)` — the backbone's final hidden states over
+            the full multimodal sequence.
+    """
+
+    presence_logits: Optional[torch.FloatTensor] = None
+    text_mask_logits: Optional[torch.FloatTensor] = None
+    image_mask_logits: Optional[torch.FloatTensor] = None
+    image_mask_grid_hw: Optional[torch.LongTensor] = None
+    last_hidden_state: Optional[torch.FloatTensor] = None
+
+
+class ConCor1PreTrainedModel(PreTrainedModel):
+    config_class = ConCor1Config
+    base_model_prefix = "concor1"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["Qwen3_5DecoderLayer", "Qwen3_5VisionBlock"]
+    _keep_in_fp32_modules_strict = [
+        "presence_head",
+        "text_segmentation_head",
+        "vision_segmentation_head",
+    ]
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _supports_flex_attn = False
+    _can_compile_fullgraph = False
+
+    def _init_weights(self, module):
+        if isinstance(module, (nn.Linear, nn.Bilinear, nn.Conv2d, nn.ConvTranspose2d)):
+            nn.init.kaiming_uniform_(module.weight, a=math.sqrt(5))
+            if getattr(module, "bias", None) is not None:
+                nn.init.zeros_(module.bias)
+        elif isinstance(module, (nn.LayerNorm, nn.RMSNorm)):
+            if module.weight is not None:
+                nn.init.ones_(module.weight)
+            if getattr(module, "bias", None) is not None:
+                nn.init.zeros_(module.bias)
+
+
+class ConCor1ForConceptCorrespondence(ConCor1PreTrainedModel):
+    """ConCor-1: bidirectional concept correspondence over an image-text pair.
+
+    Example:
+
+    ```python
+    >>> import requests
+    >>> import torch
+    >>> from PIL import Image
+    >>> from transformers import AutoModel, AutoProcessor
+
+    >>> processor = AutoProcessor.from_pretrained("UWGZQ/ConCor-1", trust_remote_code=True)
+    >>> model = AutoModel.from_pretrained("UWGZQ/ConCor-1", trust_remote_code=True, dtype=torch.bfloat16).cuda().eval()
+
+    >>> url = "http://images.cocodataset.org/val2017/000000000285.jpg"
+    >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
+    >>> text = (
+    ...     "This image depicts a close-up of a brown bear in a natural outdoor setting. "
+    ...     "The background consists of lush green grass. In the foreground, a large brown "
+    ...     "bear is positioned centrally."
+    ... )
+    >>> inputs = processor(images=image, text=text, return_tensors="pt").to("cuda")
+    >>> with torch.inference_mode():
+    ...     outputs = model(**inputs)
+    >>> correspondences = processor.post_process_correspondences(
+    ...     outputs, text=text, target_sizes=[(image.height, image.width)]
+    ... )[0]
+    >>> [(round(c["presence_score"], 3), c["text_phrases"]) for c in correspondences]
+    ```
+    """
+
+    def __init__(self, config: ConCor1Config):
+        super().__init__(config)
+
+        if not config.bidirectional_full_attention:
+            logger.warning(
+                "bidirectional_full_attention=False runs the backbone's full-attention layers "
+                "causally. ConCor-1 was trained with bidirectional full attention; predictions "
+                "will be badly degraded."
+            )
+
+        backbone_config = config.backbone_config
+        backbone_config._attn_implementation = config._attn_implementation
+        self.backbone = ConCor1VisionLanguageBackbone(backbone_config)
+
+        self.presence_head = ConCor1PresenceHead(
+            hidden_size=config.hidden_size,
+            intermediate_size=config.presence_hidden_dim,
+        )
+        self.text_segmentation_head = ConCor1BilinearCorrespondenceScorer(
+            feature_dim=config.hidden_size,
+            bridge_dim=config.hidden_size,
+            correspondence_dim=config.correspondence_dim,
+        )
+        self.vision_segmentation_head = ConCor1VisionSegmentationHead(config)
+
+        self.post_init()
+
+    def get_input_embeddings(self) -> nn.Module:
+        return self.backbone.get_input_embeddings()
+
+    def set_input_embeddings(self, value: nn.Module) -> None:
+        self.backbone.set_input_embeddings(value)
+
+    def _encode(
+        self,
+        input_ids: torch.LongTensor,
+        pixel_values: Optional[torch.FloatTensor],
+        image_grid_thw: Optional[torch.LongTensor],
+        visual_token_mask: torch.BoolTensor,
+        attention_mask: Optional[torch.Tensor],
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
+        """Run the backbone and return `(hidden_states, pre_merger_patch_features)`.
+
+        This inlines `Qwen3_5Model.forward` so that the pre-merger ViT patch
+        features (needed by the vision segmentation head) can be kept without
+        running the vision tower twice.
+        """
+        inputs_embeds = self.backbone.get_input_embeddings()(input_ids)
+
+        patch_features = None
+        if pixel_values is not None:
+            vision_output = self.backbone.get_image_features(
+                pixel_values, image_grid_thw, return_dict=True
+            )
+            if self.config.fuse_vision_encoder_features:
+                patch_features = vision_output.last_hidden_state  # (total_patches, patch_dim)
+            image_embeds = torch.cat(vision_output.pooler_output, dim=0).to(
+                inputs_embeds.device, inputs_embeds.dtype
+            )
+            image_mask, _ = self.backbone.get_placeholder_mask(
+                input_ids, inputs_embeds=inputs_embeds, image_features=image_embeds
+            )
+            inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
+
+        mm_token_type_ids = torch.zeros_like(input_ids, dtype=torch.int32)
+        mm_token_type_ids[visual_token_mask] = 1
+        position_ids = self.backbone.compute_3d_position_ids(
+            input_ids=input_ids,
+            image_grid_thw=image_grid_thw,
+            video_grid_thw=None,
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            past_key_values=None,
+            mm_token_type_ids=mm_token_type_ids,
+        )
+
+        outputs = self.backbone.language_model(
+            input_ids=None,
+            inputs_embeds=inputs_embeds,
+            position_ids=position_ids,
+            attention_mask=attention_mask,
+            use_cache=False,
+            bidirectional_full_attention=self.config.bidirectional_full_attention,
+        )
+        return outputs.last_hidden_state, patch_features
+
+    def _gather_patch_features(
+        self,
+        patch_features: torch.Tensor,
+        image_grid_thw: torch.LongTensor,
+        visual_token_mask: torch.BoolTensor,
+        num_visual_tokens: int,
+    ) -> torch.Tensor:
+        """Batch and zero-pad per-image pre-merger patch features.
+
+        The vision tower flattens all images into one sequence; split it per
+        image and pad to `num_visual_tokens * merge_size**2`, keeping the
+        interleaved child order the patch expander produces.
+        """
+        batch_size = visual_token_mask.shape[0]
+        patch_dim = patch_features.shape[-1]
+        num_children = self.config.merge_size ** 2
+        padded = patch_features.new_zeros((batch_size, num_visual_tokens * num_children, patch_dim))
+
+        per_image = patch_features.split(image_grid_thw.prod(dim=1).tolist())
+        for index in range(batch_size):
+            num_visual = int(visual_token_mask[index].sum().item())
+            if num_visual == 0:
+                continue
+            num_patches = num_visual * num_children
+            padded[index, :num_patches] = per_image[index][:num_patches]
+        return padded
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor,
+        bridge_token_mask: torch.BoolTensor,
+        text_token_mask: torch.BoolTensor,
+        visual_token_mask: Optional[torch.BoolTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        pixel_values: Optional[torch.FloatTensor] = None,
+        image_grid_thw: Optional[torch.LongTensor] = None,
+        **kwargs,
+    ) -> ConCor1Output:
+        r"""
+        Args:
+            input_ids (`torch.LongTensor` of shape `(B, S)`):
+                Flat multimodal sequence: `<vision_start>`, `<image_pad>` x N_v,
+                `<vision_end>`, text tokens, then the `Q` bridge token ids.
+            bridge_token_mask (`torch.BoolTensor` of shape `(B, S)`):
+                True at the bridge-token positions.
+            text_token_mask (`torch.BoolTensor` of shape `(B, S)`):
+                True at the text positions the text masks are predicted over
+                (the input text, excluding the vision and bridge tokens).
+            visual_token_mask (`torch.BoolTensor` of shape `(B, S)`, *optional*):
+                True at the `<image_pad>` positions. Required with an image.
+            attention_mask (`torch.Tensor` of shape `(B, S)`, *optional*):
+                1 for real tokens, 0 for padding.
+            pixel_values (`torch.FloatTensor`, *optional*):
+                Flattened image patches from the Qwen3.5 image processor.
+            image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
+                Temporal / height / width patch counts per image.
+
+        Returns:
+            [`ConCor1Output`]
+        """
+        if kwargs.get("use_cache") or kwargs.get("past_key_values") is not None:
+            raise ValueError(
+                "ConCor-1 is not autoregressive: `use_cache` / `past_key_values` are not supported."
+            )
+        if pixel_values is not None and visual_token_mask is None:
+            raise ValueError("`visual_token_mask` is required when `pixel_values` is passed.")
+        if visual_token_mask is None:
+            visual_token_mask = torch.zeros_like(input_ids, dtype=torch.bool)
+        if attention_mask is None:
+            attention_mask = torch.ones_like(input_ids)
+
+        hidden_states, patch_features = self._encode(
+            input_ids=input_ids,
+            pixel_values=pixel_values,
+            image_grid_thw=image_grid_thw,
+            visual_token_mask=visual_token_mask,
+            attention_mask=attention_mask,
+        )
+
+        bridge_features = extract_tokens_by_mask(hidden_states, bridge_token_mask)   # (B, Q, D)
+        text_features = extract_tokens_by_mask(hidden_states, text_token_mask)       # (B, N_text, D)
+
+        presence_logits = self.presence_head(bridge_features).squeeze(-1)            # (B, Q)
+        text_mask_logits = self.text_segmentation_head(text_features, bridge_features)        # (B, Q, N_text)
+
+        image_mask_logits, image_mask_grid_hw = None, None
+        if pixel_values is not None:
+            visual_features = extract_tokens_by_mask(hidden_states, visual_token_mask)
+            if patch_features is not None:
+                patch_features = self._gather_patch_features(
+                    patch_features,
+                    image_grid_thw=image_grid_thw,
+                    visual_token_mask=visual_token_mask,
+                    num_visual_tokens=visual_features.shape[1],
+                )
+            image_mask_logits, image_mask_grid_hw = self.vision_segmentation_head(
+                visual_features=visual_features,
+                bridge_features=bridge_features,
+                patch_features=patch_features,
+                image_grid_thw=image_grid_thw,
+                visual_token_mask=visual_token_mask,
+            )
+
+        return ConCor1Output(
+            presence_logits=presence_logits,
+            text_mask_logits=text_mask_logits,
+            image_mask_logits=image_mask_logits,
+            image_mask_grid_hw=image_mask_grid_hw,
+            last_hidden_state=hidden_states,
+        )
+
+
+__all__ = [
+    "ConCor1ForConceptCorrespondence",
+    "ConCor1PreTrainedModel",
+    "ConCor1Output",
+]

preprocessor_config.json

@@ -0,0 +1,23 @@
+{
+  "size": {
+    "longest_edge": 16777216,
+    "shortest_edge": 65536
+  },
+  "patch_size": 16,
+  "temporal_patch_size": 2,
+  "merge_size": 2,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "ConCor1Processor",
+  "image_processor_type": "Qwen2VLImageProcessorFast",
+  "min_pixels": 1003520,
+  "max_pixels": 1003520
+}

processing_concor1.py

@@ -0,0 +1,489 @@
+# coding=utf-8
+# Copyright 2026 The ConCor-1 authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Processor for ConCor-1.
+
+Turns an image-text pair into the flat multimodal sequence ConCor-1 expects
+(image tokens, text tokens, bridge tokens) and turns the model's per-bridge
+logits back into a set of image-text correspondences: a character-level text
+span set and a binary image mask per correspondence.
+"""
+
+from __future__ import annotations
+
+import string
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.processing_utils import ProcessorMixin
+
+ImageInput = Union["PIL.Image.Image", Sequence["PIL.Image.Image"]] 
+
+_TRIM_CHARACTERS = frozenset(string.punctuation)
+
+
+def compute_span_iou(spans_a: List[List[int]], spans_b: List[List[int]]) -> float:
+    """IoU of two character-span sets, treated as 1-D binary masks over the text."""
+    characters_a = {index for start, end in spans_a for index in range(start, end)}
+    characters_b = {index for start, end in spans_b for index in range(start, end)}
+    union = len(characters_a | characters_b)
+    if union == 0:
+        return 0.0
+    return len(characters_a & characters_b) / union
+
+
+def compute_mask_iou(mask_a: np.ndarray, mask_b: np.ndarray) -> float:
+    """IoU of two binary masks."""
+    flat_a = mask_a.astype(bool).ravel()
+    flat_b = mask_b.astype(bool).ravel()
+    union = np.count_nonzero(flat_a | flat_b)
+    if union == 0:
+        return 0.0
+    return np.count_nonzero(flat_a & flat_b) / union
+
+
+class ConCor1Processor(ProcessorMixin):
+    r"""Constructs a ConCor-1 processor from a Qwen3.5 image processor and tokenizer.
+
+    Args:
+        image_processor: the Qwen3.5 (`Qwen2VLImageProcessorFast`) image processor.
+        tokenizer: the Qwen3.5 tokenizer.
+        num_bridge_tokens (`int`, *optional*, defaults to 385):
+            Number of bridge tokens appended to every sequence.
+        bridge_token_id_start (`int`, *optional*, defaults to 248077):
+            Token id of the first bridge token.
+        patch_size (`int`, *optional*, defaults to 16): vision patch size.
+        merge_size (`int`, *optional*, defaults to 2): patch-merger factor.
+        min_pixels (`int`, *optional*, defaults to 1003520):
+            `min_pixels` forwarded to the image processor.
+        max_pixels (`int`, *optional*, defaults to 1003520):
+            `max_pixels` forwarded to the image processor. The released
+            checkpoint was trained with `min_pixels == max_pixels`, i.e. a fixed
+            ~1.0 M pixel budget.
+        presence_threshold (`float`, *optional*, defaults to 0.1)
+        text_threshold (`float`, *optional*, defaults to 0.45)
+        image_threshold (`float`, *optional*, defaults to 0.45)
+        nms_iou_threshold (`float`, *optional*, defaults to 0.5)
+
+    Example:
+
+    ```python
+    >>> text = "A close-up of a brown bear sitting in lush green grass."
+    >>> inputs = processor(images=image, text=text, return_tensors="pt")
+    >>> outputs = model(**inputs)
+    >>> correspondences = processor.post_process_correspondences(
+    ...     outputs, text=text, target_sizes=[(image.height, image.width)]
+    ... )
+    ```
+    """
+
+    attributes = ["image_processor", "tokenizer"]
+
+    image_processor_class = "Qwen2VLImageProcessorFast"
+    tokenizer_class = "AutoTokenizer"
+
+    def __init__(
+        self,
+        image_processor=None,
+        tokenizer=None,
+        num_bridge_tokens: int = 385,
+        bridge_token_id_start: int = 248077,
+        patch_size: int = 16,
+        merge_size: int = 2,
+        num_mask_upsample_blocks: int = 2,
+        min_pixels: int = 1003520,
+        max_pixels: int = 1003520,
+        presence_threshold: float = 0.1,
+        text_threshold: float = 0.45,
+        image_threshold: float = 0.45,
+        nms_iou_threshold: float = 0.5,
+        **kwargs,
+    ):
+        self.num_bridge_tokens = num_bridge_tokens
+        self.bridge_token_id_start = bridge_token_id_start
+        self.patch_size = patch_size
+        self.merge_size = merge_size
+        self.num_mask_upsample_blocks = num_mask_upsample_blocks
+        self.min_pixels = min_pixels
+        self.max_pixels = max_pixels
+        self.presence_threshold = presence_threshold
+        self.text_threshold = text_threshold
+        self.image_threshold = image_threshold
+        self.nms_iou_threshold = nms_iou_threshold
+        super().__init__(image_processor, tokenizer, **kwargs)
+
+    @property
+    def bridge_token_ids(self) -> List[int]:
+        start = self.bridge_token_id_start
+        return list(range(start, start + self.num_bridge_tokens))
+
+
+    def __call__(
+        self,
+        images: Optional[ImageInput] = None,
+        text: Optional[Union[str, List[str]]] = None,
+        return_tensors: str = "pt",
+        **kwargs,
+    ) -> BatchFeature:
+        """Build ConCor-1's flat multimodal sequences for one or more image-text pairs.
+
+        The text is *not* wrapped in a chat template: ConCor-1 consumes the raw
+        text (a caption, a list of category names, a referring expression, ...)
+        directly, and predicts its text masks over exactly these tokens.
+
+        Args:
+            images: one PIL image, or a list with one image per text.
+            text: the paired text, or a list of texts.
+            return_tensors: only `"pt"` is supported.
+
+        Returns:
+            [`BatchFeature`] with `input_ids`, `attention_mask`,
+            `visual_token_mask`, `text_token_mask`, `bridge_token_mask` and —
+            when images are given — `pixel_values` and `image_grid_thw`.
+        """
+        if return_tensors != "pt":
+            raise ValueError(f"ConCor1Processor only supports return_tensors='pt', got {return_tensors!r}")
+        if text is None:
+            raise ValueError(
+                "ConCor-1 always grounds text: pass the image's paired `text` "
+                "(a caption, category list or referring expression)."
+            )
+
+        texts = [text] if isinstance(text, str) else list(text)
+        if images is None:
+            image_list = []
+        elif isinstance(images, (list, tuple)):
+            image_list = list(images)
+        else:
+            image_list = [images]
+
+        if image_list and len(image_list) != len(texts):
+            raise ValueError(
+                f"Got {len(image_list)} image(s) and {len(texts)} text(s); pass one image per text."
+            )
+
+        pixel_values, image_grid_thw = None, None
+        num_visual_tokens = [0] * len(texts)
+        if image_list:
+            image_inputs = self.image_processor(
+                images=image_list,
+                return_tensors="pt",
+                min_pixels=kwargs.pop("min_pixels", self.min_pixels),
+                max_pixels=kwargs.pop("max_pixels", self.max_pixels),
+            )
+            pixel_values = image_inputs["pixel_values"]
+            image_grid_thw = image_inputs["image_grid_thw"]
+            num_visual_tokens = [
+                int(grid.prod().item()) // (self.merge_size ** 2) for grid in image_grid_thw
+            ]
+
+        vision_start_id = self.tokenizer.convert_tokens_to_ids("<|vision_start|>")
+        vision_end_id = self.tokenizer.convert_tokens_to_ids("<|vision_end|>")
+        image_pad_id = self.tokenizer.convert_tokens_to_ids("<|image_pad|>")
+        bridge_ids = self.bridge_token_ids
+
+        sequences, visual_masks, text_masks, bridge_masks = [], [], [], []
+        for index, sample_text in enumerate(texts):
+            ids: List[int] = []
+            visual_mask: List[bool] = []
+            text_mask: List[bool] = []
+
+            if num_visual_tokens[index] > 0:
+                ids.append(vision_start_id)
+                ids.extend([image_pad_id] * num_visual_tokens[index])
+                ids.append(vision_end_id)
+                visual_mask.extend([False] + [True] * num_visual_tokens[index] + [False])
+                text_mask.extend([False] * (num_visual_tokens[index] + 2))
+
+            text_ids = self.tokenizer.encode(sample_text, add_special_tokens=False)
+            if not text_ids:
+                raise ValueError(f"Text {index} is empty after tokenization: {sample_text!r}")
+            ids.extend(text_ids)
+            visual_mask.extend([False] * len(text_ids))
+            text_mask.extend([True] * len(text_ids))
+
+            ids.extend(bridge_ids)
+            visual_mask.extend([False] * len(bridge_ids))
+            text_mask.extend([False] * len(bridge_ids))
+
+            bridge_mask = [False] * (len(ids) - len(bridge_ids)) + [True] * len(bridge_ids)
+
+            sequences.append(ids)
+            visual_masks.append(visual_mask)
+            text_masks.append(text_mask)
+            bridge_masks.append(bridge_mask)
+
+        # Right-pad to the longest sequence, as in training.
+        max_length = max(len(ids) for ids in sequences)
+        pad_token_id = self.tokenizer.pad_token_id or 0
+        batch_size = len(sequences)
+
+        input_ids = torch.full((batch_size, max_length), pad_token_id, dtype=torch.long)
+        attention_mask = torch.zeros((batch_size, max_length), dtype=torch.long)
+        visual_token_mask = torch.zeros((batch_size, max_length), dtype=torch.bool)
+        text_token_mask = torch.zeros((batch_size, max_length), dtype=torch.bool)
+        bridge_token_mask = torch.zeros((batch_size, max_length), dtype=torch.bool)
+
+        for index, ids in enumerate(sequences):
+            length = len(ids)
+            input_ids[index, :length] = torch.tensor(ids, dtype=torch.long)
+            attention_mask[index, :length] = 1
+            visual_token_mask[index, :length] = torch.tensor(visual_masks[index], dtype=torch.bool)
+            text_token_mask[index, :length] = torch.tensor(text_masks[index], dtype=torch.bool)
+            bridge_token_mask[index, :length] = torch.tensor(bridge_masks[index], dtype=torch.bool)
+
+        data = {
+            "input_ids": input_ids,
+            "attention_mask": attention_mask,
+            "visual_token_mask": visual_token_mask,
+            "text_token_mask": text_token_mask,
+            "bridge_token_mask": bridge_token_mask,
+        }
+        if pixel_values is not None:
+            data["pixel_values"] = pixel_values
+            data["image_grid_thw"] = image_grid_thw
+        return BatchFeature(data=data)
+
+    # ── Post-processing ──────────────────────────────────────────────────────
+
+    def text_logits_to_spans(
+        self,
+        text_logits: np.ndarray,
+        offset_mapping: Sequence[Tuple[int, int]],
+        text: str,
+        text_threshold: float,
+    ) -> List[List[int]]:
+        """Convert one bridge token's text-mask logits into character spans.
+
+        Active text tokens are mapped to their character ranges, merged into
+        maximal contiguous spans and trimmed of leading/trailing whitespace and
+        punctuation. A text mask may consist of several disjoint spans, which is
+        how discontinuous and co-referring mentions are represented.
+        """
+        probabilities = 1.0 / (1.0 + np.exp(-text_logits.astype(np.float64)))
+        active = probabilities >= text_threshold
+
+        characters = set()
+        for token_index in np.where(active)[0]:
+            if token_index < len(offset_mapping):
+                start, end = offset_mapping[token_index]
+                characters.update(range(int(start), int(end)))
+        if not characters:
+            return []
+
+        ordered = sorted(characters)
+        spans: List[List[int]] = []
+        start = ordered[0]
+        end = start + 1
+        for character in ordered[1:]:
+            if character == end:
+                end += 1
+            else:
+                spans.append([start, end])
+                start, end = character, character + 1
+        spans.append([start, end])
+
+        trimmed: List[List[int]] = []
+        for span_start, span_end in spans:
+            span_start = max(0, span_start)
+            span_end = min(len(text), span_end)
+            while span_start < span_end and (
+                text[span_start].isspace() or text[span_start] in _TRIM_CHARACTERS
+            ):
+                span_start += 1
+            while span_end > span_start and (
+                text[span_end - 1].isspace() or text[span_end - 1] in _TRIM_CHARACTERS
+            ):
+                span_end -= 1
+            if span_start < span_end:
+                trimmed.append([span_start, span_end])
+        return trimmed
+
+    @staticmethod
+    def suppress_duplicate_correspondences(
+        masks: List[Optional[np.ndarray]],
+        spans: List[List[List[int]]],
+        presence_scores: np.ndarray,
+        nms_iou_threshold: float,
+    ) -> List[int]:
+        """Greedy NMS over correspondences, ranked by presence score.
+
+        A candidate is suppressed only when it duplicates a higher-scoring one on
+        *both* sides of the correspondence: image-mask IoU and text-span IoU both
+        exceed `nms_iou_threshold`.
+        """
+        count = len(spans)
+        if nms_iou_threshold <= 0.0 or count <= 1:
+            return list(range(count))
+
+        kept: List[int] = []
+        for candidate in np.argsort(-presence_scores):
+            candidate = int(candidate)
+            suppressed = False
+            for reference in kept:
+                if masks[candidate] is not None and masks[reference] is not None:
+                    if compute_mask_iou(masks[candidate], masks[reference]) < nms_iou_threshold:
+                        continue
+                if compute_span_iou(spans[candidate], spans[reference]) >= nms_iou_threshold:
+                    suppressed = True
+                    break
+            if not suppressed:
+                kept.append(candidate)
+        return sorted(kept)
+
+    def post_process_correspondences(
+        self,
+        outputs,
+        text: Union[str, List[str]],
+        target_sizes: Optional[Sequence[Tuple[int, int]]] = None,
+        presence_threshold: Optional[float] = None,
+        text_threshold: Optional[float] = None,
+        image_threshold: Optional[float] = None,
+        nms_iou_threshold: Optional[float] = None,
+        return_masks: bool = True,
+    ) -> List[List[Dict]]:
+        """Turn per-bridge logits into a set of image-text correspondences.
+
+        Bridge tokens whose presence probability passes `presence_threshold` are
+        kept; each keeps a character-span text mask and a binary image mask
+        (logits bilinearly upsampled to `target_sizes`, then thresholded), and
+        duplicates are removed with NMS.
+
+        Args:
+            outputs: the [`ConCor1Output`] returned by the model.
+            text: the same text that was passed to `__call__`.
+            target_sizes: `(height, width)` per sample the image masks are
+                resized to — normally the original image size. Defaults to the
+                processed image resolution.
+            presence_threshold / text_threshold / image_threshold /
+            nms_iou_threshold: override the defaults (0.1 / 0.45 / 0.45 / 0.5).
+            return_masks: set to `False` to skip mask upsampling and return text
+                masks only.
+
+        Returns:
+            One list of correspondences per sample, sorted by decreasing
+            presence score. Each correspondence is a dict with
+            `bridge_index`, `presence_score`, `text_spans`, `text_phrases` and
+            (unless disabled) `mask`.
+        """
+        presence_threshold = (
+            self.presence_threshold if presence_threshold is None else presence_threshold
+        )
+        text_threshold = self.text_threshold if text_threshold is None else text_threshold
+        image_threshold = self.image_threshold if image_threshold is None else image_threshold
+        nms_iou_threshold = (
+            self.nms_iou_threshold if nms_iou_threshold is None else nms_iou_threshold
+        )
+
+        texts = [text] if isinstance(text, str) else list(text)
+        presence_probabilities = torch.sigmoid(outputs.presence_logits.float()).cpu().numpy()
+        batch_size = presence_probabilities.shape[0]
+        if len(texts) != batch_size:
+            raise ValueError(f"Got {len(texts)} text(s) for a batch of {batch_size}.")
+
+        results: List[List[Dict]] = []
+        for sample in range(batch_size):
+            sample_text = texts[sample]
+            offset_mapping = self.tokenizer(
+                sample_text, return_offsets_mapping=True, add_special_tokens=False
+            )["offset_mapping"]
+
+            active = np.where(presence_probabilities[sample] >= presence_threshold)[0]
+
+            spans_per_bridge: List[List[List[int]]] = []
+            masks_per_bridge: List[Optional[np.ndarray]] = []
+            for bridge_index in active:
+                text_logits = outputs.text_mask_logits[sample, bridge_index].float().cpu().numpy()
+                spans_per_bridge.append(
+                    self.text_logits_to_spans(
+                        text_logits, offset_mapping, sample_text, text_threshold
+                    )
+                )
+                masks_per_bridge.append(
+                    self._decode_image_mask(
+                        outputs,
+                        sample=sample,
+                        bridge_index=int(bridge_index),
+                        target_size=None if target_sizes is None else tuple(target_sizes[sample]),
+                        image_threshold=image_threshold,
+                    )
+                    if return_masks
+                    else None
+                )
+
+            kept = self.suppress_duplicate_correspondences(
+                masks_per_bridge,
+                spans_per_bridge,
+                presence_probabilities[sample][active],
+                nms_iou_threshold,
+            )
+
+            correspondences = []
+            for position in kept:
+                bridge_index = int(active[position])
+                spans = spans_per_bridge[position]
+                correspondence = {
+                    "bridge_index": bridge_index,
+                    "presence_score": float(presence_probabilities[sample, bridge_index]),
+                    "text_spans": spans,
+                    "text_phrases": [sample_text[start:end] for start, end in spans],
+                }
+                if return_masks:
+                    correspondence["mask"] = masks_per_bridge[position]
+                correspondences.append(correspondence)
+
+            correspondences.sort(key=lambda item: -item["presence_score"])
+            results.append(correspondences)
+        return results
+
+    def _decode_image_mask(
+        self,
+        outputs,
+        sample: int,
+        bridge_index: int,
+        target_size: Optional[Tuple[int, int]],
+        image_threshold: float,
+    ) -> Optional[np.ndarray]:
+        """Reshape, upsample and threshold one bridge token's image-mask logits."""
+        if outputs.image_mask_logits is None:
+            return None
+
+        height = int(outputs.image_mask_grid_hw[sample, 0].item())
+        width = int(outputs.image_mask_grid_hw[sample, 1].item())
+        if height == 0 or width == 0:
+            return None
+
+        logits = outputs.image_mask_logits[sample, bridge_index, : height * width]
+        logits = logits.float().reshape(1, 1, height, width)
+
+        if target_size is None:
+            target_size = (height * self.mask_cell_size, width * self.mask_cell_size)
+
+        upsampled = F.interpolate(
+            logits, size=tuple(int(value) for value in target_size), mode="bilinear", align_corners=False
+        )
+        probabilities = torch.sigmoid(upsampled)[0, 0].cpu().numpy()
+        return probabilities >= image_threshold
+
+    @property
+    def mask_cell_size(self) -> int:
+        """Pixels per side of one image-mask cell in the processed image (4 px)."""
+        return self.patch_size // (2 ** self.num_mask_upsample_blocks)
+
+
+__all__ = ["ConCor1Processor"]

processor_config.json

@@ -0,0 +1,17 @@
+{
+  "processor_class": "ConCor1Processor",
+  "auto_map": {
+    "AutoProcessor": "processing_concor1.ConCor1Processor"
+  },
+  "num_bridge_tokens": 385,
+  "bridge_token_id_start": 248077,
+  "patch_size": 16,
+  "merge_size": 2,
+  "num_mask_upsample_blocks": 2,
+  "min_pixels": 1003520,
+  "max_pixels": 1003520,
+  "presence_threshold": 0.1,
+  "text_threshold": 0.45,
+  "image_threshold": 0.45,
+  "nms_iou_threshold": 0.5
+}

requirements.txt

@@ -0,0 +1,22 @@
+# ConCor-1 inference requirements.
+# Versions the released checkpoint was trained and evaluated with are pinned as
+# lower bounds; transformers must be new enough to ship the Qwen3.5 architecture.
+torch>=2.8.0
+torchvision>=0.23.0   # required by the fast image processor
+transformers==5.3.0
+safetensors>=0.4.0
+numpy>=1.24
+pillow>=10.0
+
+# Qwen3.5's gated-delta-net (linear-attention) kernels.
+flash-linear-attention>=0.4.1
+
+# Optional, and both need a CUDA compiler. Neither changes the set of
+# correspondences the model returns.
+#   causal-conv1d: a fused causal convolution for the linear-attention layers;
+#                  without it transformers falls back to a pure-torch path.
+#   flash-attn:    exact reproduction of the paper's logits, with
+#                  attn_implementation="flash_attention_2". The default,
+#                  attn_implementation="sdpa", needs no extra dependency.
+# causal-conv1d>=1.4.0
+# flash-attn>=2.8.0

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42
+size 12807982

tokenizer_config.json

@@ -0,0 +1,306 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "248044": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248045": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248046": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248047": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248048": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248049": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248050": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248051": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248052": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248053": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248054": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248055": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248056": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248057": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248058": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248059": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248060": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248061": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248062": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248063": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248064": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248065": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248066": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248067": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248068": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248069": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "248070": {
+      "content": "<|audio_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248071": {
+      "content": "<|audio_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248072": {
+      "content": "<tts_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248073": {
+      "content": "<tts_text_bos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248074": {
+      "content": "<tts_text_eod>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248075": {
+      "content": "<tts_text_bos_single>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248076": {
+      "content": "<|audio_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- macro render_content(content, do_vision_count, is_system_content=false) %}\n    {%- if content is string %}\n        {{- content }}\n    {%- elif content is iterable and content is not mapping %}\n        {%- for item in content %}\n            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}\n                {%- if is_system_content %}\n                    {{- raise_exception('System message cannot contain images.') }}\n                {%- endif %}\n                {%- if do_vision_count %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                {%- endif %}\n                {%- if add_vision_id %}\n                    {{- 'Picture ' ~ image_count.value ~ ': ' }}\n                {%- endif %}\n                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}\n            {%- elif 'video' in item or item.type == 'video' %}\n                {%- if is_system_content %}\n                    {{- raise_exception('System message cannot contain videos.') }}\n                {%- endif %}\n                {%- if do_vision_count %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                {%- endif %}\n                {%- if add_vision_id %}\n                    {{- 'Video ' ~ video_count.value ~ ': ' }}\n                {%- endif %}\n                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}\n            {%- elif 'text' in item %}\n                {{- item.text }}\n            {%- else %}\n                {{- raise_exception('Unexpected item type in content.') }}\n            {%- endif %}\n        {%- endfor %}\n    {%- elif content is none or content is undefined %}\n        {{- '' }}\n    {%- else %}\n        {{- raise_exception('Unexpected content type.') }}\n    {%- endif %}\n{%- endmacro %}\n{%- if not messages %}\n    {{- raise_exception('No messages provided.') }}\n{%- endif %}\n{%- if tools and tools is iterable and tools is not mapping %}\n    {{- '<|im_start|>system\\n' }}\n    {{- \"# Tools\\n\\nYou have access to the following functions:\\n\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\" }}\n    {{- '\\n\\nIf you choose to call a function ONLY reply in the following format with NO suffix:\\n\\n<tool_call>\\n<function=example_function_name>\\n<parameter=example_parameter_1>\\nvalue_1\\n</parameter>\\n<parameter=example_parameter_2>\\nThis is the value for the second parameter\\nthat can span\\nmultiple lines\\n</parameter>\\n</function>\\n</tool_call>\\n\\n<IMPORTANT>\\nReminder:\\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\\n- Required parameters MUST be specified\\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\\n</IMPORTANT>' }}\n    {%- if messages[0].role == 'system' %}\n        {%- set content = render_content(messages[0].content, false, true)|trim %}\n        {%- if content %}\n            {{- '\\n\\n' + content }}\n        {%- endif %}\n    {%- endif %}\n    {{- '<|im_end|>\\n' }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {%- set content = render_content(messages[0].content, false, true)|trim %}\n        {{- '<|im_start|>system\\n' + content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n    {%- set index = (messages|length - 1) - loop.index0 %}\n    {%- if ns.multi_step_tool and message.role == \"user\" %}\n        {%- set content = render_content(message.content, false)|trim %}\n        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}\n            {%- set ns.multi_step_tool = false %}\n            {%- set ns.last_query_index = index %}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if ns.multi_step_tool %}\n    {{- raise_exception('No user query found in messages.') }}\n{%- endif %}\n{%- for message in messages %}\n    {%- set content = render_content(message.content, true)|trim %}\n    {%- if message.role == \"system\" %}\n        {%- if not loop.first %}\n            {{- raise_exception('System message must be at the beginning.') }}\n        {%- endif %}\n    {%- elif message.role == \"user\" %}\n        {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is string %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in content %}\n                {%- set reasoning_content = content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n                {%- set content = content.split('</think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- set reasoning_content = reasoning_content|trim %}\n        {%- if loop.index0 > ns.last_query_index %}\n            {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content + '\\n</think>\\n\\n' + content }}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if tool_call.function is defined %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {%- if loop.first %}\n                    {%- if content|trim %}\n                        {{- '\\n\\n<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n                    {%- else %}\n                        {{- '<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n                    {%- endif %}\n                {%- else %}\n                    {{- '\\n<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n                {%- endif %}\n                {%- if tool_call.arguments is defined %}\n                    {%- for args_name, args_value in tool_call.arguments|items %}\n                        {{- '<parameter=' + args_name + '>\\n' }}\n                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}\n                        {{- args_value }}\n                        {{- '\\n</parameter>\\n' }}\n                    {%- endfor %}\n                {%- endif %}\n                {{- '</function>\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.previtem and loop.previtem.role != \"tool\" %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- content }}\n        {{- '\\n</tool_response>' }}\n        {%- if not loop.last and loop.nextitem.role != \"tool\" %}\n            {{- '<|im_end|>\\n' }}\n        {%- elif loop.last %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- else %}\n        {{- raise_exception('Unexpected message role.') }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is true %}\n        {{- '<think>\\n' }}\n    {%- else %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- endif %}\n{%- endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 262144,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "add_bos_token": false,
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "extra_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "processor_class": "ConCor1Processor"
+}