VISTA Lab Logo
Gazi University Logo
← Back to Blog List

SigLIP-2 From Scratch: Understanding Modern Vision Language Learning

June 10 2026 • by VISTA Lab Researcher • 7 minute read
Executive Summary: Vision Language Models (VLMs) have rapidly become one of the most influential research areas in multimodal deep learning. While traditional CLIP-like architectures achieve impressive performance, they often learn shortcut correlations instead of developing true semantic understanding, especially in tasks requiring localization and fine-grained reasoning. SigLIP-2 addresses these challenges by introducing critical training improvements like self-distillation, masked prediction, and location-aware captioning. This tutorial explores the architecture, highlights why these innovations matter, and provides a complete from-scratch PyTorch implementation.

1. Introduction

Vision Language Models have rapidly become one of the most influential research areas in multimodal deep learning. These models aim to jointly understand visual and textual information by learning a shared embedding space between images and language.

The success of architectures such as CLIP demonstrated that large-scale image-text alignment can produce highly transferable representations. These representations can then be used for tasks such as zero-shot classification, image retrieval, OCR, visual reasoning, and semantic understanding.

Despite their impressive performance, traditional CLIP-like architectures still suffer from important limitations. In many cases, models learn shortcut correlations instead of developing true semantic understanding. Rather than understanding entire objects or spatial relationships, they may focus on simple visual patterns that correlate with text labels.

This issue becomes particularly problematic in tasks requiring localization, OCR, dense scene understanding, and fine-grained semantic reasoning.

SigLIP-2 addresses these challenges by introducing several important training improvements while preserving the simplicity of the original SigLIP framework.

In this tutorial, we will examine:

The primary goal of this document is to provide an intuitive understanding of SigLIP-2 for researchers and engineers interested in modern multimodal learning.

2. Background: Vision Language Models

Vision Language Models are architectures designed to connect visual understanding with natural language understanding.

The general workflow is relatively straightforward:

This alignment process enables the model to understand semantic relationships between visual and textual information without requiring handcrafted supervision.

Modern multimodal systems typically rely on transformer-based architectures due to their strong capability for modeling long-range dependencies and contextual interactions.

However, while transformers provide strong representation learning, the training objective itself remains equally important. A model optimized only for image-text similarity may fail to learn localization, spatial reasoning, or detailed semantic structure.

This observation motivated the development of SigLIP-2.

3. What is SigLIP?

SigLIP (Sigmoid Loss for Language Image Pretraining) is a CLIP-style multimodal learning framework designed for scalable image-text representation learning.

The architecture mainly consists of:

Unlike CLIP, which uses softmax-based contrastive learning, SigLIP introduces a sigmoid-based pairwise optimization strategy.

Instead of normalizing all image-text similarities globally through softmax, SigLIP independently optimizes positive and negative pairs using sigmoid probabilities.

This modification improves scalability and optimization efficiency during large-scale multimodal training.

General architecture of SigLIP-2
Figure 1: General architecture of SigLIP-2

As illustrated in Figure 1, the architecture contains independent image and text encoders whose outputs are aligned inside a shared representation space.

4. Limitations of Traditional Contrastive Learning

Although contrastive learning is highly effective, it also introduces several limitations. The primary issue is shortcut learning. Instead of understanding complete object structures, models may rely on simple visual correlations such as:

These superficial signals can become sufficient for minimizing the training objective. As a consequence, the learned representations may fail to capture:

This becomes especially problematic in applications requiring detailed visual reasoning. SigLIP-2 was specifically designed to overcome these limitations.

5. Self Distillation

One of the most important additions introduced in SigLIP-2 is Self Distillation. Traditional Knowledge Distillation typically requires two separate models:

The student attempts to imitate the probability distribution produced by the teacher. SigLIP-2 instead adopts the self-distillation strategy inspired by DINO. Rather than using two different architectures, the same model acts as both teacher and student. The teacher parameters are updated using Exponential Moving Average (EMA):

$$\theta_t=m\theta_t+(1-m)\theta_s$$

where:

This creates two different optimization dynamics:

As a result, the model learns more stable and semantically meaningful representations.

Self-distillation significantly improves:

These improvements are especially important for OCR and fine-grained visual tasks.

6. Masked Prediction

SigLIP-2 also integrates masked prediction strategies inspired by Masked AutoEncoders (MAE).

The central idea is straightforward:

The student must then produce representations similar to the teacher despite missing information.

This training strategy forces the model to develop stronger semantic reasoning rather than relying only on superficial visual shortcuts. Masked prediction improves:

The combination of self-distillation and masked prediction creates substantially richer visual representations compared to traditional contrastive learning alone.

7. AutoRegressive Decoder

Another major innovation introduced in SigLIP-2 is the AutoRegressive (AR) Decoder. Traditional CLIP-style models only optimize image-text similarity. While effective, this objective alone is insufficient for deep semantic understanding. To solve this issue, SigLIP-2 introduces an additional caption generation objective. A lightweight transformer decoder is trained autoregressively to generate textual descriptions from image representations.

Conceptually, this process resembles language modeling:

The image encoder interacts with the decoder through cross-attention mechanisms. This enables the model to learn:

Unlike standard contrastive objectives, caption generation forces the encoder to preserve much richer semantic information.

8. LocCa: Location Aware Captioning

SigLIP-2 further improves caption supervision through LocCa (Location Aware Captioning).

The motivation behind LocCa is that visual understanding should also contain spatial awareness.

Instead of producing generic captions, the decoder is encouraged to describe:

This additional supervision dramatically improves localization capability and spatial reasoning.

As a result, the image encoder learns much more informative visual representations. This is one of the key reasons why SigLIP-2 performs significantly better on OCR and localization-sensitive tasks.

9. Sigmoid Loss

Unlike CLIP, which relies on a softmax-based contrastive objective over the entire batch, SigLIP replaces this with a pairwise binary classification formulation using a sigmoid loss. Given a batch of image-text embeddings, similarity logits are computed as:

$$s_{ij}=\frac{f_{i}^{img}\cdot f_{j}^{txt}}{||f_{i}^{img}||||f_{j}^{txt}||}$$

where \(s_{ij}\) denotes cosine similarity between image i and text j.

Instead of constructing a global softmax over all pairs, SigLIP defines a binary target matrix:

$$Y=2I-1$$

where \(Y_{ii}=1\) for matched pairs and \(Y_{ij}=-1\) for \(i\ne j\).

The optimization objective is then defined as a pairwise logistic (sigmoid) loss:

$$\mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{N}\log\sigma(Y_{ij}\cdot s_{ij})$$

where \(\sigma(\cdot)\) is the sigmoid function.

This formulation treats each image-text pair independently, avoiding global normalization and enabling more stable optimization at scale.

Compared to softmax-based contrastive learning, sigmoid-based optimization provides:

This independent pairwise supervision is a core property of the SigLIP family and a key factor behind its training efficiency.

10. Why SigLIP-2 Performs Better

The success of SigLIP-2 comes from combining multiple complementary learning objectives.

Instead of relying only on image-text alignment, the model integrates:

Together, these mechanisms enable the model to learn representations that are:

This leads to significantly stronger multimodal understanding compared to traditional CLIP-style architectures.

11. From Scratch Implementation

The complete PyTorch implementation of SigLIP-2 from scratch, including vision and text encoders, attention mechanisms, pooling heads, and the full multimodal contrastive learning objective, is available in the following repository:

View SigLIP-2 From Scratch PyTorch Implementation on GitHub
Qualitative results produced by the SigLIP-2 implementation
Figure 2: Qualitative results produced by the SigLIP-2 implementation

12. Final Thoughts

SigLIP-2 represents an important evolution of CLIP-style multimodal learning.

Rather than redesigning the transformer architecture entirely, the model improves the learning process itself through:

These additions allow the model to move beyond simple image-text alignment toward deeper semantic and spatial understanding.

As Vision Language Models continue evolving, approaches such as SigLIP-2 demonstrate that richer supervision strategies are essential for building more capable multimodal systems.

13. References