Camera Pipeline in Embedded & Automotive Systems

From Image Sensor to SoC: Serializers, GMSL/FPD-Link, Deserializers, MIPI CSI-2 & Virtual Channels

Modern embedded and automotive systems rely heavily on cameras for perception: surround view, driver monitoring, autonomous driving, robotics, ADAS, and more.
But the camera pipeline is not just “plug a camera into the SoC.” It is a carefully engineered, multi-stage data path involving sensors, serializers, high-speed links, deserializers, CSI receivers, and software pipelines.

In this blog, we walk through the entire camera pipeline end-to-end, explaining each component in detail:

• Camera sensors

• Serializers

• High-speed links (GMSL2 / GMSL3 / FPD-Link III / IV)

• Deserializers

• MIPI CSI-2

• Virtual Channels (VC)

• CSI receivers on the SoC

• Software abstraction (sensor IDs, pipelines)

• Synchronization, control, and bandwidth considerations

By the end, you should have a clear mental model of how pixels travel from the camera lens all the way into memory on your SoC.

---

1. Camera Pipeline Overview (Big Picture)

At a high level, a modern camera pipeline looks like this:

[Camera Sensor]
      ↓
[Serializer]
      ↓  (High-speed automotive link: GMSL / FPD-Link)
[Deserializer]
      ↓  (MIPI CSI-2)
[SoC CSI Receiver (NVCSI / CSI-Rx)]
      ↓
[VI / ISP / Memory / GPU / AI]

Each stage solves a specific problem:

---

2. Camera Sensor: Where Pixels Are Born

The image sensor is the starting point of the pipeline. It contains:

• A pixel array (CMOS typically)

• Analog front-end (AFE)

• ADCs (analog → digital)

• On-chip processing (binning, cropping, HDR, etc.)

• MIPI CSI-2 or parallel output interface

Common outputs:

• RAW10 / RAW12 / RAW14

• YUV422 / YUV420

• RGB

Key properties:

• Resolution (e.g., 2888 × 1508)

• Frame rate (e.g., 10 FPS, 30 FPS)

• Pixel format (RAW vs YUV)

• Lane count (2-lane / 4-lane CSI)

Sensors typically cannot drive long cables reliably. That’s where serializers come in.

---

3. Serializers: Preparing Data for Long-Distance Transport

A serializer sits physically near the camera sensor (often on the camera module PCB).

What a serializer does:

• Accepts:

  • MIPI CSI-2 or parallel pixel data from the sensor

• Converts:

  • Wide parallel or CSI data → single high-speed serial stream

• Adds:

  • Framing, encoding, error detection (CRC/ECC)

• Transmits:

  • Video + embedded data + control + clock over a single coax or STP cable

Why serializers exist:

In automotive designs, serializers are usually part of camera modules.

---

4. High-Speed Links: GMSL2 / GMSL3 / FPD-Link III / IV

To move camera data across the vehicle (often several meters), specialized links are used:

🔹 GMSL (Gigabit Multimedia Serial Link) – Analog Devices / Maxim

• GMSL2:

  • Up to ~6 Gbps

  • Widely used in ADAS and surround view

• GMSL3:

  • Higher bandwidth (multi-gigabit)

  • Designed for high-resolution multi-camera systems

🔹 FPD-Link (Flat Panel Display Link) – Texas Instruments

• FPD-Link III / IV

• Similar role as GMSL

• Point-to-point high-speed camera transport

What these links carry:

Over a single cable, they carry:

• Video data

• Control (I2C tunneling to configure sensors)

• Synchronization (FSYNC / trigger)

• Diagnostics and error reporting

• Power (in many designs)

So one cable can do: Power + Video + Control + Sync

---

5. Deserializers: Aggregation and Fan-In

A deserializer sits near the SoC. It receives multiple high-speed links from different cameras.

What a deserializer does:

• Recovers serialized data from multiple camera links

• Performs:

  • Clock recovery

  • Error detection

  • Lane alignment

• Aggregates:

  • Multiple camera streams

• Outputs:

  • Standard MIPI CSI-2 toward the SoC

Example:

• One deserializer might have 4 physical input links:

  • Camera A

  • Camera B

  • Camera C

  • Camera D

Internally, the deserializer merges these into one CSI-2 output with multiple virtual channels.

This is a key architectural pivot point:

The deserializer is where multiple physical camera links become one logical CSI interface.

---

6. MIPI CSI-2: Standard Camera Interface into the SoC

MIPI CSI-2 is the de facto standard interface between camera subsystems and application processors.

Characteristics:

• High-speed differential lanes (D-PHY / C-PHY)

• Packet-based protocol

• Supports:

  • Multiple data types (RAW, YUV, embedded metadata)

  • Multiple virtual channels

• Lane configurations:

  • 2-lane, 4-lane, 8-lane

Why CSI-2 is used:

---

7. Virtual Channels (VC): Multiplexing Multiple Cameras

This is one of the most misunderstood but powerful concepts.

What is a Virtual Channel?

A Virtual Channel (VC) is a logical stream identifier inside CSI-2 packets.

Think of it as: A label attached to each packet telling the receiver which camera this data belongs to.

Example:

All of these streams can travel over the same CSI-2 physical wires, interleaved in time.

Why VCs exist:

• Reduce pin count on SoC

• Allow aggregation at the deserializer

• Enable scalable camera architectures

Important:

• Virtual Channels are not physical links

• VC IDs are local to one CSI interface

• VC0 on CSI-A and VC0 on CSI-C are totally independent

---

8. CSI Receiver on the SoC: Demultiplexing Streams

On the SoC side (e.g., NVIDIA Thor, Orin, Qualcomm, TI, etc.):

• There are CSI receiver blocks (often called NVCSI, CSI-Rx, etc.)

• Each CSI block:

  • Receives MIPI CSI-2

  • Separates streams by Virtual Channel

  • Routes them into independent capture pipelines

Conceptually:

CSI Port (physical lanes)
   ├── VC0 → Camera Pipeline 0
   ├── VC1 → Camera Pipeline 1
   ├── VC2 → Camera Pipeline 2

From software`s point of view: Each (CSI port, VC) pair becomes a logical camera device.

---

9. Software Layer: Sensor IDs & Pipelines

In real systems, software doesn’t think in terms of “Link A” or “VC1” directly. Instead:

• Each camera is assigned a sensorId

• sensorId maps to:

  • CSI port

  • Virtual Channel

  • Deserializer instance

  • Serializer and sensor configuration

Example conceptual mapping:

This abstraction lets applications treat each camera as a clean, independent device.

---

10. Synchronization, Triggers & Frame Alignment

A real camera pipeline also includes:

Frame Synchronization

• Multiple cameras must capture frames at the same time

• Achieved via:

  • FSYNC signals

  • Triggers distributed over serializer/deserializer links

Control Path

• I2C commands from SoC → deserializer → serializer → sensor

• Used for:

  • Exposure

  • Gain

  • Mode switches

Diagnostics

• Link lock status

• CRC errors

• Frame drops

• Cable faults

All of this typically flows over the same GMSL/FPD-Link cable.

---

11. Bandwidth, Lanes, and System Design Constraints

Even if everything “connects,” systems fail if bandwidth is not planned:

Key constraints:

• Sum of all camera bandwidth ≤ CSI port capacity

• Two high-res fisheyes on one CSI port may exceed 4-lane CSI

• Deserializer output bandwidth can be the bottleneck

• ISP and memory bandwidth must be provisioned

Designers often:

• Split cameras across multiple CSI ports

• Group low-res cameras together

• Isolate high-res cameras on dedicated links

---

12. Summary: The Mental Model

Here’s the clean mental model to keep:

Physical world:
Sensor → Serializer → GMSL/FPD-Link → Deserializer

Logical world:
Deserializer → CSI-2 → Virtual Channels → SoC pipelines

And the core ideas:

• Serializers make sensors “cable-friendly”

• GMSL/FPD-Link move data long distances

• Deserializers aggregate multiple cameras

• CSI-2 is the standardized SoC interface

• Virtual Channels multiplex cameras logically

• Software maps each (CSI port + VC) to a camera device

---