Sensor Fusion for ADAS: Camera, Radar, LiDAR in 2026
A camera is excellent at seeing a red light and reading a sign, but it does not directly measure range reliably in fog. Radar measures relative velocity and remains useful in rain, yet its angular resolution is low and reflections can be ambiguous. LiDAR provides accurate 3D geometry, but costs more, becomes sparse at distance, and can still degrade through occlusion or harsh weather. Sensor fusion is not the claim that “more sensors equals safer.” It is the engineering work of turning imperfect, asynchronous observations in different coordinate systems into a decision that can be examined.
By the end, you should be able to select an architecture for a particular function: early fusion when preserving raw correspondence justifies heavy calibration and compute; late fusion when independent modules and explicit degradation matter most; or deep/BEV fusion when rich 3D perception justifies the GPU, data, and validation investment. This is a safety-case and operations decision, not a contest to adopt the newest model.
Series roadmap
- Part 1 — SAE Level 2 vs 2+ vs 3 vs 4: identify responsibility, ODD, and fallback.
- Part 2 — Sensor fusion: combine camera, radar, and LiDAR into perception that can degrade predictably when a sensor is imperfect.
First, what exactly is being fused?
An ADAS pipeline may receive camera frames, LiDAR point clouds, radar detections or point clouds, IMU, GNSS, and vehicle signals. This article focuses on external perception. Before fusion, three foundations must work:
- Time synchronization: at 20 m/s, a 50 ms mismatch means roughly one metre of vehicle motion. Timestamps, sensor-network latency, rolling shutter, and ego-motion compensation all matter.
- Spatial registration: extrinsic calibration describes where each camera, radar, and LiDAR sits and points relative to the vehicle; intrinsic calibration describes the optics. A few pixels or centimetres of error can associate a vehicle with the neighbouring lane.
- Uncertainty: each detector should expose more than a 3D box—confidence, covariance, or at least an explicit uncertain state. Fusion must not turn a weak observation into a certain conclusion.
There are three useful data levels. Raw data means pixels, range-Doppler/chirp measurements, or 3D points. A feature is an encoder tensor. An object/track is a vehicle, pedestrian, or lane hypothesis with position, velocity, class, and confidence. Early, late, and deep/BEV fusion differ mainly in the level at which modalities meet.
Architecture 1: early fusion — combine raw or low-level features early
In the strict sense, early fusion puts sensor data into a shared representation before a detector has made its conclusion. For example, calibrated camera colour/features can be projected onto LiDAR points, or radar and LiDAR can be rasterized into one grid for a single network. A practical variation is low-level feature fusion: camera and radar/LiDAR encoders produce shallow features, concatenate them or cross-attend, then use a shared detector.
camera pixels ──┐ calibrate + time align ┌─ shared encoder ─ detection / segmentation
LiDAR points ──┼───────────────────────────┤
radar returns ─┘ └─ one common representation
Strength. The model can learn an early relationship between traffic-light texture, LiDAR vehicle geometry, and radar Doppler. With stable calibration it retains detail that per-sensor boxes may discard. This is appealing for specialised perception—such as small-object detection on a platform with a fixed sensor suite and abundant training data.
Cost. Raw modalities do not naturally speak the same language. Projecting image features onto LiDAR depends directly on LiDAR points; when LiDAR degrades, useful pixels may no longer have points to attach to. The paper BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework calls out this weakness of point-level hard association: methods based on LiDAR queries can fail to produce predictions when LiDAR malfunctions. Early fusion also magnifies calibration and timing errors, is harder to debug, and can raise memory bandwidth and latency when dense image tensors meet high-resolution point clouds.
So “early” is not shorthand for “best.” Choose it when pixel-to-point correspondence is a core asset, hardware and calibration are controlled, and fault injection has demonstrated the benefit. It is risky for a function that must retain meaningful operation after one modality disappears.
Architecture 2: late fusion — understand independently, then reconcile
Late fusion runs separate camera, radar, and LiDAR detectors or segmenters. An association stage then matches object candidates by position, time, class, and uncertainty; a Kalman filter or multi-hypothesis tracker maintains state through frames. The rule can be weighted averaging, covariance intersection, learned gating, or track-to-track fusion.
camera ─ detector ─ boxes + confidence ─┐
LiDAR ─ detector ─ boxes + covariance ──┼─ association / tracker ─ fused tracks
radar ─ detector ─ range + Doppler ─────┘
Strength. This is straightforward to explain in a safety review. Radar can still supply range-rate tracks in glare; camera can still identify lights and lanes when LiDAR is temporarily unavailable; LiDAR can still localise in 3D when image classification is uncertain. Each branch can be monitored for heartbeat, quality, and latency. If LiDAR fails, the fusion engine can reduce its weight or enter a camera–radar policy rather than collapse a shared tensor. You can also replace the camera detector without retraining the full stack.
Limit. Each detector has already discarded information before fusion. An object too weak for the camera detector to emit a box cannot be prompted by LiDAR to inspect that image area again. Association becomes difficult at crowded junctions, with occlusion, mismatched classes, timestamp error, or close boxes. Three detectors are not free either: distributed compute, NMS, and tracking add latency. Never naïvely add confidences—three cameras facing the same sun have correlated error, not three independent votes.
Late fusion is often the sensible starting point for production L2/L2+ ADAS with a tight deadline, a small team, or a high need for traceability. AEB/ACC may prioritise radar range-rate, lane keeping camera cues, and LiDAR—where fitted—as geometric confirmation. In this context, “good” means a designed and tested degradation path, not merely a high sunny-day mAP.
Architecture 3: deep/BEV fusion — learn a structured common space
Deep fusion is not merely concatenation. Separate encoders create features, then a network learns to combine them with convolution, gating, or attention. Bird's-eye view (BEV) is particularly useful for driving because it maps modalities into a top-down grid around the ego vehicle; 3D boxes, lanes, and planning then share a metric coordinate system.
TransFusion is worth studying because it avoids hard matching of every LiDAR point to one camera pixel. In its CVPR 2022 paper and open-source repository, a LiDAR backbone produces BEV features; a first transformer-decoder layer uses a sparse set of object queries to propose boxes from LiDAR; its second decoder layer lets those queries attend to image features. This soft association allows attention to learn where and what image evidence matters instead of trusting calibration projection absolutely. The authors additionally use image-guided query initialization for objects difficult to see in the point cloud.
BEVFusion from MIT Han Lab follows a complementary design: multi-view cameras pass through a camera encoder and a view transform into BEV; LiDAR passes through its encoder into BEV; the two feature maps fuse before task heads for 3D detection or BEV map segmentation. The ICRA 2023 paper reports that optimised BEV pooling cuts view-transform latency by more than 40× and reaches 1.9× lower compute than its comparison baseline—an architecture-and-dataset-specific result, not a guaranteed vehicle FPS.
BEV works well because cameras bring dense semantics—colour, signs, object type—while LiDAR brings metric geometry, and both meet in one grid. But camera-to-BEV view transformation must estimate or distribute depth, which costs compute and becomes uncertain far from the ego vehicle. Transformer/cross-attention adds flexibility, while large attention maps, high-resolution multi-view images, and temporal BEV consume GPU memory. Profile, quantise, and budget these costs rather than assuming a published leaderboard implementation meets an embedded deadline.
Do not conflate the two similarly named projects. ADLab-AutoDrive/BEVFusion is the NeurIPS 2022 work emphasising a camera stream independent from LiDAR and reports a 15.7–28.9 mAP improvement in its simulated LiDAR-malfunction setting against comparison methods. MIT Han Lab's repository is the ICRA 2023 multi-task unified-BEV work with efficient pooling. Both are valuable references, but any metric is meaningful only with its stated split, augmentations, sensor suite, and failure model.
Compare to choose, not to rank
| Criterion | Early/raw fusion | Late/object fusion | Deep/BEV fusion |
|---|---|---|---|
| Fusion location | Pixels, points, or low-level features | Boxes, tracks, confidences | Learned BEV features or attention |
| Typical latency | Can be low when simple; grows with dense projection | Parallel paths, plus detector and association cost | Often heaviest at view transform/attention; can be highly optimised |
| One sensor absent | Fragile if the other branch is a prerequisite | Clearest: drop/downweight branch and retain tracks | Depends on independent encoders, masking, and training; must be fault-tested |
| Information retained | Most | Least | More than late fusion, in a learned structure |
| Debug/audit | Hard | Easiest | Medium to hard |
| Best fit | Controlled R&D with reliable calibration | Products needing modular, explicit fallback | Rich 3D perception with data and compute maturity |
Measure latency end to end, from exposure timestamp to a message ready for the planner: transfer, preprocessing, inference, fusion, tracking, and scheduling. A 25-FPS network does not automatically mean 40 ms latency; batching, queues, and one slow camera can make data much older. Set the budget by function: blind-spot warning, AEB, lane centring, and robotaxi perception have different hazards, visibility, and reaction time.
Design for faults: fusion does not create redundancy by itself
Test three conditions in replay and on vehicle: (1) a sensor is fully absent; (2) it is alive but wrong—rain, a dirty lens, radar ghosts, or calibration drift; (3) it is late or timestamped incorrectly. The second is often worse because a confident neural network can steer the fused result in the wrong direction.
A minimum policy needs an independent sensor-health monitor, per-modality quality scores, a function/speed restriction when confidence drops, and logging that reconstructs decisions. For deep fusion, training should include modality dropout, image corruption, LiDAR sparsification, and misalignment representative of the ODD; then report each failure mode, not just an average score. For late fusion, check correlation before voting. For early fusion, inspect hard dependencies before calling it fault tolerant.
Perception redundancy is also not sufficient for Level 3 and Level 4. A safety case also needs compute, power, steering, braking, a minimal-risk manoeuvre, ODD definition, and operations. Strong fusion reduces uncertainty; it cannot independently transfer legal responsibility or create a fail-operational vehicle.
A short decision tree for an engineering team
- Must the function remain useful after camera or LiDAR loss? Start with independent branches and an explicit late-fusion/fallback policy; choose deep fusion only after proving modality-dropout behaviour.
- Does the problem need dense multi-camera semantics, BEV maps, and 3D detection? Evaluate a BEVFusion-style design against a dataset such as nuScenes and OpenDRIVEVLA, then profile target hardware.
- Are calibration error or poor illumination top risks? Compare hard projection against TransFusion-style soft association and run perturbation tests; do not use only clean validation.
- Do you have synchronized data, 3D labels, GPU budget, and replay discipline? If not, late fusion can deliver safer value sooner than an uncalibrated, unmonitored BEV transformer.
A healthy roadmap is often: build per-sensor baselines and health metrics; deploy late fusion for observable fallback; then evaluate deep/BEV fusion with the same latency and fault suite. Use early fusion only when retaining raw correspondence outweighs its coupling cost.
Conclusion
Camera, radar, and LiDAR complement one another, but the architecture determines how they fail together. Early fusion preserves detail but carries heavy calibration and dependency risk. Late fusion trades some information for independent modules, auditability, and understandable degradation. Deep/BEV fusion such as BEVFusion and TransFusion can exploit semantics and geometry more effectively, but demands serious data, compute, and fault testing. Select it by function, ODD, latency budget, and behaviour when a sensor lies or falls silent—not by a model name.
