Episode Summary
Executive Summary: Sam Charrington interviews Fatih Perikli of Qualcomm AI Research about Qualcomm’s CVPR research, organized into two themes: better use of limited data and better architectures. The discussion highlights data augmentation for optical flow, domain-meaningful distortions, 3D object detection via multi-modal knowledge distillation, zero-shot 3D part segmentation with language-vision models, and edge deployment of generative AI like ControlNet and Stable Diffusion.
Main Topics: Qualcomm AI Research at CVPR (Priority: 5/5): Fatih reintroduces Qualcomm AI Research’s role in corporate R&D and summarizes a strong CVPR showing with eight main-conference papers, four workshop papers, demos, and broader research momentum. Making better use of labeled and unlabeled data (Priority: 5/5): Several papers focus on data augmentation and supervision efficiency: DistractFlow for optical flow, progressive random convolution, ReDiTrans for gaze/head-pose augmentation, and Deja Vu for conditional generation-assisted dense prediction. Optical flow as a foundation for video understanding (Priority: 5/5): Fatih explains optical flow as a core motion estimation problem that underpins tasks like tracking, compression, super-resolution, and detection; DistractFlow improves training by introducing semantically meaningful distractions instead of simple geometric augmentations. Efficient deployment of optical flow on-device (Priority: 4/5): The workshop paper DIFT shows how cost-volume-based optical flow can be made practical for edge hardware by computing only what is needed and optimizing for Snapdragon accelerator architecture. Cross-modal, cross-stage, cross-task knowledge distillation (Priority: 5/5): X3KD combines multiple distillation signals to transfer knowledge from multi-sensor teachers (e.g., LiDAR) to camera-based student models for 3D object detection, enabling strong accuracy with lower runtime cost. Zero-shot 3D part segmentation with language-vision models (Priority: 4/5): A generative AI approach uses off-the-shelf language and vision models, including GLIP, to segment fine-grained 3D object parts without manual 3D part labels, relying on prompt-driven feature alignment and multi-view consistency. Generative AI on the edge with ControlNet (Priority: 4/5): Qualcomm demonstrates running ControlNet on-device, extending Stable Diffusion with image conditioning and hardware/software/model optimization to generate images quickly and privately on an Android phone.
Key Arguments: Data is often the bottleneck in computer vision, so better augmentation and semi-supervised strategies can deliver large gains without changing the core model architecture. Optical flow remains fundamental because motion estimation supports many downstream video tasks, from tracking and detection to compression and super-resolution. Traditional augmentations are limited; semantically meaningful distortions or distractions can make models more robust and improve generalization. High-performing optical flow models often rely on cost volumes, but they are hard to deploy on edge devices unless the computation is optimized around memory and latency constraints. Knowledge distillation is most effective when multiple forms of supervision are combined jointly rather than trained separately. Using LiDAR and other modalities only during training can produce a camera-only runtime model that is cheaper and more deployable for autonomous driving. Language models and vision-language models can substitute for missing fine-grained 3D labels by turning text prompts into semantic guidance for segmentation. Generative AI is increasingly important not only for quality but for personalization, privacy, and reducing cloud dependence through edge inference.
Data Points: CVPR main-conference papers: 8 - Qualcomm AI Research had eight papers accepted to the main CVPR conference. CVPR workshop papers: 4 - Fatih said the team also had four workshop papers. Total papers discussed as a category count: 2 major categories - He grouped the work into better data usage and better architectures. DistractFlow optical flow improvement: Almost half reduction in EPE - He said the paper nearly halved endpoint error on Sintel and KITTI benchmarks. Benchmarks named for DistractFlow: Sintel, KITTI - He referenced these as standard optical flow benchmarks. DIFT speed: Around 30 milliseconds or less - He described on-device optical flow execution on cost-volume-based models. DIFT throughput improvement: 8x - He said throughput improved eight times through optimization. DIFT speedup: More than 30x - He said speed improved by more than 30 times compared with the original cost-volume approach. ControlNet model size: 1.5 billion parameters - He described ControlNet as a large model with roughly 1.5B parameters. ControlNet on-device generation time: Less than five seconds - He said Qualcomm showed image generation on an Android phone in under five seconds. Stable Diffusion/ControlNet relative size: About 50% bigger - He said the ControlNet version is about 50% larger than the Stable Diffusion version they previously demonstrated. Research horizon mentioned: Next year / 12 months - He emphasized how quickly the field changes over a year.
Pivotal Quotes: "Optical flow is a fundamental task in computer vision." — Fatih Perikli: Explaining why improving motion estimation matters across many video applications. "We are going beyond it. What we are showing is that if you go beyond by inducing semantically meaningful augmentations, distractions, and then if you use that data for training, these AI-based optical flow solutions actually become more robust and more accurate." — Fatih Perikli: Describing the core idea behind DistractFlow. "The beauty of that algorithm is the part segmentation algorithm." — Fatih Perikli: Summarizing the zero-shot 3D part segmentation approach using language-vision models.
Implications: The interview shows CV research moving toward data-efficient learning, multimodal distillation, and private edge deployment. For industry, the near-term wins are more accurate perception and generative models that run locally, with less dependence on labeled data and cloud compute.