Changes/IPU6 Camera support

IPU6 camera support

Summary

Integrate support into Fedora for Intel IPU6 attached MIPI cameras using the IPU6 CSI-receiver (isys) driver which has landed in kernel 6.10 together with libcamwera's 0.3 software ISP support and Firefox' recent support for using cameras through pipewire.

Owner

• Name: Hans de Goede
• Email: hdegoede@redhat.com

Current status

• Targeted release: Fedora Linux 41
• Last updated: 2024-10-02
• Announced
• Discussion thread
• FESCo issue: #3243
• Tracker bug: #2300336
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

Many new laptops models have a camera-sensor which is directly attached to the laptops CPU/SoC over a MIPI CSI2 databus instead of using a USB webcam module talking the standard USB UVC protocol.

These cameras require a lot of work on the software side to go from the raw Bayer data received over the CSI2 bus to an usable image. This includes both controlling things like exposure and gain settings on the sensor as well as a lot of processing of the raw data, such as debayering and whitebalancing. Adding support for these complex cameras is tricky because:

• Applications can no longer directly use /dev/video now
• Supporting ISPs (if supported in the kernel) requires ISP model specific knowledge in userspace
• Vendor's 3A algorithms for auto-exposure/gain, auto-whitebalance and auto-focus are secret and need to have open-source counterparts written and tuned
• Instead of having a single UVC driver this requires CSI-receiver + ISP + sensor drivers in the kernel
• Different IPU6 laptop models use different sensors, hw-enablement needs to be done on a laptop by laptop basis
• Good image quality requires per sensor/laptop model tuning

Parts of these challenges are solved by libcamera. For now the aim is a simple stack with good enough image quality for video-conferencing. The plan is to have a stack consisting of:

1. Mainline kernel sensor driver (currently supported: ov2740, ov01a10, hi556)
2. Mainline kernel IPU6 CSI receiver driver
3. libcamera simplepipeline-handler using software ISP for debayering + 3A
4. pipewire with pipewire libcamera plugin
5. pipewire support in Firefox (see Jan Grulich's blog)

Feedback

No feedback yet.

Benefit to Fedora

Currently IPU6 cameras do not work OOTB in Fedora, with the new libcamera software ISP stack these should work OOTB on laptops with supported camera sensors.

Scope

• Proposal owners:
  • The IPU6 CSI receiver has landed in 6.10 and some bugfixes coming to 6.11 have been added as downstream patches
  • libcamera needs a couple of small downstream-patches to enable the simple pipeline for the IPU6
  • pipewire libcamera plugin needs to be made part of the default Workstation patch-set
• Other developers: N/A (not a System Wide Change)
• Release engineering: N/A (not a System Wide Change)
• Policies and guidelines: N/A (not needed for this Change)
• Trademark approval: N/A (not needed for this Change)
• Alignment with the Fedora Strategy: Yes (better hw-support should help getting more users)

Upgrade/compatibility impact

The pipewire-plugin-libcamera needs to be automatically added on Fedora workstation updates to ensure things work. Otherwise there is no upgrade impact.

How To Test

• First of all make sure that the latest updates are install you need at least libcamera >= 0.3.1-4, pipewire >= 1.2.4 and firefox >= 130.0.1-3.

• Ensure all necessary optional dependencies are installed. This should be a no-op when using a fresh Fedora 41 install or after using an official update method: "sudo dnf install pipewire-plugin-libcamera libcamera-ipa intel-vsc-firmware".

• If you also have the rpmfusion packages for the out-of-tree / proprietary Intel IPU6 packages installed, run: "sudo ipu6-driver-select foss" from a terminal and reboot before testing. You can switch back to the proprietary stack by running "sudo ipu6-driver-select proprietary" after testing.

• Low level test, this test tests just the kernel and libcamera install qcam ("sudo dnf install libcamera-qcam") and start the qcam app. In the UI select the camera you want to test and press "Ok".

• Integration test, this test tests the entire camera stack:

1. Start Firefox
2. Go to: https://mozilla.github.io/webrtc-landing/gum_test.html
3. Press the camera button
4. In the permission dialog select the camera and press Allow
5. The GUM Test Page should now show a moving picture from the camera

User Experience

IPU6 cameras on laptops with supported camera sensors should work OOTB after this change, with the caveat that the image quality may be less then ideal. The hope is that image quality will improve over time as the software ISP and its 3A algorithms get improved. With that said it is unrealistic to expect the image quality to become as good as the proprietary stack which has extensive image quality tuning done on a per laptop basis.

Dependencies

IPU6 support not only depends on kernel and libcamera support taken care of by the proposal owner, but also on pipewire camera support and on Firefox pipewire camera support.

Contingency Plan

ATM IPU6 cameras do not work at all. So unless the new kernel driver actually causes regressions outside of the camera functionality no contingency plan is necessary.

Documentation

N/A (not a System Wide Change)

Release Notes

Fedora 41 has added support for IPU6 cameras on laptops using ov2740, ov01a10 and hi556 sensors. This support requires using applications which support accessing cameras through pipewire such as Firefox.