Changes/Passim P2P Metadata

Passim Peer-to-Peer Metadata

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

Passim is a local caching server that broadcasts specific shared metadata to other clients on your local network to reduce the amount of duplicate data downloaded from the internet.

Owner

• Name: Richard Hughes
• Email: richard@hughsie.com

Current status

• Targeted release: Fedora Linux 40
• Last updated: 2023-09-11
• Announced
• Discussion Thread
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

Much of the software running on your computer that connects to other systems over the Internet needs to periodically download metadata or information needed to perform other requests.

Everybody downloads the same file from a CDN, and although a CDN is not super-expensive, it's certainly not free. Everybody on your current network (perhaps thousands of users) has to download the same 1MB blob of metadata from a CDN over a perhaps-expensive internet link.

What if we could download the file from the CDN on one machine, and the next machine on the local network that needs it instead downloads it from the first machine? We could put a limit on the number of times it can be shared, and the maximum age so that we don't store yesterdays metadata forever, and so that we don't turn a ThinkPad X220 into a machine distributing 1Gb/s to every other machine in the office. We could cut the CDN traffic by at least one order of magnitude, but possibly much more. This is better for the person paying the cloud bill, the person paying for the internet connection, and the planet as a whole.

Feedback

IPFS is an existing project that's existed for many years and allows sharing with other users not on your local network. It's not packaged in any distributions and not trivial to install correctly. It's main drawback is that it requires a internet to IPFS "gateway" which costs a large amount of money for the LVFS, and that it's not EAR/ITAR compliant.

I've asked for feedback already on fedora-devel and have already started making changes and suggestions from that discussion -- for instance splitting out a -libs subpackage. See https://www.spinics.net/lists/fedora-devel/msg315078.html for the discussion.

Benefit to Fedora

Fedora will consume less bandwidth when checking for firmware updates. Per user there is only a 2MB/day saving, but for millions of Fedora users this adds up to a huge amount of saved data (and money) globally.

Scope

The code is already written, tested and ready to go. The passim package is already included in Fedora (although not installed by default) and fwupd 1.9.5 includes this functionality.

Upgrade/compatibility impact

Old versions of fwupd will be updated and start sharing metadata with other local users with no changes required.

How To Test

Install two physical machines or VMs with Fedora 40 that are both on the local network. Run fwupdmgr refresh on the first, and observe that passim dump or https://localhost:27500/ lists the published metadata file . Then run fwupdmgr refresh --verbose on the second machine and see that the file is downloaded from https://localhost:27500 rather than cdn.fwupd.org. Avahi needs to be enabled and running, as does passimd although both are autostarted as required.

User Experience

Each user will use 2MB less bandwidth per day when there are other users on the local network with the same metadata file. There is no user-visible difference to any operation.

Dependencies

None; fwupd will recommend passim to be installed by default and autolaunch it as required.

Contingency Plan

Change Recommends: passim to Suggests: passim in the fwupd.spec file so that it's not autoinstalled by default. In this case fwupd will fall back to downloading from the CDN every day.

Documentation

* https://github.com/hughsie/passim/blob/main/README.md

Release Notes

Fedora now uses a peer-to-peer service called Passim to reduce the amount of bandwidth used when downloading metadata.