Fedora Atomic Host installations use the rpm-ostree deployment method, and do not use RPM-based package management at all. Consequently, they do not use the DNF system upgrade mechanism which is used to upgrade between Fedora releases for other Fedora installations. Instead, use the procedure described here.
Let's assume you're upgrading from Fedora 40 to 41. First, you'll need to add a remote for the new major version:
ostree remote add fedora-41 --set=gpg-verify=false https://dl.fedoraproject.org/pub/fedora/linux/atomic/41
Then, rebase to it:
rpm-ostree rebase fedora-41:fedora-atomic/f41/x86_64/docker-host
That should be all!