Fedora Atomic Host installations use the rpm-ostree deployment method, where packages are assembled on the Fedora release engineering side and delivered as atomic units, rather than the yum/DNF method of client side assembly. Consequently, they do not use the DNF system upgrade mechanism which is used to upgrade between Fedora releases for other Fedora installations. Instead, use the procedure described here.
Upgrading from Fedora 25 Atomic Host to Fedora 26 Atomic Host:
ostree remote add --if-not-exists --gpg-import /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-primary fedora-atomic-26 https://kojipkgs.fedoraproject.org/atomic/26 rpm-ostree rebase fedora-atomic-26:fedora/26/x86_64/atomic-host
Like any other rpm-ostree update, this is staged for the next reboot, so to finally apply the update:
sudo systemctl reboot
That should be all!