Infrastructure/F9LessonsLearned

Challenges during F9 release

• directory permissions on /pub/fedora/linux/releases/9 were set wrong twice. First from 700 to 750 before we were ready for mirrors to get the bits; again to 755 a few hours later, so the bits were open for public download for a few hours. Unfortunately, dir permissions are the only mechanism we have in place to handle access restrictions before we're ready.

• be careful respinning jigdo images by hand. Finish pyjigdo so hand-editing isn't necessary.

• Get updates/ and updates/testing/ posted, empty, at the same time as the release bits get posted.

• Be sure bodhi is ready to drop packages into the above dirs. It really is OK for them to be in updates/testing while the release is being staged, there shouldn't be that many 0-day updates in general, and it gives mirrors a chance to pick them up.

• MM's database needed a vacuum to become usable. A MM fix is in the works to reduce this need, but is probably a good idea to full vacuum the necessary tables in the days leading up to the release so there are no surprises on release day.

• When trying to push F9 updates with bodhi, we hit a couple of issues
• dist-f9-updates{,-testing} tags were not created in koji
• the f9-updates.mash configuration files were not created

• Several mirrors didn't catch the bitflip for quite a while. How can we improve the speed of this action? (push mirroring?)

• Overloaded mirrors return http error 427 (try another mirror). But download.fp.o URLs do a simple redirect, so the end user winds up seeing the error, but no way to fix it. Can we use HTTP 300 redirects (instead of 302 redirects) to present the user with a choice if the redirect fails?

• would like to have stats from mirrors as to bits moved, users served etc.

• make sure web links are live before sending release announcement.

• after marking the -Preview release to not be displayed on the publiclist chooser, restart start-mirrors on its app server so it drops its cache.

• It worked really well to schedule 'at' jobs 30-40 minutes ahead of release. Bitflip on the mirrors themselves (so they don't wait to rsync it), removing repository redirects, hiding -Preview, displaying the new release number. Then restart start-mirrors.

• 'at' doesn't handle UTC / DST conversions properly.