|
|
| (14 intermediate revisions by 7 users not shown) |
| Line 2: |
Line 2: |
| {{shortcut|ISOP:RELEASE}} | | {{shortcut|ISOP:RELEASE}} |
|
| |
|
| This SOP contains all of the steps required by the Fedora Infrastructure team in order to get a release out. Much of this work overlaps with the Release Engineering team (and at present share many of the same members). Some work may get done by releng, some may get done by Infrastructure, as long as it gets done, it doesn't matter.
| |
|
| |
|
| == Contact Information ==
| | This SOP has moved to the fedora Infrastructure SOP git repo. Please see the current document at: http://infrastructure.fedoraproject.org/infra/docs/fedora-releases.rst |
| Owner: Fedora Infrastructure Team, Fedora Release Engineering Team
| |
|
| |
|
| Contact: #fedora-admin, #fedora-devel, sysadmin-main, sysadmin-releng group
| | For changes, questions or comments, please contact anyone in the Fedora Infrastructure team. |
|
| |
|
| Location: N/A
| |
|
| |
| Servers: All
| |
|
| |
| Purpose: Releasing a new version of Fedora
| |
|
| |
| == Description ==
| |
|
| |
|
| |
| == Preparations ==
| |
|
| |
| Before Alpha for a release ships, the following items need to be completed. *Please open a ticket for all of them individually*
| |
|
| |
| # New website from the websites team (typically hosted at http://fedoraproject.org/_/), edit syncStatic.sh to freeze the website in preparation for release updates.
| |
| # Verify mirror space (for all test releases as well)
| |
| # Release day ticket and milestone (keep a log of who is doing what and notes)
| |
| # Verify with rel-eng permissions on content are right on the mirrors. Don't leak.
| |
| # Add MirrorManager repository redirects. ${version}-Alpha => ${version}. ${version}-Beta => ${version}. ${version+1} => rawhide. This directs Alpha and Beta users at the pending release, and the next version users at rawhide. MM's manage-repo-redirects script is used to do this.
| |
| # Communication with Red Hat IS (Give at least 2 months notice, then reminders as the time comes near) (final release only)
| |
| # Infrastructure change freeze
| |
| # Modify Template:FedoraVersion to reference new version. (Final release only)
| |
| # Re-review Lessons Learned pages from previous release, be sure we aren't forgetting things we meant to do after the last release.
| |
| # Enable Caching for the wiki for release (Final release only)
| |
| # Disable Caching for the wiki day after release. (Final release only)
| |
| # Add the new release to stats gathering scripts on log1 (modules/scripts/files/fedoraUsage.sh) (Final release only)
| |
|
| |
| === Change Freeze ===
| |
|
| |
| The rules are simple. If we are in a pre release freeze or a full freeze, take a look at [http://git.fedorahosted.org/git/fedora-infrastructure.git?p=fedora-infrastructure.git;a=blob_plain;f=architecture/Environments.odg;hb=HEAD the architecture document]. If the hostname is in the freeze area, that entire host is frozen. So making global changes is also forbidden. If, for some reason, a change needs to be made, a request must go to the fedora-infrastructure-list requesting it. It will require at least two people to signoff on the change from either sysadmin-main or the releng group. Exceptions to this include:
| |
|
| |
| * Planned changes as part of the release
| |
| * Emergencies / outages
| |
|
| |
| Change freezes will be sent to the fedora-infrastructure-list and begin 2 weeks before each release and the final release. The freeze will end one day after the release. Note, if the release slips during a change freeze, the freeze just extends until the day after a release ships.
| |
|
| |
| To see the latest environments map run:
| |
|
| |
| <pre>
| |
| sudo yum -y install openoffice.org-draw
| |
| git clone git://git.fedorahosted.org/git/fedora-infrastructure.git/
| |
| cd fedora-infrastructure/architecture
| |
| oodraw Environments.odg
| |
| </pre>
| |
|
| |
| While the instructions above are the canonical location, a png has been provided for ease of reference. See [https://fedorahosted.org/fedora-infrastructure/browser/architecture/Environments.png here]
| |
|
| |
| === Static / cached urls ===
| |
|
| |
| The following URL's should be setup to be cached via a static page or with mod_cache_mem as needed. These are the links that will be hit hardest during the release.
| |
|
| |
| # http://get.fedoraproject.org/
| |
| # http://help.fedoraproject.org/
| |
| # http://join.fedoraproject.org/
| |
| # http://fedoraproject.org/wiki/Releases/{{FedoraVersion||next}}/ReleaseOverview
| |
| # http://fedoraproject.org/wiki/Bugs/F{{FedoraVersion||next}}Common
| |
| # http://docs.fedoraproject.org/release-notes/index.html
| |
| # http://docs.fedoraproject.org/release-notes/f{{FedoraVersion||next}}/index.html
| |
| # http://docs.fedoraproject.org/release-notes/f{{FedoraVersion||next}}/en_US/index.html
| |
| # http://docs.fedoraproject.org/install-guide/index.html
| |
| # http://docs.fedoraproject.org/install-guide/f{{FedoraVersion||next}}/index.html
| |
| # http://docs.fedoraproject.org/install-guide/f{{FedoraVersion||next}}/en_US/index.html
| |
| # http://fedoraproject.org/wiki/Releases/{{FedoraVersion||next}}/ReleaseSummary
| |
| # http://docs.fedoraproject.org/
| |
|
| |
| Each one of these tasks except for the release day ticket must be completed and closed prior to the release.
| |
|
| |
| == Notes about release day ==
| |
|
| |
| Release day is always an interesting and unique event. After the final sprint from test to the final release a lot of the developers will be looking forward to a bit of time away, as well as some sleep. Once Release Engineering has built the final tree, and synced it to the mirrors it is our job to make sure everything else (except the bit flip) gets done as painlessly and easily as possible.
| |
|
| |
| All communication is typically done in #fedora-admin though we may start also having communications in sip:infrastructure@fedoraproject.org. Typically these channels are laid back and staying on topic isn't strictly enforced. On release day this is not true. We encourage people to come, stay in the room and be quiet unless they have a specific task or question releated to release day. Its nothing personal, but release day can get out of hand quick.
| |
|
| |
| During normal load, our websites function as normal. This is especially true since we've moved the wiki to mod_fcgi. On release day our load spikes a great deal. During the Fedora 6 launch many services were offline for hours. Some (like the docs) were off for days. A large part of this outage was due to the wiki not being able to handle the load, part was a lack of planning by the Infrastructure team, and part is still a mystery. (There are questions as to whether or not all of the traffic was legit or a ddos.
| |
|
| |
| The Fedora 7 release went much better. Some services were offline for minutes at a time but very little of it was out longer then that. The wiki crashed, as it always does. We had made sure to make the fedoraproject.org landing page static though. This helped a great deal though we did see load on the proxy boxes as spiky.
| |
|
| |
| == Day Prior to Release Day ==
| |
|
| |
| === Step 1 (Torrent) ===
| |
| Setup the torrent. All files can be synced with the torrent box (exists at duke) but just not published to the world. Verify with sha1sum. Follow the instructions on the http://fedoraproject.org/wiki/Infrastructure/SOP/TorrentRelease page up to and including step 4.
| |
|
| |
| === Step 2 (Website) ===
| |
| Verify the website design / content has been finalized with the websites team. Update the [[Template:FedoraVersionNumber|Fedora version number]] wiki template.
| |
|
| |
| === Step 3 (Mirrors) ===
| |
| Verify enough mirrors are setup and have Fedora ready for release. If for some reason something is broken it needs to be fixed. Many of the mirrors are running a check-in script. This lets us know who has Fedora without having to scan everyone. Hide the Alpha, Beta, and Preview releases from the publiclist page.
| |
|
| |
| === Step 4 (Enable Updates) ===
| |
| The new Fedora release will likely have 0day updates. Making sure we can push those updates is important.
| |
|
| |
| Move MirrorManager repository tags from the development/$version/ Directory objects, to the releases/$version/ Directory objects.
| |
|
| |
| Remove MirrorManager repository redirects from to-be-released version to rawhide repos. This lets people get the proper updates.
| |
|
| |
| As far as bodhi is concerned
| |
| * Create the new release in bodhi, to allow developers to queue up 0day updates. Make sure this is 'locked' so they don't get pushed. See the bodhi [https://fedorahosted.org/bodhi/wiki/Administration administration] page for instructions.
| |
| * Create mash config files for the new release
| |
| * Make sure the dist-fX-updates{,-testing} tags are creating in Koji
| |
| * Make sure the rsync script is updated
| |
| * When it's time to push, simply unlock the release in bodhi and you should be good to go.
| |
|
| |
| === Step 5 (Lessons Learned page) ===
| |
| Set up a new page like [[Infrastructure/F9LessonsLearned| ]] to record findings we'll use next release cycle.
| |
|
| |
| === Step 6 (preupgrade releases.txt) ===
| |
| At the mass branch point, rel-eng filed a ticket asking FI to update the releases.txt file used by preupgrade. Ensure this ticket is complete. Ensure release code name and URLs are correct for pre-release vs final release locations.
| |
|
| |
| == Release day ==
| |
|
| |
| === Step 1 (Prep and wait) ===
| |
| Verify the mirrors are ready and that the torrent has valid copies of its files (use sha1sum)
| |
|
| |
| {{ admon/warning | Do not move on to step two until the Release Engineering team has given the ok for the release. It is the releng team's decision as to whether or not we release and they may pull the plug at any moment.
| |
| }}
| |
|
| |
| === Step 2 (Torrent) ===
| |
| Once given the ok to release, the Infrastructure team should publish the torrent and encourage people to seed. Complete the steps on the http://fedoraproject.org/wiki/Infrastructure/SOP/TorrentRelease page after step 4. Also, upload the files to the SixXS IPv6 torrent seeds, using the form linked from http://www.sixxs.net/tools/tracker/
| |
|
| |
| === Step 3 (Bit flip) ===
| |
| The mirrors sit and wait for a single permissions bit to be altered so that they show up to their services. The bit flip (done by the releng team) will replicate out to the mirrors. Verify that the mirrors have received the change by seeing if it is actually available, just use a spot check. Once that is complete move on.
| |
|
| |
| === Step 4 (Preupgrade releases.txt) ===
| |
| Preupgrade requires us to update the contents of http://mirrors.fedoraproject.org/releases.txt to include the new release in the preupgrade-ok list. The git repository containing this file is at puppet1:/var/lib/puppet/application/mirrors/.git. Add the new release and 'git push' it into place. This gets picked up every 30 minutes and populated on the proxies.
| |
|
| |
|
| |
| === Step 5 (Website) ===
| |
| Once all of the distribution pieces are verified (mirrors and torrent), all that is left is to publish the website. At present this is done by making sure the master branch of fedora-web is pulled by the syncStatic.sh script in puppet (which contains instructions for switching this). It will sync in an hour normally but on release day people don't like to wait that long so do the following on puppet1:
| |
|
| |
| <pre>
| |
| sudo /etc/cron.hourly/syncStatic.sh && sudo func proxy\* call command run "/usr/local/bin/syncFiles.sh fedoraproject.org /srv/web/fedoraproject.org/"
| |
| </pre>
| |
|
| |
| Verify http://fedoraproject.org/ is working.
| |
|
| |
| Also, ensure that /get-prerelease now redirects to /get-fedora. The appropriate configuration is available in the puppet configs in modules/fedora-web/files/redirects.conf.
| |
|
| |
| === Step 6 (Docs) ===
| |
| Just as with the website, the docs site needs to be published. Just as above follow the following steps:
| |
|
| |
| <pre>
| |
| /root/bin/docs-sync
| |
| </pre>
| |
|
| |
| === Step 7 (Monitor) ===
| |
| Once the website is live, keep an eye on various news sites for the release announcement. Closely watch the load on all of the boxes, proxy, application and otherwise. If something is getting overloaded, see suggestions on this page in the "Juggling Resources" section.
| |
|
| |
| === Step 8 (Done) ===
| |
| Just chill, keep an eye on everything and make changes as needed. If you can't keep a service up, try to redirect randomly to some of the mirrors.
| |
|
| |
| === Priorities ===
| |
|
| |
| Priorities of during release day (In order):
| |
|
| |
| # Website - Anything related to a user landing at fedoraproject.org, and clicking through to a mirror or torrent to download something must be kept up. This is distribution, and without it we can potentially lose many users.
| |
| # Linked addresses - We do not have direct control over what Digg, Slashdot or anyone else links to. If they link to something on the wiki and it is going down or link to any other site we control a rewrite should be put in place to direct them to http://fedoraproject.org/get-fedora.
| |
| # Torrent - The torrent server has never had problems during a release. Make sure it is up.
| |
| # Release Notes - Typically grouped with the docs site, the release notes are often linked to (this is fine, no need to redirect) but keep an eye on the logs and ensure that where we've said the release notes are, that they can be found there. In previous releases we sometimes had to make this available in more than one spot.
| |
| # docs.fedoraproject.org - People will want to see whats new in Fedora and get further documentation about it. Much of this is in the release notes.
| |
| # wiki - Because it is so resource heavy, and because it is so developer oriented we have no choice but to give the wiki a lower priority.
| |
| # Everything else.
| |
|
| |
| == Juggling Resources ==
| |
| In our environment we're running different things on many different servers. Using Xen we can easily give machines more or less ram, processors. We can take down builders and bring up application servers. The trick is to be smart and make sure you understand what is causing the problem. These are some tips to keep in mind:
| |
|
| |
| * IPTables based bandwidth and connection limiting (successful in the past)
| |
| * Altering the weight on the proxy balancers (see: [[Infrastructure/SOP/Balancers| Balancers]] )
| |
| * Create static pages out of otherwise dynamic content
| |
| * Redirect pages to a mirror
| |
| * Add a server / remove un-needed servers
| |
|
| |
|
| [[Category:Infrastructure SOPs]] | | [[Category:Infrastructure SOPs]] |
| [[Category:Release Engineering SOPs]]
| |
