Staging Infrastructure SOP

Fedora has a partially implemented staging environment. This document is intended to be used by people who need to use the staging environment.

Contact Information

Owner: Fedora Infrastructure Team

Contact: #fedora-admin, sysadmin-main

Location: Mostly in PHX2

Servers: app[1-2].stg.phx2.fedoraproject.org proxy1.stg.phx2.fedoraproject.org

Purpose: Provides testing to our admins and community members

Basic workflow and rules

The staging environment replicates as closely as we can the production environment. We don't have full budget to duplicate every single host so we generally pick and choose and do other oddities, for example in staging we combine all the database servers onto one host.

We do use the same username and password in staging and production and regularly merge puppet configs from the master (production) branch to the staging branch. It is also a common practice to dump production databases and pull them into staging. As such no one is given access to staging who doesn't also have it in production.

Rules

Pretty much anything goes in staging. If you break something, please fix it. If you're going to do something that breaks other people's work, make sure to let them know first and coordinate accordingly. We try to completely rebuild the staging environment every other release or so.

Addresses

The addresses in production are identical instaging with the the change s/fedoraproject.org/stg.fedoraproject.org/ So, for example, the staging account system is at https://admin.stg.fedoraproject.org/accounts/. The staging smolts is at http://stg.smolts.org/

Puppet

In order to make changes to the staging environment you must first checkout the staging branch. This is a pretty standard git branch. From your normal puppet repo (see [Infrastructure/SOP/Puppet]) make sure you are on the right branch by running git branch:

$ git branch
* master

Another good way to ensure you're on the right branch is to add the current branch to your shell prompt.

Next we're going to check out the staging branch.

$ git checkout -b staging --track origin/staging
Branch staging set up to track remote branch staging from origin.
Switched to a new branch 'staging'

the -b is the local name in your personal checkout, the --track origin/staging tells git the name of the branch as it is in the master repo. You will see if you run git branch again, git has automatically changed to the staging branch for you:

$ git branch
  master
* staging

Always make sure to know what branch you are going to commit to. If you accidentally send something to master that you intended for staging, it is quite possible you will cause downtime.

To switch back to the master branch just run:

$ git checkout master
Switched to branch 'master'

Cherry Picking

Often times a commit made in staging needs to get pulled into master. You don't want to pull all of staging into master, after all someone else might be working in staging and may not want their changes to go into production yet. To do this cherry-picking is a good alternative. Take the following example where I make a test commit to staging, then cherry-pick it into production:

$ vi README
$ git commit -a -m "Made readme more accurate"
[staging 52a5150] Made readme more accurate
 1 files changed, 1 insertions(+), 2 deletions(-)
$ git checkout master
Switched to branch 'master'
$ git cherry-pick 52a5150
Finished one cherry-pick.
[master 8e65e49] Made readme more accurate
 1 files changed, 1 insertions(+), 2 deletions(-)

There are 3 git commands run there. First I commit to staging, then I checkout master. Then I cherrypick the git commit into master. From there I can push. Note: I could have pushed the staging commit and tested it in staging which will probably be the most common.

Master Merging

We regularly pull from master into staging to make sure staging doesn't get fully out of sync. In the perfect world the work flow would always go from staging first, into production (master branch) second. Since we cannot fully replicate our production environment, and because we use our staging environment for integration, this isn't possible. So the following command helps keep things in sync:

$ git checkout staging
Switched to branch 'staging'
$ git merge master

When doing a merge it is quite possible that a conflict will come up. When resolving a conflict between master and staging, just do your best. If confused about what the file should look like, contact the people who made the last commits to the file.

Nodes and the bootstrap

Due to a bootstrap issue in our environment, all staging nodes need to be in both the staging and master branch. To designate a staging node in the staging environment, simply define this variable at the top of the node definition.:

$puppetEnvironment='staging'

When adding a new node make sure /etc/hosts is all filled out properly for that host and the others (see /etc/hosts below).

Firewalls and /etc/hosts

Because the configs in production are identical to those in staging, we have decided to use some DNS voodoo to get the environment to work correctly. For example, in production a proxy server may contact app[1-7] or bapp1. Since we only have two app servers in staging, we've decided to point app1 prod to app1.stg in staging. Likewise with app2. But app3 points to app1.stg, app4 points to app2.stg, etc.

Also, since the staging environment is on the same physical network as production, we have specific IP based firewall rules in production to deny connections from any staging hosts with a few exceptions (like access to the production yum repos).

What does all of this mean? Well, when creating a new service. Make sure you just use "app1" when referring to app1 or just "db2" when referring to db2. /etc/hosts should take care of the rest. If you are adding a new host. Make sure to add it to every other staring host's node definition. See other staging nodes for examples.