Infrastructure/AppBestPractices

Coding style

• Python code should be PEP8 compliant

Tools to improve style

You can run pylint over your code once in a while to help catch PEP8 violations. pyflakes and pep8 (the program) are fast enough that you could setup your editor to run them whenever you save a python file (but they don't catch nearly as many things as pylint does).

Common Programming Issues

Centralized logging

Most of our apps use the standard Python logging module, which usually ends up logging to /var/log/httpd/error_log on the app server.

To have your app logs shipped to our central logging server, you can configure the SysLogHandler to do so.

For config-file based logging setups, you can do something like the following:

[logging]                                                                                                                                                             
                                                                                                                                                                          
[[handlers]]                                                                                                                                                          
                                                                                                                                                                         
[[[syslog_out]]]                                                                                                                                                      
class='handlers.SysLogHandler'                                                                                                                                        
args="('/dev/log', handlers.SysLogHandler.LOG_LOCAL4,)"                                                                                                               
formatter='full_content'                                                                                                                                              
                                                                                                                                                                          
[[loggers]]                                                                                                                                                           
                                                                                                                                                                          
[[[bodhi]]]                                                                                                                                                           
level='DEBUG'                                                                                                                                                         
qualname='bodhi'                                                                                                                                                      
handlers=['syslog_out']                                                                                                                                               
propagate=0                                                                                                                                                           

Here is an example of doing it in pure Python:

import logging
import logging.handlers

syslogger = logging.getLogger('bodhi')
syslogger.setLevel(logging.DEBUG)
handler = logging.handlers.SysLogHandler(address='/dev/log', facility=logging.handlers.SysLogHandler.LOG_LOCAL4)
syslogger.addHandler(handler)
syslogger.info('Hello SysLog!')

The app logs will then appear in /var/log/hosts/<HOST>/<YEAR>/<MONTH>/<DAY>/apps.log as well as the merged log /var/log/merged/apps.log on our central rsyslog server.

Translations

Use transifex to manage translations and add the projects to the Fedora Project Team so the Fedora i18n team can translate.

Do not mark Exception messages for translation. We want those to remain untranslated for several reasons:

1. It's easier to do a web search for the exceptions if the messages aren't translated.
2. Translators don't know what all the technical information in exception messages are so the translations aren't always accurate.
3. The exception messages aren't intended for end users -- they're to help with debugging or other coders. So it's not very useful to translate them.

fedmsg

Think about how to add fedmsg hooks into every event that changes data. However, also make it so that fedmsg is optional (ie: the app doesn't fail if fedmsg is not available/installed). This makes it easier to test apps outside of infrastructure and makes us more robust in case fedmsg starts failing sometime.

Libraries to use

• kitchen: A library of common code that we've needed in Fedora. Lots of code to deal with unicode issues and localization. A few other things as well. Docs yum install python-kitchen

Web Frameworks

Use one of Flask, Pyramid, or TurboGears2. Limiting the number of web frameworks we're responsible for will greatly help with our long term maintenance burden.

Common Development Tasks

Use git flow

We would like to have our upstream repository reflect what's deployed in infrastructure as well as what we are working on at the moment. git flow's ideas of separate branches maps well to this vision. The master branch in git flow should be very close to the code we have deployed in production. It should contain the code from the latest release plus any hotfixes that we've applied over the top (in infrastructure we use a puppet module to deploy the hotfix. In the upstream repo, we can use git flow hotfixes to manage applying the hotfix to the master and develop branch).

Getting started

Quickstart to using git flow.

yum install gitflow.

Then get clone the repository and initialize it for use:

# First thing when you initially clone a repo
$ git clone $URL
$ git flow init
# This will ask you questions.  The defaults are almost always
# fine because we've setup the repositories to work with it

You should always work on the 'devel' or 'develop' branch if just doing small maintenance. Changes on develop will get pushed out as the next major release.

New major feature

# Working on a major feature
# git flow creates feature branches that are prefixed with feature/

$ git flow feature start my-new-feature
# make edits, commit

# Share that feature with others for review.  This pushes your branch up to github
$ git flow feature publish my-new-feature

Go to the github interface and open a pull request. (In the future, perhaps we could package http://defunkt.io/hub/ and get a more automated way to do this step).

Ping someone(s) to review the change for you and +1 on the pull request. We aren't very strict about how thorough a code review must be at this point. The code review is partly to catch errors in the code but it's also to expose new contributors to how our code works. So a perfunctory code review by someone who's new to this infrastructure project has value just as a review by someone who's a better coder than you. If you want the change to be reviewed by someone specific for specific things that you're unsure about, be sure to ping them specifically.

# When you're done.. you can merge it on github with a click, or if you'd like to do so from the CLI
$ git flow feature finish my-new-feature

Hotfix

hotfixes end up on both the master and develop branches. They branch off of master.

$ git flow hotfix start fix-traceback-on-login
# make some edits
$ git flow release publish fix-traceback-on-login

Ping the #fedora-apps channel or specific people to review and +1 your change. Note that small hotfixes are especially good for getting people who aren't core contributors to this code base to start becoming familiar with it whereas core committers might be a little bored with looking at them. If we have new contributors wanting to join, this is a good way to introduce them to the project.

hotfixes sometimes have a time element so remember that we deploy hotfixes in infrastructure independently of when the code is merged upstream. If you're confident of hte fix, you may want to commit the hotfix to our production (or staging) deployments in infrastructure's puppet repo and create the hotfix ticket in trac while you're waiting for the hotfix to be reviewed.

# When you're done.. you can merge it from the CLI like this:
$ git flow hotfix finish my-new-feature

# If you want to do it on github, you need to both merge the pull request to
# the develop and the master branches.

Making a release

# Releases end up on the master branch
$ git flow release start 0.3.32
$ # make some edits, bump version numbers, commit, release to pypi.
$ git flow release finish 0.3.32
$ git push --all
$ git push --tags

Further Reading

You can also read this quick overview and a more in depth overview of what the various branches are.

Using Vanilla git

If you're more comfortable using vanilla git without the gitflow addon, the things to be aware of are:

• new commits should go to the devel branch (the default on the repos we set up on github).
• master is used to make releases from and should reflect what's in production (or in an rpm, ready to be pushed to production) at any given time.
• hotfixes have their own branches that need to be merged into both master (for release) and devel.