OpenStack status report 2012-03-20

Test Day

We held a test day for OpenStack Essex in Fedora 17 on March 8.

The event was fairly well attended with 11 people reporting their test results on the wiki page.

The set of test cases we used were heavily based on the Fedora 16 Test Day but included testing Keystone, Horizon and Quantum (with openvswitch) integration. We also had instructions detailing how to run the Tempest test suite. Finally, for the first time, we had a Fedora 16 guest image that everyone could download and use for testing.

Running through the various tests highlighted some of the implications of using keystone authentication e.g.

1. nova-manage's user and project commands become obsolete
2. Equivalent of nova-manage project zipfile is non-trivial
3. nova-manage image convert doesn't work anymore, you need to use glance directly

Worryingly, we hit a number of serious SELinux issues e.g.

1. #801299 - AVC when first starting mysqld
2. #801330 - AVC denials starting OpenStack glance services
3. #801352 - SELinux policy for OpenStack's new nova-cert service
4. #801746 - SELinux AVC denial executing from /tmp
5. #760055 - SELinux policy for keystone

Horizon mostly "just worked", which is really great. We did file a number of bugs, though:

1. #801745 - intermittent database connection errors
2. #801690 - horizon: download ec2 credentials fails
3. #801688 - horizon: cannot delete a user or project
4. #801686 - horizon: failed to detach or delete a volume
5. #801685 - horizon: no VNC console
6. #801684 - horizon: action dropdowns don't appear to work
7. #801208 - tenant chooser doesn't work

Some other bugs filed include:

1. #803354 - keystone returns 500 errors after a while
2. #801452 - euca-describe-instances does not show IP addresses
3. #801312 - webob deprecation warnings
4. #801302 - sqlalchemy-migrate warnings during openstack-nova-db-setup
5. #801366 - Invalid X-Auth-Token breaks API service
6. #800704 - keystone endpoint-list tracebacks

We also stumbled across this cheeky little libvirt regression introduced by switching to systemd:

1. #802475 - libvirt in a VM occasionally brings up 'default' network when it shouldn't, kills vm networking

One idea for future improvement is for us to use a dedicated yum repository for the test day to remove any ambiguity about which updates have been pushed to the mirrors and to allow us to quickly push out fixes on the day itself.

Fedora 17

Essex Release Progress

The essex-4 milestone was released on March 1 and quickly pushed into Fedora 17, as was swift 1.4.7 and Quantum essex-4.

The Essex release now enters its release candidates phase in the lead up to the final release on April 5.

So far, Nova rc1 and Quantum rc1 have been released, and Swift has tagged its 1.4.8 release.

Getting Started Wiki

In preparation for the Fedora 17 release, we have updated the Getting_started_with_OpenStack_Nova wiki page and moved it to Getting_started_with_OpenStack_on_Fedora_17.

The new instructions include details on how to use Keystone authentication and the Horizon dashboard.

Also Quantum installation and troubleshooting steps have been prepared for testing nova and quantum in a multi-node setup using openvswitch as a plugin.

F-17 Package Updates

Since the last status report, the following notable updates have been pushed to F-17:

• openstack-nova-2012.1-0.7.e4.fc17
  1. update to essex-4
  2. switch to new .ini style conf file format
  3. depend on new bridge-utils package
• openstack-nova-2012.1-0.8.e4.fc17
  1. support non-blocking libvirt operations
  2. suppress power state errors
  3. fix nova-compute failing to start
  4. fix sqlalchemy-migrate warnings during openstack-nova-db-setup
  5. fix osapi v1.1 returns errors when getting server status
• openstack-glance-2012.1-0.5.e4.fc17
  1. Update to essex-4
  2. Require pyxattr rather than python-xattr
  3. Add python-iso8601 dependency
• openstack-keystone-2012.1-0.9.e4.fc17
  1. Update to essex-4
• openstack-keystone-2012.1-0.10.e4.fc17
  1. Change default catalog backend to sql
  2. Add missing keystoneclient dep
  3. Add openstack-config-set script
• python-django-horizon-2012.1-0.1.rc1.fc17
  1. Update to RC1 snapshot
  2. Change default URL to http://localhost/dashboard
  3. Fix static content
• openstack-quantum-2012.1-0.3.e4.fc17
  1. Depend on Open vSwitch
• openstack-quantum-2012.1-0.3.e5.fc17
  1. Update to Essex RC1 candidate tarball

Open vSwitch Package Review

Chris Wight and Dan Berrange worked openvswitch through the review process.

Open vSwitch is a major Fedora 17 feature that is going to be hugely beneficial to OpenStack and Quantum in Fedora. Very exciting!

Multiple Instances of Swift Services

When swift moved from SysV init to systemd, we lost the ability to launch multiple instances of the same service on a machine.

Derek stumbled across systemd's "instances" support and we'll soon re-instate this support. See #805149 for more details.

Keystone LDAP Support

Adam Young had his LDAP backend merged into keystone for essex-4.

Adam blogged about using keystone's LDAP driver with FreeIPA.

Misc Fedora News

F-16 Guest Images

Dan Berrange posted a F-16 guest image which can be used with OpenStack.

The image is based on the Fedora EC2 images, includes cloud-init and is a 200Mb download.

Essex Preview Repo For F-16

Alan Pevec has started maintaining a "preview" repository for Essex on Fedora 16.

If you're running Fedora 16 and you want to try out the Essex release, this is the repo for you!

Also Steve Dake summarized steps for updating the OpenStack Diablo release to Essex on Fedora 16.

Devstack F-16 Support

Russell Bryant has been hard at working improving devstack's F-16 support in his fedora-support branch in github. This branch also adds support for using Qpid instead of RabbitMQ.

Keystone Fedora PAM Support

Russell also added PAM authentication support for Keystone in Fedora.

Fedora Support in Puppet Labs Recipes

Derek Higgins has been working on adding Fedora support to Puppet Labs' recipes for OpenStack.

Support was added to Nova. Work is underway for Swift. And a whole bunch of dependent modules are also gaining Fedora support.

These recipes are used by upstream's Smokestack instance when testing on Fedora.

iSCSI tgtd Issue With Systemd

A recent systemd update caused tgtd to hang for 5 minutes on startup. Derek pushed this simple fix for the problem.

cloud-init and OpenStack

We're now testing cloud-init with OpenStack a bit more. Pádraig Brady and Joe Brue filed these bugs:

1. #795998 - run-parts is run with the non-existent --regex option
2. #750979 - cloud-init scripts do not make hostname changes permanent in /etc/sysconfig/network

and both have been fixed in recent updates.

Broken python-boto Update

OpenStack folks identified a serious python-boto issue before update hit stable.

Upstream News

OpenStack Governance Elections

The OpenStack project held its spring governance elections recently and elected technical leads for Nova, Swift, Glance, Keystone and Horizon. Two new members of the Project Policy Board were also elected.

Two Fedora developers - Mark McLoughlin and Eoghan Glynn - were nominated for positions but, despite a hard-fought and emotional campaign, neither were elected. Next time!

Rewritten libvirt Driver XML Generation

Dan Berrange has posted a massive patch set to Nova for comments. The patches replaces Nova's usage of Cheetah templates for generating libvirt XML with a safer approach of de-serializing a DOM.

Since this is such a large change, it will not be proposed until Folsom opens up.

Dan also had these interesting fixes merged lately:

1. Remove the <acpi/> feature from UML/LXC guests
2. Simply & unify console handling for libvirt drivers
3. Use cache='none' for all disks

libvirt Driver Image Handling

Pádraig continued improving the libvirt driver's image handling with these fixes:

1. ensure atomic manipulation of libvirt disk images
2. allow the compute service to start with missing libvirt disks

libvirt Issue Fixed in F-16, Broken in Ubuntu Oneric

This report of a libvirt issue with OpenStack is rather interesting. It turns out that this was an issue found upstream, fixed in Fedora 16 but is still broken in Ubuntu Oneric.

Kudos to our libvirt package maintainers!

Rootwrap in Quantum

Rootwrap is a helper script added to Nova in Essex to help lock down the sudo commands that Nova can run. Bob Kukura has now added rootwrap to Quantum and, while doing so, ensured that the Quantum agents no longer need to run as root.

Blogs etc.

Keystone Blog Posts

Adam also blogged on some other topics related to keystone:

1. Keystone should move to Apache HTTPD
2. PKI for Keystone
3. HATEOAS Openstack Keystone

OpenStack, Deltacloud and CIMI

Marios Andreou, a developer on the deltacloud project, wrote this interesting blog post on OpenStack Networking and CIMI.

CIMI is a cloud API standard being developed by the DMTF and supported by the deltacloud project.