FedoraXenQuickstartFC5

README FIRST

Fedora Core 5 reached its End of Life on Friday June 29th, 2007. This documentation is preserved for historical purposes only. In other words, don't assume anything in this document will work on your current version of Fedora.

About Xen

Xen is an open source virtual machine system. More information on Xen itself can be found at http://wiki.xensource.com/xenwiki/ and the Fedora Xen page.

Fedora is following the 3.0.x Xen line. Xen 3.0.0 was released in December of 2005 and is incompatible with guests using the previous Xen 2.0.x releases.

Quick Start

Setting up Xen and guests should be significantly more straight-forward with Fedora Core 5 than it previously was with Fedora Core 4. Here's a run-through of what is required to configure a system as a Xen host as well as setting up Xen guests.

System Requirements

• Your system must use GRUB, the default boot loader for Fedora[[FootNote(This is required because you actually boot the Xen hypervisor and it then starts the Linux kernel. It does this using the MultiBoot standard.)]
• Sufficient storage space for the guest operating systems. A minimal command-line Fedora system requires around 600MB of storage, a standard desktop Fedora system requires around 3GB.
• Generally speaking, you will want to have 256 megs of RAM per guest that you wish to install. This is largely due to memory requirements for the installation and will hopefully be less by the time FC5 is released.
• Support is present for both x86 and x86_64 systems

Conventions

Commands which require root privileges are prefixed with the character '#'. To become root, issue the command 'su' as a normal user and supply the root password. Alternatively, use another system such as 'sudo' if you have it setup.

Installing the Xen Software

When doing a fresh install of Fedora Core 5, you can specify that Xen should be installed by adding 'xen0' to your kernel command line. This will ensure that the Xen kernel and userspace support packages are installed.

If you already have a Fedora Core 5 system installed, you can install the Xen kernel by running the following command:

This installs the required packages and their dependencies. 'kernel-xen0' contains the Xen-enabled host system kernel (domain 0) as well as the hypervisor. Also, the 'xen' package will be installed, which contains the user-space tools for interacting with the hypervisor.

Once this is done, you should have an entry set up in the file /boot/grub/grub.conf for booting the xen0 kernel. The xen0 kernel is not set as the default boot option.

To set GRUB to boot with xen0 by default, edit /boot/grub/grub.conf and set the default to the xen0 [[FootNote(Note that you can change so that future kernel-xen0 packages are the default kernel by editing /etc/sysconfig/kernel)]

default=0

If the graphical boot screen or X-Windows fails on the next bootup, you may need to disable the graphical boot (rhgb) and switch to using runlevel 3 (command-line only mode).

This is an example /boot/grub/grub.conf, with the 3 option to start the host system on runlevel 3:

default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Fedora Core (2.6.15-1.29_FC5hypervisor)
root (hd0,0)
kernel /xen.gz-2.6.15-1.29_FC5
module /vmlinuz-2.6.15-1.29_FC5xen0 ro root=/dev/VolGroup01/LogVol00 3
module /initrd-2.6.15-1.29_FC5xen0.img

The Xen facilities in Fedora Core uses VNC to provide a graphical interface that may be accessed from remote systems, as well as the host. Install the vnc package on those systems that will be used to manage Xen:

This provides the vncviewer client.

Enabling Xen

Once you've rebooted with the xen-hypervisor GRUB option, you should be running with a Xen-enabled kernel [[FootNote(To check that you are running a xen0 kernel use 'uname -r' - the name of the kernel should contain xen0)] . Also, the Xen service should already be running. To check that Xen is running successfully, enter the following command:

You should see Domain-0 (the Xen-enabled host) running.

Note that for the default setup, networking for guest OS's is bridged. This means that they will get an IP address on the same network as your host, thus, if you have a DHCP server providing addresses, you will need to ensure that it is configured to give addresses to your guests. You can change to another networking type by editing /etc/xen/xend-config.sxp

Creating A Guest System

These instructions set up a simple Fedora installation as a Xen guest system.

Building a Fedora Guest System

With Fedora Core 5, installation of Xen guests using anaconda is supported, although with a few known bugs. The xen package includes a script for starting guest installs. You can run it with

You will then be prompted to answer a few questions regarding the virtual machine you are trying to create. This information can all be passed as command line options; run with an argument of --help for more details. In particular, kickstart options can be passed with -x ks=options.

1. What is the name of your virtual machine? This is the label used for display in the various xm commands as well as being used as the name of the domain config file in /etc/xen
2. How much RAM should be allocated (in megabytes)? This is the amount of RAM to be allocated for the guest instance in megabytes (eg, 256). Note that installation with less than 256 megabytes is not recommended (or likely to work with test2).
3. What would you like to use as the disk (path)? The local path and file name of the file to serve as the disk image for the guest (eg, /home/joe/xenbox1). This will be exported as a full disk to your guest.
4. How large would you like the disk to be (in gigabytes)? The size of the virtual disk for your guest if the file specified above does not already exist. The amount to specify depends on what you intend to install. 4.0 gigabytes is a reasonable guess for a "default" install
5. What is the install location? This is the path to a Fedora Core 5 installation tree in the format used by anaconda. NFS, FTP, and HTTP locations are all supported. Examples include:
   • nfs:my.nfs.server.com:/path/to/test2/tree/
   • http://my.http.server.com/path/to/tree/
   • ftp://my.ftp.server.com/path/to/tree

If you get an error at this point saying 'The privileged domain did not balloon!', you may need to resize one or more of your existing VMs' memory limits in order to free up RAM for the new guest VM. For instance, to resize dom0 to use only 256 megs of RAM, you could run:

xm mem-set 0 256

Once you've shrunk your existing domains, you should be able to re-try the xenguest-instally.py script.

After this, the install kernel and initrd will be grabbed from your install tree and the installer will start.

• Once the questions are answered, you should be prompted with the standard anaconda text screen to choose the installation language and then keyboard.
• Configure TCP/IP as you normally would for a system.
• You are now about to install the guest. You can proceed with either a text mode or graphical mode via VNC.

Graphical Mode Installation via VNC

• Select Start VNC
• Now provide a password for a secure VNC connection or No Password for an open connection
• The VNC server information will be presented. Connect with your choice of VNC client (either running on the local machine or another system).
• VNC will display a standard Fedora graphical installation screen. Proceed with the installation of your xen guest.
• Notes
• The xvda device is the file identified during the install-script.py process. It appears to the guest as an entire disk, not just as a partition; you can partition it or carve it up with LVM just like a normal disk. The first time you use it, you will need to partition and format it during the install process.

After Installation

Once the installer exits, you'll want to power your guest OS on with the following command.

foo is the name you gave for your guest. The -c will attach a xen console to your vm. A variety of other commands are available via xm including

for a list of commands that can be run

for a display showing what your virtual machines are doing similar to that provided by top.

to nicely shut down a guest OS where foo is the name of your guest.

to save the state of the guest 'foo' to the file /root/foo.restore

to restore the above saved guest.

Automatically Loading Domains On Startup

If you want your domains to automatically start when rebooting the system make sure that you enable the xendomains service, and place the Xen configuration file for the domains in the /etc/xen/auto/ directory.

To enable the xendomains service:

Note that the /etc/init.d/xendomains script is broken with respect to its use of the LSB init functions. This is arguably due to a bug in Fedora's LSB support, and has been filed in Fedora's bugzilla as bug 171052 and bug 171056 . The latter bug includes a modified version of the /etc/init.d/xendomains script as an attachment which you can use instead.

Troubleshooting

If you're running into issues with your Xen installation there are two log files on the host system to help you out. The file /var/log/xend.log holds the same information as you receive when running 'xm log', and contains basic information. Unfortunately these log messages are often very short and contain little useful information. The following is the output of trying to create a domain running the kernel for NetBSD/xen.

[2005-06-27 02:23:02 xend]  ERROR (SrvBase:163) op=create: Error creating domain:(0, 'Error')
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvBase.py", line 107, in _perform
val = op_method(op, req)
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", line 71, in op_create
raise XendError("Error creating domain: " + str(ex))
XendError: Error creating domain: (0, 'Error')

The second file, /var/log/xend-debug.log usually contains much more detailed information. Trying to start the NetBSD/xen kernel will result in the following log output:

ERROR: Will only load images built for Xen v3.0
ERROR: Actually saw: 'GUEST_OS=netbsd,GUEST_VER=2.0,XEN_VER=2.0,LOADER=generic,BSD_SYMTAB'
ERROR: Error constructing guest OS

If you are reporting errors or need help with your Xen setup you should always include both the xend.log and xend-debug.log messages that you receive.

Serial Console

For more difficult problems, serial console can be very helpful. If the Xen kernel itself has died and the hypervisor has generated an error, there is no way to record the error persistently on the local host. Serial console lets you capture it on a remote host.

You need to set up the Xen host for serial console output, and set up a remote host to capture it. For the console output, you need to set appropriate options in /etc/grub.conf, for example:

title Fedora Core (2.6.16-1.2080_FC5xen0)
root (hd0,2)
kernel /xen.gz-2.6.16-1.2080_FC5 com1=38400,8n1 sync_console
module /vmlinuz-2.6.16-1.2080_FC5xen0 ro root=LABEL=/ rhgb quiet console=ttyS0 console=tty pnpacpi=off
module /initrd-2.6.16-1.2080_FC5xen0.img

for a 38400-bps serial console on com1 (ie. /dev/ttyS0 on Linux.) The "sync_console" works around a problem that can cause hangs with asynchronous hypervisor console output, and the "pnpacpi=off" works around a problem that breaks input on serial console. "console=ttyS0 console=tty" means that kernel errors get logged both on the normal VGA console and on serial console. Once that is done, you can install and set up ttywatch (from fedora-extras) to capture the information on a remote host connected by a standard null-modem cable.

Accessing data on a guest disk image

There are two tools which can help greatly in accessing data within a guest disk image: lomount and kpartx. Remember never to do this while the guest is up and running, as you could corrupt the filesystem if you try to access it from the guest and dom0 at the same time!

• lomount

lomount only works with small disk images and cannot deal with LVM volumes, so for more complex cases, kpartx (from the device-mapper-multipath RPM) is preferred:

• kpartx

add map guest1p1 : 0 208782 linear /dev/xen/guest1 63
add map guest1p2 : 0 16563015 linear /dev/xen/guest1 208845

Note that this only works for block devices, not for images installed on regular files. To use file images, you'll need to set up a loopback device for the file first:

/dev/loop0
add map loop0p1 : 0 208782 linear /dev/loop0 63
add map loop0p2 : 0 12370050 linear /dev/loop0 208845

In this case we have added an image formatted as a default Fedora install, so it has two partitions: one /boot, and one LVM volume containing everything else. They are accessible under /dev/mapper:

brw-rw---- 1 root disk 253,  6 Jun  6 10:32 xen-guest1
brw-rw---- 1 root disk 253, 14 Jun  6 11:13 guest1p1
brw-rw---- 1 root disk 253, 15 Jun  6 11:13 guest1p2

To access LVM volumes on the second partition, we'll need to rescan LVM with "vgscan" and activate the volume group on that partition (named "VolGroup00" by default) with "vgchange -ay":

Reading all physical volumes.  This may take a while...
Found volume group "VolGroup00" using metadata type lvm2
2 logical volume(s) in volume group "VolGroup00" now active
LV        VG         Attr   LSize   Origin Snap%  Move Log Copy%
LogVol00  VolGroup00 -wi-a-   5.06G
LogVol01  VolGroup00 -wi-a- 800.00M
...

Note
Always remember to deactivate the logical volumes with "vgchange -an", remove the partitions with "kpartx -d", and (if appropriate) delete the loop device with "losetup -d" after you are finished. There are two reasons: first of all, the default volume group name for a FC install is always the same, so if you end up activating two disk images at the same time you'll end up with two separate LVM volume groups with the same name. LVM will cope as best it can, but you won't be able to distinguish between these two groups on the command line.

And secondly, if you don't deactivate it, then if the guest is started up again, you might end up with the LVM being active in both the guest and the dom0 at the same time, and this may lead to VG or filesystem corruption.

Hints

Q: I am trying to start the xend service and nothing happens, then when I do a xm list1 I get the following:

Error: Error connecting to xend: Connection refused.  Is xend running?

Alternatively, I run xend start manually and get the following error:

ERROR: Could not obtain handle on privileged command interface (2 = No such file or directory)
Traceback (most recent call last):
File "/usr/sbin/xend", line 33, in ?
from xen.xend.server import SrvDaemon
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line 21, in ?
import relocate
File "/usr/lib/python2.4/site-packages/xen/xend/server/relocate.py", line 26, in ?
from xen.xend import XendDomain
File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line 33, in ?
import XendDomainInfo
File "/usr/lib/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 37, in ?
import image
File "/usr/lib/python2.4/site-packages/xen/xend/image.py", line 30, in ?
xc = xen.lowlevel.xc.xc()
RuntimeError: (2, 'No such file or directory')

A: You have rebooted your host into a kernel that is not a xen-hypervisor kernel. Yes I did this myself in testing :)

You either need to select the xen-hypervisor kernel at boot time or set the xen-hypervisor kernel as default in your grub.conf file.

Installation of guest FC5 freezes

If your installation of FC5 appears to start normally, but after a long wait it seems to go nowhere, you may need to update your installation tree's images/xen/vmlinuz and images/xen/initrd.img.

In this example, the most current kernel is 2.6.16-1.2133 and the original kernel version in the FC5 install tree images/xen directory is 2.6.15-1.2054. You may have to adjust these accordingly. Make sure your 'xen0' kernel matches your 'xenU' kernel version. This worked for the i686 architecture, there is no reason why it shouldn't work for x86_64 as well just by replacing i686 with x86_64 in the example commands. I'll get arround to testing this soon.

Install the package 'kernel-xenU':

If necessary, update your 'xen0' kernel:

Before starting the guest installation, reboot to your updated xen0 kernel.

Go to the top level of the FC5 installation tree and change directory to images/xen:

Move (do not delete) the old vmlinuz and initrd.img out of the way:

Updating vmlinuz:

Simply copy the new vmlinuz from /boot:

Updating initrd.img:

Generate modules.dep for the new 'xenU' kernel:

Make a temporary working directory and change to it:

Extract the contetns of the old initrd.img:

Go to the modules directory:

Copy the new versions of modules.alias and modules.dep:

Extract the contents of modules.cgz:

Create a directory tree for the new modules:

Find the same modules from the new kerenl and copy them in directory created in previous step:

#done

Remove the old modules.cgz:

Repakage modules.cgz with the new modules:

Remove the temporary modules directories:

Back to the top level of the extracted old/initrd.img:

Repackage 'initrd.img':

Back to images/xen:

Cleanup by removing the tmp directory created above:

You can now follow the guest installation procedure described above with the updated vmlinuz and initrd.img in your FC5 installation tree.

Getting Help

If the Troubleshooting section above does not help you to solve your problem, check the Red Hat Bugzilla for existing bug reports on Xen in FC5. The product is "Fedora Core", and the component is "kernel" for bugs related to the xen0 kernel and "xen" for bugs related to the tools. These reports contain useful advice from fellow xen testers and often describe work-arounds.

For general Xen issues and useful information check the Xen project documentation , and mailing list archives .

Finally, discussion on Fedora Xen support issues occur on the Fedora Xen mailing list

Footnotes