About Xen
The Xen virtual machine system is now available in Fedora Core 4 and above versions. More information on Xen itself can be found at http://wiki.xensource.com/xenwiki/ and the Fedora Xen page.
Fedora is following the -unstable (3.0.x) Xen development line at the moment, which does occasionally lead to things being broken but also lets us track a lot of the more interesting work going on there. It also prevents the current NetBSD/xen, FreeBSD 5.3 and Solaris 11beta ports from running because they are all based on Xen 2.0.x.
About This Quick Start
Since setting up to run Xen isn't entirely straight-forward, here's a run-through of what should work for configuring a Fedora Core 5 system as a Xen host, and setting up a single Xen guest.
This Quick Start is an update version of the Fedora Core 4 Xen Quick Start page. A number of things have changed since FC4 and Xen 2 such as package names and configuration options, this guide is aimed at being a fresh start rather than an upgrade guide.
System Requirements
- Your system must use GRUB, the default boot loader for Fedora [[FootNote(This is required because you actually boot the Xen hypervisor and it then starts the Linux kernel. It does this using the Multi
Boot standard.)] - 256 MB of RAM with the default setup [[FootNote(You can conceivably get by with less, but you'll have to reduce the dom0_memory line in
/boot/grub/grub.conf. The Xen hypervisor itself requires ~32 MB. Plus the memory required for your guest systems.)] - Sufficient storage space for the guest operating systems. A minimal command-line Fedora system requires around 600MB of storage, a standard desktop Fedora system requires around 3GB.
- SELinux must not be enforcing it's policy, or some of the operations below will fail. Use
system-config-securitylevelto set SELinux to not enforce, and reboot to let this change take effect.
Conventions
Commands which require root privileges are prefixed with the character '#'. To become root, issue the command 'su' as a normal user and supply the root password. Alternatively, use another system such as 'sudo' if you have it setup.
Installing the Xen Software
Before we start we need to set up the basics of the Xen system. To install the main software, type the command:
The xen package contains the Xen hypervisor and tools needed to run virtual machines on x86 systems; kernel-xen-hypervisor is the Xen-enabled host system kernel (domain 0) and kernel-xen-guest is the Xen guest system kernel.
Once this is done, you should have an entry set up in the file /boot/grub/grub.conf for booting the xen-hypervisor kernel. The xen-hypervisor kernel is not set as the default boot option.
To set GRUB to boot with xen-hypervisor by default, edit /boot/grub/grub.conf and set the default to the xen-hypervisor kernel:
default=2
xen-hypervisor is the third entry in /boot/grub/grub.conf, these entries are numbered starting from zero.
rhgb) and switch to using runlevel 3 (command-line only mode).This is an example /boot/grub/grub.conf, with the init 3 option to start the host system on runlevel 3:
default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title Fedora Core (2.6.15-1.29_FC5hypervisor) root (hd0,0) kernel /xen.gz-2.6.15-1.29_FC5 module /vmlinuz-2.6.15-1.29_FC5hypervisor ro root=/dev/VolGroup01/LogVol00 init 3 module /initrd-2.6.15-1.29_FC5hypervisor.img
Enabling Xen
Once you've rebooted with the xen-hypervisor GRUB option, you should be running with a Xen-enabled kernel [[FootNote(To check that you are running a xen-hypervisor kernel use 'uname -r' - the name of the kernel should contain xen-hypervisor)] .
Now start the Xen service:
To check that Xen is running successfully, enter the following command:
You should see Domain-0 (the Xen-enabled host) running:
[root@xen-hypervisor ~] # xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 940 1 r----- 267.8
To enable the Xen service to automatically start on bootup, enter the following:
Creating A Guest System
These instructions set up a simple base Fedora installation as a Xen guest system.
Create a Virtual Disk for the Guest System
{{Admon/warning | This example system uses the kernel from the kernel-guest package, as the kernel for your guest domain must currently be kept outside of the guest itself.
First, create an image file to use as the storage for our Fedora guest system. This acts as a virtual hard drive for the guest system. This command creates an image file with a size of 1 GB. To create a larger filesystem increase the seek= value.
dd if=/dev/zero of=fedora-guest1.img bs=1M count=1 seek=1024
Now, format the image file by creating an ext3 filesystem on it.
/sbin/mke2fs -F -j fedora-guest1.img
You should now be able to mount the image on a temporary mountpoint, say /mnt
Setup the Base Environment
To install a guest system on to the image we need to set up some basic files on the filesystem of the image that have to be different for Xen (and for the system installation with yum).
First up create some diectories that we will need in the following steps:
Create some required device nodes in the image. [[FootNote(Using an initrd and thus udev seems to work as well with a Fedora Core development tree as of 2005-09-16. Just run depmod and mkinitrd from xen-hypervisor for the xen-guest kernel; make sure to install a xen-guest kernel into the guest system; perform a special depmod run as this is still missing from the kernel rpm.)] :
Create the guests /etc/fstab file inside the image (i.e. /mnt/etc/fstab), something simple like the following should work:
/dev/sda1 / ext3 defaults 1 1 none /dev/pts devpts gid=5,mode=620 0 0 none /dev/shm tmpfs defaults 0 0 none /proc proc defaults 0 0 none /sys sysfs defaults 0 0
Finally, make sure /sys/ is mounted inside the image so that mkinitrd won't fail
Configure YUM
We need to set up a temporary configuration for yum so we can install our guest system. Create a yum-install.conf file that we will use to find our repositories, this file can be created in a temporary location as it will not be needed after we successfully install our guest.
For a stable release the contents of the yum-install.conf file should contain:
[main] cachedir=/var/cache/yum debuglevel=2 logfile=/var/log/yum.log exclude=*-debuginfo gpgcheck=0 obsoletes=1 reposdir=/dev/null [base] name=Fedora Core 5 - $basearch - Base mirrorlist=http://fedora.redhat.com/download/mirrors/fedora-core-5 enabled=1 [updates-released] name=Fedora Core 5 - $basearch - Released Updates mirrorlist=http://fedora.redhat.com/download/mirrors/updates-released-fc5 enabled=1
For a rawhide development system the contents of the yum-install.conf file should instead contain:
[main] cachedir=/var/cache/yum debuglevel=2 logfile=/var/log/yum.log exclude=*-debuginfo gpgcheck=0 obsoletes=1 reposdir=/dev/null [development] name=Fedora Core Development Tree mirrorlist=http://fedora.redhat.com/download/mirrors/fedora-core-rawhide enabled=1
If you want to use a local Fedora mirror, then you'll want to set the baseurl= parameter and comment out the mirrorlist= parameter before using it; otherwise, it will point to the default set of mirror Websites.
Install a Base Operating System
Firstly we need to install the kernel-xen-guest package into the image. This will pull in dependencies that total to a bit over 60 MiB.
After the the kernel-xen-guest package and dependencies have installed you should update the initrd:
Then, decide which package group(s) you want to install. Start with Core and Base (or for the space constrained just Core, however this is more difficult as you will be missing utilities such as yum).
In Fedora Core 4 installing Base would pull in the Core group as a dependency. As of Fedora Core 5 this is no longer the case.
Install the guest system with the package groups by running yum. For example, to install with Core and Base use the command:
Grab a coffee and have a snack, it's going to take a little while depending on your connection speed to the yum repository :-)
SELinux must be disabled to do this. SELinux does path based checks for whether or not things are allowed which do not apply when you're doing installs into a chroot environment.
Come back and if everything went okay, you'll have a minimal install in /mnt.
Do any other configuration you want to on the file system and then unmount it:
Registering a Guest System as a Domain
Now, we just have to create a configuration file in /etc/xen/. As a simple example, the following configuration file is /etc/xen/fedora-guest1:
kernel ="/boot/vmlinuz-2.6.15-1.29_FC5guest" memory = 64 name = "fedora-guest1" vif = [''] disk = ['file:/root/fedora-guest1.img,sda1,w'] root = "/dev/sda1" extra = "ro selinux=0 3"
Substitute /boot/vmlinuz-2.6.15-1.29_FC5guest with the xen-guest kernel you installed. Additionally, the device sda1 listed here as the disk is related to the /etc/fstab you created above.
Before actually creating the guest domain you may want to decrease the amount of memory being allocated for your host system (domain0), in order to leave adequate room in RAM for your guest domains. This is especially important as the Xen system won't page your guest domains to your host domain's swap space. Constraining the RAM space used by your host domain can help assure that adequate RAM is available for your guests.
The way to do this is to run the following commands as root:
xm mem-max 0 384 xm mem-set 0 384
The mem-max command restricts the host domain (the 0 parameter signifies domain 0) from ever increasing its memory usage beyond 384 megabytes. The mem-set command forces the host domain to immediately shrink its memory allocation down to 384 megabytes.
It may take a bit of time for the domain 0 kernel to page out memory over the new limit, so you should give it a minute or so to complete.
When this is done, your system should have enough RAM free to allow you allocate one or more pretty good sized guest domains.
Now, create a new domain:
xm create -c fedora-guest1
You should now see the ever familiar boot process, at the end you should see the login prompt at which point you can login as 'root' and begin playing around.
If you get errors about a missing shell, despite
/etc/passwd and /bin/bash (or whatever) being present, you need to disable SELinux for this new domain, as well as domain 0.If you want to disconnect from the running Xen guest console, this can be done with the key control key combination:
^
To reconnect to any running xen guest console type:
xm console fedora-guest1
To restart the guest domain can be done by connecting to the host domain with another window and running the following:
xm shutdown fedora-guest1 xm create -c fedora-guest1
This is pretty early and rough, but it's enough to starting playing with. The next step for development is getting to where you can do actual installations in a Xen guest environment and then being able to boot kernels which are on the guest's file system.
Automatically Loading Domains On Startup
If you want your domains to automatically start when rebooting the system make sure that you enable the xendomains service, and place the Xen configuration file for the domains in the /etc/xen/auto/ directory.
To enable the xendomains service:
Note that the /etc/init.d/xendomains script, as of RPM xen-3.0-0.20050912.fc4, seems to be broken with respect to its use of the LSB init functions. This is arguably due to a bug in Fedora's LSB support, and has been filed in Fedora's bugzilla as bug 171052 and bug 171056 . The latter bug includes a modified version of the /etc/init.d/xendomains script as an attachment which you can use instead.
Troubleshooting
If you're running into issues with your Xen installation there are two log files on the host system to help you out. The file /var/log/xend.log holds the same information as you receive when running 'xm log', and contains basic information. Unfortunately these log messages are often very short and contain little useful information. The following is the output of trying to create a domain running the kernel for NetBSD/xen.
[2005-06-27 02:23:02 xend] ERROR (SrvBase:163) op=create: Error creating domain:(0, 'Error')
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvBase.py", line 107, in _perform
val = op_method(op, req)
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", line 71, in op_create
raise XendError("Error creating domain: " + str(ex))
XendError: Error creating domain: (0, 'Error')
The second file, /var/log/xend-debug.log usually contains much more detailed information. Trying to start the NetBSD/xen kernel will result in the following log output:
ERROR: Will only load images built for Xen v3.0 ERROR: Actually saw: 'GUEST_OS=netbsd,GUEST_VER=2.0,XEN_VER=2.0,LOADER=generic,BSD_SYMTAB' ERROR: Error constructing guest OS
If you are reporting errors or need help with your Xen setup you should always include both the xend.log and xend-debug.log messages that you receive.
Q: I am trying to start then xend service and nothing happens, then when I do a xm list I get the following:
Error: Error connecting to xend: Connection refused. Is xend running?
Alternatively, I run xend start manually and get the following error:
ERROR: Could not obtain handle on privileged command interface (2 = No such file or directory) Traceback (most recent call last): File "/usr/sbin/xend", line 33, in ? from xen.xend.server import SrvDaemon File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line 21, in ? import relocate File "/usr/lib/python2.4/site-packages/xen/xend/server/relocate.py", line 26, in ? from xen.xend import XendDomain File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line 33, in ? import XendDomainInfo File "/usr/lib/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 37, in ? import image File "/usr/lib/python2.4/site-packages/xen/xend/image.py", line 30, in ? xc = xen.lowlevel.xc.xc() RuntimeError: (2, 'No such file or directory')
A: You have rebooted your host into a kernel that is not a xen-hypervisor kernel. Yes I even did this myself in testing :)
You either need to select the xen-hypervisor kernel in the grub menu at boot time or set the xen-hypervisor kernel as default in your grub.conf file.
Getting Help
If the Troubleshooting section above does not help you to solve your problem, check the Red Hat Bugzilla for existing bug reports on Xen in FC5. The product is "Fedora Core", and the component is "Xen". These reports contain useful advice from fellow xen testers and often describe work-arounds.
Discussion on Fedora Xen support issues occur on the Fedora Xen mailing list
The Xen 3 User Manual can be found at http://www.cl.cam.ac.uk/Research/SRG/netos/xen/readmes/user/user.html
For general Xen issues and useful information check the Xen project documentation , and mailing list archives .