Moji

This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

Description

Moji is a simple build system in which the client grabs a package from the server, builds it locally using mock, and then uploads the results back to the original server. Moji stands for "mini koji" and it was written to support the Fedora-ARM pre-koji build effort (for example, F15-armv5tel).

Using Moji for the Fedora ARM Build Effort

Setting Up

To use a system as a Moji client for the ARM F15 armv5tel bringup:

1. Install the Moji repo file: rpm -Uvh http://scotland.proximity.on.ca/fedora-arm/moji-repo/rpm/moji-release-1-1.noarch.rpm
2. Install the Moji preconfigured client: yum install -y moji-client
3. Start the Moji server: service moji-client start
4. (Optional) Configure the Moji server to start at next boot: chkconfig moji-client on

Please see the arm@lists.fedoraproject.org mailing list for more information.

Starting and Stopping the Moji Client

• To start the Moji client: service moji-client start

• To stop the Moji client gracefully, at the end of the current build: service moji-client disable
• To stop the Moji client immediately: service moji-client stop

Moji Source Code

The source code for Moji is in the Fedora ARM git repository at http://fedorahosted.org/arm/ and can be viewed at https://fedorahosted.org/arm/browser/moji/moji.py

Application Information

Server

python /path/to/moji.py server /path/to/source /path/to/working /path/to/finished /path/to/repo [password]

• "source" is the folder containing source RPM files
  • the "source" folder can contain a file called "priority.txt" which is a simple plaintext file where each line is the filename of a source pkg that should be built first (ex. nled-2.52-7.fc15.src.rpm)
• "working" is a folder that contains any source RPM files currently being worked on by a builder
• "finished" is a folder containing source RPM files that have been built (failure or success)
• "repo" is the directory which contains the mock build result output of any "finished" builds
  • the "repo" directory should also be publicly accessible (ex. via httpd)
  • createrepo needs to also be run via some other means (ex. shell script via incrontab)

For the ARM F15 armv5tel startup, the Moji server is australia.proximity.on.ca.

Client

python /path/to/moji.py client <hostname or ip> <mock config name> [password]

• the mock option should be a config file name (ex fedora-15-arm)
  • the actual mock file (ex. /etc/mock/fedora-15-arm.cfg) should point to at least 2 repos: one to the moji server repo dir and another to a pre-established "main-line" repo

Design/Protocol

• defaults to TCP port 6789
• non-threaded and does not fork per child
  • does fork before any request that could take a while (ex. sending files or building packages)
• if a client disconnects, their build is considered failed and sent to another builder later on
• clients send dummy keep-alive ping messages thru its socket object
• more details to come soon...

Repo Management

$ incrontab -e

/tmp/builds/done IN_CREATE touch /tmp/builds.flag

$ vim builds.sh

#!/bin/bash

while true
do
	if [ -e /tmp/builds.flag ] # check the lock flag
	then
		echo "Starting on [ `date` ]" | tee -a /home/builder/builds.log
		
		rm /tmp/builds.flag
		mkdir /tmp/builds/repo 2> /dev/null # make sure the temp repo directory exists
		rm -frv /tmp/builds/repo/* 2> /dev/null # clear the repo directory
		
		ls /tmp/builds/done | while read line
		do
			if [ ! -e "/tmp/builds/repo/${line}" ]
			then
				ln -s "/tmp/builds/done/${line}" "/tmp/builds/repo/${line}"
			fi
		done
		
		rm -frv /tmp/builds/repo/repodata # remove any repo data links
		cp -frv /tmp/builds/done/repodata /tmp/builds/repo/ # copy over the old repo data
		createrepo --update -d /tmp/builds/repo # create/update the new repo data
		cp -frv /tmp/builds/repo/repodata /tmp/builds/done/ # copy back the new repo data
		
		echo "Ending on [ `date` ]" | tee -a /home/builder/builds.log
	fi
	
	sleep 5 # sleep loop or risk fire!
done

Mock Config

fedora-15-arm.cfg

config_opts['root'] = 'fedora-15-arm'
config_opts['target_arch'] = 'armv5tel'
config_opts['legal_host_arches'] = ('armv5tel', 'armv6l', 'armv7l')
config_opts['chroot_setup_cmd'] = 'groupinstall buildsys-build'
config_opts['dist'] = 'fc15'
config_opts['plugin_conf']['root_cache_enable'] = False
config_opts['plugin_conf']['yum_cache_enable'] = False

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=1
reposdir=/dev/null
logfile=/var/log/yum.log
retries=20
obsoletes=1
gpgcheck=0
assumeyes=1
syslog_ident=mock
syslog_device=

# repos

[custom]
name=fedora-custom
baseurl=http://server.hostname.com/repo
enabled=1

### NOTE : USE A VAR HERE FOR THE PATH INSTEAD ###

[f15-arm-koji]
name=fedora15-arm-koji
baseurl=http://arm.koji.fedoraproject.org/repos/dist-f15-build/latest/arm/
enabled=1

[fedora]
name=fedora
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-13&arch=arm
failovermethod=priority

[updates-released]
name=updates
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f13&arch=arm
failovermethod=priority
"""

Base Package Set

priority.txt

• Use these package names only to get a list of full package file names (ENVRA)
• Each package entry must be on its own line in the text file

acl alsa-lib apr apr-util arts ascii aspell atk attr audiofile audit audit-libs avahi basesystem bash bc bind binutils bison byacc bzip2 bzip2-libs ca-certificates ccache cdparanoia celt check chkconfig chrpath cloog cloog-ppl cluster cmake compat-readline5 ConsoleKit coreutils coreutils-libs corosync cpio cpp cracklib cracklib-dicts crda cronie cryptsetup-luks cups curl cvs cyrus-sasl cyrus-sasl-lib dash db4 db4-utils dbus dbus-glib dbus-python deltarpm desktop-file-utils dev dhcp diffutils dos2unix doxygen dvipdfm dvipdfmx dvipng e2fsprogs ed elfutils elfutils-libelf elfutils-libs enchant esound expat expect fakechroot fakeroot fedora-release file file-libs filesystem findutils firebird flac flex fontconfig freeglut freetype gamin gawk gc gcc gcc-c+ gcc-host gcc-libgcc GConf2 gd gdb gdbm gdk-pixbuf2 gettext git glib2 glibc glibc-common glibc-devel glibc-headers gmp gnome-keyring gnupg gnupg2 gperf gpm grep gstreamer gtk2 gtk3 guile gzip hostname httpd hunspell icu imake info initscripts iproute iputils iw jasper java-1.5.0-gcj kbd kernel kernel-headers keyutils keyutils-libs krb5 krb5-libs lapack less lftp libacl libart_lgpl libassuan libasyncns libattr libavc1394 libblkid libbonobo libcap libcap-ng libcgroup libcom_er libconfig libcurl libdaemon libdbi libdmx libdrm libedit libffi libfontenc libfreebob libFS libgcc libgcrypt libglade2 libgomp libibverbs libICE libIDL libidn libiec61883 libieee1284 libjpeg-turbo libksba libmcrypt libmng libmodman libmount libmpc libnet libnl libnotify libogg liboil libotf libpaper libpciaccess libpng libraw1394 librdmacm librelp libselinux libsemanage libsepol libSM libsndfile libssh2 libstdc+ libstdc++-deve libtalloc libtasn1 libtdb libthai libtiff libtool libusb libusb1 libuser libutempter libuuid libvisual libvoikko libvorbis libX11 libXau libXaw libxcb libXcomposite libXcursor libXdamage libXdmcp libXevie libXext libxfce4util libXfixes libXfont libXft libXi libXinerama libxkbfile libxml2 libXmu libXNVCtrl libXp libXpm libXrandr libXrender libXres libXScrnSaver libxslt libXt libXtst libXv libXvMC libXxf86dga libXxf86misc libXxf86vm libyubikey linux-atm llvm lm_sensors lua lvm2 lynx lzo m2crypto m4 make man-db mingetty module-init-tools mpc mpfr mysql ncurses ncurses-base ncurses-libs netpbm net-snmp net-tools newt nkf nspr nss nss-softokn nss-softokn-freebl nss-sysinit nss-util oniguruma openjade openldap opensp openssh openssl ORBit2 pam pango passwd patch pcre pcsc-lite pcsc-lite-ccid perl perl-HTML-Parser perl-libs perl-Module-Pluggable perl-Pod-Escapes perl-Pod-Simple perl-threads perl-threads-shared perl-Tk perl-XML-Parser php pinentry pkgconfig plymouth polkit poppler popt postgresql ppl procps psmisc psutils pth pygpgme pyOpenSSL python python3 python-enchant python-krbV python-libs python-pycurl python-simplejson python-sqlalchemy PyXML qt3 rarian readline recode redhat-rpm-config rpm rpm-build rpm-libs rrdtool rsync ruby sed setup shadow shadow-utils shared-mime-info sharutils slang speex sqlite startup-notification strace stunnel sysfsutils system-config-keyboard systemd system-setup-keyboard systemtap sysvinit t1lib tar t-binutils tcl tcp_wrappers tcsh teckit telnet texinfo texlive t-gcc tidy tix tk tmpwatch transfig ttmkfdir tzdata u-boot udev unique3 unixODBC unzip usbutils usermode ustr util-linux util-linux-ng uuid vala vim wget which wireless-tools Xaw3d xcb-util xfconf xfig xinetd x-loader xmlto xorg-x11-drv-evdev xorg-x11-drv-fbdev xorg-x11-drv-keyboard xorg-x11-drv-v4l xorg-x11-drv-vesa xorg-x11-font-utils xorg-x11-utils xorg-x11-xauth xorg-x11-xdm xorg-x11-xfs xorg-x11-xfwp xorg-x11-xinit xorg-x11-xkb-utils xz xz-libs yum yum-metadata-parser zip zlib