Docs/Drafts/MockSetupUsingLocalMirror

Mock Setup Using Local Mirror

Author: Parag Nemade Zdeng

Revision: 0.4

Initial Draft: Friday Jun 30, 2006

Last Revised: Saturday Aug 21, 2006

Last Modified: Wednesday Nov 12, 2014

Note: This page describes my personal mock setup using local mirror setup for FC6.

This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

Setting Local Mirrors For Fedora Development version

Prior required steps are to have at least 10GB free space on your hard drive for setting Fedora development repository.

To setup local mirror, create following files

1)copy /etc/yum.repos.d/fedora-development.repo to /etc/yum.repos.d/mirror-fedora-development.repo

2)copy /etc/yum.repos.d/fedora-extras.repo to /etc/yum.repos.d/mirror-fedora-extras.repo

Say if you have /repo directory having minimum free space 10GB then download fedora-mirror.py from http://www.fedoraproject.org/wiki/Projects/fedora-mirror to /mirror_repo.

Change /srv/mirror to /mirror_repo in fedora-mirror.py file. For development version we need only development, extras and groups repository. We don't need any updates repository as its not exist.

Modified version of fedora-mirror.py

Final Modified version of fedora-mirror.py for development version is

#!/usr/bin/python

# License: GPL
# Author:  Rudolf Kastl
# default target: /mirror_repo
# how?: use config file and special mirror .repo files

# todo:

## code:
# - ConfigParser
# - logging
# - legacy and updates-testing awareness
# - fix hierachy into the fedora mirror dir scheme
# - use separate configs for each release
# - progressbars
# - eta (complete, per repo, per file(?!))
## support files/packaging/integration:
# - creation of specially crafted .repo files
# - cron integration
# - 0 conf apache serving as split package
# - mock config files as split package
# - yum localmirror config files as split package
# - discuss access rights on /mirror_repo


################
# special cases:
# - the fedora core repos differ in logic -> core repos have updates, updates-testing, legacy -> to be investigated
# - development is another special case not having updates updates-testing and legacy
# - other repos (3rd party) never have updates and updates-testing
# - SRPMS are another special case -> same for each arch of the same release
###########################


import os

flavourlist=["core", "extras", "updates"] 
versionlist=["development"] 
archlist=["i386"] 
subflavourlist=["binary"] 

### block for parsing commandline switches
from optparse import OptionParser

parser = OptionParser()
# var can be accessed with options.filename

parser.add_option("-d", "--dest", dest="destination", help="Sync Destination", metavar="dir", default="/mirror_repo")
parser.add_option("-n", "--newonly", action="store_true", dest="newonly", help="Sync Destination", default=False)
parser.add_option("-t", "--testonly" , action="store_true", dest="testonly", help="No Sync Transaction", default=False)
parser.add_option("-q", "--quiet", action="store_false", dest="verbose", default=True, help="don't print status msgs to stdout")

(options, args) = parser.parse_args()
### end bloc


targetlocation = options.destination

def syncMirror (version, flavour, arch, subflavour, targetlocation):
print "syncing: Fedora " + version, flavour, arch, subflavour, "to", targetlocation

# handling the special cases

if subflavour == "binary":
subflavourid = ""
subflavourpath = ""
elif subflavour == "source":
subflavourid = "-"+subflavour
subflavourpath = "/SRPMS"
else:
subflavourid = "-"+subflavour
subflavourpath = "-"+subflavour

# devel is special

if version == "development":
repoid="mirror-"+flavour+"-"+version+subflavourid
else:
repoid="mirror-"+flavour+subflavourid

if subflavour == "source":
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+subflavourpath
else:
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+"/"+arch+subflavourpath# handling the newonly option of reposync

if options.newonly == True:
newonly = " -n"
else:
newonly = ""

# building the command for syncing a single repo
syncdistro="mirrorreleasever="+version+" mirrorbasearch="+arch+" /usr/bin/reposync"+newonly+" -r "+repoid+" -p "+targetdir

# for testing we just print instead of sync

if options.testonly == True:
print syncdistro
else:
# create potentially missing paths

try:
if os.path.isdir(targetdir) == False:
os.makedirs(targetdir)
except:
print "Error: Cannot create targetdirectory:", targetdir

os.system(syncdistro)

## main loop

for version in versionlist:
for flavour in flavourlist:
for arch in archlist:
for subflavour in subflavourlist:
# stuff that just doesent happen

if version == "development" and flavour == "updates":
continue
if version == "development" and flavour == "legacy":
continue
if version == "development" and flavour == "updates-testing":
continue
else:
syncMirror(version, flavour, arch, subflavour, targetlocation)

Creating Directories for Local Repository Path

Now start downloading all development repository to /mirror_repo by giving

python fedora-mirror.py

Once all core and extras development repository downloaded, Now move core and extras repository to parent directory /mirror_repo.

fedora-mirror.py will download core repository at

/mirror_repo/fedora/development/core/i386/mirror-core-development

fedora-mirror.py will download extras repository at

/mirror_repo/fedora/development/extras/i386/mirror-extras-development

Create following directories

/mirror_repo/fedora/development

/mirror_repo/fedora/development/i386/

/mirror_repo/fedora/development/i386/core

/mirror_repo/fedora/development/i386/extras

/mirror_repo/fedora/development/i386/groups

Setting up core repository directory

Then create clean i386 repository by doing following

mv /mirror_repo/fedora/development/core/i386/mirror-core-development/* /mirror_repo/fedora/development/i386/core

cd /mirror_repo/fedora/development/i386/core

createrepo .

Setting up extras repository directory

mv /mirror_repo/fedora/development/extras/i386/mirror-extras-development /mirror_repo/fedora/development/i386/extras

cd /mirror_repo/fedora/development/i386/extras

createrepo .

Setting up groups repository directory

cd /mirror_repo/fedora/development/i386/groups

create minimal.xml and add its contents as

<?xml version="1.0"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">

<comps>
<group>
<id>build</id>
<uservisible>true</uservisible>
<name>Minimal Install</name>
<packagelist>
<packagereq type="mandatory">bash</packagereq>
<packagereq type="mandatory">bzip2</packagereq>
<packagereq type="mandatory">coreutils</packagereq>
<packagereq type="mandatory">cpio</packagereq>
<packagereq type="mandatory">diffutils</packagereq>
<packagereq type="mandatory">fedora-release</packagereq>
<packagereq type="mandatory">gcc</packagereq>
<packagereq type="mandatory">gcc-c++</packagereq>
<packagereq type="mandatory">gzip</packagereq>
<packagereq type="mandatory">make</packagereq>
<packagereq type="mandatory">patch</packagereq>
<packagereq type="mandatory">perl</packagereq>
<packagereq type="mandatory">rpm-build</packagereq>
<packagereq type="mandatory">redhat-rpm-config</packagereq>
<packagereq type="mandatory">sed</packagereq>
<packagereq type="mandatory">tar</packagereq>
<packagereq type="mandatory">unzip</packagereq>
<packagereq type="mandatory">buildsys-macros</packagereq>
</packagelist>
</group>
</comps>

download buildsys-build and buildsys-macors rpms under /mirror_repo/fedora/development/i386/groups

wget http://buildsys.fedoraproject.org/buildgroups/development/i386/buildsys-build-0.5-1.noarch.rpm

wget http://buildsys.fedoraproject.org/buildgroups/development/i386/buildsys-build-0.5-3.fc6.noarch.rpm

wget http://buildsys.fedoraproject.org/buildgroups/development/i386/buildsys-macros-6-2.fc6.noarch.rpm

wget http://buildsys.fedoraproject.org/buildgroups/development/i386/buildsys-macros-6-2.fc6.src.rpm

Final contents of directory groups will look as

ls /mirror_repo/fedora/development/i386/groups

minimal.xml buildsys-macros-6-2.fc6.noarch.rpm buildsys-macros-6-2.fc6.src.rpm buildsys-build-0.5-1.noarch.rpm buildsys-build-0.5-3.fc6.noarch.rpm

now do createrepo -g minimal.xml . under /mirror_repo/fedora/development/i386/groups

Thus you will find all repository set under /mirror_repo/fedora/development/i386.

Mock Setup for Local Mirror

I have setup mock environment for development(FC6) version. My mock configuration for i386 is /etc/mock/fedora-devel-i386-core.cfg

import os
config_opts['root']  = 'fedora-development-i386-core'
config_opts['target_arch']  = 'i386'

config_opts['yum.conf']  = """
[main] 
cachedir=/var/cache/yum
debuglevel=1
reposdir=/dev/null
logfile=/var/log/yum.log
retries=20
obsoletes=1
gpgcheck=0
assumeyes=1

[core] 
name=core
baseurl=file:///mirror_repo/fedora/development/i386/core

[updates-released] 
name=updates
baseurl=file:///mirror_repo/fedora/development/i386/core

[groups] 
name=groups
baseurl=file:///mirror_repo/fedora/development/i386/groups

[extras] 
name=extras
baseurl=file:///mirror_repo/fedora/development/i386/extras

[local] 
name=local
baseurl=http://extras64.linux.duke.edu/plague-results/fedora-5-extras
enabled=0

"""

Building packages in Mock Environment

To build packages in mock you need to go through http://fedoraproject.org/wiki/PackageMaintainers/MockTricks

Troubleshooting for fedora-mirror.py

If you face any problems using fedora-mirror.py script then you can use /usr/bin/reposync command to download binary RPMS for extras repository.

Core repository can be setup by copying FC6 installation DVD/CDROMs to /mirror_repo/fedora/development/i386/core. Detailed Procedure to do this can be found at http://www.city-fan.org/tips/YumRepoFromImages

reposync command without any options will download extras repository. Just move all those RPMS to /mirror_repo/fedora/development/i386/extras and use createrepo . command

Thus still you can download and create local repository without fedora-mirror.py script.

Mock Setup Using Local Mirror For EPEL (taking RHEL6.x as a example)

Set Up Local Mirror For RHEL/Centos Development

RHEL yum online update is not free, i.e. we could not install any packges using yum in RHEL. So we need to setup other repos or using local mirror for repo.

Remove the orginal yum installation

# rpm -qa|grep yum|xargs rpm -e --nodeps
warning: /etc/yum.conf saved as /etc/yum.conf.rpmsave

Downloading Centos yum packages

python-iniparse：http://mirror.centos.org/centos-6/6.4/os/i386/Packages/python-iniparse-0.3.1-2.1.el6.noarch.rpm

yum-metadata-parser：http://mirror.centos.org/centos-6/6.4/os/i386/Packages/yum-metadata-parser-1.1.2-16.el6.i686.rpm

yum：http://mirror.centos.org/centos-6/6.4/os/i386/Packages/yum-3.2.29-40.el6.centos.noarch.rpm

yum-plugin-fastestmirror：http://mirror.centos.org/centos-6/6.4/os/i386/Packages/yum-plugin-fastestmirror-1.1.30-14.el6.noarch.rpm

RPM-GPG-KEY：http://mirror.centos.org/centos-6/6.4/os/i386/RPM-GPG-KEY-CentOS-6

# wget http://mirror.centos.org/centos-6/6.4/os/i386/Packages/python-iniparse-0.3.1-2.1.el6.noarch.rpm
# wget http://mirror.centos.org/centos-6/6.4/os/i386/Packages/yum-metadata-parser-1.1.2-16.el6.i686.rpm
# wget http://mirror.centos.org/centos-6/6.4/os/i386/Packages/yum-3.2.29-40.el6.centos.noarch.rpm
# wget http://mirror.centos.org/centos-6/6.4/os/i386/Packages/yum-plugin-fastestmirror-1.1.30-14.el6.noarch.rpm
# wget http://mirror.centos.org/centos-6/6.4/os/i386/RPM-GPG-KEY-CentOS-6

Install the downloaded yum packages

Note: Import RPM-GPG-KEY-CentOS-6 before yum install

     yum and yum-plugin-fastermirror need to be install together

# rpm -ivh python-iniparse-0.3.1-2.1.el6.noarch.rpm
# rpm --import RPM-GPG-KEY-CentOS-6
# rpm -ivh yum-metadata-parser-1.1.2-16.el6.i686.rpm
# rpm -ivh yum-3.2.29-40.el6.centos.noarch.rpm yum-plugin-fastestmirror-1.1.30-14.el6.noarch.rpm

Now, we can usring yum for online installation.

Setup Local Mirror Using ISO Image

Mount on the ISO Image

# mount -t iso9660 -o loop /mnt/RHEL-6.6-20140926.0-Workstation-i386-dvd1.iso /mnt/RHEL-6.6-Localrepo

Create repo file

# vi /etc/yum.repos.d/local.repo
-------------------------------------
[rhel6]
name=rhel6-local
baseurl=file:///mnt/RHEL-6.6-Localrepo
gpgcheck=1
gpgkey=file:///mnt/RHEL-6.6-Localrepo/RPM-GPG-KEY-redhat-release
enabled=1

You can use yum clean all and yum makecache to update the repos. Now the local repo is ready for use.

Mock Setup for Local Mirror for EPEL

My mock configuration file is "epel-6-i386".Make configuration as following:

# vi /etc/mock/epel-6-i386.cfg
-----------------------------------
config_opts['root'] = 'epel-6-i386'
config_opts['target_arch'] = 'i686'
config_opts['legal_host_arches'] = ('i386', 'i586', 'i686', 'x86_64')
config_opts['chroot_setup_cmd'] = 'groupinstall buildsys-build'
config_opts['dist'] = 'el6'  # only useful for --resultdir variable subst
...
...
...
[local]
name=local
baseurl=file:///mnt/RHEL-6.6-Localrepo
cost=2000
enabled=1
...
...

Now mock can using local mirror for packages installations.