Infrastructure/fedora-mirror

fedora-mirror.py

A tool for synchronizing a local mirror of fedora rpms.

Goals

• Being able to cleanly maintain a local mirror for fedora packages.
• Zero config as far as possible

Non-Goals

• This is not a replacement for rsync - it just makes it possible for people w/o access to an rsync server to sensibly synchronize to a local repository. If you have access to rsync servers, use them.

Status

• Using reposync as backend
• Basics are working

Todo

Code

• make everything configureable to the bone and internally use a nested dictionary. needs code addition to the main loop.
• ConfigParser
• logging
• legacy and updates-testing awareness
• fix hierachy into the fedora mirror dir scheme
• use separate configs for each release
• progressbars
• eta (complete, per repo, per file(?!))
• more exceptions

Support files/packaging/integration

• creation of specially crafted .repo files
• cron integration
• 0 conf apache serving as split package
• mock config files as split package
• yum localmirror config files as split package
• discuss access rights on /srv/mirror

Source Code

#!/usr/bin/python

<!--# License: GPL
-->
<!--# Author:  Rudolf Kastl
-->

<!--# default target: /srv/mirror
-->
<!--# how?: use config file and special mirror .repo files
-->

<!--# todo:
-->

<!--## code:
-->
<!--# - ConfigParser
-->
<!--# - logging
-->
<!--# - legacy and updates-testing awareness
-->
<!--# - fix hierachy into the fedora mirror dir scheme
-->
<!--# - use separate configs for each release
-->
<!--# - progressbars
-->
<!--# - eta (complete, per repo, per file(?!))
-->

<!--## support files/packaging/integration:
-->
<!--# - creation of specially crafted .repo files
-->
<!--# - cron integration
-->
<!--# - 0 conf apache serving as split package
-->
<!--# - mock config files as split package
-->
<!--# - yum localmirror config files as split package
-->
<!--# - discuss access rights on /srv/mirror
-->

<!--################
-->
<!--# special cases:
-->
<!--# - the fedora core repos differ in logic -> core repos have updates, updates-testing, legacy -> to be investigated
-->
<!--# - development is another special case not having updates updates-testing and legacy
-->
<!--# - other repos (3rd party) never have updates and updates-testing
-->
<!--# - SRPMS are another special case -> same for each arch of the same release
-->
<!--###########################
-->



import os

flavourlist=["core", "extras", "updates", "updates-testing", "legacy"] 
versionlist=["development", "5"] 
archlist=["i386", "x86_64", "ppc"] 
#targetlocation="/home/che/mirror"
subflavourlist=["debuginfo", "source"] 

<!--### block for parsing commandline switches
-->
from optparse import OptionParser

parser = OptionParser()
<!--# var can be accessed with options.filename
-->
parser.add_option("-d", "--dest", dest="destination", help="Sync Destination", metavar="dir", default="/srv/mirror")
parser.add_option("-n", "--newonly", action="store_true", dest="newonly", help="Sync Destination", default=False)
parser.add_option("-t", "--testonly" , action="store_true", dest="testonly", help="No Sync Transaction", default=False)
parser.add_option("-q", "--quiet", action="store_false", dest="verbose", default=True, help="don't print status msgs to stdout")

(options, args) = parser.parse_args()
<!--### end bloc
-->

targetlocation = options.destination

def syncMirror (version, flavour, arch, subflavour, targetlocation):
print "syncing: Fedora " + version, flavour, arch, subflavour, "to", targetlocation

<!--# handling the special cases
-->
if subflavour == "binary":
subflavourid = ""
subflavourpath = ""
elif subflavour == "source":
subflavourid = "-"+subflavour
subflavourpath = "/SRPMS"
else:
subflavourid = "-"+subflavour
subflavourpath = "-"+subflavour

<!--# devel is special
-->
if version == "development":
repoid="mirror-"+flavour+"-"+version+subflavourid
else:
repoid="mirror-"+flavour+subflavourid

if subflavour == "source":
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+subflavourpath
else:
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+"/"+arch+subflavourpath

<!--# handling the newonly option of reposync
-->
if options.newonly == True:
newonly = " -n"
else:
newonly = ""

<!--# building the command for syncing a single repo
-->
syncdistro="mirrorreleasever="+version+" mirrorbasearch="+arch+" /usr/bin/reposync"+newonly+" -r "+repoid+" -p "+targetdir

<!--# for testing we just print instead of sync
-->
if options.testonly == True:
print syncdistro
else:
<!--# create potentially missing paths
-->
try:
if os.path.isdir(targetdir) == False:
os.makedirs(targetdir)
except:
print "Error: Cannot create targetdirectory:", targetdir

os.system(syncdistro)

<!--## main loop
-->
for version in versionlist:
for flavour in flavourlist:
for arch in archlist:
for subflavour in subflavourlist:
<!--# stuff that just doesent happen
-->
if version == "development" and flavour == "updates":
continue
if version == "development" and flavour == "legacy":
continue
if version == "development" and flavour == "updates-testing":
continue
else:
syncMirror(version, flavour, arch, subflavour, targetlocation)