From Fedora Project Wiki

FedoraHostedv2

Hosted needs a bunch of work done to it to make it all happy for everyone.

Leading problems:

  • performance
  • scaling out
  • managing older projects
  • what interfaces we want to offer/maintain
  • automation

The goal right now is to support sharding out as much of the projects into smaller batches to make them manageable on smaller instances/vms and to make them easier maintain b/c of scale, more or less.

The general idea is that every project will be listed by:

$projectname.fedorahosted.org

This should point to a cname of the host where the project's dvcs and web app live (dvcs at minimum).

This should allow us to move around projects more or less at will.

Right now our 2-instance infrastructure using gluster is not performing very well and we really need to increase our horizontal scaling.

Short term plans

  • change everything to use $projectname.fedorahosted.org
    • cgit needs to reflect this
    • any web pages or wiki pages need updating.
    • project trac sites need updating
    • any place else?

* Enable trac-openid-plugin on all instances and retire auth_pg * Create 2 new instances at osuosl02 to migrate to. One live, one 'warm spare'. * Migrate all git projects to new instance, leave others in current site? ** would require changes to 'index' site

  • look at using rdiff-backup or similar to do more frequent backups of data.

Medium term / time permits plans

  • test scaling out some git projects to another instance
  • test gitolite

ideas container

  • setup fedorahosted.org's domain zone as a template and allow it to handle a map of project name to destination
  • data we need from each project:
    • name
    • contact/leader
    • scm(s) used
    • using trac?
    • fas group(s) related
    • last commit to dvcs
    • last ticket or wiki change
    • (we should have all this info in various places, just need to gather it together)
  • backups - how should we be backing up these smaller hosted instances
    • Data is trac, releases, git/scm.
    • rdiff-backup? rsync?
    • storage local to the instances would be much faster.
  • git:// proxy - we need to setup a test a way of proxying the connections to git://fedorahosted.org/git/project.git to git://project.fedorahosted.org/git/project.git
    • this may be gitpod or dulwich based but we have to do SOMETHING here
    • we may want a sunset on this after a year or something.
  • trac needs to be setup to use the authopenid auth mechanism by default for new instances
  • trace authopenid needs to be tested against older trac repos to see if it breaks ticket/wiki ownership
  • the /usr/bin/run-git user script used for folks connecting needs to be cleaned up and tested against projectname.fedorahosted.org urls
  • figure out how fasClient will need to be configured on each shard.
  • git.fedorahosted.org/cgit/ needs to have its index page populated and pointing to the right set of urls/repos for all projects
    • Or could we have project.fedorahosted.org/cgit just show that single projects scm?
    • we do likely need an index of all projects tho.
  • when making new projects, how do we figure out where something should go?
  • locating a project - assumedly just ssh'ing to it's projectname.fedorahosted.org hostname
  • document most/all of the above in new fedorahosted docs - both for infradocs but also for users
  • reskinning the trac main webpage
  • making the index of projects at fedorahosted.org a little more manageable and a lot more fedora-y
    • maybe searchable?
  • How do we make a new project - how 'self service' can we make it?
    • I think we can visit that down the road some, since it's not critical right now.
  • How to document and handle some optional web apps for projects
  • Enable way for projects to have html/web space. (existing ticket already on this)
  • reviewboard - what do we do with it and who maintains it? Where should it live? Which projects use it?
    • I think we can ask them to move to openshift instances.
    • sgallagh maintains it.
  • wildcard ssl cert
    • Being ordered.
  • the ssh key(s) we'll need for things (probably the same one for all of hosted, I'd bet)
  • eventually test out gitolite instead of locally created shell accounts for users
  • Is there a way to redirect users commiting via git+ssh to fedorahosted.org to projectname.fedorahosted.org
  • How do we handle outages if these pieces are all over the place?
  • managing user expectations?
  • what fedorahosted.org should look like and have on it.
  • when we're ready we need some way of redirecting people from: fedorahosted.org/projectname/somepage to projectname.fedorahosted.org/somepage/

things to think about to make the future more manageable

  • switching to gitolite instead of restricted shell accounts?
    • dealing with /releases/ uploads if we did
  • gitlab-public - talk to osas people?
  • supporting only git cloning via the 'smart' http protocol and via ssh?
  • supporting git:// urls only for our already existing projects and decomming the rest
  • stopping taking projects which are:
    • svn?
    • bzr?
    • hg?
    • or should we continue supporting the world?
Retrieved from "https://fedoraproject.org/w/index.php?title=Infrastructure_FedoraHosted_Version2_Notes&oldid=334327"