Infrastructure FedoraHosted Version2 Notes

FedoraHostedv2

Hosted needs a bunch of work done to it to make it all happy for everyone.

Leading problems:

• performance
• scaling out
• managing older projects
• what interfaces we want to offer/maintain
• automation

The goal right now is to support sharding out as much of the projects into smaller batches to make them manageable on smaller instances/vms and to make them easier maintain b/c of scale, more or less.

The general idea is that every project will be listed by:

$projectname.fedorahosted.org

This should point to a cname of the host where the project's dvcs and web app live (dvcs at minimum).

This should allow us to move around projects more or less at will.

Right now our 2-instance infrastructure using gluster is not performing very well and we really need to increase our horizontal scaling.

Here are a number of items we need in order to make this happen

• setup fedorahosted.org's domain zone as a template and allow it to handle a map of project name to destination
• data we need from each project:
  • name
  • contact/leader
  • scm(s) used
  • using trac?
  • fas group(s) related
  • last commit to dvcs
  • last ticket or wiki change
• backups - how should we be backing up these smaller hosted instances
• git:// proxy - we need to setup a test a way of proxying the connections to git://fedorahosted.org/git/project.git to git://project.fedorahosted.org/git/project.git
  • this may be gitpod or dulwich based but we have to do SOMETHING here
• trac needs to be setup to use the authopenid auth mechanism by default for new instances
• trace authopenid needs to be tested against older trac repos to see if it breaks ticket/wiki ownership
• the /usr/bin/run-git user script used for folks connecting needs to be cleaned up and tested against projectname.fedorahosted.org urls
• git.fedorahosted.org/cgit/ needs to have its index page populated and pointing to the right set of urls/repos for all projects
• when making new projects, how do we figure out where something should go?
• locating a project - assumedly just ssh'ing to it's projectname.fedorahosted.org hostname
• document most/all of the above in new fedorahosted docs - both for infradocs but also for users
• reskinning the trac main webpage
• making the index of projects at fedorahosted.org a little more manageable and a lot more fedora-y
  • maybe searchable?
• How do we make a new project - how 'self service' can we make it?
• How to document and handle some optional web apps for projects
• reviewboard - what do we do with it and who maintains it? Where should it live? Which projects use it?
• wildcard ssl cert
• the ssh key(s) we'll need for things (probably the same one for all of hosted, I'd bet)
• eventually test out gitolite instead of locally created shell accounts for users
• Is there a way to redirect users commiting via git+ssh to fedorahosted.org to projectname.fedorahosted.org
• How do we handle outages if these pieces are all over the place?
• managing user expectations?
• what fedorahosted.org should look like and have on it.
• when we're ready we need some way of redirecting people from: fedorahosted.org/projectname/somepage to projectname.fedorahosted.org/somepage/

things to think about to make the future more manageable

• switching to gitolite instead of restricted shell accounts?
  • dealing with /releases/ uploads if we did
• gitlab-public - talk to osas people?
• supporting only git cloning via the 'smart' http protocol and via ssh?
• supporting git:// urls only for our already existing projects and decomming the rest
• stopping taking projects which are:
  • svn?
  • bzr?
  • hg?
  • or should we continue supporting the world?