|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| {{DISPLAYTITLE:Modularity/Developer Notes}}
| | {{admon/important|This page is deprecated| All Fedora Modularity Documentation has moved to the new [https://docs.pagure.org/modularity/ Fedora Modularity Documentation website] with source hosted along side the code in the [https://pagure.io/modularity Fedora Modularity website git repository]}} |
| <div style="float: right;" class="toclimit-2">__TOC__</div>
| |
| | |
| == Abstract ==
| |
| | |
| The purpose of this document is to describe systems and services used for generic [[Modularity]] development, research and possibly future infrastructure deployments, and is intended as a reference guide for the involved engineers and the so-called ''doers-of-things''.
| |
| | |
| If you're new to Modularity, start with the following instead:
| |
| | |
| * [[Modularity]]
| |
| * [[Modularity/Getting_involved]]
| |
| * [[Modularity/Infra]]
| |
| | |
| == Systems ==
| |
| | |
| We use a number of systems for Modularity work. Some of those are dedicated installations for our cause, some are generic and shared with other Fedora initiatives. This section focuses on the former.
| |
| | |
| === dev.fed-mod.org ===
| |
| | |
| The main and currently the only dedicated system we have.
| |
| | |
| This is an OpenStack instance running Fedora. In case the <code>dev.fed-mod.org</code> domain name no longer works, the IPv4 address is <code>209.132.184.168</code>. We use a shared user account named <code>fedora</code>. If you need access, contact any of the current engineers and provide them with your public SSH key.
| |
| | |
| ==== Automatic COPR rebuilds ====
| |
| | |
| We run automatic [https://copr.fedorainfracloud.org/groups/g/modularity/coprs/ COPR rebuilds] of certain modularity projects (fm, modulemd and modulemd-resolver) with a cron job every 15 minutes. Edit <code>~fedora/rebuild_packages.sh</code> to add yours.
| |
| | |
| ==== Automatic documentation rebuilds ====
| |
| | |
| We also run automatic [http://readthedocs.org/ readthedocs.org] documentation builds every 15 minutes for fm and modulemd with simple cron jobs.
| |
| | |
| ==== Metadata service ====
| |
| | |
| The metadata-service is deployed on this system and handles all <code>^/fm/(.+)</code> HTTP requests. This API isn't really defined yet. Browse the project's sources to see how it works.
| |
| | |
| ==== Status reports and agile tools ====
| |
| | |
| Our [http://www.fed-mod.org/taiga-report/ Taiga status reports] are also hosted on this sytem and regenerated every 15 minutes, again, with a cron job.
| |
| | |
| Other agile tools such as the Modularity Bot or sprint-tools for Trello/Taiga synchronization are also hosted here. See <code>~fedora/fm-trello-taiga-sync</code>, <code>~fedora/sprint_tools</code> and the various related cron jobs.
| |
| | |
| ==== Webhosting in general ====
| |
| | |
| The host is running a generic httpd webserver. The configuration is kept in <code>/etc/httpd/conf.d/fm.conf</code> and we use the default web root for our stuff, <code>/var/www/html</code>. For example, the experimental [http://fed-mod.org/dev/modules/ modules repository] is hosted there.
| |
| | |
| Feel free to use this for whatever you need. However, keep in mind the available disk space on this machine is fairly limited.
| |
| | |
| == Services ==
| |
| | |
| The high-level purpose of the majority of the services listed below is described in the [[Modularity/Infra]] document.
| |
| | |
| ==== The input ====
| |
| | |
| Module input data consists of two main parts — the [https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules module definition file in the modulemd format] and the components, such as RPMs (coincidentally the only format we currently support but expect that to change at some point in the future). In Fedora, both the modulemd files and the components' SPEC files are stored in dist-git and the associated ACLs are stored in pkgdb (Package Database).
| |
| | |
| We use namespaces to distinguish between modules and RPMs in those two systems, aptly named <code>modules</code> and <code>rpms</code>.
| |
| | |
| {{admon/note|How To Build A Module In Staging|A more up to date workflow is [[Modularity/HowToBuildAModuleInStaging|available]]}} | |
| | |
| ===== dist-git =====
| |
| | |
| For development and testing purposes we use the [http://pkgs.stg.fedoraproject.org/cgit staging dist-git instance] which supports the above mentioned namespaces. The recommended way to interact with dist-git is using the <code>fedpkg</code> tool and configuring it to interact with this instance.
| |
| | |
| Once you have installed fedpkg, edit <code>/etc/rpkg/fedpkg.conf</code> (or wherever your <code>fedpkg.conf</code> is located) to change all occurrences of the default <code>pkgs.fedoraproject.org</code> to <code>pkgs.stg.fedoraproject.org</code>.
| |
| | |
| Here is an example:
| |
| | |
| Install pag via <code>$ sudo dnf install -y pag</code>. Once you have done this, download and set up the custom rpkg repo:
| |
| <pre>
| |
| $ cd /tmp
| |
| $ pag clone karsten/rpkg
| |
| $ cd rpkg/src
| |
| $ git checkout origin/modularity_new
| |
| </pre>
| |
| | |
| Next, set fpkg to point to stg/rida. To do this, add <code>fedpkg-stage.conf</code> to <code>/tmp/rpkg</code> and input the following content:
| |
| <pre>
| |
| [fedpkg]
| |
| lookaside = http://pkgs.stg.fedoraproject.org/repo/pkgs
| |
| lookasidehash = md5
| |
| lookaside_cgi = https://pkgs.stg.fedoraproject.org/repo/pkgs/upload.cgi
| |
| gitbaseurl = ssh://%(user)s@pkgs.stg.fedoraproject.org/%(module)s
| |
| anongiturl = git://pkgs.stg.fedoraproject.org/%(module)s
| |
| tracbaseurl = https://%(user)s:%(password)s@fedorahosted.org/rel-eng/login/xmlrpc
| |
| branchre = f\d$|f\d\d$|el\d$|olpc\d$|master$
| |
| kojiconfig = /etc/koji-stage.conf
| |
| build_client = koji
| |
| clone_config =
| |
| bz.default-tracker bugzilla.redhat.com
| |
| bz.default-product Fedora
| |
| bz.default-version rawhide
| |
| bz.default-component %(module)s
| |
| sendemail.to %(module)s-owner@fedoraproject.org
| |
| distgit_namespaced = True
| |
| ridaurl = https://dev.fed-mod.org:5000/rida
| |
| </pre>
| |
| | |
| Next, get the sources:
| |
| <pre>
| |
| $ fedpkg --config /etc/rpkg/fedpkg-stage.conf co modules/testmodule --anonymous
| |
| </pre>
| |
| | |
| Here, <code>--anonymous</code> is used in case you are not a packager on stg envt.
| |
| | |
| Now build:
| |
| <pre>
| |
| $ cd testmodule
| |
| $ fedpkg --config /etc/rpkg/fedpkg-stage.conf module-build
| |
| </pre>
| |
| | |
| | |
| Next, review the [http://dev.fed-mod.org/ BPO overview], then check [http://dev.fed-mod.org/modularity/logs/logs.txt the logs] (in case that nothing happens). All repositories dumped by [http://dev.fed-mod.org/modularity/repos/ pungi-signed-repo]
| |
| | |
| ===== pkgdb =====
| |
| | |
| A [https://admin.stg.fedoraproject.org/pkgdb/ staging pkgdb instance] is also available, storing the modules' ACL entries.
| |
| | |
| Contact people with the admin ACL privileges for the given module for commit access. Contact [[User:Ralph]] if you need a new module pkgdb entry & dist-git repository.
| |
| | |
| ==== The message bus ====
| |
| | |
| We expect to use [http://www.fedmsg.com/en/latest/ fedmsg] extensively for inter-component communication in all stages of module build, testing and distribution.
| |
| | |
| However, since we don't do any of those things yet, there's not much to say about this. Read the upstream documentation to see what Fedora Messaging is about.
| |
| | |
| ==== The module builder ====
| |
| | |
| The module builder consists of three main components: the build orchestrator, the koji build system and the pungi compose tool. The basic, overly simplified idea of building modules is:
| |
| | |
| # The client (e.g. the module packager) contacts the orchestrator and requests a build.
| |
| # The orchestrator does all the heavy lifting -- prepares the buildroot as a koji target, clones and builds all the components in koji in the correct order, run CI checks for components and modules, tracks the build states and rebuilds all dependant modules, if required.
| |
| # After every module build, the orchestrator notifies pungi which in turn creates module deliverables.
| |
| | |
| There are still many open questions regarding this process. We will fine-tune the details on the go.
| |
| | |
| ===== orchestrator =====
| |
| | |
| The orchestrator is a service with a publicly available interface that the clients can interact with, for example by issuing <code>fedpkg module-build</code> as a module packager, that ''orchestrates'' the complete build of modules as noted above. Note the orchestrator doesn't yet exist and although we don't have any specific design in mind, we expect to have ''something'' ready in the near future.
| |
| | |
| The orchestrator will emit fedmsg messages to interact with other infrastructure components. It will work with PDC to both store (via pdc-updater) and retrieve module dependency graphs. It will also require its own database to track module build states. The public interface will ''not'' be an XMLRPC.
| |
| | |
| And the service will most likely be hosted on dev.fed-mod.org.
| |
| | |
| ===== koji =====
| |
| | |
| The module RPM content will be built in koji. The current idea is to use koji tags and targets to represent modules and tag inheritance to define buildroots for RPM components. The orchestrator needs to be able to manage these.
| |
| | |
| We expect to use the [http://koji.stg.fedoraproject.org/koji/ staging koji instance] once we have a usable and somewhat stable orchestrator. Until then, and to allow more flexibility when developing against koji, we also have our own Fedora 24-based koji virtual machines you can play with. They're too large to be shared on dev.fed-mod.org. Contact [[User:Psabata]] if you're interested in getting them.
| |
| | |
| ===== pungi =====
| |
| | |
| Once all the components in the module are built, the orchestrator will signal pungi to create deliverables, such as RPM repositories or container images, from the respective koji tags. pungi will also store compose information in PDC (again, via pdc-updater), push the deliverables to mirrors (either directly or via an update system) and interact with various other currently nonexistent RCM tools.
| |
| | |
| We're considering putting all or most of this functionality directly into koji.
| |
| | |
| ==== The module knower ====
| |
| | |
| We would like to store certain practical bits about modules in PDC (Product Definition Center) so that it can be easily viewed and queried by both humans and other infrastructure tools. Specifically, we're interested in two kinds of information:
| |
| | |
| # Source inter-modular dependencies — useful for tracking what modules need to be rebuilt
| |
| # Compose information — for listing contents of deliverables
| |
| | |
| The first use case requires a new data type to be created in PDC.
| |
| | |
| ===== PDC =====
| |
| | |
| There's a [https://pdc.stg.fedoraproject.org/ staging PDC instance] available we hope to use later in the development cycle.
| |
| | |
| We might set up our own instance on dev.fed-mod.org, if necessary.
| |
| | |
| ===== pdc-updater =====
| |
| | |
| pdc-updater is a simple, stateless service that responds to fedmsg events and populates the PDC database. It already exists but needs some patching to support the new data type we require.
| |
| | |
| It is unclear whether a staging pdc-updater is available. We might as well deploy our own on dev.fed-mod.org.
| |
| | |
| ==== The compose magic ====
| |
| | |
| Once built, modules could be composed into products such as ''Fedora Workstation'', ''Fedora Server'' or maybe even anything the user defines either for integration QA purposes or building custom-tailored system images for and by the end users. See the [[Modularity/Infra]] document for more information about these concepts.
| |
| | |
| ===== CaaS =====
| |
| | |
| CaaS is not yet properly designed or implemented. Therefore we haven't thought about deployment either.
| |
| | |
| ===== Pixie dust =====
| |
| | |
| The same for the so-called ''Pixie dust''.
| |
| | |
| ==== The update system and distribution ====
| |
| | |
| Both modules and module composes will need to be held by some unspecified service before they get pushed to the master mirror. This is necessary for two reasons:
| |
| | |
| # Human testing — verifying the builds and composes work fine beyond the capabilities of our CI
| |
| # Pushing mass rebuild results as one update — simply to avoid broken states on the end users' systems
| |
| | |
| This might be implemented by patching the current Fedora update system and/or introducing yet another service/layer. We could use [https://bodhi.stg.fedoraproject.org/ the staging bodhi instance] for development. Note we have no staging mirrors or systems capable of serving this kind of content at the moment.
| |
| | |
| ==== Miscellanous ====
| |
| | |
| We also use a number of other systems and services.
| |
| | |
| ===== pagure.io =====
| |
| | |
| All of our own code is hosted at [https://pagure.io/ pagure.io]. Many of the projects grant commit access to the [https://pagure.io/group/modularity @modularity group]. If you'd like to a part of it, just ask any of the current members to add you.
| |
| | |
| ===== COPR =====
| |
| | |
| There's a [https://copr.fedorainfracloud.org/groups/g/modularity/coprs/ @modularity project space] on COPR where we build & share our tools such as fm, modulemd or modulemd-resolver. All [https://admin.fedoraproject.org/accounts/group/members/modularity-wg/* modularity-wg FAS group members] should have permissions to create new projects there. Contact any modularity-wg sponsor or administrator to join the group.
| |
| | |
| ===== Jenkins =====
| |
| | |
| We have [http://jenkins.fedorainfracloud.org/job/fm-modulemd/search/?q=fm a number of Fedora Infra Jenkins jobs] set up for the CI of our tooling. Contact [[User:James]] if you'd like something added there.
| |
| | |
| [[Category:Modularity]]
| |
| [[Category:Modularization]]
| |
| | |
| ===== GitHub =====
| |
| | |
| In order to collect our changes and submit them as pull requests, we have our own forks for some projects which are hosted on GitHub. For grooming prior to submitting things upstream (e.g. reviewing and merging pull requests from individual contributors against our forks), contributors are added to the [https://github.com/orgs/fedora-modularity/teams/committers <code>committers</code>] team in the [https://github.com/fedora-modularity <code>fedora-modularity</code>] organization. Please ping [mailto:nils@redhat.com Nils Philippsen] ([https://github.com/nphilipp github]) to get yourself added.
| |
| | |
| Our forks:
| |
| | |
| * [https://github.com/fedora-modularity/product-definition-center Product Definition Center] ([https://github.com/product-definition-center/product-definition-center upstream])
| |
| * [https://github.com/fedora-modularity/pdc-updater pdc-updater] ([https://github.com/fedora-infra/pdc-updater upstream])
| |
