From Fedora Project Wiki

Fedora New Init System

The following is a analysis of alternative init systems and why Fedora is planning on extending the existing system based on dbus.

Development

Development will take place in the initscripts CVS repository. 

$ unset CVS_RSH
$ export CVSROOT=:pserver:anonymous rhlinux.redhat.com:/usr/local/CVS
$ cvs -z3 login
$ cvs co initscripts

Existing Sys V based implementation

Please refer to any of the varied references on the net for System V init to get a general idea of how the current init system works.

The current initialization and bootup system has generated various complaints:

  • it's not very fast, or at least appears that way
  • it does not have full LSB support, or dependency support
  • there's not a good global view of what services that are configured for a particular runlevel are actually running
  • there's no mechanism for respawning of services except by init
  • there's no structured logging of serviced, and if they failed, etc

Over time, various alternative frameworks have popped up, including, but not limited to:

  • minit
  • runit
  • initng
  • serel

Looking at how these stack up:

  • Bootup speed

A lot of these claim "look how much faster we boot!". However, in most cases they miss the point. For example:

  1. The new-fangled init system has:
    1. exec nscd
  2. The current init system has:
    1. make sure nscd directories are in place
    2. check to make sure it's not already running
    3. set ulimits correctly
    4. set nice level
    5. setuid to a specified user, if necessary
    6. exec nscd
    7. log whether or not it succeeded

Telling me that A. is faster is meaningless, and, frankly, not useful. What's needed is to make B. faster. In some cases, this can mean eliminating some of those steps. But blindly eliminating all of them is silly.

The other aspect of bootup speed is parallelization. Initially, this seems like a big win. However, testing of simple naive implementations show that, at least initially, parallelization isn't a huge benefit. Generally, this is because disk seek time and other I/O limitations can dominate. Moreover, a not insignificant portion of boot time is the parts handled by /etc/rc.sysinit; this is almost entirely linear in nature (need to load modules first, then check filesystems, then clean out /tmp, etc.)

Currently, LSB support is wedged in via chkconfig. chkconfig can parse LSB standard headers for start and stop levels. For LSB dependencies, a conversion to the current Sxx/Kxx priorities is done at script installation time by computing a valid start priority relative to the dependencies specified.

This does have some problems: - priorities aren't recomputed if other dependencies are added/changed - adoption of other LSB features is slow

Moreover, very few of the current initscripts correspond to LSB standards such as for exit codes - in fact, they're very much ad-hoc in that regard.

  • No global view of state

Services can be queried individually for status, and there's a global view of what services are configured to start via system-config-services. There's no combination of this, however. (And, the implementation of system-config-services leaves much to be desired.)

  • No respawn mechanism

Some of the newer implementations actually do handle this.

  • No structured logging

Earlier (prior to FC4) initlog was used. This was a inefficient and badly coded (by me!) program that basically dumped a program's output to syslog if it failed, and otherwise logged a simple '<foo> succeeded'. However, this only happened on startup; there was no monitoring of services to see if they failed later, or respawning of services if they did fail. Moreover, since it was just logging program output, it didn't log in a way that could be easily monitored or picked out.

So, how to solve these problems?

Switch to alternative systems or extend the existing one. This would include, at a minimum, /etc/rc. It may also include /sbin/init, and could extend to crond, atd, xinetd, and other service frameworks as well.

Features required by this implementation:

  • Proper *runtime* dependency support. Starting a service should start its dependencies, in-order. Changes to dependent services should be immediately handled. Out of this work, parallelization can easily fall out. Without this work, parallelization is pointless.  :)
  • Full backwards compatibility. Old-style init scripts aren't going to go away anytime soon.
  • Full LSB support for LSB init scripts. Makes the spec groups happy, even though the number of LSB scripts is basically nil.
  • D-BUS support. Services should be exposed via D-BUS, and should be available for querying to:
  • check what's configured
  • check what's actually running
  • start them
  • stop them
  • etc.

The services should also post notifications via D-BUS when they've started successfully, and (more importantly) when they haven't, or when they've unexpectedly exited.

  • Support for respawning services, if necessary. Complete with rate limits and other fun stuff.

This should handle most of the concerns above. More speed hopefully will fall out of this, but if it doesn't, oh well.

Looking at these features, the best way to do this is almost certainly to add the D-BUS, etc support into the services themselves, and provide a wrapper for legacy LSB and other initscripts that provides the D-BUS interface to them.


Steps to solve these problems

  1. Clean up all initscripts
  2. Clean up /etc/init.d/functions
  3. Replace /etc/rc
  4. Make gdm/prefdm an initscript with early login
  5. D-Bus support

Backwards compatibility and choice of solution

After reviewing most of the projects mentioned below it became clear that the amount of work needed to put into them to solve the problems we're facing at the moment surpased the work we would need to put into the existing system to extend it to what we needed. So instead of jumping onto one of the new projects which, all in their own way make a lot of sense and solve different problems we decided to extend the current system following the KISS principle: Keep it Simple, Stupid. The major points are:

  • Backward compatible. You can install any 3rd party package and it will smoothly integrate into the extended framework we're working on and might even profit from it (see LSB conform initscripts).
  • Very simple. Expanding and modifying the existing system allows us to keep the changes small, simple and understandable while still providing a lot of flexibility and power (e.g. injected dependencies, which allow us to do some really neat tricks).
  • Fast. As the new framework builds on the old one but allows us to either start fewer services and/or start them in parallel we can be sure that the new system will be faster than what we had before.
  • Easy to extend. As the new framework is so simple, extending it becomes just as easy.
  • No massive changes. While some might argue this is a bad thing, it has it's positive sides: The old initscript framework worked very nicely and stable for the last 10-20 years. And instead of throwing it all away we're evolving it instead of doing a revolution.
  • Flexible. As pointed out already, as the new framework is really simple it still allows us to be very flexible in what we do.

Alternative options

Things to look at when implementing this:

  • initng

http://www.initng.org/

Package available in Fedora Extras. The package maintainer is fixing problems upstream as well. A GUI is also available as rpm , although not in any repository yet.

  • monit (GPL license)

http://www.tildeslash.com/monit/index.php

Looks interesting and has a compatible license. Someone more knowledgeable on the subject should investigate this - Rahul

  • SystemServices

http://www.gnome.org/~seth/blog/2003/Sep/27

Beat code out of Seth if necessary.  :) He's got the right idea, even if I quibble with some of his implementation. (Not sure that python is a good idea here, at least for parts of it.)

  • launchd (Apache license)

http://www.macgeekery.com/tips/all_about_launchd_items_and_how_to_make_one_yourself http://developer.apple.com/documentation/Darwin/Reference/ManPages/man8/launchd.8.html http://arstechnica.com/reviews/os/macosx-10.4.ars/5

  • Murdur - init script basically handling the rest of the scripts

http://www.pardus.org.tr/eng/projeler/comar/SpeedingUpLinuxWithPardus.html

One major Python script is called, this one starts the other, in Pyhton re-written scripts. The scripts are improved I/O wise (also with the advantages of Python vs shell script) but are also parallelized.

  • runit

http://smarden.org/runit/

Two modes of operation: run hand in hand with SysV init or replace init completely. Provides the tools that enable service supervision, parallel startup and per service logging. Documentation on the tools is detailed but implementation schemes are poorly documented. Can be compiled with dietlibc to produce very, very small executables with tiny memory footprints. The current init scripts can be used as runit scripts with trivial changes (using the services own/internal option to run in the foreground instead of forking into the background). Requires some upstream work to get (a few) service developers to provide the foreground option that does not produce extra debug output. runit extends the ideas developed in deamontools (and monit?). Compiles and runs on Linux, *BSD, MacOSX and Solaris. runit has been kicking around since 2000 and is quite stable and mature. It is used as the default init scheme on two distros: Annvix and Zinix. Most importantly it meets close to 80% of the requirements listed above and can do 100% with a bit of glue logic layered around it (especially with regards to dependancies and oneshot (non-daemon) scripts). Initial testing shows a 50%-60% boot speedup on average running everything installed on a default Fedora 8 DVD install (everything selected). Oddly enough runit can also be used with Upstart as well as Init.

  • (notting) Have tested runit in the past. It marked dependencies as 'satisfied' whenever it started to run a script, not when it completed successfully. Ergo, not very useful.
  • (RehanKhan)Still I believe that was the fault of the script, not of the runit tools itself. I have posted a 'proof of concept' here --> http://fedorafastboot.wiki.sourceforge.net. It is a pre-liminary attempt to meet the goals described above but it does produce a marked reduction in boot times (perhaps not as much as initng but it isn't a re-write of the init process) whilst maintaing the depth, breadth and investment in current init scripts. The wheel doesn't have to be invented again. It could be a reasonable interim solution whilst other projects mature.
  • (notting) No, that is incorrect. If runit is saying 'service "blah"' is available for dependencies *as soon as it runs exec() on whatever* (be it a daemon, or a network script), that is not the fault of the script. That is a fundamental design flaw. If runit has changed to actually implement sane behavior since I last looked at it, that's a different story.
  • (RehanKhan) I agree with you. It was an issue when I was converting haldaemon to run as a supervised service using runit. Haldaemon takes a few seconds to get initialised but runit comes back saying it is available. However runit is only saying the services' process exists (runit's job is done). It would require deep knowledge of the service itself for runit to say that the service the process provides is available (an open socket on the dbus) which is unrealistic for a generalised service management framework. runit does provide for this situation however. Using a 'check' script the services' developer (who does have an intimate knowledge of the service) can implement a check tailored to the service. For haldaemon I look for the socket file and wait until it is available before reporting the service is ready. Using this and other refinements one has a good chance of being able to manage any service. Please take a look at the link above, the 'glue' has been applied to runit and it meets nearly all the criteria (just no dbus but that is also discussed). The documentation in the project wiki covers this and the other issues that came up when I converted more than 80 init scripts of the 99 in my init directory. Converting init scripts is also very easy as the runit script does not differ much from the original init script (which was an issue above). My laptop runs EVERY persistent boot process as a supervised, automatically restartable service (even prefdm/gnome/kde) with optional per-service logging. Init only runs the 3 or 4 init scripts that do not run persistent processes (smolt, firstboot and the like.). With very little extra work this will work on RHEL as well.
  • (notting) Aside: discussions via wiki are silly. That being said... it is still a fundamental flaw in the process. Saying that 'just becuase the process exists', that it's ready is broken. Wait for the daemon to return. Or the script that you launch to exit. Moreover, about your other claims - if you have to modify and convert 80 scripts, there is no way that that is an *interim* solution. We only want to change things once. Also, your link is broken.
  • (RehanKhan) lol, wiki conversations are kinda weird. The link seems to work in (dare I say it?) IE7 and firefox 2. Just search for fedorafastboot on sourceforge. By thier very nature daemons don't return. They start and should just carry on. In a supervised environment they should just run in the foreground like any other app. All the pid finding and tracking code becomes un-neccessary (and thankfully goes away). The init scripts need to be modified because you don't have to do all that backgrounding (daemon function), pid tracking and return code processing (which is not usually the return code of the service but the daemon function). If a supervised process returns it has stopped working. In the init environment you do get return codes from the service because they fork (and they are saying they have forked successfully not that they are ready) and the original process returns (this is bad for supervision). Runit never says the service is ready, that is up to the service because only the service knows when it is ready. It just says it has launched a process, is tracking it and will restart it if it fails. For example haldeamon should jump on dbus right after it is exec'ed and tell requesting processes that it is up but busy for the moment and that they should try again in a while not just panic like crazy and exit (e.g. bluetooth and avahi and wpa supplicant). This would fix the panel apps as well. No generalised service supervision scheme can know to do that. Have a look at the project and maybe post on the forums and we can talk about it further. Look at the screenshots page and then the bootcharts in the wiki (on the 'Fastest start to the desktop' page).
  • (notting) Again, you're saying that the fact that runit doesn't work without modifying every service isn't a 'runit' problem. I don't believe that. Also, you discuss 1) having to change daemons to support runit 2) that the way you achieved greatest startup speed was by not starting critical desktop services, which ends up meaning very little. I fail to see anything in runit that makes it seem like a good idea compared to other upstream solutions. Sorry.
  • (RehanKhan) To resolve these issues every init script has to be visited anyway. From what I see, every init script has been visited and changed since FC6. No service itself has to be modified. With the exception of 1 out of the 80 i've looked at, every daemon application supports running in the foreground. They do need to start a certain way to get service supervision (highly desirable). To get fast start init is not the problem. Nash, X, I/O and fragmentation need to be resolved (and are being elsewhere). When I drop some services I am squeezing an improvement of an extra 2 - 3 seconds. Nash, udev-start and fsck still take 25 seconds. These are universal issues for any init system. What I am saying is that a runit based system can check off virtually every issue on this page and is here today, was here yesterday, will be here tomorrow. I have gone from a standard F8 system to a full service supervision, parallel startup, per-service logged system with all the bells and whistles of full init scripts in three weeks (even for software I don't use). No sofware has been changed from the standard F8 rpm's except for the init script. Init is still around so software updates don't mess anything up and as a bonus it starts faster than init for the same services (including X and gnome). This page has been up for around two and a half years and runit could have solved this problem then. It would be unfortunate (but not catastophic) to overlook runit again.


Things we'd like to look at, but can't (GPL License incompatability) :

  • Solaris SMF (planned - Sun CDDL License)

http://docs.sun.com/app/docs/doc/817-1985/6mhm8o5n0?a=view http://www.filibeto.org/aduritz/truetrue/solaris10/sol-smf.pdf http://mediacast.sun.com/share/lianep/t-smf-general-march-2005.pdf

You can look at the docs. You can probably even play with a OS X and Solaris 10 implementation.

You can not look at the code.

Required languages of implementation: C, potentially a little python

Not much flexibility here. Obviously, the lower level in the bootup process you get, the more you're confined to C.

See also

[[Navigation(children)]

Retrieved from "https://fedoraproject.org/w/index.php?title=FCNewInit&oldid=32455"