Friday, October 29, 2010

Upstart, a better init or a more painful one

For some time now Upstart has slowly been taking over the old SysV init scripts on most Linux distributions, for better or worse. Now some of the things they are trying to do with Upstart are good and have some merit, though at the cost of simplicity and ease of use. I fear the developers and supporters are worrying more about the 5% use case rather than the 95% use case and they have thrown the old mantra of keep it simple and stupid out the window. The problem when you give the 5% too much focus, you end up creating a solution that is horribly broken and resembles solutions from the little company up in Redmond Washington.

For those struggling to find your way through the new hell of Upstart here are a few pointers:

1. Dependency hell is brought to a whole new level of pain and frustration. Imagine a system where everything can emit and event and things “should” start or stop based on those events. Hey that is a great theory. But how on earth do you troubleshoot it. How do you get your system to start things in an exact order without building a chain like what we had back with SysV init scripts. In the old days of SysV, you gave all of your scripts a number, and they started in order from lowest to highest. If you wanted something to start sooner, you just lowered the number. Simple. Now you have to try and go through and figure out what starts when and what needs to happen before everything else. There is no tool that will print out a dependency map and there is no logging or debugging to speak of.

2. The location of Upstart configuration files can be found at /etc/init/. The old remaining SysV init scripts that can not yet be converted to Upstart can be found at /etc/init.d/.

3. The scripting support in the “script” blocks of Upstart is very limited and extremely basic. Basic things like echoing out data to the console does not work which makes it hard to try and use the scripts for anything other than shotgun starting a service or kill-9ing it.

4. The default answer to stopping a daemon in Upstart is to sent it a SIGTERM and wait a few seconds and if it is not stopped to send it a SIGKILL, a kill -9. Holy crap batman, there goes my database server that takes a while to shutdown cleanly.

5. You can not create any custom function blocks to pull extra status or details from a server that might support it. So your only option is to create a separate tool to manage the daemon. If you are going to do that, why not just stay with the SysV init scripts.

6. Most Linux Distributions are pushing this down our throats fast and hard, even when it is not fully baked or fully functional yet. This reminds me of the tactics that come from a little company in Redmond Washington.

7. Upstart can not manage all services yet, and some just do not start at all with Upstart. In the effort to build the most advanced leaky abstraction, some times a daemon will fork once or twice and Upstart can not track it and thus you get in to a state where Upstart “thinks” the service is stopped or started when it really has not. So imagine if that was your database server again, it would suck if the system went ahead and shutdown the filesystem even though the database server is not even close to shutting down.

I have spoken with the lead developer of Upstart several times. And he seems reasonable and desires to fix at least some of the problems and weaknesses in Upstart. However, I fear that there are some that he will not fix. Some of the things that we need it to do are not in the vision of it.

The things I would change about Upstart, sooner rather than later are:

1) Make the scripting engine inside the script blocks work with all script commands and be a fully functional scripting engine.

2) Provide a tool that prints out the exact order that things will start at boot with a parenthetical list next to each entry with its dependencies.

3) Provide the ability to add arbitrary functions/scripts to the Upstart configuration files so you can do something like “initctl apache super-details” and have it run a script block inside the configuration file called “super-details” which does what ever you need it to do.


I hope that Upstart and other proposed replacements to SysV init scripts can focus on making a better solution and not focus on trying to solve a 5% use case problem that causes pain and frustration for the other 95%.

No comments:

Post a Comment