bangmoney.org

Over the past two weeks, I've been working on figuring out where all the memory goes during installation. As many people have no doubt noticed, we've increased the memory requirements a good bit for F15. This is mostly due to my single large initrd change. wwoods is working on further refinements for F16 that should lower that requirement by a good bit. But, that alone doesn't tell the story of install-time memory usage.

The first thing I needed was data. So, I wrote a little program to poll /proc/meminfo and record statistics, as well as check which programs are using the most memory. A little experimenting showed my initial frequency of once every ten seconds was not at all fine-grained enough, so I cranked it up to once per second. My graphs were still showing some bizarre behavior towards the end, so I bumped memory available to the VM way up and got rid of swap to remove a variable.

All those changes got me a graph that I was happy with. So then I set out to perform a full install and get good data. I tested a VM with 2 GB of memory, installing from a DVD, doing media check, changing language to German in the GUI, and doing the default package set. Here is my graph, annotated with times at the obvious spikes:



Apologies for the tiny graph thumbnail there. Please just open it in a new tab to see the enormous one.

The A and B spikes are due to anaconda running localedef during installation. Turns out it's much smaller to ship the source files for locale data and build them at install time than it is to ship the compiled locales. Spike A is us doing localedef during loader for English, which seems unnecessary. Spike B is because I selected German later on in the GUI.

Spikes C through H are various packages running update-mime-database as part of their %post script. These aren't big spikes, but they're noticeable enough in an otherwise boring stretch. It seems to me like this could be run once as part of something's %posttrans script to save a bit of time and effort.

The enormous spike is I, followed by J. These are both from the selinux-policy-targeted package's %post running semodule. I have some data on this by running the script through valgrind, but that's a topic for another day.

Spike K is firefox (or something else, but firefox certainly does this) running gtk-update-icon-cache from a %posttrans script, and spike L is the kernel running depmod out of its %posttrans as well. While both pale in comparison to what's happening earlier, it seems like both are using an awful lot of memory for what should be simple tasks. Some investigation here is warranted.

Two things are unlabeled. First, the graph starts out with a baseline usage of somewhere around 175 MB. This is due to the giant initrd being loaded into memory. Doing so sets a pretty high base, but like I said earlier, it's being worked on. Second, there's a large climb starting at 20:47:00. This is repo setup and dependency resolution. As you can see, the memory consumed there is never freed so the big selinux-policy-targeted spike is made even worse. This is either anaconda or yum doing something goofy.

Update: At least part of this is rpm - not yum. After some further tracing, I determined approximately 100 MB of memory usage comes from the populateTs call in anaconda's yuminstall.py. In yum, that doesn't do anything particularly interesting so I dug down into the call into rpm. It looks like the memory is getting consumed by storing all the provides and file lists from each package in memory. I bet this portion of the graph would change with the number of packages to install. I don't yet see a way around this.

Also unlabeled is the fact that everything between about 20:47:00 and 21:03:00 is out of anaconda's control. This time is repo setup and depsolving (which is largely yum) and then package installation (which is whatever crazy things each individual package wants to do).

So, where do we go from here? First, decreasing baseline memory usage will do a lot of good. Second, figuring out what can be done in semodule is critical. Until we figure that out, there's not much point in doing anything else in anaconda. That is far and away our biggest driver of memory consumption and it's not even under our control. If you have swap configured, it'll be getting hit like crazy at this point. Third, we need to figure out if depsolving memory can be freed at any point. That would also help lower the severity of spike I.

I suppose it's also worth testing a minimal install and an everything install to see what other crazy packages there are. It's also worth me getting this patch committed to anaconda so we can occasionally re-run and make sure we haven't introduced any new problems. At the least, seeing what happens after we're no longer using the giant initrd will be interesting.

This past weekend was FUDCon and for the first time since expanding to other offices, we had the entire crew in one place. That includes our remotees in Washington, Alabama, and Hawaii; people in other US offices, and the whole group from Brno (which is a very long flight). This was very nice as many of us had never met others before. In particular, I'd never met bcl, mgracik, or akozumpl before this weekend.

Saturday, we gave two anaconda-related presentations. The first was simply a session to meet the team and talk about what we'd done recently:

• I mentioned the storage test stuff (which is in mainline autoqa right now and will be running as soon as they rebuild).


• dlehman talked about his image install support that was mentioned here a couple weeks ago.


• mgracik talked about his lorax project, which completely replaces the way we build installation images. Along the way, someone also suggested lorax could replace livecd-creator or perhaps some of the appliance creator tools. This is pretty brilliant, given that all lorax does is create trimmed down images based on a config file. I'm excited to see some experimentation here.


• akozumpl talked about making the anaconda UI run in a separate thread, an idea he previously mailed out. So far there's been no responses to his mail, but I think that's all because we've been stunned into silence. Personally, I think there's some good to come out of doing this (like having a more responsive UI) but threading is always a little scary.


• Speaking of UI, wwoods talked about moving from pygtk to pygi, adding more glade into anaconda, and so forth. Expect to see a whole lot more UI talk in the coming year - we've ignored this part of anaconda for far too long.


• rvykydal talked about all his recent networking changes.


• Speaking of livecd, bcl talked about his desire to merge livecd-creator and image install support.


• We did not get to pjones, msivak, or dcantrell. Sorry.

Mo posted a more thorough description of all this, complete with picture.

Sunday we skipped the lightning talks and beginning of the hackfest to have an anaconda feature planning throw down. The end result was that every feature on the feature page now has a target release, owner, and list of associated components. Well, okay, not every feature has a target release. Some we pushed off because they're either too hard or not well enough understood yet. Overall, it's a pretty ambitious list, touching almost every area of anaconda (though there's almost nothing for storage which I suppose is a testament to the quality of the rewrite). I'll be surprised if we get half these features done, but even if that's all we do it will still be serious work.

As mentioned before, it looks like the UI is going to be the biggest piece of work in the next year or two. Will already has some interesting ideas for completely changing anaconda away from a wizard style, but I'll let him expand on that when he's ready. There's also all the UI stuff that Mo has previously talked about and mocked up. For whatever reason, we tend to gravitate towards the backend projects and neglect the UI which means we end up looking very dated. I'm hoping we will take care of that this time around.

Other features that generated a lot of discussion were adding wireless support (do we need to make our own NM-aware UI? run a panel? what?), replacing our own custom init with systemd, obsoleting the loader, and replacing package selection with the spin selector.

Monday, we again had our own big anaconda hackfest which was very informal. The biggest thing to come out of that was a lot of discussion about wwoods's proposed UI, as well as akozumpl's ideas about replacing dispatch and threading the UI. I'm especially interested in the dispatch work. The dispatcher is a block of code that's basically a very fancy layer around a basic list, controlling which steps anaconda will run. We spread step adding/skipping all over the place in anaconda so it's very hard to follow. msivak had a great idea about treating the dispatcher as a queue where each step simply pushes in what to run later. This would mean you're never really moving back and forth through anaconda, which would seriously clean up our code.

So as you can see, this was a very productive FUDCon for us. We didn't write any code, but we got a huge number of ideas flowing . It looks to be yet another very busy year in anaconda. And you thought system installation was all figured out.

No news is good news

I don't really have anything to talk about for this week. Since it's almost Christmas break here, everyone's been taking it pretty easy. For those of you who don't know, much of Red Hat will be shut down starting tonight through New Year's. Given that, there's not really any reason in starting anything new this week only to put it down for ten days and then try to remember what you were doing. I did a new build yesterday and since only translations have changed since, that will be the last build until 2011. Please keep all this in mind if you file bugs or make mailing list posts. Someone might get back to you before January, but don't count on it. Have a good break, and remember to step away from the computer for a while. I know I will.

Creating Images

Before you can run the installer, you need to create some boot media that contains the installer. The boot media contains the installer which is written in Python, plus a whole lot of Python libraries, a pile of storage utilities, a bunch of base system utilities people like to have for their kickstart scripts, a lot of libraries that all these things require, and a ton of data files like translations and timezone data. Overall, creating the installation environment is a pretty slow and involved process understood only by a few. For a very long time, that process has lived in anaconda. Now, however, we are about ready to break it out into its own project. Why would we do such a thing?

• anaconda's image building scripts contain whitelists of every file that should be included in the image. There are so many ways this can go wrong. Rebuilding a package and moving its data files from one location to another might mean they're no longer included which could be a fatal problem. Just renaming a file could result in the same thing. This used to hit us all the time when libraries updated, but we are less susceptible to library moves anymore. Still, the whitelist is not at all foolproof.


• Adding or removing something from the image means rebuilding anaconda. For obvious reasons, this is less than ideal.


• It's hard to tell what the image building scripts do. I'll be honest: I'm prejudiced against large shell script projects. I find all the quoting and here documents utterly unreadable. It's also difficult to debug.


• We don't do a very good job of cleaning up after ourselves.

The project to redo our image building scripts is called Lorax and mgracik has been working on it for quite a while now. The main goals are to move it to Python so it'll be more easily comprehended and debugged as well as importable in all our rel-eng tools; and use a blacklist instead of whitelist so things moving around doesn't break the images. As was reported on anaconda-devel-list this week, we're just about good to start using it, too. A little surprisingly, it appears lorax runs quite a bit faster too. That'll be a welcome change since it means speeding up tree composition, which also helps for anyone testing tree changes.

So what is left before we can start using this miracle new project? Well, lorax needs to get into Fedora first. I worked on the review ticket yesterday and today and got it into shape so now we're just waiting on the rest of the process to magically finish up. Then, mgracik needs to do some work to make it support other architectures. While we really only do i386 and x86_64 in Fedora now, it'd be nice for lorax to support everything that anaconda (theoretically) does - ppc and s390, and some half-baked support for older processors. And then finally, we can go into the scripts directory in anaconda and remove a bunch of old crud.

While this may not be the most exciting user-level development, I'm very pleased to see some of these fragile scripts get replaced with what looks to be a more readable and less error-prone system. Hopefully this translates into a less frequently busted install image.

Image Installation

There are a whole lot of tools that take some input file (often but not always kickstart) and generate a disk image that's got some filesystems and packages in it. The livecd creator stack of stuff is a prime example of this. You may also recognize this as basically what anaconda does. It stands to follow, then, that anaconda could do the livecd creator job as well. The reason we don't is largely historic: for the longest time, anaconda was not really easy for anything else to make use of. It wasn't installed like a real Python module and even if you managed to get it imported, the code wasn't really organized in a way at all easy to make fun of. Essentially, anaconda was a ton of code off in its own world.

It's taken us a couple years, but we've gotten to a point where anaconda is something you can import and use like any other Python module (albeit a pretty strange one). First, there was that whole storage rewrite that made the storage code much more modular and easy to add crazy new stuff to. Second, we reorganized the source tree and package layout. Third, for all my storage test code I worked on making it really importable outside the special anaconda environment.

With all that done, dlehman was able to crank out a long series of patches allowing image installs in just about a week. The way it works is that you first create some disk images (using dd, qemu-img, whatever) and then you pass them to anaconda as arguments. anaconda will ignore every other disk on the system and only use what you pass.

For now, you'd do something like this:

LIVECMD="anaconda --image=/opt/disk1.img --image=/opt/disk2.img -m http://cannonball/install/rawhide/20101110/x86_64/os/"; liveinst

If you're doing a kickstart install, you may also want to use your "disks" as --ondisk= arguments. Instead of having to know what loopback device anaconda will assign, you can use an alternate syntax when starting anaconda that allows you to give the images whatever name you want, then refer to that in the kickstart.

None of this is really documented yet, and I haven't given it a test run. However it's looking promising so far and I'm hoping we can soon start to work it in as the guts of some of these image installation tools.

Storage testing

A huge part of testing any release is testing the installer. And a huge part of testing the installer is testing the partitioning code. This is probably the single most intensely tested area of the entire distribution. Unfortunately, this is also mostly manual labor.

If we could automate this testing, there are tons of benefits. First, we could run the tests after every git commit, or every night, or every week, or continuously. It doesn't have to wait for someone to have free time before a release. We'd also be starting from a known on-disk state which means problems are reproducible. With clever enough tests, we could figure out problems in seldom-used code paths. And if run frequently enough, it could help us make sure our storage-related commits weren't breaking things elsewhere. All told, it seems incredibly useful.

Well, turns out there IS a way to do it and I have it working as a part of autoqa. It's pretty complicated, but the basic idea is that each test case first creates whatever disk images it needs in whatever state it needs them (pre-existing partitions? encrypted disk? garbage RAID metadata?), then starts up a VM with those disk images and runs a small program that runs a kickstart file through anaconda's storage code. Then, it examines the final state of the disks and compares it to what's expected and reports the results.

It's really into the finishing touches stage now. Except for resizing, I have the partitioning section of the Fedora test matrix automated plus some more complicated tests. jlaska has been busy chewing through his list of related autoqa tasks to get this merged into master. In addition, I've put out a call for further test ideas and signed up to lead a training session at FUDCon Tempe in January.

4k sector disks

Speaking of storage, pjones has been hard at work on adding support for installing and booting from hard drives with 4k sector sizes. Currently, just about every single shipping hard drive has a 512 byte sector size. However, we expect this to change over time as disks with different sector sizes slowly come onto the market.

The first part of making this work is of course making sure the bootloader can recognize and read the disk so it can find the kernel. Much to no one's surprise, the first part also ended up being the hardest part. The second part would have been making sure the kernel and user space tools (parted and anaconda, most importantly) worked. However, they'd all either been updated earlier or required nothing, so everything just worked.

Along the way, he also discovered that booting from a single 3 TB large drive works fine, though making a filesystem that large takes long enough to let you go grab lunch and the 3 TB vs. 3 TiB difference is now about 200 GB in size, which itself is a pretty large hard drive.

There's no TWIA this week because it's Thanksgiving and I'm off work both Thursday and Friday. Personally, I've been plenty busy with merging my storage tests into autoqa. That's probably the thing I'll talk about next week. So for now, go off and do something that doesn't involve computers for a while. Your body and mind will thank you.

Maintain

This first entry came up last week, but I ran out of space and got tired of typing so it got pushed off to this week's report. bcl sent out a proposal for doing official anaconda updates for past releases. This is something that we've resisted in the past, with various degrees of success. We've also had the Fedora Unity guys help us with this from time to time.

The basic problem is that after a Fedora release, anaconda never gets updated though various other tools that anaconda does use do get updated. This isn't a problem on most installation media because we never remake the media. It is, however, a problem for people doing livecd respins. What'll happen is that anaconda will assume a particular library call for NetworkManager (just to pick on a component at random) that exists when the release happens, but changes in an update. Or something similar in udev. Or metacity. Or many other components. Then when you go to respin, it won't work because anaconda's never been updated to use the new function call.

Basically, bcl's proposal is that over the course of a release, someone on the anaconda team ("Maintenance Release Manager", as he calls it) will be responsible for backporting build fixes from master to the release branch, then doing an update. It's also very important to note what his proposal is not. It's not a way to get features into previous releases. It's not a way to get every single bugfix backported. It's not a way to get new official release media made. This is really only helpful to people doing respins.

I'm always a little skeptical of these proposals since they've come up so frequently and not gone anyway. I can see the need for it now (respins didn't used to be a thing we did in Fedora), but no one wants to do updates because they're hard and boring. Hopefully this has just been a case of people not being motivated enough and bcl will be able to turn this trend around. Good luck!

Interface

Our user interface redesign is in full swing, judging by the main thread started last week and the networking thread started this week. mizmo has been focusing on the syslinux boot screen over the past week and has some nice looking mockups ready.

Meanwhile, rvykydal posted another set of patches for configuring the network. This time, he's trying to get the DVD installer to work more like the live CD installer by putting a panel in the DVD installer and running nm-applet out of there. I'm not sure how much traction those patches are going to get, but it's an interesting idea and heads in the direction of bringing the two paths closer together.

As for me, I tossed out some comments on mizmo's UI walkthroughs to explain why things are the way they are or where we can get rid of stuff entirely. For example, one idea I've got is killing the root password screen and setting up the first created user with sudo. It seems like a lot of other distributions are doing that these days.

Want even more UI? Check out Dustin Kirkland's post on the matter. And if you've got any ideas, now is the time to jump in on anaconda-devel-list. Don't wait until after the release to do your ranting. Help us out now while it's still open to discussion.

Preserve

It wouldn't be a week in anaconda without a partitioning problem. This time, it's some weirdness with the kickstart --noformat option. Kickstart's part, logvol, and raid commands all take a --noformat option that does exactly what it says: don't format this mount point, but mount it anyway. This is useful if you've got a /home you don't want reformatted but would like included in the finished system.

It's also useful if you have really particular partitioning requirements. anaconda (and kickstart) don't support every single option that mkfs, lvm tools, etc. take. This is on purpose, as they take a whole lot of options that would make for a UI nightmare. For those people, we suggest they make their partitioning layout in a kickstart %pre script and then use --noformat to preserve whatever they did. It works pretty well and everyone seems to be happy.

Unfortunately, sometimes we get people who have a previously installed / and they choose to not format it through --noformat or the UI checkbox. As you can expect, doing an install over top of that creates a giant mess and leads to really weird bug reports. So, dlehman has worked up a patch to prevent people from using an unformatted /. That's good, oh except it breaks everything I just talked about in the previous paragraph.

dcantrell came up with a pretty good - if slightly complicated - proposal. His proposal basically involves a special %pre script option that would tell anaconda what mountpoint this script makes. Then, anaconda could check all --noformat lines against a list of things that must be formatted and then make sure there was a %pre script that generated them. While a little roundabout, this should take care of both cases above. No code yet, as we are still talking about the proposal.

Substitution

Last week, I promised I'd talk about how substituting the $releasever and $basearch variables in repo lines played out. I'm sure many people have been waiting anxiously for the results. Well, this extremely involved commit from bcl fixed it all up. As I suspected, it was a problem where we had all the pieces but needed to hook them up right.

Interface

The user interface discussions are up and running thanks to mizmo's thorough screenshotting and questioning as linked to from the UX Redesign wiki page. She's raised quite a few questions about most every screen in anaconda. We're going to be making a lot of decisions about the purpose of each screen and UI element in the next release or two. I've wanted this kind of discussion for years but something else has always preempted it. Now I think we finally have enough people working on anaconda to where we can afford to have someone go off and work on the UI. Just don't expect instant changes.

Since we've just started, there's little to talk about here. We haven't decided or changed anything. I don't even think we know what direction we should be moving in. Basically right after I finish this post, I'm going to go respond to everything on the wiki page to get us moving there. If you have thoughts, feel free to join in the conversation. However, I'd like to keep this focused on Fedora. Looking at other distributions is useful, but we need to be making decisions based upon what Fedora needs and where Fedora wants to go, and not just because some other distribution does something.

Filesystems

This subthread on fedora-devel-list is all about whether we're moving to btrfs as the default filesystem. The short answer is no, not for F15. We will, however, allow you to create btrfs volumes without passing any secret command line parameters.

One reason for me not wanting to make btrfs the default is that I don't yet know what all it's going to be capable of doing. There's been a lot of talk about it being a potential replacement for LVM, which makes me think we need a real strategy around it. What, exactly, do we want to do? Add to this that wwoods wants to quit using LVM by default anyway and it seems like we really need to come up with a plan. After all, I'd hate to kill LVM this release, switch to btrfs by default next release, and then move in some fancy btrfs LVM-like features for the release after that. That's a whole lot of disruptive filesystem changing.

Who wants to talk about this at FUDCon in Tempe? I'd love to hear what people have to think about where we should be going.

Growth

As you may have read elsewhere, Will Woods is moving over to the anaconda side. I'd hate to speak for him before he's even gotten a chance to start and look around, but it seems like he'll still be working on testability and all that kind of stuff. It's just that he'll be looking at it from a different side and have commit access to make changes. So, welcome! Now he can be personally responsible too.

Release

Another week, another major release. Once again, congrats are in order to the ridiculous number of fixes and features done by the installer guys. And just in case you were wondering:

clumens@exeter:~/src/anaconda$ git log -p rhel5-branch..rhel6-branch | diffstat | tail -1

 1390 files changed, 2364004 insertions(+), 2425494 deletions(-)

The biggest news this week is that Fedora 14 finally escaped from its holding cell into the wild. Congratulations to everyone who put in so much time on this release! I've summarized the relevant anaconda F13->F14 changes on the changes wiki page. As you can see, it's largely internal stuff or tweaks around the edges. Lots of old crud got removed too. You can also see the changes from F14 to rawhide.

Releases are always kind of weird for me. By the time everyone else starts playing with the final release, I've already been working on the next one for a month or two. The bugs and quirks of F14 anaconda already seem so long ago. It's just me, but I find it strange that people are still using install.img when I've had that killed for weeks.

libarchive is better than whatever we've got

A long battle in anaconda has been replacing our own implementations of things with whatever the rest of the system uses. I keep thinking we're pretty close to done, but we keep finding more stuff that can go. This week, dcantrell proposed removing our own cpio code, since libarchive can do all that for us. It hasn't been pushed yet but I expect to see it in pretty soon. This is just another example of how we can add features in a release and still shrink the code size. The result is fewer places where something can go wrong.

yum variables in kickstart

If you've ever looked at the yum repo config files, you have seen this:

mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch

The $releasever and $basearch variables are substituted in by yum when it processes the config files. This keeps them nice and generic, so every release and architecture can get the same ones. Unfortunately, you can't use the same trick in your kickstart files as was pointed out on kickstart list this week. As of this moment, we don't have it tracked down quite yet but it should be fairly easy to solve.

We already have code to substitute $releasever, and we should be able to figure out $basearch too. And while we can probably use yum to figure that second part out, we're on our own for the releasever. That's because yum checks the version of the installed system-release package, and there's no rpmdb in the installation environment to check. But still, this has all the makings of a problem where we have all the pieces, we just need to hook them up correctly.

This is also a great example of a problem identified by a user that we're able to solve once we are made aware of it. I'll let you know next week how it turns out.

What else?

Is there anything in particular that's happened in anaconda recently you'd like to know more about? Something that just doesn't make sense or needs a better explanation? Let me know. This isn't a solicitation for bug reports - we get enough of those already - but for something that could benefit from discussion.

Quick bug fix round up

Nothing too exciting happened this week in anaconda land so I'm going to recap a couple commits and then get to something completely different. dlehman committed all his action sorting patches I discussed last week and we will shortly have a build including them. akozumpl got us away from the nasty old pile of timezone setting code and closer to what's in system-config-date. Now that we're running in full screen, we have space for stuff like that. We also had a bunch of build fixes and logging changes.

wiki

I thought I'd spend a minute talking about the anaconda wiki, which is surprisingly useful and reasonably up-to-date. In particular, there are three pages that you may find the most helpful:

• Building - This page describes how you do a source build of anaconda, how you build an anaconda package (including the directions for what we do when making official builds), and how to create a new tree to test out your modifications. Because anaconda is a special program, building and testing it is never really straightforward. Our wiki page attempts to at least explain all the steps though it's still a little intimidating.


• Kickstart - Kickstart is the anaconda automation language. You write up a text file detailing all the installation parameters, then point the installer at that file (commonly by putting it on a web site and passing anaconda ks=http://whatever), and anaconda runs without further interaction. This wiki page describes the kickstart file syntax and all the various commands and options you can use. We frequently update kickstart commands so the wiki page describes the absolute latest version. For information on a specific Fedora release, you'll need to consult the installation guide.


• Options - You can pass a ton of command line options to anaconda via the bootloader prompt. They look like kernel command line options because of that, but they end up getting passed to anaconda. With them, you can control network parameters, where to install from, debugging levels, what hardware to probe, and much more. Like the Kickstart page, we attempt to keep this one reasonably up to date.

/usr is not recommended

The surprise controversy this week was that I removed /usr from the list of default mount points in the UI. That doesn't mean you can't still use it. It just means you have to type it in manually as opposed to choosing it from a drop down. The rationale is that Fedora in general does a pretty bad job of supporting /usr on its own mount point. We do such a bad job that even the install guide recommends against it.

Well, someone actually read the anaconda changelog (which is probably the most surprising of all) and decided to comment. The whole mail thread was focused on whether or not Fedora as a whole should allow /usr on its own partition so it was really only tangentially about anaconda. We largely stayed out of the conversation and it kind of died without any real conclusion. The more interesting parts of the thread were about per-user /tmp which doesn't really have anything to do with the initial post.

Personally, I feel that a separate /usr is an historical relic and a niche use case, but again I haven't done anything to prevent you from using it. And if Fedora one day has proper support, I can always add it back into the drop down as a default suggestion.

There is only tsort

dlehman dropped a big pile of patches on the list yesterday that I am pretty excited about. This is real deep down internal stuff that no user will ever notice, but is important stuff. Several Fedora releases ago, he led us in a massive storage rewrite that involved getting rid of almost all of our old crufty storage code and replacing it with something much more OO in design. You may remember this as a time of great upheaval and bug reports as we figured out what we were doing. But in the end, I think it's resulted in a much higher quality partitioning system.

Well one of the main abstractions added in the storage rewrite was the concept of an Action. There are lots of kinds of Actions (as you can see for yourself), but they really come down to creating, deleting, and resizing filesystems and partitions. Despite all the fancy technologies these days, that's really all the storage code does.

Now, Actions are not done as soon as they are requested. When you tell anaconda to create a /dev/sda1 partition of 500 MB and put /boot on it, it doesn't go off and do that right away. It creates at least two actions (a create for /dev/sda1 and a create for the /boot filesystem) and adds them to a tree of actions to be performed. It's only after you do all your other partitioning selections and press "Write to Disk" that the Actions get performed. And of course, it's not quite that simple. As you can imagine, Actions have some specific order to them. You can't create /boot before you create /dev/sda1. Creating /boot might also require deleting whatever was there beforehand. This ordering can get quite complex in a hurry, depending on what you've got on the disks and what you tell anaconda to do. RAID and encryption adds a whole new layer of Actions on top, for instance.

Figuring this order out requires a sort function. Unfortunately, our existing Action sorting function is pretty rough. We were in a hurry for a lot of the storage rewrite towards the end, so the sort function just sort of grew and grew until it became really hard to follow. Also, as I have implied from calling it a "sort function", it was really just one giant function instead of using proper OO design to separate out the sorting concepts into each individual Action class. The latter is how you commonly see things designed, especially in Python by giving every class a _cmp_ method.

dlehman's patches finally kill the horrors of the old pruneActions and sortActions, moving the logic of action comparison into requires() and obsoletes() methods on each Action class. It's then a simple matter of using the One True Sort - the topological sort (or tsort) - on all the Actions and running the result. tsort is a very well understood sorting algorithm that converts a tree into a list, so it does exactly what we want.

The patch set introduces many more lines than it removes, but you need to look closer. The majority of the additions are test cases, while the actual code size shrinks. I also expect this to fix up a lot of bugs where we try to make things out of order, too. It should at least make them much easier to spot.

A crazy new idea

What's a work week without some big new idea that will change everything around? This time, it's killing the split media installs. Split media is simply the six or seven disc CD set. You may be wondering what could possibly be wrong with split media and if we're just bored and looking for code to delete. While it's true that we're always looking for code to delete (fewer lines = fewer bugs), split media does have some serious problems.

First, it causes problems for anaconda. When you install packages, they get grouped together into an RPM transaction. Within that transaction, we can check that there aren't two packages with the same files (a file conflict), that you have enough space to install everything in the set, and so on. Installing from the DVD is one single large transaction. Every use of yum to install on the command line is an individual transaction. CDs complicate the situation. Because each CD only contains some of the packages, each CD ends up being its own transaction.

This is a big problem. It means that if a package on CD#2 has the same file as a package from CD#1, we can't detect that until we're processing CD#2. Or, we could be in the middle of installing from CD#5 when you run out of disk space. Oops - nothing we can do about it now except tell you to reboot and pick less stuff next time. There's a bug for this somewhere, but I cannot find it.

Second, it causes problems for rel-eng. The above transaction stuff gets even more annoying when you consider that certain packages have scripts that run at the end of the transaction and expect everything needed for the script to run to have been installed. A prime example of this is the kernel and dracut. In order to make the initrd so you can boot after installation, every program needed to access your root filesystem must have been installed as part of the same transaction as the kernel and dracut. This includes LVM (default partitioning does LVM, after all), perhaps RAID tools, perhaps encryption tools, perhaps multipath tools, and so on. If all that stuff isn't on CD#1, you are left with a system that doesn't boot.

The fix is to have the distribution composing tools force certain packages to be on CD#1. And any time you have a hard-coded list in a program, you're asking for maintenance troubles in the future. There's also a cascading problem here where forcing some packages onto CD#1 makes that disc larger than the 700 MB CD size limit. Hopefully some packages can spill over to CD#2, but this may not be possible due to package dependencies. In that case, the only option is to remove some packages from the release media.

We always hit these problems at the very last minute, too. It's just time for split media to go. We currently offer the complete DVD, a single live install CD, and a minimal boot.iso that can be used to network install. My hope is that the split media user base is getting smaller all the time as they look at these other methods that require fewer discs.

Followup

As a followup to last week's edition, it's worth mentioning that I committed my series of install.img killing patches. So, all rawhide trees and F15 trees will ship without that file. Don't panic if you don't see an install.img.

This is my first installment of This Week in Anaconda, where I try to explain what we've been up to and why things are the way they are. I hope you find it interesting, if not exactly entertaining. Hopefully I'll get better with this over time.

anaconda is ugly and unusable

The most exciting thing to happen this week in anaconda land was the fedora-devel thread where it was basically suggested we just switch to the Ubuntu installer. I'm not going to get too into this here because I think it's already been discussed to death.

Basically, there's no way we will be switching to anyone else's installer. We have a whole team working on anaconda that represents decades of experience on this code base. Why would we just throw that away? Second, Fedora uses RPMs whereas other distributions use .debs. So even switching installers would involve a lot of work to get it working with yum. Third, anaconda is an extremely capable installer. I don't think most people understand this. We install on everything from netbooks to mainframes. You can install off a DVD, NFS, URL, or hard drive source. You can do RAID, LVM, iSCSI, multipath, FCoE, and a whole lot more. Oh, and you can completely automate the entire process. Fitting all that stuff into someone else's code is not exactly a trivial task. I estimate it would take at least a full year to get it up to speed and what do people do in the meantime? Just not install with any of that support?

I'll give you the point that anaconda is ugly. I've thought it needs a UI overhaul for years now, and we might finally be getting one. My recent changes to run in full screen mode (it's 2010, what took so long?) really show off how much work we have ahead of us. But this is all fixable stuff. It doesn't require throwing out the whole code base just to make it look and feel nicer.

One point that came up was why we support all these installation paths in the same program. Well, whatever we do it's not going to involve maintaining one installer for enterprise users, one installer for laptop users, one installer for livecds, and so on. Maintaining RHEL5, RHEL6, F14, and rawhide (which is at least two, perhaps even three installers) is hard enough without adding a multiplier.

To be fair, anaconda already does kind of do multiple experiences from the same code base. If you do a livecd install, you don't get any package selection. If you choose to only use basic storage devices, you don't get any of the filter UI. If you don't choose custom partitioning, you don't get a bootloader screen. If you do a text install, you don't get much of anything. Please don't do a text install - use VNC instead. See, there's another thing anaconda supports that I doubt many other installers do as well. We've also got install classes which allows deciding which steps to skip or show. anaconda searches for install classes in the tree it's installing, so we can leave it up to the spin to decide.

So we've got a lot of capabilities here. We just need to come up with a new UI design, implement it, and tell distributors what they can do to customize the experience to the product.

No more install.img patches

I dropped a huge set of patches on the list this week that will really change installation all around. Understanding why is going to take a little back story. For as long as I can remember, anaconda has been broken up into two parts: the loader, which is a C-based program in an initrd, and stage2, which is the familiar graphical interface. stage2 has been shipped in various forms over the years, but right now it's in install.img.

The reason for this split has been space savings. The initrd gets loaded by the kernel at bootup and sits in memory. It only contains enough stuff to find and download the much larger stage2 image. How it finds stage2 can be modified by the askmethod, stage2=, and repo= parameters in addition to booting off the DVD or CD#1. In total, it makes for some very complicated and error-prone code.

Over time, the initrd has grown and grown. This is because we try to use system components wherever possible and things like NetworkManager are necessarily much larger than our own special-purpose networking code. Also, software just tends to get larger over time. Add this to the complexity of never having the full system when we boot, and we have wanted to merge the initrd and stage2 images together for a long time.

Well I finally did it. Basically, I first found a bunch of stuff we're doing that is unnecessary and removed it to save space. This includes shipping a 30 MB locale-archive instead of a 100 MB one, removing icon archives, and removing certain X plugins. Then, I just merged everything that was left in install.img into the initrd. Of course, the code to do all that was much larger but it did involve removing more lines than I added.

So what does this mean for the user? First, the install.img is gone and the initrd.img is much larger. Instead of 30 MB, it's more like 100 MB. Some tftp implementations may have trouble with this. That's about the only downside I can see. The overall set of stuff to download is significantly smaller which should make for faster downloads and reduced memory requirements. It gets rid of the obnoxious stage2= parameter that you never should have been exposed to in the first place. And most importantly, it means less code which means fewer bugs. It frees up developer time to focus on more important stuff, and it begins to make me think about getting rid of the loader entirely since we now have python in the initrd.

One final note - I originally wanted to move translations out of the initrd entirely and into their own language-specific images, but I was unsuccessful in that. The splitting out patch was easy but fetching was difficult. If you do a pxeboot and don't give askmethod or repo=, anaconda uses the Fedora mirror system but we don't know that information until we're way ahead at package selection. We may need to bring up the network to fetch the language image, but we need to be able to prompt for the network in their native language. Oh well. I'm hoping to get some new ideas for this after pushing the first patch set.

F14 blocker hunt

F14 is almost over which means it's time to panic over blocker bugs. In the past week, everyone on the team has been putting in a lot of time knocking bugs off the list. We luckily haven't had too many this release but we also haven't done much besides just move files around.

akozumpl cranked through a couple annoying x-related bugs, including one where hitting the up arrow tried to run gnome-screenshot. Oops. That part was new, while the other bug was caused by not cherry-picking a fix from rhel6-branch onto master. Process is hard.

I did a quick fix to make sure to bring up the network before saving a bug report via any method. Back when we used python-meh for this, we could prompt just before save-to-bugzilla or save-to-scp. But using report meant a loss of some coupling, so now we can only prompt before even asking to save. This wasn't a very exciting patch.

As usual, there was a ton of storage stuff. dlehman and pjones worked on a whole series to a pile of components to allow anaconda to set the UUID on mpaths and RAID devices. bcl fixed rescanning disks when going backwards from the upgrade screen, which is the sort of bug we see a whole lot of. And finally, dlehman also made sure we add RAID containers before RAID members.

In case you were wondering what happened to the other days of beers, the short story is that writing about beer gets really old really quickly. There's only so many things you can say, so many adjectives you can use. I got tired of feeling compelled to find and drink a new beer every day. Basically, drinking beer became work. Now maybe if I were able to turn it into a book deal, it would be worth it.

Anyway, it's time for something new on here. As some of you may know, I have worked at Red Hat since 2004, specifically working on the installer. Over time, I've worked my way up until I'm now the technical lead. That means I'm supposed to be coming up with crazy new ideas and helping to set the overall direction.

In all this time, I've never been particularly good at self-promotion or promoting anaconda in general. So starting this Friday, I'm going to work on a once-a-week wrap up of what we did, what interesting new ideas we have, and where we're going. I'll try to respond to any big mailing list threads. I'll occasionally give my rationale for why things in anaconda are the way they are. And hopefully, this will be interesting to someone.

Magic Hat Odd Notion Summer

Frequently, Magic Hat puts some crazy miscellaneous beer into their mixed cases as a test. It's usually named something like "Batch 47" and if enough people like it, it ends up as a regular beer with a real name in the future. This time, it looks like they've been experimenting with three drastically different beers all under the name "Odd Notion".

The summer one pours a golden straw color with a pale, short lived head. The taste is very citrus spicy with plenty of funky Belgian yeast characteristics. What do you know, the website says it uses a Belgian yeast strain. I never could have guessed. There's a hint of wheat malts in amongst all the spice. Odd Notion Summer is not at all shy about its 5.9% alcohol content. It's a very smooth, easy to drink beer. I can see enjoying a couple in a session.

Overall, I am impressed. This is a solid Belgian pale ale from a brewery that I want to succeed but always end up feeling let down by. So many of their beers are unremarkable to me, but this is a real stand out. Too bad it's in the experimental slot in the summer mixed case. I think picking up the case is worth it just to try this beer. Hopefully it'll be moving into regular rotation.

Magic Hat Wacko

I bought a mixed case of Magic Hat beers, so expect a couple reviews of those in the days ahead. I started off with Wacko, which I knew nothing about in advance. Magic Hat isn't really known for telling you a lot about their beers on the bottle. You stand very little chance of knowing what style one of their beers might be. You'll be lucky if it's even got a style at all. They take a very lax attitude towards crazy things like style, which can be fine sometimes. On the other hand as David Shea mentioned weeks and weeks ago, they also tend to focus on the hype more than the beer sometimes. So with that said, here we go.

When you pour a beer into your glass, you have a certain expectation. If the bottle tells you it's a stout or a porter, you expect something brown or black. If it calls itself a hefeweizen, you expect something very lightly colored. If it's a red ale, you expect something on the red side. Having no style information on the bottle, I didn't know what to expect from Wacko. Even if I had known, I wouldn't have expected what I got. It pours a bright, BRIGHT, iridescent, unnatural red color with a pink head. It's seriously the most ridiculous beer color I have ever seen. It also set up the expectation from Sarah for cherry soda.

Luckily, it does not taste like cherry soda. It tastes malty without being overly sweet, a little bit spicy (salty, even?) when swished around, and more sweet in the finish. It's not a very heavy beer which is understandable since it's supposed to be a summer beer. The taste fades fairly quickly, leaving you wondering exactly what that bright red stuff in your glass is doing there. So to recap... style? I'd have to say it's just a spiced beer, probably based on some sort of simple pale ale. I would drink it again which is good because there's at least three more in the fridge, but I'm going to be wondering what's going on with it the entire time.