Gentoo Stage1/Stage3 Building: Fact -vs- Fiction

Over the last few months of 2005 a meme was floated out to the Gentoo community, that installing Gentoo from a pre-packaged stage3 tarball is "better" than the previously supported method of bootstrapping from a stage1 tarball. In spite of the fact that many Gentoo users still prefer (1) the stage1 installation method, support for this installation method has been abandoned (2). The primary reasons for dropping support of the stage1 bootstrapping installation method appear to be that users are too stupid to comprehend it, there is no advantage to using it, it is too buggy, and that it is a waste of time (3).

What is most interesting to me about this discussion is the fact than no one has bothered to offer any facts to back up these assertions. I have never installed gentoo any other way than bootstrapping from a stage1 tarball, so I was very interested in investigating the issues further. I decided to reach my own conclusions by putting it to a test, and quite frankly, I am astonished at just how broken the stage3 install process is. How Gentoo developers can justify stage3 installs as "better" in any way is beyond me. Perhaps they don't eat their own dog food?

Testing the Claims

To test the claims that there were no advantages to using stage1 installs, I created some automated test scripts. The scripts do a simple task, start with a 2005.1-r1 official release tarball and get it updated to the portage snapshot dated 2006-01-16, utilizing USE flags suitable for a typical Linux/Apache/Mysql/PHP server. The scripts test the two different methods to achieve this - bootstrapping from a stage1 and merely updating a stage3. The scripts automate the steps described in the handbook to perform each installation type.

Test Conditions

  1. The final build should use the most recent stable toolchain and system packages available.
  2. No "unstable" packages.
  3. Each test build utilizes the exact same USE flags and conservative processor optimizations, customized for a typical LAMP server (no X, nptl, no pam, no nls).
  4. The test should emulate as closely as possible the installation process described by the instructions in the Gentoo Handbook.
  5. The entire process should be easily duplicated by others (entire build is done in a chroot).

Test Outline

Stage1 Stage3
  1. retrieve stage tarball
  2. extract stage tarball
  3. retrieve portage snapshot
  4. extract portage snapshot
  5. prepare build environment (1)
  6. modify bootstrap (2)
  7. fetch bootstrap distfiles
  8. fetch system distfiles
  9. bootstrap.sh (Part 1 - gcc upgrade)
  10. purge gcc
  11. bootstrap.sh (Part 2 - resume bootstrap)
  12. emerge -e system
  13. purge python
  1. retrieve stage tarball
  2. extract stage tarball
  3. retrieve portage snapshot
  4. extract portage snapshot
  5. prepare build environment
  6. fetch system distfiles
  7. fetch world distfiles
  8. gcc upgrade (3)
  9. emerge libtool
  10. emerge libstd++-v3
  11. purge pam (4)
  12. emerge -e system
  13. selective depclean/purge (5)
  14. emerge -e world
  15. purge old gcc
  16. purge python
(1) /etc/{resolv.conf,localtime} and from the host is used in the chroot build environment, /proc/mounts is copied to the chroot build /etc/mtab, /dev is bind mounted, proc is mounted within the chroot build environment.

The following settings are created with the chroot build /etc/make.conf: 
USE="-X -alsa -apm -arts -avi -cups -doc -eds -emboss -gnome -gpm -gstreamer
-gtk -gtk2 -imlib -info -ipv6 -kde -mad -man -mikmod -motif -mp3 -mpeg -nls
-ogg -oggvorbis -opengl -oss -pam -qt -quicktime -sdl -vorbis -xmms -xv
apache2 mysql nptl nptlonly php userlocales"
CHOST="i686-pc-linux-gnu"
CFLAGS="-O2 -march=pentiumX -pipe -fomit-frame-pointer"

FEATURES="-ccache -distcc nodoc noman noinfo"
The following /etc/locales.build is created in the chroot build: 
en_US/ISO-8859-1
en_US.UTF-8/UTF-8
(2) A patch is applied to bootstrap.sh which splits the bootstrap into two parts, part 1 installs up to gcc, part 2 resumes after gcc. This allows the user to manully run gcc-config to verify the latest gcc profile is being used, then purge the old gcc. The patch also removes STAGE1_USE from the ENV_EXPORTS on line 273, a bug which prevents the nptl, -nls, and userlocales USE flags from consideration during the bootstrap.

(3) The Gentoo Handbook does not mention that a gcc migration is required, nor is there a link on the documentation pages to the gcc migration guide. There are no specific instructions on how to proceed when you have changed USE flags that require the system to be completely rebuilt while simultaneously there is a gcc upgrade in portage. The author experienced an undocumented feature that automatically switched him over to a new binary incompatible gcc during a previous emerge -Du world, in spite of what the published documentation stated. For this reason it is assumed that the new user magically discern the need for the gcc upgrade, divine the location of the migration guide, and correctly follow the "safe" migration procedures.

(4) As a result of the -pam USE flag, an emerge -e system shows pam-login as a blocker. For this reason pam-login and pam have to removed before the emerge -e system (undocumented feature).

(5) Due to the -gpm and -nls USE flags, Locale-gettext and gpm are reported by emerge --depclean as being unneeded packages. During testing the author purged these before running the emerge -e system and ended up with an unusable build - the ls command quit functioning without libgpm and certain ebuilds/autoconfig/automake requires a functioning ls (undocumented feature).

Test Results

I ran the stage1/stage3 test builds on several different servers. A side by side summary of results is listed below. The total build time does not include the time required to actually fetch package source files, it is the actual time taken to compile the packages.

CPU(s) Memory Total Emerges Total Unmerges Total Build Time Full Reports
Intel P4 3GHz (ht) 2G 99/226 25/50 2h 32m 31s/4h 48m 31s stage1 stage3
Intel P3 1GHz 256M 99/226 25/50 5h 50m 52s/11h 56m 0s stage1 stage3
PowerMac 7455 1GHz 128M 100/225 27/54 6h 14m 56s/12h 30m 22s stage1 stage3
Intel PIII 600MHz x 2 2G 99/226 25/50 10h 11m 17s/20h 45m 19s stage1 stage3

My Conclusions

Anyone who claims that there is no advantage to the stage1 bootstrapping method, that it is a waste of time, or that it is too difficult to comprehend is downright delusional.

  1. Installing from stage3 takes over twice the time uselessly emerging the exact same number of packages multiple times.
  2. The stage3 gcc migration itself first upgrades glibc, then gcc 3.4.4-r1, then upgrades the gcc 3.3.5 on the official release tarball to gcc-3.3.6. This makes absolutely no sense.
  3. The same developers who are stating that it is pointless to recompile gcc/glibc more than once are advocating an installation method that compiles three different gcc's for a total of 4 gcc packages being compiled and builds glibc 3 times.
  4. Applying customizations to USE flags during a stage3 install either produced blockers (pam) or rendered the build unusable (gpm). Avoiding or working around these problems require searching out undocumented fixes from unofficial (wiki) sites. The same USE flags had no ill effects during a stage1 install.
  5. With relatively minor tweaks to the bootstrap.sh script, only two commands are needed to install from a stage1 - bootstrap.sh and emerge -e system. The stage3 process is inadequately documented, the gcc migration is not mentioned in the handbook, if I had followed the handbook explicitely I likely would have ended up with a severely broken build.
  6. All of the so-called "bugs" in bootstrap.sh over the last few months have actually been problems with ebuilds that have circular dependencies. This bootstrap.sh script is used by catalyst to generate the official release tarballs, if it is broken then so is catalyst.
  7. Building from a trimmed down stage1 bootstrap tarball is a much simpler, cleaner, and a much faster method to build gentoo.
  8. There are actually no advantages to a stage3 install unless you are installing for a desktop environment, you don't need to update or add any packages beyond what are already present in the stage3 tarball, and you are not planning on customizing Gentoo at all.

A Concession

One developer posted about the real problem in this post to the gentoo-doc mailing list:

"As more and more USE flags are added, especially to packages in "system", it has become increasingly complex to ensure that the bootstrapping process even works."
I have to agree, too much crap is leaking into the "system" target. I disagree, however, with his conclusions. The solution in my mind is not to blame users for being too stupid to bootstrap, or to abandon the stage1 bootstrapping installation method altogether. The real problem is that system packages can have dependencies other "non-system" packages at all. The base system should only provide the basic packages required to bootstrap compile the toolchain itself, bootstrap portage, and boot the system. The system packages should not depend on or pull in dependencies from non system packages. If you cannot provide a consistent baseline method for building, upgrading, and maintaining the toolchain and tools required to build it, you are going to get bitten no matter what hacked up solution you come up with.

References:
1)
Gentoo Forum Topic: What is your favorite Gentoo installation method?
Gentoo Forum Topic: What exactly happened to stage1? And bootstrapping?
2)
Gentoo FAQ, How do I Install Gentoo Using a Stage1 or Stage2 Tarball?:
"The Gentoo Handbook only describes a Gentoo installation using a stage3 tarball. However, Gentoo still provides stage1 and stage2 tarballs. This is for development purposes (the Release Engineering team starts from a stage1 tarball to obtain a stage3) but shouldn't be used by users: a stage3 tarball can very well be used to bootstrap the system. You do need a working Internet connection. Bootstrapping means building the toolchain (the C library and compiler) for your system after which you install all core system packages. To bootstrap the system, perform a stage3 installation. Before you start the chapter on Configuring the Kernel, modify the bootstrap.sh script to suit your needs and then run it: "

NOTE: No Gentoo documentation exists to shed light on modifying the bootstrap.sh script.

3)
Gentoo gentoo-doc Mailing List Topic: Stage1/2 deprecation from Gentoo Handbook:
"The Gentoo Release Engineering team is pushing us to remove stage1 and stage2 references from the Gentoo Handbook. They are getting too many errors because the user is not able to fully comprehend the implications of a stage1 or stage2 installation."
--Sven Vermeulen
"Using a stage1 simply has no benefit, it does not teach you anything. All that typing bootstrap.sh and emerge -e system does is enlarge the size of your ePenis. It also wastes your time because stage3 is faster, even if you change the CFLAGS/USE flags and recompile it all."
--Xavier Neys
"The real root of this problem is the misconception in our user base that a stage1 install actually benefits the user in any way. It does not."
--Chris Gianelloni