Kicking off 2012 with some downstream fun

Red Hat makes it a not-impossible task to remove the Red Hat branding from their flagship product and make your own distribution. In a comedically over-simplified way, essentially you can:

  • download the freely-available RHEL source RPMs
  • de-brand them (essentially replace redhat-logos and a few other key packages)
  • build out all of your altered RPMs and make a distro out of it (WAY out of this scope)
  • release your distribution under the same open source license as RHEL

There are several players in the universe of downstream, RHEL-derived releases, currently, and their situations are always changing. That is, of course, because they are all-volunteer efforts. Last year CentOS ( had some serious issues with their 6.0 release, and have always had a reputation of being an opaque and non-communicative group. Scientific Linux seized on the CentOS issues and attempted to make some real in-roads into their user base and community visibility. Another group actually formed in 2011, tryin to take the lessons learned from CentOS and Scientific Linux and make a new distribution with transparency built-in to the community. So how is every one doing today?

CentOS (

In North America, CentOS has been the goto RHEL-derived Linux distribution since before I ever got into IT. The releases are normally stable, but have been plagued in the past by delays in security updates, point upgrades, and almost always a lack of communication from the CentOS team. There was talk in 2011 of CentOS finally moving to a rolling release methodology, but I can’t find any confirmation that it ever happened. Their site also has no links I could find to 6.x documentation or process. This isn’t to say that it isn’t happening, it’s just to say that they’re not telling anyone about it. They do announce 6.2 being released, which brings them more or less up to their upstream source. With all of the turbulence and lack of communication and confusion, the mirrors stay up and I have to say I’ve got a 6.x CentOS ISO sitting on my desk… somewhere.

Scientific Linux (

Scientific jumped pretty hard at CentOS last year when it took CentOS a LONG time to get their 6.0 release out. For the first time, people who had never really considered anything other than CentOS were scratching their head and wondering who this upstart from CERN was. Of course, SL is no upstart, being around since 2004. This 2011 inroad into CentOS-land was relatively short-lived, however.  In August, Troy Dawson announced that he was leaving the SL project to work for Red Hat on their new OpenShift project. Not long after that, rumors of lack of team cohesion and direction started to bubble up. Currently, SL’s most recent release is Scientific Linux 6.2 beta 2 (as of January 9, 2012).

AscendOS (

AscendOS is the new kid on the block in this neighborhood, hoping to have their first production release based on RHEL 6.3. I actually had a conversation via email with Andrew, one of the AscendOS team leaders. He confirmed that AscendOS is going through a lot of the growin pains of any new open source project. Those problems are, of course, the number of contributors and the amout of time those contributors can allocate to a project. AscendOS has some developer builds available for download, and are still actively working to refine their build process and environment. If you can and want to help an interesting new project, I highly encourage you to give this project a look.

While these aren’t by any stretch all of the RHEL-derived downstream distributions, these are the ones that most interest me currently. Are there any other interesting ones out there?

Enterprise Linux by any other name

I’ve been doing a fair bit or reading and writing and talking about different versions of “Enterprise Linux”, and the more I talk and think about it, the more I come to realize that I’m not as comfortable with the definition of that phrase as I would like.

The current working definition of “Enterprise Linux” is a Linux distribution based off of the Source RPMs and build methods of RedHat Enterprise Linux. Essentially a group of people get together, put together a build infrastructure, and make a distribution using RedHat’s released sources. However, there are interpretations and changes made that make each of these distributions unique in their own right. CentOS has its issues, of course, but tries to stay as faithful as it can to the RedHat product. Scientific Linux seems to be a livelier group right now, but they’re truly making their distribution their own.

For example, SL doesn’t install the RedHat/Fedora Auto Bug Reporting Tool by default. While this is perfectly fine, it does make me wonder how if SL wants to push any bugs their community finds upstream or if they want to patch them within their own community. I was instructed by of the moderators at the SL forum site to contact one of the primary SL contributors about this, and I will. I’ll of course report back what I hear. I look forward to it.

I don’t know much about PISA’s philosophy, really. If someone knows more about the project team and their goals, I’d love to know.

AscendOS is just now getting off the ground, and their vision of building out an Enterprise Linux distribution without all of the baggage of the older communities. Noble, sure, but they’re still doing one thing like all the rest that may be worth examining. They’re all based off of what RedHat decides to do in their distribution.

I get the conventional wisdom, that if you follow RedHat’s lead, it’s easier to be compatible with RedHat. But it also piegon holes a distribution in to making mistakes that RedHat makes.

Now if you know me or have ever worked with me or talked to me for more than 3 minutes or been to a Richmond LUG meeting,  you’ll know that I’m a huge RedHat fanboy. I love how Shadowman goes about his business, and think they’re an admirable example of how to be a really good open source company. The RedHat product that really moved them into the limelight was RHEL 5. After coming out in 2007, it became the Linux standard in the Enterprise. But it had some serious shortcomings that proved at least annoying, and sometimes painful, for admins running it on a daily basis. The decisions that pop into my head are

  • holding on to Python 2.4 for so long
  • not including syslog-ng
  • not moving up to openldap 2.4
These are some of the first changes that happen to most any RHEL/CentOS 5.x installation that I control. I know RedHat had a good reason to maintain the older versions of python and ldap, and not including syslog-ng. But is it the best for an enterprise?
Should their be a distribution out there that relies more on sound principles than a specific company to guide their product? A community that tried to make the best distro available to run in a corporate environment.
Should we change how we define “Enterprise Linux” from “a RHEL-derived Linux distribution” to “A downstream Linux distribution that is optimized and hardened to work best in a corporate environment”. I guess that’s a pretty open-ended question, but I think it’s worth thinking.

Clone Wars 2 – Revenge of the Install DVDs

Last week I found out some interesting information when I took an initial look at two of the most popular RHEL-based Linux distributions, CentOS and Scientific Linux (SL). The next obvious step is to install the two, side by side, and continue this comparison. To be clear, I’m not running benchmarks of any sort on the two operating systems. I have no doubt that they will perform consistently enough to make that exercise un-interesting. What I’m looking for is what makes these distributions different, and anything that would make me particularly like or dislike one of these distributions.

The Setup

I set up downloads from public mirrors of both using their install DVD images. I was initially going to take all default settings, but their default settings turned out to be significantly different. So I picked the “Desktop” option for both setups, and took all other defaults.

I chose the Desktop version because I assumed it would provide the most “stuff”. A minimal runlevel 3 install, while more practical for my everyday life, would also be a short and boring talk.

They were both installed into VMWare Fusion 3 VM’s on my work-provided Macbook Pro. Each has a 20GB hard disk file and 1GB of memory.

Downloading the DVD Images

Downloading the DVD ISOs was relatively painless for both distributions. CentOS was a little faster, but I’m sure that fluctuates from mirror to mirror and over time.

Scientific Linux Installation

I installed SL first, and took all of its default settings, including installing the Desktop profile. It then prompted me that I’d require both DVD1 and DVD2 for installation. Odd, I thought. This didn’t sound familiar from the last time I installed RHEL6. So I tried other profiles, including “minimal”. It wanted both DVDs for all of them. So off I went to download another 1GB of data. Certainly not horrible but it seemed a little odd.

Installation looked identical to RHEL6, with different images, of course. The SL logo is a stylized Bohr-style atomic model. The graphics are simple and clean.

After reboot I got a black desktop with the atom logo large and on the right. The only immediately noticeable difference from the default RHEL desktop is that they’ve added a launcher button for the Gnome terminal app to the top Panel bar. While not Earth-shattering, it’s always task #3 I do to a Linux box with a GUI, so it’s a timesaver. They’ve also enabled the weather options on the Panel clock for me. While I like the idea and applaud the effort, it did make me wonder “What else are they doing to tweak the UI?”, and sent me off on a minor snipe hunt.

CentOS Installation

The CentOS install was also simple and clean. Their default install profile is “minimal”, so I picked Desktop to be consistent. Other than that the only difference were the graphics, and I took all default settings.

Oddly, CentOS never warned about nor asked me for the DVD number 2. It was happy to install only from the larger DVD image.

Rebooting after install landed me at a dark blue desktop with some stylized circle/bubble things. I like dark desktops, so this is fine with me. Clean and professional looking, it’s a huge improvement from the CentOS teal era. They don’t appear to have installed any desktop modifications, just like in previous CentOS installs.

Compare and Contrast

So how are these two distributions different? Well, those differences are few and far between at the levels I was looking into, but a few of them do make them distinctly different products.

The method I used to generate a way to figure out how they differed was to run a diff on a sorted list of their installed packages. While a little simplistic, this seemed like the way to get the most info for my buck. I’ve attached the raw sdiff at the bottom of this post.

Desktop Tweaks

SL installed SL_desktop_tweaks-6-3.noarch.rpm, which installs /usr/share/config/panel-default-setup.SL.entries. This is where my terminal shortcut, weather, and any other UI bits are coming from.

Automatic Bug Reporting Tool (ABRT)

SL doesn’t install this Fedora project at all. While I’m not a huge fan of this app, if I were newer to Linux, or not as comfortable with it as I am, I would certainly use this to push bugs into the RH bugzilla, generate error reports, etc. I really think excluding this app weakens the entire community, and I’d love to hear SL’s take on why they’ve left it out.

YUM differences

CentOS uses the fastestmirror YUM plugin, like it has in the past, to try and get you the fastest mirrors for your updates, etc. It’s not a perfect tool, but I applaud the effort, mainly because it means I’m not adding/removing mirrors to my config all the time as they come and go.

SL doesn’t use fastestmirror. Rather it hard-codes a list of ftp servers into its YUM repo configs:

SL yum repository configThis seems a little antiquated and static. I’d like to know why they haven’t moved to something more dynamic.

Also, for reasons I couldn’t quite put together, SL doesn’t include libtar as a default installation. It’s out there in its repositories, just not included on my test install. Why not?

The biggest difference, however, is that SL installs and ENABLES BY DEFAULT yum-autoupdate. Without an exclude statement of some sort (which they didn’t add in), this little gem will automatically update your, well, everything. If I were using this distribution in a production environment and didn’t notice this little guy was flipped on, it would be an almost guaranteed headache. Again, I’d like to know the rationale behind the decision. It has been mentioned on the SL forums, but as of my last reading no “why” answer or opinion has been offered by the team.


Well, this time I actually have a few. After digging a little deeper into these two RHEL-derived products (I won’t call them clones any more because neither truly clones RHEL), I find that they are both of high-quality, and their teams have put effort and thought into their creation. However, for me, right now, the edge still goes to CentOS. This is primarily because of what they do include (the ABRT, to help keep information flowing upstream) and what they don’t (auto-updating my system). I’d love to speak to people from both communities, and keep this conversation going on.

Raw sdiff output

Jamie-Duncans-MacBook-Pro-4:~ jduncan$ sdiff sl-list-sorted.txt cent-list-sorted.txt -s
SL_desktop_tweaks-6-3.noarch                                  | abrt-1.1.13-4.el6.x86_64
                                                              > abrt-addon-ccpp-1.1.13-4.el6.x86_64
                                                              > abrt-addon-kerneloops-1.1.13-4.el6.x86_64
                                                              > abrt-addon-python-1.1.13-4.el6.x86_64
                                                              > abrt-cli-1.1.13-4.el6.x86_64
                                                              > abrt-desktop-1.1.13-4.el6.x86_64
                                                              > abrt-gui-1.1.13-4.el6.x86_64
                                                              > abrt-libs-1.1.13-4.el6.x86_64
                                                              > abrt-plugin-logger-1.1.13-4.el6.x86_64
                                                              > abrt-plugin-rhtsupport-1.1.13-4.el6.x86_64
                                                              > abrt-plugin-sosreport-1.1.13-4.el6.x86_64
                                                              > centos-indexhtml-6-1.el6.centos.noarch
                                                              > centos-release-6-0.el6.centos.5.x86_64
firefox-3.6.9-2.el6.x86_64                                    | firefox-3.6.9-2.el6.centos.x86_64
gnome-desktop-2.28.2-8.el6.x86_64                             | gnome-desktop-2.28.2-8.el6.centos.x86_64
httpd-2.2.15-5.sl6.x86_64                                     | httpd-2.2.15-5.el6.centos.x86_64
httpd-tools-2.2.15-5.sl6.x86_64                               | httpd-tools-2.2.15-5.el6.centos.x86_64
initscripts-9.03.17-1.el6.x86_64                              | initscripts-9.03.17-1.el6.centos.x86_64
                                                              > libtar-1.2.11-16.el6.x86_64
nss-3.12.7-2.el6.0.sl6.x86_64                                 | nss-3.12.7-2.el6.x86_64
nss-sysinit-3.12.7-2.el6.0.sl6.x86_64                         | nss-sysinit-3.12.7-2.el6.x86_64
opal-3.6.6-4.el6.0.sl6.x86_64                                 | opal-3.6.6-4.el6.x86_64
pilot-link-0.12.4-6.el6.0.sl6.x86_64                          | pilot-link-0.12.4-6.el6.x86_64
plymouth-0.8.3-17.sl6.2.x86_64                                | plymouth-0.8.3-17.el6.centos.x86_64
plymouth-core-libs-0.8.3-17.sl6.2.x86_64                      | plymouth-core-libs-0.8.3-17.el6.centos.x86_64
plymouth-gdm-hooks-0.8.3-17.sl6.2.x86_64                      | plymouth-gdm-hooks-0.8.3-17.el6.centos.x86_64
plymouth-graphics-libs-0.8.3-17.sl6.2.x86_64                  | plymouth-graphics-libs-0.8.3-17.el6.centos.x86_64
plymouth-plugin-label-0.8.3-17.sl6.2.x86_64                   | plymouth-plugin-label-0.8.3-17.el6.centos.x86_64
plymouth-plugin-two-step-0.8.3-17.sl6.2.x86_64                | plymouth-plugin-two-step-0.8.3-17.el6.centos.x86_64
plymouth-scripts-0.8.3-17.sl6.2.x86_64                        | plymouth-scripts-0.8.3-17.el6.centos.x86_64
plymouth-system-theme-0.8.3-17.sl6.2.noarch                   | plymouth-system-theme-0.8.3-17.el6.centos.noarch
plymouth-theme-rings-0.8.3-17.sl6.2.noarch                    | plymouth-theme-rings-0.8.3-17.el6.centos.noarch
plymouth-utils-0.8.3-17.sl6.2.x86_64                          | plymouth-utils-0.8.3-17.el6.centos.x86_64
redhat-logos-60.0.14-1.sl6.1.noarch                           | redhat-bookmarks-6-1.el6.centos.noarch
redhat-lsb-4.0-2.1.el6.x86_64                                 | redhat-logos-60.0.14-10.el6.noarch
redhat-lsb-graphics-4.0-2.1.el6.x86_64                        | redhat-lsb-4.0-2.1.el6.centos.x86_64
redhat-lsb-printing-4.0-2.1.el6.x86_64                        | redhat-lsb-graphics-4.0-2.1.el6.centos.x86_64
                                                              > redhat-lsb-printing-4.0-2.1.el6.centos.x86_64
report-0.18-7.sl6.x86_64                                      | report-0.18-7.el6.centos.x86_64
report-config-ftp-0.18-7.sl6.x86_64                           | report-config-ftp-0.18-7.el6.centos.x86_64
report-config-localsave-0.18-7.sl6.x86_64                     | report-config-localsave-0.18-7.el6.centos.x86_64
report-config-scp-0.18-7.sl6.x86_64                           | report-config-scp-0.18-7.el6.centos.x86_64
report-gtk-0.18-7.sl6.x86_64                                  | report-gtk-0.18-7.el6.centos.x86_64
report-newt-0.18-7.sl6.x86_64                                 | report-newt-0.18-7.el6.centos.x86_64
report-plugin-ftp-0.18-7.sl6.x86_64                           | report-plugin-ftp-0.18-7.el6.centos.x86_64
report-plugin-localsave-0.18-7.sl6.x86_64                     | report-plugin-localsave-0.18-7.el6.centos.x86_64
report-plugin-scp-0.18-7.sl6.x86_64                           | report-plugin-scp-0.18-7.el6.centos.x86_64
sl-bookmarks-6-1.sl6.noarch                                   <
sl-indexhtml-6-1.sl6.noarch                                   <
sl-release-6.0-6.0.1.x86_64                                   <
sl-release-notes-6.0-1.noarch                                 <
xfsdump-3.0.4-2.el6.x86_64                                    <
xfsprogs-3.1.1-4.el6.x86_64                                   <
xulrunner-                                | xulrunner-
yum-3.2.27-14.el6.noarch                                      | yum-3.2.27-14.el6.centos.noarch
yum-autoupdate-2-1.noarch                                     <
                                                              > yum-plugin-fastestmirror-1.1.26-11.el6.noarch

The Clone Wars – CentOS vs. Scientific Linux

Update: The conversation continues HERE.

With Linux in the Enterprise, RHEL is king. Sure there are people who love and use Debian, or Suse. I would imagine that if you looked hard enough you could likely find somebody who’s using Slackware or Gentoo in a business somewhere. But I think it can safely be said that RHEL is currently the dominant enterprise Linux distribution. Then, of course, there are the clones.  If you so choose, you can forgo Shadowman’s Support team and either compile the freely available Redhat Source RPMs, or choose to use a community-supported RHEL clone. Currently, the two most popular of those clone distributions are CEntOS (Community Enterprise Operating System) and Scientific Linux (SL).

So if you have decided to not utilize Redhat support, which of these downstream clones is the better choice? With the recent (much delayed) release of CentOS 6.0 in the past week, many companies are looking to move up to the RHEL 6.0 family of operating systems. But is CentOS still the right choice? Being a primarily CentOS shop, and being more than a little OCD myself, I decided to compare the two in as practical as a manner as I could. Below are the results.


When it’s running on production, you don’t have time to wait on a tiny community to figure out how to backport in some obscure cross-site scripting vulnerability in an even more obscure module in your favorite language, even if you’re part of that community. An enterprise operating system needs to have an active and robust community to support itself, paid or not.

CentOS has been around for a long time and has a huge following. There have been murmurs of late about the core contributors getting tired, and the delay in CentOS 6.0 was the evidence. I don’t believe that fully, but I do believe the project could do with some fresh blood and possibly some new ideas.  But I don’t think it’s going anywhere anytime soon.

Scientific Linux hasn’t been around nearly as long, at least on the scale that it is currently enjoying. The community, however, is vibrant, and is backed by several large research labs such as CERN and Fermilab. Big plusses.

Advantage: Push


In Open Source software, the process is often times as important as the product. While I don’t believe there is anything massively different in how these 2 projects are doing what the do, SL is certainly better at talking about it and making the community aware of how it’s working. This presentation(PDF) is a pretty great one, even if it’s a little dated. SL Community, I’d love to see an update, for the record.

Advantage: Scientific Linux

RHEL Compatability:

This used to be a much larger difference, as late as version 5.x. Scientific made some pretty large changes to the RHEL repository structure, and added in some packages of their own. CentOS has always been as faithful a clone as was possible at the time. This is largely cleaned up in version 6.0, with the extra SL packages moving out to external repos, but much like the workflow advantage above, perception is still a strong influence.

Why is this important? Well, like lots of people, we’re a mixed RHEL/CentOS shop. It just makes life SO MUCH EASIER.

Advantage: CentOS

Mirror Speed and Availability:

I couldn’t find any perceivable difference in this category. Both networks are robust and highly available.

Advantage: Push

Community Support:

This is one of the most important factors when adopting a distribution, and sadly the one answer I’m not able to fully answer. I utilize CentOS support all the time, via the web, forums, and IRC. I’ve only occasionally sought support for SL, and this was way back in version 5.2. So I’m not really qualified to answer this one fully right now. However, I see active forums off of their home page and a 10 minute visit to the IRC channel on freenode saw plenty of conversation for a Tuesday night. I don’t think SL would have grown so much without good community support.

Advantage: Push

Lifecycle Support:

This was the one that surprised me.

As expected, CentOS mirrors the RHEL lifecycles. RHEL/CentOS 5.x will be supported through 2014. They haven’t updated their wiki yet, but I’m sure 6.x will be the same, with a full 7-year lifecycle.

Scientific only plans on a three year lifecycle. But on their forums they also mention supporting things in theory as long as Redhat does. So I’m a little confused on this one.

While I don’t typically plan on using the same OS for longer than 3 years, if it ain’t broke, I’m certainly not fixing it.

Advantage: CentOS

So those are my thoughts on the situation. Scientific Linux is definitely on the rise, and CentOS certainly needs to air out themselves a little. But at least with version 6.0, we’re still going to be going with our tried and true CentOS. I’m just not comfortable enough, yet, with the Scientific Linux community, mainly because they still don’t quite know how long they plan to keep their products alive. Out of this look at RHEL clones, though, the single biggest thing I’ve discovered is that I’m going to have to keep evaluating this choice down the road.