All posts by jduncan

Red Hatter and chronic tinkerer

kpatch – my kneejerk reaction

Oracle gobbled up a company called KSplice 50 ITYA (IT Years Ago – or July 2011). They then shoe-horned it into their downstream clone of RHEL so people could slip in kernel upgrades without rebooting systems sort of like how magicians yank table cloths out from under dishes on a table. It’s scary on any number of levels.

Now there is a new-ish project called kpatch that has the backing of Red Hat (full disclosure – I work for Shadowman). I’ve only had a little time to look at the incomplete documentation on how it works. That said, it looks to be a huge step forward over ksplice. From it’s Red Hat Blog announcement:

With respect to granularity, kpatch works at the function level; put simply, old functions are replaced with new ones.  It has four main components:

  • kpatch-build: a collection of tools which convert a source diff patch to a hot patch module. They work by compiling the kernel both with and without the source patch, comparing the binaries, and generating a hot patch module which includes new binary versions of the functions to be replaced.
  • hot patch module: a kernel module (.ko file) which includes the replacement functions and metadata about the original functions.
  • kpatch core module: a kernel module (.ko file) which provides an interface for the hot patch modules to register new functions for replacement.  It uses the kernel ftrace subsystem to hook into the original function’s mcount call instruction, so that a call to the original function is redirected to the replacement function.
  • kpatch utility: a command-line tool which allows a user to manage a collection of hot patch modules.  One or more hot patch modules may be configured to load at boot time, so that a system can remain patched even after a reboot into the same version of the kernel.

That’s way cooler than just doing some fancy RAM voodoo and slipping new kernels in like ksplice.

But I still don’t see where it has a place on a company’s production server or in their security plans.

I believe that if a system cannot sustain the reboot of a single instance of Linux (physical or virtual) then there is a serious flaw in its architecture. To further that I think something like kpatch could end up being a strong crutch to bad architects out there; allowing them to  keep working in this flawed manner.

I know that my crazy idealism doesn’t represent the current reality everywhere (or almost anywhere). But if this is the only justification for its existence then I think we could have and should be using our cycles better somewhere else.

More details as I discover them.

Managing Purpose as much as Procedure

There are thousands of apps and books and philosophies out in the world designed to help people prioritize their work day and get more done.  Searching for ‘productivity books’ on Amazon provides 38,699 results. It is an entire industry, and a very lucrative one. What doesn’t seem to be an industry is managing the purpose behind all of those beautifully organized tasks. I think that is just as or more important than the task itself.

It sounds obvious, I know. Of course you should think about why you are doing the things that you have been tasked to complete. But I have seen so many examples of people not truly doing that exact thing that I don’t think it is given a lot of thought by professionals.

The definition of ‘why’ is the biggest problem. Most people report to someone else in their company; we all have a boss. From that boss we are given our share of the company’s picture and asked to perform our task to further the company’s agenda. That is awesome, and I’m all for it. A company certainly thinks through why it is doing something. The successful companies think it through, at least. Unfortunately, ‘because the company said so and I don’t want to get griped at (or worse)’ isn’t a justification that will easily help you move your career forward.

‘Why are YOU performing that task?’ is a better question to ask. Unless you are 100% satisfied and happy in your current position and have no desire to grow or move on, you should be performing that task to move forward in your career as well as to help your company realize its goals. Moving forward in your career requires expanding your skills and experiences so you can take on more responsibility and handle larger issues.

Again, it sounds like a no-brainer, but how often do you truly ask yourself that question:

How is this task on my [calendar/TODO list/productivity app/sticky note] helping me move forward toward a defined goal that I have set for myself?

Sometimes it’s obvious how it aligns to your goals. Conferences, Training, chances to present to others, etc. are easily identified as contributing to anyone’s toolbox. But the devil is in the details. Looking through my daily TODO list (currently kept in Evernote, but that changes all the time because I’m hooked on productivity apps), I can honestly say that I can align all of my daily tasks to a goal that moves me forward in my career.

Having that knowledge is powerful. Once you are consciously aware of how a task can help you move forward, you can take full advantage of the opportunity to hone those skill(s). Over time, this change in philosophy can help turbo-charge your career path. I have driven co-workers and supervisors to the edge of insanity asking these questions over the years. But I can say that this philosophy, which is what it is more than anything else I think, has helped push my career forward faster than most people I got started in IT with 7 years ago (for the record, I was 29 years old when I started my first IT gig).

I firmly believe that there is almost no such thing as ‘busywork’. If it doesn’t align with your goals in some way, then you probably aren’t the right person to be doing it or maybe it’s time to start the old job search again. 

a humble take on the Red Hat and CentOS agreement

The news is everywhere. Red Hat’s community website has a good summary of it at As soon as the announcement came out that Red Hat was going to step in and help the CentOS project with day jobs and some semblance of project management the interwebs were ablaze with opinions. They ranged from “wow!” to typical conspiracy theory nonsense (I can say on good authority that Red Hat Tower does NOT contain a secret control room from which Red Hat is trying to overtake the FOSS world).

This is NOT at Red Hat Tower, but there is air hockey and cookies.

After talking it over with a lot of the IT folks that I know (mostly the less conspiracy theory-ish ones), here is my take on what’s going on with this whole cooperative effort.

  1. Cooperation between Red Hat and CentOS is not new. Red Hat intentionally makes it not-so-hard to legally de-brand all of the trademarked stuff inside the source code that we distribute for RHEL.
  2. The CentOS core team is small. I mean 6 people small. The community is obviously much larger, and the QA team has a lot of influence. And those 6 people have day jobs. So any real expansion of the CentOS project would take multiple extra hours in the day for these people to actually get it done.
  3. Red Hat is helping out with #2. Helping with the web site and some project management / governance (and of course daytime salaries).

Number 3 draws the inference that the CentOS wants to make at least some sort of change in how CentOS operates. That would make sense, given the incredible speed that projects like Openstack are developing.

*Just in case you didn’t know*

For Openstack to work properly, a very (very) upstream kernel has to be used. Rebuilding a new kernel from source takes new build environments and QA processes and infrastructure and time and effort and money. Before this agreement, the CentOS team didn’t have that kind of time because they were up nights with bug and security fixes coming in all the time.

So CentOS couldn’t build an Openstack release prior to this agreement (IMHO).

*Downstream Innovation – What I think is happening*

Right now innovation is hard with RHEL. Customers use it for… you know… production stuff.  That’s not to say it doesn’t happen. I see innovation in Enterprise Linux everyday.

But with this agreement, there is now a project full of people working full time every day on making Enterprise Linux more innovative. I think it’s a win for the entire Open Source Community.