VMWare – A Cautionary Tale for Docker?

Of course VMWare has made a ton of money over the last ~12 years. They won every battle in ‘The Hypervisor Wars‘.  Now, at the turn of 2015 it looks to me like they’ve lost the wars themselves.

What? Am I crazy? VMWare has made stockholders a TON of money over the years. There’s certainly no denying that. They also have a stable, robust core product. So how did they lose? They lost because there’s not a war to fight anymore.

Virtualization has become a commodity. The workflows and business processes surrounding virtualization is where VMWare has spent the lion’s share of their R&D budgets on over the years. And now that is the least important part of virtualization. With kvm being the default hypervisor for OpenStack, those workflows have been abstracted higher up the Operations tool chain. Sure there will always be profit margins in commodities like virtualization. But the sizzle is gone. And in IT today, if your company doesn’t have sizzle, you’re a target for the wolves.

Of course docker and VMWare are very different companies. Docker, inc. has released its code as an open source project for ages. They also have an incredibly engaged (if not always listened to) community around it. They had a the genius idea, not of containers, but of making containers easily portable between systems. It’s a once-in-a-lifetime idea, and it is revolutionizing how we create and deliver software.

But as an idea, there isn’t a ton of money in it.  Sure Docker got a ton of VC to go out and build a business around this idea. But where are they building that business?

I’m not saying these aren’t good products. Most of them have value. But they are all business process improvements for their original idea (docker-style containers).

VMWare had a good (some would call great) run by wrapping business process improvements around their take on a hypervisor. Unfortunately they now find themselves trying to play catch-up as they shoehorn new ideas like IaaS and Containers into their suddenly antiquated business model.

I don’t have an answer here, because I’m no closer to internal Docker, Inc. strategy meetings than I am Mars. But I do wonder if they are working on their next great idea, or if they are focused on taking a great idea and making a decent business around it. It has proven to be pennywise for them. But will it be pound-foolish? VMWare may have some interesting insights on that.



and you think the code is the hardest part

Well, you’re pretty much right. BUT.

I’ve been working, on and off, on a project called soscleaner since last December-ish. It’s a pretty straight-forward tool. It takes an existing sosreport and obfuscates data that people don’t typically like to release like hostnames and IP addresses. The novel part is that it maintains the relationships between obfuscated items and their counterparts. So a hostname or IP address is obfuscated with the same value in all of the files in an sosreport. It allows the person looking at the ‘scrubbed’ report to still perform meaningful troubleshooting.

It’s not a big enough problem to get a true company or engineer’s attention, but it’s too big for a hack script. So I decided to try and tackle it. And I have to say that the current iteration isn’t too bad. It doesn’t what it’s supposed to pretty reliably, and all of the artifacts to make it a ‘real program’ are in place. Artifacts like:

  • issue tracking
  • wiki
  • README and Licensing Decisions
  • publishing binary packages (RPMs in this case, right now)
  • publishing to language-specific repositories (PyPi in this case, since it’s a Python application)
  • creating repositories (see RPM’s link)
  • submitting it to a Linux distro (Fedora in this case, for now)
  • writing unittests (a first for me)
  • creating some sort of ‘homepage
  • mailing lists

All of this has been an amazing learning experience, of course. But my biggest take away, easily, is that all of the things that wrap around the code to actually ‘publish’ an application is almost as hard as the coding itself. I am truly stunned, and I have a new appreciation now for the people who do it well every day.

Old Sysadmin Dog and New Monitoring Tricks

For as long as I’ve been paid to know stuff about computers, I’ve been a fan of Zabbix (http://www.zabbix.com). It’s been my go-to monitoring application because it’s incredibly powerful, tune-able and isn’t hard to get up and running (especially considering the power and tune-ability).

However, recently I’ve been presented with a new product that is making me re-think my stance that there is one best monitoring application in modern IT. I was introduced to New Relic (http://www.newrelic.com).  Normally this ‘cloudy’, ‘monitoring-as-a-service’ application wouldn’t get my attention.  But I was made to use recently at a customer site and I have to admit that it has some really good points.

Big Bang For Your Network Traffic

New Relic condenses down the data into a JSON object that you POST into their API.  So if you have 1000 characters in your JSON post and you send the data once per minute, you have 1000 bytes of data uploaded per minute per server for your monitoring solution.  1K per minute isn’t too bad for 1 minute granularity with your monitoring application.

For comparison, a Linux Zabbix agent monitoring 40-50 items with various frequency averages 2-3Kbps of data per second to its server.

Simple to Configure

You can monitor anything you like, with a caveat.  As long as you can gather the data as an numeric value (integer/float), then you can upload it to their API and track it over time.  The server plugin is a little thin (I’d imagine because it has to work with any OS), so I wrote my own RHEL-centric plugin (end of personal plug).

Presentation is Key

The New Relic UI is really gorgeous.  It’s also intuitive and pretty powerful. You can customize dashboards for your plugin to present data in ways that make sense to you and your purpose.  All very web 2.0.

It’s Free, as in food.

You can register for a New Relic account at their free level and get a pretty usable monitoring platform. Of course $$ unlocks cooler features.

It’s Not ALL Cupcakes and Sunshine

New Relic is what I call a ‘track and graph’ monitoring application.  Simply, you can track data and graph it to analyze and look for performance trends. With their application-level analytics they can do all sorts of neat stuff. I’ve seen some Java thread analysis that would knock your socks off. But from a DevOps perspective, New Relic can be likened to a lightweight 1-minute granularity replacement for ‘sar’ data and all of those scripts we’ve all written over the years to track stuff.

The alerting is somewhat limited as well.  You can set threshold alerts, but that’s about all.  There’s no staggered alerts and as far as I know, there’s not the ability to script alerts from within New Relic either.


New Relic is not a replacement for a monitoring solution that is as robust as something like Zabbix.  It’s not designed to be, and it doesn’t try to be. From a DevOps perspective (and that’s not even really its best use case), it can be a great way to get useful data on your systems quickly and with very little overhead.  On top of that, a box only needs to be able to use New Relic is outbound web access to their API or to an HTTP/HTTPS web proxy.

My Own Private Cloud Part One – The Hardware


I work from home. That means I often need (or at least desire) to be a relatively self-sustained node. For a lot of my work and research, that means I need a pretty well stocked and stable home lab environment. I’ve recently decided to refresh and improve my home/work lab. This series of posts will document what I have going on now (and why), and what I plan to have going on in the future.

In The Beginning

I love technology. I’m literally paid dollars to stay on top of it and help people get the most out of it. But for a long time my own home lab was a pretty pathetic creation. Before coming to work at Red Hat, it was nothing more than what I could fit on my laptop. Since coming to Red Hat it was a Dell T3500 Workstation running RHEL 6 and KVM. On and off, I would have a Satellite server up to help provision systems faster, but it wasn’t a mainstay.

Initial Hypervisor / Lab System

Welcome the New (to me) blood

***Everything going forward has to be prefaced with the fact that I am a complete and utter cheapskate. </full_disclosure>.***

After hemming and hawing about it for a few months I decided to pull the trigger. I needed 3 things:

  • A control server / utility server
  • A second hypervisor (along with my T3500) so I can high-availability tests with things like Openstack RDO and also survive downtime.
  • A NAS so I can do at least NFS-style sharing

So off to Google shopping I go. I finally decided on the following:

Second Hypervisor

Utility Server

NAS Storage

  • Western Digital MyBook Live 3TB appliance
  • I know. No redundant disks. The stuff I really care about is backed up to multiple ‘net-based services. This lab isn’t to serve pictures and be a media hub for my house. It’s a research and development lab/playground.

Gigabit Switch

  • TrendNet 8-port gigabit switch
  • The first one died in a puff of smoke when i plugged in the power supply. A quick replacement by amazon and its replacement seems to be working really well.

After months on the request list, my company also approved my request for a second hypervisor (a little late, but awesome). So now I have three.

Third Hypervisor

Next Up

So all of the boxes are unpacked and set with the recycling. Now what? Now we get to the fun part is what. It took me a few iterations to arrive at a toolchain that I really liked and worked well. The next few posts will be dedicated to talking about that toolchain and how I got it all set up and glued together. Here’s a pic of my not-very-well-organized-yet lab up and running.

My refreshed Home Computer Lab
My refreshed Home Computer Lab

I <3 OpenStack! Now What do I do?

OpenStack (http://www.openstack.org/) truly has the imagination of a lot of the IT world right now. It has the power to be an incredible tool that changes how modern IT business is performed. It is actually a collection of tools that can run on a single or multiple systems and often have at least some overlapping functionality.

  • OpenStack Compute (code-name Nova) – core project since Austin release
  • OpenStack Networking – core project since Folsom release
  • OpenStack Object Storage (code-name Swift) – core project since Austin release
  • OpenStack Block Storage (code-name Cinder) – core project since Folsom release
  • OpenStack Identity (code-name Keystone) – core project since Essex release
  • OpenStack Image Service (code-name Glance) – core project since Bexar release
  • OpenStack Dashboard (code-name Horizon) – core project since Essex release

Projects Incubated in the Grizzly release:

These tools allow you to see multiple types of hypervisors and storage systems through a single pane of glass. It is already a very powerful tool. The inclusion of the concept of metering and Orchestration in the latest release point in the direction of a solution, but they are (at least today) intended to be gateways into OpenStack more than complete solutions for those problems.

And that’s the problem that Enterprises will soon be facing. Even with Red Hat working on their supported version of OpenStack whose community is called RDO (http://openstack.redhat.com/Main_Page), OpenStack doesn’t provide all of the abilities that are needed to truly manage an enterprise installation and achieve a true ROI. So how do you do that?

A company has to be truly aware of what’s going on in their datacenters to really make money using a hybrid cloud approach at the level OpenStack can do it. It’s a matter of looking at a price sheet to see how much a given system would cost on Amazon for a length of time. But how much does it cost to add that same machine to one of your internal hypervisors? A company would have to be truly aware of heating and colling costs, and the changes made by increased load on a given hypervisor system to be able to accurately charge back cost to its various clients.

Once a company is aware at that level, there are a few ways to go. The one I’m currently very impressed with is from Red Hat. It’s the 2.0 version of their CloudForms product (http://www.redhat.com/solutions/cloud-computing/red-hat-cloud/private-clouds-cloudforms.html). When Cloudforms 1.0 came out I didn’t really see the goal or the point. However, Red Hat purchased a company called ManageIQ last year, and set their team on integrating their product in and working on the now-released 2.0 product.

It’s pretty nice for several reasons:

  • It’s delivered easily as a virtual appliance (a first for Red Hat)
  • It gives you the single pane of glass to automate and meter all of your virtualization environments
  • It uses no hacky access methods. No database hacks. It goes through the front door and manages everything properly.
  • It’s extremely easy to build out complex logic-driven events within your infrastructure

Laying CloudForms on top of OpenStack and your virtualization solutions gives enterprise users an incredibly powerful management platform for managing a truly hybrid cloud through a very few “panes of glass”.

1 – from http://www.openstack.org/software/roadmap/

taking my toys and going to the sandbox

One of the coolest new toys that I was recent told about that’s available in Linux (at least Fedora/RHEL 6, not sure about Debian-based stuff or Suse) is the sandbox application. A massive hat tip to Dan Walsh, the author of this app, for showing it off in a recent SELinux talk.

From its man page:

Run the cmd application within a tightly confined SELinux domain. The default sandbox domain only allows applications the ability to read and write stdin, stdout and any other file descriptors
 handed to it. It is not allowed to open any other files. The -M option will mount an alternate homedir and tmpdir to be used by the sandbox.
If you have the policycoreutils-sandbox package installed, you can use the -X option and the -M option. sandbox -X allows you to run X applications within a sandbox. These applications will start
 up their own X Server and create a temporary home directory and /tmp. The default SELinux policy does not allow any capabilities or network access. It also prevents all access to the users other
 processes and files. Files specified on the command that are in the home directory or /tmp will be copied into the sandbox directories.
If directories are specified with -H or -T the directory will have its context modified with chcon(1) unless a level is specified with -l. If the MLS/MCS security level is specified, the user is
 responsible to set the correct labels.

That all sounds well and good. But how does this help me? WELL… the first one that popped into my head is opening up documents from the web that I’m not really sure about. A PDF from someone or some site that you think might contain a script that does bad things, but I REALLY want to look at it. Take a peek in a sandbox!

$ sandbox -X firefox ~/test.pdf

and *poof*, Firefox opens up the PDF for me. BUT, when looking through this instance of Firefox can’t see my home directory contents, or my normal temp folder. It’s also running inside it’s own X server instance. And most importantly, the instance is running in an extremely limited SELinux context.

So if this PDF contains a nasty of some sort that is trying to read my plaintext file that contains my web passwords and creditcard data, it can’t see it to get to it.

It’s all nice and easy and locked down and safe(r) and way easier than creating a chroot jail.

Is Ubuntu getting itself ready for the big F word?

FULL DISCLOSURE: I’m an active member of the Fedora Project community, and I also happen to work at Red Hat, Inc. (not in a code development engineering role).

No.  Not that F word. In an open source project, especially one with large corporate backing, there’s an even worse utterance out there…


Is the Ubuntu project setting itself up for a fork down the road? 

Obviously if I could tell the future I would be doing much more productive things than sorting through Canonical’s messy handling of their Linux distribution. Winning on the ponies at Aqueduct jumps to mind. At any rate, I have no idea if the SABDFL will hold his ship together or not, but there is evidence to suggest that it could easily be heading for a messy fork in the river.

What is Upstream? What is Downstream?

This has always confused me, but where does “derived” fit in the standard model of how an open source project lives. It’s derived from Debian, but what is the actual relationship? Is it upstream from Debian, Downstream? Were they just standing in line together at the DMV one Saturday? It’s a small issue, but it nags at me to no end.

The “Community” Organization

Back in the days of Dapper Drake, it was funny to refer to Mark Shuttleworth as the “Self-Appointed Benevolent Dictator For Life”. It was cute. It was quirky. It was sort of orangey-brown. Just like Ubuntu. But Ubuntu was about togetherness and community, and it was the coolest Linux distro out there at the time. Upstream drivers and a 6-month release cycle and holy crap, it supports my video card!

Ubuntu’s dictator also has the large task of staffing Ubuntu’s Community and Technology governing entities.

The Community Council’s charter is to:

The social structures and community processes of Ubuntu are supervised by the Ubuntu Community Council. It is the Community Council that approves the creation of a new team or project, along with team leader appointments. The council is also responsible for the Code of Conductand tasked with ensuring that community members follow its guidelines.

The Technology Board is responsible for:

The Ubuntu Technical Board is responsible for the technical direction that Ubuntu takes. It makes decisions on package selection, packaging policy, installation systems and processes, kernel, X server, library versions and dependencies. The board works with the relevant team to try to establish a consensus on the right direction to take.

Fast forward to 2012 and Canonical is tryin to monetize Ubuntu, squeezing it into anything that someone has the guts to ask them about. Sadly squeezing into places where Linux itself has been for ages is the most common use case I’ve been able to find (Ubuntu TV and Ubuntu in your car).  You also have 8 years of Mark Shuttleworth picking the people on and direction of the two major governing bodies within Ubuntu itself. Fun examples of this attempt to monetize Ubuntu can be found in the latest releases Amazon “integration” (shame on you, Amazon) and also in this bug, talking about searches run from the Unity dash.


I’ve used Unity for a grand total of 8 minutes. But I know that one of two scenarios about it is true:

1. the minority of unsatisfied Unity users is exponentially more vocal than the satisfied majority


2. there are a LOT of Ubuntu users out there that are NOT HAPPY WITH UNITY.

A search of Mark Shuttleworth’s blog for “unity” shows one early conceit that the original versions “sucked”, but that they are now “well positioned”. Whatever the community wants, it looks like Unity isn’t going anwwhere except into Ubuntu.

Pulling Bits out of the Community’s Hands

I’ve seen spin on this article and the blog post that caused it to be written, but I can only read it one way that makes sense to me.

Canonical (read Mark Shuttleworth) believes that a small group of people in a closed environment can do better work than a large community.

I’ve “done” FLOSS for a while now, and if there is a single immutable truth it is this: Open Source is noisy and often messy. But it’s the noise and mess where you find the genius that changes the world. The off-hand idea in a minor listserv. The idea floated as “impossible” or “impractical”, just like the GCC was back in the day.

I don’t care if he releases all of the code under the GPL after he makes it. When he decided to create skunkworks teams of his hand-selected people, Ubuntu stopped being a community project. And maybe it never was. I’m not saying that Ubuntu is invalidated as a product because it’s not community-driven. I’m just saying stop talking the talk. One thing I do know is that if happens within Fedora (a little response to this), it happens out in the light of day, on a mailing list or in IRC or in the web tools. If I were a contributor to the Ubuntu project I’d be seriously thinking about offering up my time and talents.