VMWare – A Cautionary Tale for Docker?

Of course VMWare has made a ton of money over the last ~12 years. They won every battle in ‘The Hypervisor Wars‘.  Now, at the turn of 2015 it looks to me like they’ve lost the wars themselves.

What? Am I crazy? VMWare has made stockholders a TON of money over the years. There’s certainly no denying that. They also have a stable, robust core product. So how did they lose? They lost because there’s not a war to fight anymore.

Virtualization has become a commodity. The workflows and business processes surrounding virtualization is where VMWare has spent the lion’s share of their R&D budgets on over the years. And now that is the least important part of virtualization. With kvm being the default hypervisor for OpenStack, those workflows have been abstracted higher up the Operations tool chain. Sure there will always be profit margins in commodities like virtualization. But the sizzle is gone. And in IT today, if your company doesn’t have sizzle, you’re a target for the wolves.

Of course docker and VMWare are very different companies. Docker, inc. has released its code as an open source project for ages. They also have an incredibly engaged (if not always listened to) community around it. They had a the genius idea, not of containers, but of making containers easily portable between systems. It’s a once-in-a-lifetime idea, and it is revolutionizing how we create and deliver software.

But as an idea, there isn’t a ton of money in it.  Sure Docker got a ton of VC to go out and build a business around this idea. But where are they building that business?

I’m not saying these aren’t good products. Most of them have value. But they are all business process improvements for their original idea (docker-style containers).

VMWare had a good (some would call great) run by wrapping business process improvements around their take on a hypervisor. Unfortunately they now find themselves trying to play catch-up as they shoehorn new ideas like IaaS and Containers into their suddenly antiquated business model.

I don’t have an answer here, because I’m no closer to internal Docker, Inc. strategy meetings than I am Mars. But I do wonder if they are working on their next great idea, or if they are focused on taking a great idea and making a decent business around it. It has proven to be pennywise for them. But will it be pound-foolish? VMWare may have some interesting insights on that.


Virtualization vs. Containers – a fight you should never have

I have some time to while away before I get on to a plane to head back home to my amazing wife and often-amazing zoo of animals. Am I at the Tokyo Zoo, or in some ancient temple looking for a speed date with spiritual enlightenment? Of course not. I came to the airport early to find good wifi and work on an OpenStack deployment lab I’m running in a few weeks with some co-workers. Good grief, I’m a geek. But anyway.

Auburn the beagle in her natural habitat
Auburn the beagle in her natural habitat

But before I get into my lab install I wanted to talk a little bit about something I saw way too much at OpenStack Summit. For some reason, people have decided that they are going to try to make money by making Linux Containers ‘the next Virtualization’. Canonical/Ubuntu is certainly the worst example of this, but they are certainly not the only one. To repeat a line I often use when I’m talking about the nuts and bolts of how containers work:

If you are trying to replace you virtualization solution with a container solution, you are almost certainly doing both of them wrong.

First off, at the end of the day it’s about money. And the biggest sinkhole inside a datacenter is not fully utilizing your hardware.

Think about how datacenter density has evolved:

  1. The good ole days – a new project meant we racked a new server so we could segregate it from other resources (read: use port 80 again). If the new project only needed 20% of the box we racked for it, we just warmed the datacenter with the other 80%.
  2. The Virtualization Wars – Instead of a new pizza box or blade, we spun up a new VM. This gives us finer grain control of our resources. We are filling in those resource utilization gaps with smaller units. So that same 20% could be set up multiple times on the same pizza box, giving up closer to 100% resource consumption. But even then, admins tended to err on the side of wasted heat, and we were only using a fraction of the VM’s allocated resources.
  3. The Golden Age of Containers – Now we can confidently take a VM and run multiple apps on it (zomg! multiple port 80s!) So we can take that VM and utilize much more of it much more of the time without the fear that we’ll topple something over and crash a server or a service.

This is where someone always shoves their hand in the air and says

<stinkeye>But I need better performance than a VM can give me so I’m running MY CONTAINERS on BAREMETAL.</stink_eye>

My response is always the same. “Awesome. Those use cases DO exist. But what performance do you need?”.

Here’s the short version.

A properly tuned KVM virtual machine can get you withing 3-4% of bare-metal speed.

Leaving out the VM layer of your datacenter means that once you consume that extra 3-4% of your baremetal system that KVM was consuming, you have to go rack another system to get past it. You lose a lot of the elastic scalability that virtualization gives you. You also lose a common interface for your systems that allow you to have relatively homogeneous solutions across multiple providers like your own datacenter and AWS.

Containers bring some of that flexibility back. but they only account for the Dev side of the DevOps paradigm. What happens when you need to harden an AWS system and all you care about are the containers?

Secondly, hypervisors and Linux containers are FUNDAMENTALLY DIFFERENT TECHNOLOGIES.

A hypervisor virtualizes hardware (with QEMU in the case of KVM), and runs a completely independent kernel on that virtual hardware.

A container isolates a process with SELinux, kernel namespaces, and kernel control groups. There are still portions of the kernel that are shared among all the containers. And there is ONLY ONE KERNEL.

All of that to say they are not interchangeable parts. They may feel that way to the end user. But they don’t feel that way to the end user’s application.

So take the time to look at what your application needs to do. But also take some time to figure out how it needs to do it. All of the use cases are valid under the right circumstances.

  • Straight virtualization
  • Straight baremetal
  • containers on VMs
  • containers on baremetal

A well thought out It infrastructure is likely to have a combination of all of these things. Tools like OpenStack, Project Atomic, and CloudForms make that all much much easier to do these days.

Open Stack Summit Day 3 – Closing Thoughts

wow. I’m exhausted.

OpenStack Summit 2015, Tokyo Edition is over. It was amazing. I have a handful of ideas for follow up technical posts after I have time to get home and dig into them a little bit. But I want to get a few thoughts down on the conference as a whole while I’m sitting in my incredibly small room in Tokyo being too tired to go out on the town.

There could have been a container summit inside OpenStack Summit. Everywhere I turned, people were talking about containers. How to use them effectively and innovate around scaling them. It was awesome. These 2 technologies (IaaS and Containers) are going going to collide somewhere not very far up the road. When they do it is going to be something to behold. I can’t wait to be part of it.

The conference on the whole was incredible. I can’t give enough credit to the team who put it all together. It was stretched out across (at least) 4 buildings on multiple floors, and it worked the vast majority of the time. The rooms were a little over-crowded for the biggest talks (or any talk that had the words ‘container’ or ‘kubernetes’ or ‘nfv’ in the title), and they tended to be a little too warm. The warm seems to be common for most public areas in Japan. I guess that’s just how they roll here.

Probably my biggest criticism of the conference is angled at most of the keynote speakers. They were, on the whole, not great. When I am at a large IT conference like this, I expect the keynote presentations to be motivational and polished. Too many of these were history lessons and needed a few more rounds in front of a mirror. There were exceptions of course (particular kudos to the IBM BlueBox folks!). But that was my biggest ‘needs improvement’ factor the OpenStack Summit Tokyo.

Out of 10, I would give this conference a solid 8. My score for Tokyo would be similar, if not higher.

I can’t wait to see what happens in Austin. I’m already working on ideas for talks. 🙂

OpenStack Summit Day 2 – Wrapping my head around NFV

Day 3 isn’t technically over yet. But I’m exhausted. And jet lag is hard. So I’m sitting in one of the conference hotels in a very low chair with my laptop in my lap. Don’t judge.

One of the biggest ideas at OpenStack Summit this year is NFV (Network Function Virtualization). A straw poll of the talk descriptions reveals approximately 213245 talks on the topic this week in Tokyo.

A thousand years ago I worked for a crappy phone company in Richmond and I got to know how a telephone company works. I got to spend some time in telephone central offices and helped solve carrier-level problems. With NFV, I understood why people wanted to get into that world (there’s a TON of money sitting in those dusty places). But I didn’t quite understand the technical plan. What is going to be virtualized? Where is the ceiling for it? It just didn’t make engineering sense to me.

To help combat that ignorance I’ve gone to 4 or 5 of those 213245 sessions today. I’ve also asked pretty much every booth in the Marketplace ‘how are we doing this stuff?’. At the HP Helion booth, I got my engineering answer. My disconnect was that the logistics of defining a big pipe (OC-48 or something like that) in software would just be an exercise in futility. Going all the way up the software stack with that many packets would require a horizontal scale that wasn’t cost-effective.

Of course that’s not the goal. There IS a project name CORD (Central Office Re-imagined as a Datacenter) that is intriguing. But it’s also very new, and mostly theory at this point.

But If we can take some of the equipment that is currently out in the remote sites (central offices, cell towers) and virtualize it, we can then move it into the datacenter instead of having it out in the wild. That makes maintenance and fault-tolerance a lot cheaper. It also saves on man-hours since you don’t have to get in a truck and drive out to the middle of nowhere to work on something.

Another added benefit is that it would disrupt the de facto monopoly that currently exists with the companies that provide that specialized equipment. Competition is a good thing.

That’s the gist. And it’s a good one. We can take commodity hardware and use it to virtualize specialied equipment that normally lives in the remote locations. And we can virtualize it in a datacenter that’s easier and cheaper to get to.

OpenStack Summit Day 1 – The Big Tent is BIG, and Tokyo Lessons Learned

My morning started off with a few lessons learned about being in Tokyo, where I speak 0 words of the language.

  • have cash, Japanese cash, if you plan on getting on Tokyo public transit. After learning this lesson I spent 45 minutes looking for a 7-11 (they use Citi ATMs which apparently easier for us gringoes) before getting in a cab to get me to the Summit on time. We passed 4 7-11’s in the first 1/2 mile of my trip. Of Course.

    This guy laughed at me multiple times
    This guy laughed at me multiple times
  • It is a serious walking city. the walking. omg the walking. and then the walking.

But on to the OpenStack Summit stuff, of which there is a lot.

After getting registered with the required keynote addresses. They are all on the schedule, so I won’t go into the who and what, but a few observations.

  1. The production quality is incredibly high. Like giant tv cameras on platforms high. Like 5 big monitors so us in the back can see too, high.

    big crowd filing in before the keynotes started
  2. The speakers were, on the whole, a little unpolished. They usually had good things to say, but could have used a few more dry runs for a crowd this big.
  3. ZOMG the crowd. Well over 5000 people from 56 countries. The big tent really is big these days. It is awesome, in a word. It is also the most inclusive conference I’ve ever attended. That is also very awesome.
  4. Double ZOMG THE HEAT. The conference is stretched out over 3 (4?) hotels plus a conference center. All of the thermostats seem to be set on ~81 Farenheit (Celsius?). Take that and toss in an overcrowded room full of sweaty geeks and things can get a little uncomfortable. Especially in the middles of the aisles. Especially especially after lunch.
  5. The Marketplace (vendors tables) is utter chaos. With that said, Mirantis easily wins this year. They have
  6. There is now an OpenStack Certification. Or there will be soon, at least. You can be a Certified OpenStack Administrator (COA). I don’t know how this is going to play with the existing Red Hat Certification, but I’m interested in finding out.
  7. Openstack has a new way of visualizing it constituent parts. http://www.openstack.org/software is way WAY better than the old wiki-style nastiness.
  8. Bitnami COO Erica Brescia took some pretty awesome shots at Docker Hub and its lack of curation. It’s the wild west out there, and it comes with consequences. I’m not a huge fan of Bitnami. But I am a huge fan of how Erica Brescia does her job.

My least favorite observation on the day was Canonical’s slogan for LXD. They had an ad on the spashes before the keynotes started and it was something along the lines of “Ubuntu/Canonical has the fastest hypervisor on the planet with lxd $something $something $something”

Hey Canonical, you are aware that containers and virtual machines are different things, right? So are you trying to re-define the word, or are you trying to pass off a container manager as a hypervisor? Huh? At any rate, it’s an awful slogan and even worse marketecture. I’m debating a drive by of their booth tomorrow.

After lunch I went to a talk held by Mirantis where the compared a base install of their offering to a GA(ish?)-release of RHEL OSP 7. They were more fair and balanced than I thought they would be. Their product, Fuse, is 3 or 4 years old at this point and very polished. OSP 7 uses OSP Director, which is based on TripleO. OSP 7 is Red Hat‘s first release based on this installer. It suffers from exactly the warts you think it would.

With that said, I was surprised they had to pic some pretty small nits to make their presentation work. A lot of their documentation issues were already addressed. But they correctly identified the biggest areas of need for OSPd as Red Hat works to mature it in OSP 8 and beyond.

All in all Day 1 was great fun. I’m looking way forward to Day 2. On top of that I’m PRETTY SURE I can get to and return from the conference using Tokyo Public Transport.

OpenStack Summit Day 0 – a mode B train ride

This year I get to go to Open Stack Summit in Tokyo. It will be my first time visiting Japan. Right now I am in a very small hotel room at 3am (local time), wide awake because I went to bed at 7pm. Such is jet lag, I guess. My personal goal for this even is to create a short post daily with some initial thoughts / reactions / fun things I learned.

Day 0 was just getting here. I got on a plane in Richmond, VA just before 9am Eastern on Sunday morning. I got off a plane an hour outside of Tokyo at 3:40pm Monday. I was in the air for ~16 hours total. Time zones and datelines will forever befuddle me. I took the Narita Express train from the airport to downtown Tokyo.

The practical thing I learned is that GPS on your phone SUCKS in downtown Tokyo. I was going to walk the ~1 mile from Tokyo Station to my hotel. Google Maps took me in 4 different directions while thinking I was on 3 different roads before I gave up and got into a cab. I’m pretty sure I would still be walking if not for that nice man.

The other thing that jumped out at me was during the train ride in. Space is so much more utilized in Japan. At first I thought it was just sort of stacked up and haphazard. But as I rode by it I began to see the organization and beauty in how the space in the Tokyo area is utilized. It’s pretty amazing.

It made me start thinking about my own house and the 5 acres of trees that it sits on. Not in a better/worse sort of way. Obviously I have different goals than someone who lives near downtown Tokyo. But when I give a talk about containers I talk a lot about them being the ‘next layer of density’ in computing. Bimodal IT is one of the biggest concepts in that area.

Over the next few days, I will definitely be a mode A guy walking around in a mode B country. Wish me luck!


Multi-Node OpenStack on your laptop in about an hour

OpenStack is just about the hottest thing going in IT today.  Started in 2010 as a joint project between Rackspace and NASA, it is quickly maturing into the premier solution for anyone who wants to make their infrastructure more self-service without having to go behind and clean up after developers all day every day.

It’s biggest hurdle is still its learning curve and the associated pain you are almost guaranteed to suffer during your first install.  After approximately 243535 tries, I have a pretty solid process to stand up an OpenStack demo setup across multiple nodes on your laptop. It could also be altered (if any alterations are even needed) to deploy it across multiple physical or virtual systems in any environment.

Depending on your bandwidth to patch the servers, it takes about an hour, soup to nuts.

Which Flavor?

RHEL OSP 7 comes with an awesome tool called OSP Director, which is based on TripleO. It is essentially a canned OpenStack install that you then use to deploy production OpenStack nodes. It’s called an ‘undercloud’.

For two reasons, I’m not using OSP Director for this demo.

  1. It takes more time and more resources. If I were doing this in my environment and I was an everyday engineer, I’d totally use it. But this is an exercise in tire-kicking.
  2. I haven’t had time yet to play with it very much.

Instead I’m using RDO’s Quickstart tool, which is based on Packstack.

OpenStack in 60 seconds

The goal when OpenStack was started was to engineer an FOSS alternative to Amazon Web Services. What they came up with were an ever-growing list of services that each perform a task required (or that is optionally neat) to build out a virutalized infrastructure.

The services are all federated together with RESTful APIs. Python is the language of choice.

Core Services

  • Nova – compute services. The core brains of the operation and the initial product
  • Neutron – Software-defined Networking service. Nova also has some less flexible networking components built in.
  • Cinder – provides block devices to virtual machines (instances in OpenStack parlance)
  • Glance – manages images used to create instances
  • Swift – provides object/blob storage
  • Keystone – Identity and Identification services for all other services as well as users
  • Horizon – a Django-based web frontend that’s customizable and extensible
  • Heat – Orchestration services
  • Ceilometer – Telemetry

Optional Fun Services

  • Trove – Database-as-a-service
  • Ironic – Bare metal provisioning – treat your racked stuff like your virtual stuff
  • Elastic Map Reduce – Big-Data-as-a-Service (?!?!)
  • $instert_awesome_project_here

All Openstack modules that are currently canon are listed on their roadmap.


Cluster Setup

The demo setup I’ll be going through was setup on my laptop (Fedora 21) using good ol’ KVM Virtual Machines running RHEL 7. The laptop has 8 cores and 16GB of RAM total.

  • rdo0.localdomain ( – 4GB RAM, 2 VCPU
    • Controller Node (all services except Nova Compute)
  • rdo1.localdomain ( – 2GB RAM, 2 VCPU
    • Nova Compute Node
  • rdo2.localdomain ( 2GB RAM, 2 VCPU
    • Nova Compute Node

Host OS Setup

NOTE – since these are VMs, the single NIC I assigned them was designated eth0. We all know the naming convention has changed in RHEL 7

subscription-manager register --username=$RHN_USERNAME --password=$RHN_PASSWORD
subscription-manager attach --pool=$SUBSCRIPTION_MANAGER_POOL_ID
subscription-manager repos --disable=\* --enable=rhel-7-server-rpms --enable=rhel-7-server-optional-rpms --enable=rhel-7-server-extras-rpms
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -y
sudo yum install -y https://rdoproject.org/repos/rdo-release.rpm
yum install -y openstack-packstack vim-enhanced
yum update -y
systemctl stop NetworkManager
systemctl disable NetworkManager
systemctl enable network
#confirm the network setup is working
ifdown eth0 && systemctl start network && ifup eth0 
#reboot to apply any patches that require it, etc.

The above snippet will

  • register your system to Red Hat via subscription-manager
  • attach the proper subscription pool (supplied by you)
  • enable the needed channels
  • install the RDO package repository
  • install a few things (I’m a Vim guy, feel free to edit)
  • disable NetworkManager (required for OpenStack) and replace it with a legacy network service script
  • activate the new network setup

Once this is set up on each host (I did it on one and cloned it twice to create the new other VMs) , you are ready to get OpenStack rolling.

Creating an answers file

On rdo0.localdomain, run the following command. It willThe next command will generate a default answers file that you can then edit and keep up with over time as you deploy various OpenStack incarnations.

packstack --gen-answer-file rdo.txt

The following changes were made.

NOTE – if you create 2 answer files and diff them, you will see many other changes, as passwords are randomized each time.

# diff -u rdo.txt rdo-edited.txt 
--- rdo.txt 2015-08-23 15:41:45.041000000 -0400
+++ rdo-edited.txt 2015-08-21 20:17:05.538000000 -0400
@@ -64,7 +64,7 @@
 # Specify 'y' to install Nagios to monitor OpenStack hosts. Nagios
 # provides additional tools for monitoring the OpenStack environment.
 # ['y', 'n']
 # Comma-separated list of servers to be excluded from the
 # installation. This is helpful if you are running Packstack a second
@@ -84,7 +84,7 @@
 # List of IP addresses of the servers on which to install the Compute
 # service.
 # Specify 'y' to provision for demo usage and testing. ['y', 'n']

And then you run packstack. Depending on your install targets and have much horsepower is available, this can take a while. On my laptop, it takes the better part of an hour.

packstack --answer-file=rdo.txt

Getting Networking Working

The next part of the setup will borrow heavily from This RDO blog post about setting up Neutron with an existing network.

After packstack does its thing, assuming you have a ‘Success!’ sort of output on your screen, you will then have a 3-node OpenStack cluster with 2 Nova Compute nodes and 1 node doing pretty much everything else. Unfortunately, out of the box you need to make a few tweaks so you can see your new instances from your libvirt networking (or the network in your lab or whatever your use case is).

NOTE – this needs to happen on each host

Create your bridge and set up your NIC

On my VMs the only NIC is named eth0 (benefit of using a VM). So you may need to edit this slightly to fit your set up’s naming conventions.

We want to use a bridge device to get our VMs on to our network so we create a device named br-ex. We then edit $YOUR_NIC to

[root@rdo0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br-ex 
IPADDR= # Old eth0 IP since we want the network restart to not 
 # kill the connection, otherwise pick something outside your dhcp range
NETMASK= # your netmask
GATEWAY= # your gateway
DNS1= # your nameserver

[root@rdo0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 

Tell Neutron about your bridge

We then run the following to tell Neutron to use a bridge called ‘br-ex’, and to use the proper plugins

openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini ovs bridge_mappings extnet:br-ex
openstack-config --set /etc/neutron/plugin.ini ml2 type_drivers vxlan,flat,vlan

You could probably restart Neutron and be OK here, but I prefer the belts and suspenders.

Define your software network

After the reboot, you should be able to ssh back into your normal IP address. We now have a host infrastructure that is ready to serve our OpenStack instances. So let’s define our SDN components so we can get going!

NOTE – This should be done on your controller node, rdo0.localdomain in my case

Provider network

# source keystonerc_admin
# neutron net-create external_network --provider:network_type flat --provider:physical_network extnet --router:external --shared

Public subnet and router

# neutron subnet-create --name public_subnet --enable_dhcp=False --allocation-pool=start=,end= \
 --gateway= external_network
# neutron router-create router1
# neutron router-gateway-set router1 external_network

Private subnet

# neutron net-create private_network
# neutron subnet-create --name private_subnet private_network

Connect the two networks with the router

# neutron router-interface-add router1 private_subnet

Wrapping Up

And that’s it! You should now have 3 nodes ready to handle your OpenStack demo loads.

My current plan is to keep evolving this setup and re-deploying to do things like

  • take advantage of Ceph
  • Further distribute loads (maybe a virtual/physical hybrid setup)
  • handle multiple NICs
  • ???

If you have grief or ideas for improvement please feel free to comment below.