Category: vmware

Containers and persistent storage

Containers and persistent storage

Containers are a method of operating system virtualization that allow you to run an application and its dependencies in resource-isolated processes. Containers allow you to easily package an application’s code, configurations, and dependencies into easy to use building blocks that deliver environmental consistency, operational efficiency, developer productivity, and version control. Containers are immutable, meaning they help you deploy applications in a reliable and consistent way independent of the the deployment environment.

As containers continue to rise in popularity beyond the developer populous the way these constructs are being used becomes increasingly varied, especially (but not exclusively) in light of enterprise applications the questions of persistent storage comes up more and more. It is a fallacy to think only stateless application can or should be containerized. If you take a look at you’ll see that about half of the most popular applications on Docker Hub are stateful, like databases for example. (Post hoc ergo propter hoc?). If you think about monolithic applications versus micro-services, a monolithic application typically requires state, if you pull this application out into micro-services, some of these services can be stateless containers but others will require state.

I’ll mainly use Docker as the example for this post but many other container technologies exist like LXD, rkt, OpenVZ, even Microsoft offers containers with Windows Server Containers, Hyper-V isolation, and Azure Container Service.

Running a stateless container using Docker is quite straightforward;

$ docker run --name demo-mysql mysql

When you execute docker run, the container process that runs is isolated in that it has its own file system, its own networking, and its own isolated process tree separate from the (local or remote) host.

The docker container is created from a readonly template called docker image. The “mysql” part in the command relates to this image, i.e. containerized application, that you want to run by pulling it from the registry. The data you create inside a container is stored on a thin writable layer, called the container layer, that sits on top of the stack of read-only layers, called the image layers, present in the base docker image. When the container is deleted the writable layer is also deleted so your data does not persist , in docker the docker storage driver is responsible for enabling and managing both the read-only image layers and the writable layer, both read and write speeds are generally considered slow.

Image result for docker layers

Assuming you want persistent data for your containers there are several methods to go about this. You can add a storage directory to a container’s virtual filesystem and map that directory to a directory on the host server. The data you create inside that directory on the container will be saved on the host, allowing it to persist after the container shuts down. This directory can also be shared between containers. In docker this is made possible by using volumes, you can also use bind mounts but these are dependent on the directory structure of the host machine whereas volumes are completely managed by Docker itself. Keep in mind though that these volumes don’t move with container workloads as they are local to the host. Alternatively you can use volume drives (Docker Engine volume plugins) to store data on remote systems instead of the Docker host itself. If you are only interested in storing data in the container writeable layer (i.e. on the docker host itself) you can use Docker storage drivers which then determine which filesystem is supported.

Typically you would create a volume using the storage driver of your choice in the following manner;

$ docker volume create -—driver=pure -o size=32GB testvol1

And then start a container and attach the volume to it;

$ docker run -ti -v testvol1:/data mysql

Storage Vendors and Persistent Container Storage

Storage vendors have an incentive to make consuming their particular storage as easy as possible for these types of workloads so many of them are providing plug-ins to do just that.

One example is Pure Storage who provide a Docker Volume Plugin for their FlashArray and FlashBlade systems. Current they support Docker, Swarm, and Mesos. Most other big name storage vendors also have plugins available.

Then there are things like REX-Ray which is an open source, storage management solution, it was born out of the now defunct {code} by Dell EMC team. It allows you to use multiple different storage backends and serve those up as persistent storage for your container workloads.

On the virtualization front VMware has something called the vSphere Docker Volume Service which consists of two parts, the Docker Volume Plugin and a vSphere Installation Bundle (VIB) to install on the ESXi hosts. This allows you to serve up vSphere Datastores (be it Virtual SAN, VMFS, NFS based) as persistent storage to your container workloads.

Then there are newer companies that have been solely focusing on providing persistent storage for container workloads, one of them is Portworx. Portworx want to provide another abstraction layer between the storage pool and the container workload. The idea is that they provide a “storage ” container that can then be integrated with the “application” containers. You can do this manually or you can integrate with a container scheduler like Docker Swarm using Docker Compose for example (Portworx provides a volume driver).

Docker itself has built specific plugins as well, Cloudstor is such a volume plugin. It comes pre-installed and pre-configured in Docker swarms deployed through Docker for AWS. Data volumes can either be backed by EBS or EFS. Workloads running in a Docker service that require access to low latency/high IOPs persistent storage, such as a database engine, can use a “relocatable” Cloudstor volume backed by EBS. When multiple swarm service tasks need to share data in a persistent storage volume, you can use a “shared” Cloudstor volume backed by EFS. Such a volume and its contents can be mounted by multiple swarm service tasks since EFS makes the data available to all swarm nodes over NFS.

Container Orchestration Systems and Persistent Storage

As most enterprise production container deployments will utilize some container orchestration system we should also determine how external persistent storage is managed at this level. If we look at Kubernetes for example, that supports a volume plugin system (FlexVolumes) that makes it relatively straightforward to consume different types of block and file storage. Additionally Kubernetes recently started supporting a implementation of the Container Storage Interface (CSI) which helps accelerate vendor support for these storage plug-ins as volume plugins are currently part of the core Kubernetes code and shipped with the core Kubernetes binaries meaning that vendors wanting to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) must align themselves with the Kubernetes release process. With the adoption of the Container Storage Interface, the Kubernetes volume layer becomes extensible. Third party storage developers can now write and deploy volume plugins exposing new storage systems in Kubernetes without having to touch the core Kubernetes code.

When using CSI with Docker, it relies on shared mounts (not docker volumes) to provide access to external storage. When using a mount the external storage is mounted into the container, when using volumes a new directory is created within Docker’s storage directory on the host machine, and Docker manages that directory’s contents.

To use CSI, you will need to deploy a CSI driver, a bunch of storage vendors have these available in various stages of development. For example there is a  Container Storage Interface (CSI) Storage Plug-in for VMware vSphere.

Pre-packaged container platforms

Another angle how vendors are trying to make it easier for enterprises to adopt these new platforms, including solving for persistence, is by providing packaged solutions (i.e. making it turnkey), this is not new of course, not too long ago we saw the same thing happening with OpenStack through the likes of VIO (VMware Integrated OpenStack), Platform9, Blue Box (acquired by IBM), etc. Looking at the Public Cloud providers these are moving more towards providing container as a service (CaaS) models with Azure Container Service, Google Container Engine, etc.

One example of packaged container platforms though is the Cisco Container Platform. This is provided as an OVA for VMware (meaning it is provisioning containers inside virtual machines, not on bare metal at the moment), initially this is supported on their HyperFlex Platform which will provide the persistent storage layer via a Kubernetes FlexVolume driver. It then can communicate externally via Contiv, including talking to other components on the HX platform like VMs that are running non containerized workloads. For the load-balancing piece (between k8s masters for example) they are bundling NGINX and for logging and monitoring they are bundling Prometheus (monitoring) and an ELK stack (logging and analytics).

Another example would be VMware PKS which I wrote about in my previous post.


Containers are ready for enterprise use today, however there are some areas that could do with a bit more maturity, one of them being storage. I fully expect to see continued innovation and tighter integrations as we figure out the validity of these use-cases. A lot progress has been made in the toolkits themselves, leading to the demise of earlier attempts like ClusterHQ/Flocker. As adoption continues so will the maturity of these frameworks and plugins.

VMware’s Cloud Nativeness

VMware’s Cloud Nativeness

At first glance it feels like VMware’s efforts related to cloud native applications are meant to cater to the traditional admins, people who are mainly running virtual machines today but want, or need, to support other types of workloads as well. People who secretly don’t “get” this new container hotness* and really want to manage their environment with the tools and knowledge that got them where they are today and are very comfortable with. Or to put it another way, VMware is ultimately trying to cater to both the infrastructure/operations person needing to support these new types of workloads and the developer building them.

Before looking at VMware’s portfolio related to cloud native applications let’s take a step back and examine both what CNA means and what drives it. According to Pivotal, Cloud-native is an approach to building and running applications that fully exploits the advantages of the cloud computing delivery model. Cloud-native is about how applications are created and deployed, not where. “Where” meaning the underlying infrastructure is not as important as it once was, granted, virtualisation already decreased our reliance on static infrastructure, but if we are honest with ourselves infrastructure was still the thing we got excited about the most as VMware admins. Depending on where you see yourself on the “Public Cloud” is the only thing that makes sense going forward scale, you’ll be more or less inclined to accept the idea that cloud computing can be considered a delivery model that is appropriate for both public and private cloud like environments alike. Core to the cloud computing model is the ability to offer seemingly unlimited computing power, on-demand, along with modern data and application services for developers. It is this last piece, I believe, where private cloud solutions fall down the most compared to public cloud vendors, the onslaught of services delivered at more and more rapidly increasing pace by companies like Amazon (AWS), Microsoft (Azure), and Google (GCP) are creating an understandable pull, attracting new workloads. Therefore it seems that when we discuss Cloud Native Applications from the lens of VMware, we again tend to mostly argue the needs of these applications related to the infrastructure piece. If we agree that products and services delivered through software are becoming major competitive differentiators of the business then it is precisely this what is driving the need of Cloud Native Applications. This is where Marc Andreesen’s software is eating the world quote usually comes in…

The way VMware describes it’s portfolio for CNA on their own website is;

VMware offers cloud-native solutions across container networking, security, storage, automation, operations and monitoring for enterprises and service providers.

Since VMware is trying hard to reach beyond it’s traditional enterprise base, including forays into multiple open-source type projects, I wanted to try and provide a somewhat comprehensive overview of everything it’s doing in those spaces and it’s potential use cases. Generally speaking VMware’s cloud native portfolio includes VMware vSphere Integrated Containers, for running containerised workloads with existing infrastructure. But please don’t interpret this as a silver bullet to “lift and shift” all your legacy applications to a more modern environment, start by asking why you can’t re-architecture your existing apps, and only then ask yourself if you will truly benefit from moving to this new approach, maybe for easier maintenance or improved scalability, but I’m willing to bet that mostly this wont be the best approach for your old and crusty stuff. For applications that naturally fit a containerised approach the story if different of-course. Secondly VMware Photon Platform is targeted to building new cloud-native infrastructure solutions from the ground up.

VMware VIC - Page 1 (2)

Very simplified view of components fitting together

vSphere Integrated Containers

VIC is meant for people who want to stick with the more traditional vSphere infrastructure but also provide Container based services to internal developers. It is available with vSphere Enterprise Plus 6.0 or later, or with vSphere with Operations Management Enterprise Plus. It consists of 3 main components, the vSphere Integrated Container Engine (VIC Engine), Admiral, the Container Management Portal, and Harbor, the Container Registry (based on Docker Distribution). The portal provides a user interface and an API for managing container repositories, images, hosts, and instances. The registry furnishes a user interface and an API so developers can make their own container repositories and images. The engine is a container runtime integrated with vSphere.

vSphere, as opposed to Linux, acts as the container host, containers are deployed as VMs (not in VMs), every container is fully isolated from the host and from the other containers.

 VIC Marketecture

With VIC the idea is that you can still leverage the existing management toolsets in your vSphere based environment, since these tools typically have an understanding about the virtual abstraction as a VM, the idea is that you run a Container as a VM (C-VM in the picture above). The goal is to run both traditional and containerised applications side-by-side on a common infrastructure, this, if you will, is it’s main use case, take what you know and extent it so you can support container based workloads in a way that is transparent for the developer in theory. It’s meant to avoid having to build a completely separate environment.

VMware Photon Platform 

VMware Photon Platform is a container-optimised cloud platform, at first it seems to have a somewhat similar goal to VIC, which is to enable developers to build, test, and run container based workloads while still allowing “traditional IT” to maintain security, control and performance of data center infrastructure. But the use cases have a different focus, namely;

  • Kubernetes as a Service: Developers can deploy, resize, and destroy Kubernetes clusters to develop, test, and automate containerised applications.
  • Platform as a Service: Photon Platform integrates with Pivotal Cloud Foundry to build and scale applications.
  • Continuous integration and delivery: Try to improve the CI/CD pipeline with uniformity and reusability, especially in environments with high container churn.
  • API-managed on-premises cloud: IT can deploy resources and automate their management through a RESTful API.
  • Security: The VMware Lightwave security service can protect applications, Kubernetes clusters, and Photon Platform components.

Screen Shot 2017-09-14 at 08.45.45.png

Photon Platform Marketecture

Note that the Photon Controller is no longer a part of the Photon Platform, VMware has made the decision to stop driving this project and instead lean on the open source community to continue this effort. It was originally envisioned to replace vCenter as the control plane for these new type of workloads since vCenter’s monolithic design was never expected to serve these types of applications. More recently of-course the community has moved to other control plane architectures and container orchestration systems.

NSX-T as the secret sauce.

With VIC you can select vSphere networks that appear in the Docker client as container networks. NSX-T intends to extent the networking capabilities and make them more enterprise appropriate. By default a container sits behind a NAT’d interface and as such form a operations point of view you struggle with visibility. An other challenge is consuming services outside the container realm, if you have services that need to be consumed from your traditional VMs or even bare metal systems you typically need to pass via bridging nodes potentially choking traffic. NSX-T tries to overcome these challenges by supporting native container networking and micro-segmentation for CaaS (Container-as-a-Service ) and PaaS (Platform-as-a-Service), providing tools with which network and security teams can operationalise containerised apps in a vSphere environment.

Statefulness and persistent storage

When discussing containers and container storage you cannot ignore the difference between stateless and stateful workloads and the impact that has on storage persistency. Containers are ephemeral, so if you want to run stateful applications that require persistent storage you need to take some additional steps to ensure your data is maintained through the lifecycle of the containers. VMware provides two persistent volume offerings for situations like these, one for Docker and one for Kubernetes. For Docker VMware provides the vSphere Docker Volume Service (vDVS), this plug-in abstracts the underlying storage of the vSphere environment and makes it available as Docker volumes, it also integrates with Docker Swarm. For stateful containers orchestrated by Kubernetes you can also leverage persistent vSphere Storage (vSAN, VMFS, and NFS) with Kubernetes persistent volume, dynamic provisioning and StatefulSet primitives. Both of these projects are now under the Project Hatchway umbrella.

Operationalising a Cloud Native environment with VMware

Again going back to using the tools you know as a VMware admin, VMware has tried to extend vRealize Automation to make it more relevant for these new types of workloads, it leverages the open source Project Admiral, a scalable and lightweight container management platform, to deploy and manage containers through virtual container hosts on VMware vSphere Integrated Containers. The idea is still to provide a separation between the operational folks and the developers. Developers can provision container hosts from the vRealize Automation service catalog as well as model containerised applications using unified service blueprints or Docker Compose.

Open Source components

Photon OS

Version 2.0 of Photon OS was announced at the beginning of November, Photon OS is a Container-Optimized Linux Operating System. It is an open source minimal Linux container host optimized for cloud-native applications, cloud platforms, and VMware infrastructure. Think of container Operating Systems in general as an OS that provides only the minimal functionality required for deploying applications inside software containers, but typically including built-in mechanisms for shared services consumption and orchestration. Photon OS even comes in 2 flavours, the minimal version is a lightweight host tailored to running containers when performance is paramount, whilst the full version includes additional packages to help deploy containerized applications. Other notable container OSes include a.o. CoreOS, RancherOS, Atomic, and to some extent Intel Clear Linux.

For easy consumption Photon OS is available as a pre-built image for the 3 main public cloud providers as well as an ISO and OVA image. Photon OS includes the open source version of the Docker daemon. After installing Photon OS and enabling Docker, you can immediately run a Docker container.

I’ve written about Photon (and AppCatalyst, and Bonneville) on this blog before if you want some more background.

vSphere Integrated Containers Engine

vSphere Integrated Containers Engine (VIC Engine) is the key element of vSphere Integrated Container, it is a container runtime for vSphere, allowing developers familiar with Docker to develop in containers and deploy them alongside traditional VM-based workloads on vSphere clusters, and allowing for these workloads to be managed through the vSphere UI in a way familiar to existing vSphere admins. The VIC Engine provides lifecycle operations, vCenter support, logs, basic client authentication, volume, and basic networking support.


Harbor is an enterprise-class container registry server based on Docker Distribution, it is embedded in both vSphere Integrated Containers and Photon Platform, it provides advanced security, identity, role based access control, auditing, and management services for Docker images. With Harbor, you can deploy a private registry to maintain your data behind the company firewall. In addition, Harbor supports AD/LDAP integration and the setup of multiple registries with images replicated between registries for high availability.


Admiral is a key component of vSphere Integrated Containers solution. It is a container management solution with an accent on modelling containerised applications and providing placement based on dynamic policy allocation. It manages Docker hosts, policies, multi-container templates, and applications to simplify and automate resource utilisation and application delivery. Developers can use Docker Compose, Admiral Templates, or the Admiral UI to compose their app and deploy it using the Admiral provisioning and orchestration engine. VMware admins can manage container host infrastructure and apply governance to its usage, including resource grouping, policy based placement, quotas and reservations, and elastic placement zones.

Photon controller

Photon controller is a distributed, multi-tenant host controller optimised for containers. The Photon Controller exposes RESTful APIs, SDKs, and CLI tooling to automate infrastructure resources. It is built to support open container orchestration frameworks such as Kubernetes and Pivotal Cloud Foundry as well virtualised environments. Photon Controller functions as the brain of Photon Platform.

NOTE:  Photon Controller is no longer actively maintained by VMware.


Xenon is a decentralised Control Plane framework for writing small REST-based services (Microservices). The photon controller project makes heavy use of Xenon to build a scalable and highly available Infrastructure-as-a-Service fabric, composed of stateful services, responsible for configuration (desired state), work flows (finite state machine tasks), grooming and scheduling logic.

Project Lightwave 

Project Lightwave offers enterprise-grade, identity and access management services such as single sign-on, authentication, authorization, and certificate authority, as well as certificate key management for container based workloads. It is designed for environments that require multi-tenant, multi-master, scalable LDAP v3 directory services. Lightwave authentication services support Kerberos, OAuth 2.0/OpenID Connect, SAML and WSTrust which enable interoperability with other standards-based technologies in the datacenter.

Photon OS + Project Lightwave

Lightwave is an open source security platform from VMware that authenticates and authorises users and groups with AD or LDAP. Through integration between Photon OS and Project Lightwave, organisations can enforce security and governance on container workloads, for example, by ensuring only authorised containers are run on authorised hosts, by authorised users.

Evolution of the Software Defined Data Center

One of the core tenets of the SDDC is the speed at which you can adapt your IT infrastructure to the needs of the business, so what if those needs are not foreseen in the original plans for the SDDC, in other words what if they go beyond the capabilities of the software that is powering your SDDC environment. To this end VMware is partnering with Pivotal to deliver something called a Developer-Ready Infrastructure. Integrating Pivotal Cloud Foundry (PCF) with VMware SDDC infrastructure solutions. PCF will provide a PaaS to the developer community so they don’t have to worry (or care) about the underlying infrastructure at all, and at the same time the VMware admin is still empowered by his VMware toolset to maintain and provide services to make this PaaS run.

VMware PKS

At VMworld 2017 VMware announced their collaboration with Pivotal to deliver Kubernetes-based container services for multi-cloud enterprises and service providers. It pulls in many of the technologies mentioned before to make a Kubernetes environment more “enterprise grade”, like NSX-T for the networking piece and Project Hatchway for persistent storage. Additionally PKS will include a service broker that provides out-of-the-box access to GCP services. It will enable a VMware admin to expose certain GCP services so that development teams can provision and consume GCP services by creating and managing “service instances” with the kubectl CLI or API. The GCP service broker supports offering GCP subscription services such as Google Cloud Storage, Google BigQuery, and Google Stackdriver. These services will be able to be consumed by applications running on-premises or from within GCP.

In summary, VMware is trying to provide a bridge between multiple different parties, a bridge between developers and VMware admins, a bridge between traditional VMware infrastructure and Public Cloud based services (incidentally, this was also the premise of VMware’s former vCloud Air platform, now sold to French cloud hosting provider OVH.) The question you should pose yourself is, I think, whether it makes sense to incorporate these new architectures, to what extend are your applications unique snowflakes that do not fit this new world of micro-services, have you tended to pets all your life or can some of them be considered cattle. Furthermore it remains to be seen if the pendulum will swing back at some point. Legacy will be here for a long time yet.

*That was tongue in cheek, just in case that wasn’t clear

VMworld US – Sunday

VMworld US – Sunday

This is my 7th VMworld event, but the first time attending VMworld in the US, I’ve been to Vegas before but arriving this Saturday evening it felt different somehow, maybe it was the scorching heat or the onslaught of UFC/Boxing fans at the MGM Grand, both where very “in your face”.


VMware signage welcoming you to Las Vegas

I decided to check out the OpeningActs event at the Beerhaus which was a pretty neat experience. The first panel session, moderated by John Troyer (aka John Trooper) was on VMware’s continued relevance in todays rapidly changing tech world which spurred a lot of Amazon AWS remarks. I think the overall consensus was that VMware might be in “sustainability” mode but that it is still an excellent foundation in a lot of customer environments. “Basic infrastructure” might be boring, but it is of great importance nevertheless. I think the other angle that might have been overlooked a bit here is that Amazon needs this relationship with VMware too, it needs a way into that “boring” enterprise market.


Second panel session was on “How failing made me better”, moderated by Jody Tyrus.
My favourite quote from the session was, and I’m paraphrasing here; “I sometimes still feel like I’m a 12 year old in a body of a 42 year old that is about to be exposed”. Rebecca Fitzhugh wrote and excellent blog on her experience with dealing with failure here.



Unfortunately I had to skip the 3rd session because I needed to get to the Expo for a Rubrik all-hands briefing before the Welcome Reception. After the reception I headed to the VMUG party at the House of Blues, which was pretty cool (because they served Stella Artois 😉 ), Michael Dell made an appearance and talked about his love for the VMUG and hinted at some cool Pivotal announcements coming this week.


After the party I needed to get jet-lagged self to bed. Looking forward to a great VMworld week.



VMware NSX and Palo Alto NGFW

VMware NSX and Palo Alto NGFW

VMware NSX is a platform for network and security virtualization, and as such it has the capability to integrate onto it’s platform certain functionalities that are not delivered by VMware itself. One such integration point is with Palo Alto Networks’s Next-Generation Firewall.

VMware NSX has built-in L2-4 stateful firewall capabilities both in the distributed firewall running directly in the ESXi hypervisor for east-west traffic, and in the Edge Services Gateway VM for north-south traffic. If L2-4 is not sufficient for your specific use case we can use VMware’s NSX Service Composer to steer traffic towards a third party solution provider for additional inspection.

At a high level the solution requires 3 components, VMware NSX, The Palo Alto Networks VM-series VM-1000-HV, and Palo Alto Networks’ central management system, Panorama.

Screen Shot 2015-05-04 at 09.47.20

Currently the VM-1000-HV supports 250.000 sessions (8000 new sessions per second) and 1Gbps firewall throughput (with App-ID enabled). The VM-series firewall is installed on each host of the cluster where you want to protect virtual machines with Palo Alto’s NGFW. Each VM-series firewall takes 2 vCPU’s and 5GB RAM.

Screen Shot 2015-05-04 at 09.53.28

If you look (summarize-dvfilter) at each ESXi host after installation you should see the VM-series show up in the dvfilter slowpath section.

Screen Shot 2015-05-04 at 09.58.59

We can also look at the Panorama central management console and verify that our VM-series are listed under managed devices.

Screen Shot 2015-05-04 at 10.03.06

Deciding which traffic to pass to the VM-series is configured using the Service Composer in NSX. The Service Composer provides a framework that allows you to dictate what you want to protect by creating security groups, and then deciding how to protect the members of this group by creating and linking security policies.

Screen Shot 2015-05-04 at 10.07.13

It is perfectly feasible to use security policy to first enable NSX’s distributed firewall to deal with certain type of traffic (up to layer 4) and only steer other “interesting” traffic towards the Palo Alto VM-series, this way you can simultaneously benefit from the distributed throughput of the DFW and the higher level capabilities of Palo Alto Networks NGFW.

Using the Service Composer, we create a security policy and use the Network Introspection Service to select which external 3rd party service that we want to steer traffic to. In this case we select the Palo Alto Networks NGFW and can further select the source, destination, and specific traffic (protocol/port) that we want to have handled by the VM-series.


Today only the traffic is passed to the external service but it is feasible to pass on more metadata that additionally could be acted upon by the third party provider. For example what if we could pass along that the VM we are protecting is running Windows Server 2003 and thus needs to have certain additional security measures applied.

So now that we have a policy that redirects traffic to the VM-series we need to apply this to a specific group. The power of combining NSX with Palo Alto Networks lies in the fact that we can use dynamic groups (both on NSX and in Panorama) and that members of the dynamic groups are sync’ed (about every 60 seconds) between both solutions. This means that if we add or remove VM’s from groups, the firewall rules are automatically updated. No more dealing with large lists of outdated firewall rules relating to decommissioned applications that nobody is willing to risk deleting because no one is sure what the impact would be.

For example we could create a security group using dynamic membership based on a security tag, this security tag could easily be applied as metadata by a cloud management platform (vRealize Automation for example) at the time of creation of the VM. (or you can manually add/remove security tags using the vSphere Web Client).

Screen Shot 2015-05-04 at 10.27.43

In Panorama we also have this concept of dynamic address groups, these are linked in a one-to-one fashion with security groups in NSX.


If we look a the group membership of the address groups in Panorama we will see the IP address of the VM, this can then be leveraged to apply firewall rules in Palo Alto Networks.

Screen Shot 2015-05-04 at 10.34.19

NOTE: if I would remove the VM from the security group in NSX about 60 seconds later the IP address in Panorama would disappear.

Traffic is redirected by using the filtering, and traffic redirection module that are running between the VM and the vNIC. The filtering module is an extension of the NSX distributed firewall, the traffic redirection module defines which traffic is steered to the third party services VM (VM-series VM in our case).


If we use the same dvfilter command (summarize-dvfilter) on the ESXi host as before we can see which slots are occupied;

Slot 0 : implements vDS Access Lists.
Slot 1:  Switch Security module (swsec) capture DHCP Ack and ARP messages, this info then forwarded to the NSX Controller.
Slot 2: NSX Distributed Firewall.
Slot 4: Palo Alto Networks VM-series

Screen Shot 2015-05-04 at 11.00.00

So as we are now able to steer traffic towards the Palo Alto Networks NGFW we can apply security policies, as an example we have built some firewall rules blocking ICMP and allowing SSH between two security groups.

Screen Shot 2015-05-04 at 10.36.52

As you could see from the picture earlier the VM in the SG-PAN-WEB group has IP address (matching the member IP seen above in the dynamic group DAG-WEB in Panorama).

We are not allowed to ping a member of the dynamic group DAG-APP as dictated by the firewall rules on the VM-series firewall.

Screen_Shot_2015-05-04_at_10_39_54Since SSH is allowed we can test this by trying to connect to a VM in the DAG-APP group.


We can also verify if this session shows up on the VM-series firewall by opening the console on the vSphere web client.

Screen Shot 2015-05-04 at 10.45.38

And finally if we look at the monitoring tab on Panorama we can verify that our firewall rules are working as expected.

Screen Shot 2015-05-04 at 10.47.07

So that’s it for this brief overview of using Palo Alto Networks NGFW in combination with VMware NSX. As you can see from the screenshot below, NSX allows for a broad list of third party solutions to be integrated, so the solution is very extensible and true to it’s goal of being a network and security platform for the next generation data center.

Screen Shot 2015-05-04 at 10.48.30

vSphere 6 – vMotion nonpareil

vSphere 6 – vMotion nonpareil

I remember when I was first asked to install ESX 2.x for a small internal project, must have been somewhere around 2004-2005, I was working at a local systems integrator and got the task because no one else was around at the time. When researching what this thing was and how it actually worked I got sucked in, then some time later when I saw VMotion(sic) for the first time I was hooked.

Sure I’ve implemented some competing hypervisors over the years, and even worked for the competition for a brief period of time (in my defence, I was in the application networking team 😉 ). But somehow, at the time, it was like driving a kit car when you knew the real deal was parked just around the corner.

mercy-4vsRealCar2So today VMware announced the upcoming release of vSphere 6, arguably the most recognised product in the VMware stable and the foundation to many of it’s other solutions.

A lot of new and improved features are available but I wanted to focus specifically on the feature that impressed me the most soo many years ago, vMotion.

In terms of vMotion in vSphere 6 we now have;

  • Cross vSwicth vMotion
  • Cross vCenter vMotion
  • Long Distance vMotion
  • Increased vMotion network flexibility

Cross vSwitch vMotion

Cross vSwitch vMotion allows you to seamlessly migrate a VM across different virtual switches while performing a vMotion operation, this means that you are no longer restricted by the network you created on the vSwitches in order to vMotion a virtual machine. It also works across a mix of standard and distributed virtual switches. Previously, you could only vMotion from vSS to vSS or within a single vDS.

Screen Shot 2015-01-19 at 17.52.25

The following Cross vSwitch vMotion migrations are possible:

  • vSS to vSS
  • vSS to vDS
  • vDS to vDS (including metadata, i.e. statistics)
  • vDS to vSS is not possible

The main use case for this is data center migrations whereby you can migrate VMs to a new vSphere cluster with a new virtual switch without disruption. It does require the source and destination portgroups to share the same L2. The IP address within the VM will not change.

Cross vCenter vMotion

Expanding on the Cross vSwitch vMotion enhancement, vSphere 6 also introduces support for Cross vCenter vMotion.

vMotion can now perform the following changes simultaneously:

  • Change compute (vMotion) – Performs the migration of virtual machines across compute hosts.
  • Change storage (Storage vMotion) – Performs the migration of the virtual machine disks across datastores.
  • Change network (Cross vSwitch vMotion) – Performs the migration of a VM across different virtual switches.

and finally…

  • Change vCenter (Cross vCenter vMotion) – Performs the migration of the vCenter which manages the VM.

Like with vSwitch vMotion, Cross vCenter vMotion requires L2 network connectiviy since the IP of the VM will not be changed. This functionality builds upon Enhanced vMotion and shared storage is not required.

Screen Shot 2015-01-19 at 17.58.38

The use cases are migration of Windows based vCenter to the vCenter Server Appliance (also take a look at the scale improvement of vCSA in vSphere 6) i.e. no more Windows and SQL license needed.
Replacement of vCenters without disruption and the possibility to migrate VMs across local, metro, and cross-continental distances.

Long Distance vMotion

Long Distance vMotion is an extension of Cross vCenter vMotion but targeted to environments where vCenter servers are spread across large geographic distances and where the latency across sites is 150ms or less.

Screen Shot 2015-01-19 at 18.03.38The requirements for Long Distance vMotion are the same as Cross vCenter vMotion, except with the addition of the maximum latency between the source and destination sites must be 150 ms or less, and there is 250 Mbps of available bandwidth.

The operation is serialized, i.e. it is a VM per VM operation requiring 250 Mbps.

The VM network will need to be a stretched L2 because the IP of the guest OS will not change. If the destination portgroup is not in the same L2 domain as the source, you will lose network connectivity to the guest OS. This means that in some topologies, such as metro or cross-continental, you will need a stretched L2 technology in place. The stretched L2 technologies are not specified (VXLAN is an option here, as are NSX L2 gateway services). Any technology that can present the L2 network to the vSphere hosts will work, as it’s unbeknown to ESX how the physical network is configured.

Increased vMotion network flexibility

ESXi 6 will have multiple TCP/IP stacks, this enables vSphere to improve scalability and offers flexibility by isolating vSphere services to their own stack. This also allows vMotion to work over a dedicated Layer 3 network since it can now have it’s own memory heap, ARP and routing tables, and default gateway.


In other words the VM network still needs L2 connectivity since the virtual machine has to retain its IP. vMotion, management and NFC networks can all be L3 networks.

Introducing VMware vCloud Air Advanced Networking Services

Introducing VMware vCloud Air Advanced Networking Services

VMware just announced some additions to it’s public cloud service, vCloud Air, one of the additions is advanced networking services powered by VMware NSX. Today the networking capabilities of vCloud Air are based on vCNS features, moving forward these will be provided by NSX.

If you look at the connectivity options from your Data Center towards vCloud Air today you have:

  • Direct connect which is a private circuit such as MPLS or Metro Ethernet.
  • IPSec VPN
  • or via the WAN to a public IP address (think webhosting)

By switching from vCNS Edge devices to NSX Edge devices vCloud Air is adding SSL VPN connectivity from client devices to the vCloud Air Edge.


VMware is adding, by using the NSX Edge Gateway, dynamic routing support (OSPF, BGP), and a full fledged L4/7 load-balancer (based on HA Proxy), that also provides SSL offloading.
As mentioned before SSL VPN to the vCloud Air network, for which clients are available for Mac OS X, Windows, and Linux is also available.
Furthermore the number of available interfaces have also been greatly increased from 9 to 200 sub-interfaces, and the system now also provides distributed firewall (firewall policy linked to the virtual NIC of the VM) capabilities.


The NSX Edge can use BGP to exchange routes between vCloud Air and your on-premises equipment over Direct Connect. NSX Edge can also use OSPF to exchange internal routes between NSX edges or a L3 virtual network appliance.


NSX also introduces the concept of Micro-Segmentation to vCloud Air. This allows implementation of firewall policy at the virtual NIC level. In the example below we have a typical 3-tier application with each tier placed on its own L2 subnet.

Screen Shot 2015-01-20 at 11.16.45

With micro-segmentation you can easily restrict traffic to the application- and database-tier while allowing traffic to the web-tier, even though, again just as an example all these VM’s sit on the same host. Assuming that at some point you will move these VM’s from one host to another host, security policy will follow the VM without the need for reconfiguration in the network. Or you can implement policy that does not allow VM’s to talk to each other even though they sit on the same L2 segment.


If you combine this with security policy capabilities in the NSX Edge Gateway you can easily implement firewall rules for both North-South and East-West traffic. The system will also allow you to build a “container” by grouping certain VM’s together and applying policies to the group as a whole. For example you could create 2 applications, each consisting of 1 VM from each tier (web, app, db), and set policies in a granular level. As a service provider you can very easily create a system that supports multiple tenants in a secure fashion. Furthermore this would also allow you to set policies and move VM’s from on-premises to vCloud Air while still retaining network and security configurations.

New Year, New Job.

New Year, New Job.

I’m super excited to be taking on a new role in the NSBU at VMware, as of the 1st of January I’ll officially be joining the team as a Sr. Systems Engineer for the Benelux. I’ll be focused mainly on VMware NSX, including it’s integrations with other solutions (Like vRA and OpenStack for example).

Unofficially I’ve been combining this function with my “real” job for a couple of months now ever since a dear and well respected colleague decided to leave VMware. Recently I was fortunate enough to get the opportunity to attend a 2 week training at our Palo Alto campus on NSX-v, NSX-MH, OpenStack, VIO, OVS,…


The experience was face-meltingly good, I definitely learned a lot and got the opportunity to meet many wonderful people. One conclusion is that the NSX team certainly is a very interesting and exciting place to be in the company.

In the last few months I got my feet wet by training some of our partner community on NSX (most are very excited about the possibilities, even the die-hard hardware fanatics), staffing the NSX booth at VMworld Europe, and by having some speaking engagements like my NSX session at the Belgian VMUG.


So why NSX?

In the past I’ve been working on a wide variety of technologies (being in a very small country and working for small system integrators you need to be flexible, and I guess it’s also just the way my mind works #squirrel!) but networking and virtualisation are my two main fields of interest so how convenient that both are colliding!
I’ve been a pure networking consultant in the past, mainly working with Cisco and Foundry/HP ProCurve and then moved more into application networking at Citrix ANG and Riverbed.

The whole network virtualisation and SDN (let’s hold of the discussion of what’s what for another day) field are on fire at the moment and are making the rather tedious and boring (actually I’ve never really felt that, but I’m a bit of a geek) field of networking exciting again. The possibilities and promise of SDN have lot’s of potential to be disruptive and change an industry, and I’d like to wholeheartedly and passionately contribute and be a part of that.

As NSX is an enabling technology for a lot of other technologies it needs to integrate with a wide variety of solutions. 2 solutions from VMware that will have NSX integrated for example are EVO:RACK and VIO. I look forward to also work on those and hopefully find some time to blog about it as wel.

Other fields are also looking to the promise of SDN to enable some new ways of getting things done, like SocketPlane for example, trying to bring together Open vSwitch and Docker to provide pragmatic Software-Defined Networking for container-based clouds. As VMware is taking on a bigger and bigger role in the Cloud Native Apps space it certainly will be interesting to help support all these efforts.

“if you don’t cannibalise yourself, someone else will”
-Steve Jobs

I’m enjoying a few days off with my family and look forward to returning in 2015 to support the network virtualisation revolution!


NSX Logical Load Balancer

NSX Logical Load Balancer

VMware NSX is a network virtualisation and security platform that enables inclusion of 3rd party integration for specific functionality, like advanced firewalling, load-balancing, anti-virus, etc. Having said that VMware NSX also provides a lot of services out of the box that depending on the customer use-case can cover a lot of requirements.

One of these functionalities is load balancing using the NSX Edge Virtual Machine that is part of the standard NSX environment. The NSX Edge load balancer enables network traffic to follow multiple paths to a specific destination. It distributes incoming service requests evenly (or depending on your specific load balancing algorithm) among multiple servers in such a way that the load distribution is transparent to users. Load balancing thus helps in achieving optimal resource utilization, maximizing throughput, minimizing response time, and avoiding overload. NSX Edge provides load balancing up to Layer 7.


The technology behind our NSX load balancer is based on HAProxy and as such you can leverage a lot of the HAProxy documentation to build advanced rulesets (Application Rules) in NSX.

Like the picture above indicates you can deploy the logical load balancer either “one-armed” (Tenant B example) or transparently inline (Tenant A example).

To start using the load balancing functionality you first need to deploy an NSX edge in your vSphere environment.

Go to Networking & Security i your vSphere Web Client and select NSX Edges, then click on the green plus sign to start adding a new NSX edge. Select Edge Services Gateway as an install type and give your load balancer an indicative name.


Next you can enter a read-only CLI username and select a password of your choice. You can also choose to enable SSH access to the NSX Edge VM.

Screen Shot 2014-12-03 at 20.37.57

Now you need to select the size of the Edge VM, because this is a lab environment I’ll opt for compact but depending on your specific requirements you should select the most appropriate size. Generally speaking more vCPU drives more bandwidth and more RAM drives more connections per second. The available options are;

  • Compact: 1 vCPU, 512 MB
  • Large: 2 vCPU, 1024 MB
  • Quad Large: 4 vCPU, 1024 MB
  • X-Large: 6 vCPU, 8192 MB

Click on the green plus sign to add an Edge VM, you need to select a cluster/resource pool to place the VM and select the datastore. Typically we would place NSX edge VMs in a dedicated Edge rack cluster. For more information on NSX vSphere cluster design recommendations please refer to the NSX design guide available here:

LB002Since we are deploying a one-armed load balancer we only need to define one interface next. Click the green plus sign to configure the interface. Give it a name, select the logical switch it needs to connect to (i.e. where are the VMs that I want to load balance located) configure an IP address in this specific subnet (VXLAN virtual switch) and you are good to go.


Next up configure a default gateway.

Screen Shot 2014-12-03 at 21.03.50Configure the firewall default policy and select Accept to allow traffic.

Screen Shot 2014-12-03 at 21.04.22Verify all setting and click Finish to start deploying the Edge VM.

Screen Shot 2014-12-03 at 21.05.13After a few minuted the new Edge VM will be deployed and you can start to configure load balancing for the VMs located on the Web-Tier-01 VXLAN virtual switch (as we selected before) by double clicking the new Edge VM.

Click on Manage and select Load Balancer, next click on Edit and select the Enable Load Balancer check box.


Next thing we need is an application profile, you create an application profile to define the behavior of a particular type of network traffic. After configuring a profile, you associate the profile with a virtual server. The virtual server then processes traffic according to the values specified in the profile.

Click on Manage and select Load Balancer, then click on Application Profiles, give the profile a name. In our case we are going to load balance 2 HTTPS webservers but are not going to proxy the SSL traffic so we’ll select Enable SSL Passthrough.

LB006Now we need to create a pool containing the webservers that will serve the actual content.

Select Pools, give it a name, select the load balancing algorithm.

Today we can choose between;

  • Round-robin
  • IP Hash
  • Least connection
  • URI
  • HTTP Header
  • URL

Then select a monitor to check if the severs are serving https (in this case) requests.
You can add the members of the pools from the same interface.


Now we create the virtual server that will take the front-end connections from the webclients and forward them to the actual webservers. Note that the Protocol of the virtual server can be different from the protocols of the back-end servers. i.e. You could do HTTPS from the client webbrowser to the virtual server and HTTP from the virtual server to the back-end for example.

Screen Shot 2014-12-04 at 18.29.43

Now you should be able to reach the IP (VIP) of the virtual server we just created and start to load balance your web-application.

LB008Since we selected round-robin as our load-balancing algorithm each time you hit refresh (use control and click on refresh to avoid using the browsers cache) on your browser you should see the other webserver.

VMware OpenStack Virtual Appliance

VMware OpenStack Virtual Appliance

VMware provides an OpenStack Virtual Appliance, VOVA for short, to help VMware admins get some hands-on experience with using OpenStack in a VMware environment. It is however purely a proof of concept appliance and is not supported in any way by VMware. To find out more about the OpenStack effort at VMware in general please visit

You can download the VOVA appliance OVF package (OpenStack Icehouse release) here.

Screen Shot 2014-08-01 at 10.17.05

VOVA only works with vCenter 5.1 and above and only supports a single Datacenter, you should also not run production workloads on this cluster as a precaution. If you are running multiple hosts in your cluster you should enable DRS in “fully automated mode”, the cluster should also have only one shared datastore for all the hosts in the cluster. It is recommended to deploy the VOVA appliance on a host that is not part of the cluster managed by the appliance. (so you can manage multiple clusters in your vCenter instance).

When the OVF package is deployed it will show you the OpenStack Dashboard URL on first boot.

Screen Shot 2014-08-01 at 10.18.53

You can login with the credentials demo/vmware

Screen Shot 2014-08-01 at 10.19.38

The appliance comes pre-loaded with a Debian disk image which allows you to easily launch new instances.

Screen Shot 2014-08-01 at 10.21.47

Spawning the first VM can take a while because the 1 GB Debian disk image must be copied from the file system of the VOVA appliance to your cluster’s Datastore. All subsequent instances should be significantly faster (under 30 seconds).

VOVA also allows you to test the OpenStack CLI tools which directly acces the OpenStack APIs, you need to SSH (root/vmware) into the VOVA’s console and run the CLI commands from there.

Screen Shot 2014-08-01 at 10.25.21

The vCenter Web Client plug-in for OpenStack is also included allowing you to see OpenStack instances from the Web Client

Screen Shot 2014-08-01 at 10.49.18

Currently the VOVA appliance has some limitations;

  • No Neutron support: Neutron with vSphere requires the VMware NSX solution. We plan to release a future version of VOVA that can optionally leverage NSX.
  • No Security Groups support. With vSphere, VMware NSX is required for security groups network filtering. We plan to release a future version of VOVA that can optionally leverage NSX.
  • No exposed options to configure floating IPs. This is possible with the current appliance, but it has not been exposed via the OVF options.
  • No support for sparse disks. If you try to upload your own disk images, the images must be flat, not sparse.
  • No support for Swift (object storage). VOVA has no plans to leverage OpenStack swift for object storage. You are free to deploy swift on your own in another VM.

Also keep in mind that VOVA is not a product and will likely be discontinued once production-quality solutions with similar ease-of-use are made available (remember that there is nothing “special” about VOVA, it is just the open source OpenStack code running on Ubuntu, with proper configuration for using vSphere). However in the months following, expect to see updated versions of VOVA with the option of using Neutron + Security groups via VMware NSX.