Tag: docker

Unifying Docker Container and VM networking

Unifying Docker Container and VM networking


Most environments are not homogeneous, typically you have multiple types of workloads and I believe this will only increase in the near future with the rise of containers, PaaS, VM’s, bare metal,… In this brief overview I wanted to demonstrate how you can connect Virtual Machines and Containers on the same overlay network in an automated manner via our SDN solution. This way every time you spin up a new workload it will automatically get its network and security policy applied and behaves like any other endpoint on the network.

Screen Shot 2015-11-06 at 13.01.15

Docker networking

There are multiple options to do networking in Docker, typically a container (running a specific service) can be exposed externally by mapping an internal port to external one. When you install Docker, it creates three networks automatically (bridge, null, and host), when you run a container you can use the –net flag to specify which network you want to run a container on. By default the Docker daemon connects containers to the bridge network. If you run ifconfig on the host you can see the bridge as part of the host’s network stack.

The none network adds a container to a container-specific network stack, this is what we will use in the case of Nuage Networks to connect the Docker container to our VRS (Open vSwitch)

The host network adds a container on the hosts network stack. You’ll find the network configuration inside the container is identical to the host.

Nuage Networks and Docker containers

In the case of Nuage Networks we will attach every container to a tenant (overlay) network which is provided by our centralised management (VSD) and control (VSC) plane and configured on the Docker host in our VRS (Open vSwitch). This allows us to use our centralised networking and security policies providing IP configuration, firewall rules, QoS, etc. If traffic leaves the Docker host it is encapsulated in VXLAN so from a management point of view this no different then how we work with Virtual Machines.


I’ve created a L2 network (called DockerSN below) in Nuage (synced to OpenStack) where I’m connecting both my containers and VM workloads. The subnet has a range of

Screen Shot 2015-11-06 at 09.11.28

So when I spin up a new container on my Docker host and connect it to the Nuage VRS I’ll automatically get the policies from that construct applied.

Screen Shot 2015-11-06 at 09.23.22

So as you can see above, my new container (gloomy_jang) has gotten the IP address, if we go back to our Virtualized Services Architect interface we can see 2 containers (one I created earlier) and a VM attached to the same subnet (could also be to a separate subnet ofc).

Screen Shot 2015-11-06 at 09.28.40

We can drill down on the newly created container and get all the network and security policy details

Screen Shot 2015-11-06 at 09.29.48

We now have connectivity between our container workloads and VM (

[[email protected] ~]# docker exec 152f5660e56a ping -c 3
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.650 ms
64 bytes from icmp_seq=2 ttl=64 time=0.450 ms
64 bytes from icmp_seq=3 ttl=64 time=0.450 ms

Networking Field Day 10 – Nuage Networks

Networking Field Day 10 – Nuage Networks


Networking Field Day 10 was held from August 19 till 21 2015, in Silicon Valley, NFD is part of the Tech Field Day series of events organized by Stephen Foskett and team, and aims to bring together independent bloggers and IT product vendors to share information and opinions in a presentation format. In this setting demo’s and whiteboard sessions are usually appreciated more than slide-ware and marketing pitches. Assuming you are an independent (e.g. you don’t work for a vendor) blogger/influencer you can ask to become a TFD delegate and when selected get to experience these sessions first hand.

One of the vendors at NFD 10 was Nuage Networks, Nuage has appeared at various TFD events on numerous (10 times by my count) occasions and that’s how I first heard about and got interested in their solutions.

So what did they show at NFD 10?

First up was Sunil Khandekar, Founder and CEO, with an introduction to Nuage Networks and an update on the products. He talked about building software defined, programmable, automated data centers and how Nuage through its declarative policy based automation is applicable to all types of workloads, be it bare metal, virtual machines, or containers. Further he mentioned how Nuage is a complete SDN solution seeing that it hits on all the key tenets, namely; abstraction, automation, control, and visibility. Then he went on to give an update on Nuage Networks.

Screen Shot 2015-09-02 at 10.22.05

Nuage was started January 2012 with the idea that networking should be as instantaneous and consumable as compute has historically become, the solution, VSP, was launched about a year later in April 2013 delivering on this thesis. Currently Nuage is on its 4th release and has seen great customer traction.

The Virtual Services Platform (VSP) has 3 main components, the Virtual Services Directory (VSD), the Virtual Services Controller (VSC), and the Virtual Routing & Switching engine (VRS). Additionally Nuage also optionally provides a hardware VTEP gateway, the 7850 Virtualized Services Gateway (VSG), to connect legacy networking to VXLAN overlays.

Screen Shot 2015-09-02 at 10.32.19

To get a more detailed overview of the solution please see my Nuage Compendium page*.

Sunil also announced the VSP SDK (VSPK) available on https://github.com/nuagenetworks with the idea of fostering open collaboration around the platform. Through this proposed github collaboration, custom scripts for network automation, control or visibility can be developed and shared by its customer community

It is also important to understand that Nuage takes the concept of network automation beyond the data center and extends it to the WAN (branch office) with Virtual Network Services (VNS).

Virtual Network Services (VNS) basics

Next up was Rotem Salomonovitch, head of product management for VNS, talking about how it works in some more detail. The idea of VNS is to make setting up a new branch stupidly easy and independent of the underlying (carrier) technology, essentially SDWAN with a lot of automation. VNS is available as a hardware box, a VM, and as a software only package to install on a bare metal server.

Screen Shot 2015-09-02 at 15.25.36

VNS uses the same control plane components from the data center (VSD / VSC) but has a different data plane, e.g. forwarding entity called the Network Services Gateway (NSG), reason being is that data plane’s in branches typically differ from those in data centers where you have GigE, 10GigE, 40GigE and beyond. In the branch you have different interface types and things like encryption in the WAN, additional security requirements, etc. The NSG is meant to be a platform, initial applications of this platform are networking services (routing, QoS, FW,…) but the goal is to go beyond network connectivity and enable application flexibility. (see section below).
The appliance has multiple WAN ports and also USB connectivity so you could do things like extend it with external LTE connectivity for example.

Rotem then moved on to talk about different abstractions for different types of audiences, e.g. a developer just wants his application to be “connected” whereas the network design team are concerned with connection points, bandwidth consumption etc. To support this he talked about both vertical abstractions, for multi-tenancy, and horizontal abstractions for different audience types within each tenant. The idea is to expose only those abstractions that a particular audience is interested in (until we break all IT silos and everyone is a unicorn of course).

Screen Shot 2015-09-02 at 15.42.34

The idea of abstractions is not to have each team work in their own silo but rather have the system interpret a certain audience’ set of abstractions and translate those to complete the end-to-end policy setup e.g. the application developer uses the Application Designer to create a new app, he only defines the app concepts (e.g. front-end, middle-tier, database) and then the system translates those to subnets, ACLs, etc.

Another example is the VPN designer where you can bring up a new site by linking the device object in the GUI to a location, then, depending on the authentication method, the only thing that needs to happen is physically connecting the WAN and LAN ports on the device at the branch and the device will “bootstrap” automatically and pick up its configuration. The forwarding engine in the device is multi-tenant capable, each subnet is created as a L2 EVPN construct (remember that the idea of VNS is to be independent of the WAN technology), the access ports of the box are piped into those.

Application Flexibility at the Branch

The VNS does not only provide network connectivity but also allows you to run containerized workloads on top of it (it’s an Intel Atom based system). The main idea is to automate the attachment of existing container based applications to the branch network. One example would be to enable running external network operations tools to perform local logging of LAN elements and then setup a single encrypted connection over the WAN or do local auditing of running configurations. Another would be to run a user simulation at the branch before go-live to validate user experience and adjust as needed.

Screen Shot 2015-09-02 at 16.51.00

Theoretically you can run any containerized app (pull it from docker hub) on VNS (provided that you have enough resources to run it), Nuage takes care of the multi-tenant network aspects of running multiple containers on a single host.

Boundary-less Wide Area Networking

Next up was Hussein Khazaal, solution director, talking about extending connectivity from the data center to the branch, to the public cloud, making a VPC your own personal branch office with consistent business policies between all of them.

Screen Shot 2015-09-02 at 17.11.26

The way it works is you get (in this case) a virtual NSG from Nuage which comes as an Amazon AMI (Amazon Machine Image) and deploy it as any other instance to your VPC. Now you can use the NSG-v in Amazon just like any other NSG from your application designer / vpn designer. The classic use case would be to load-balance your front-end (web) application between your data center and Amazon in times of increased load, essentially cloud-bursting made real. If you would want to do something like this without Nuage you would need to figure out how to translate your business policies to the available constructs at each public cloud provider, with the NSG-v it consumes the centralized policies from the VSD just like any other forwarding engine.

*The Nuage Compendium page is under construction, and will be expanded over time.

VMware AppCatalyst, Bonneville, and Photon.

VMware AppCatalyst, Bonneville, and Photon.

VMware has lot’s and lot’s of customers, running lot’s and lot’s of workloads, both dev and test workloads and production workloads, you know like, super duper important stuff that cannot, under any circumstance break.

As the whole DevOps “movement” makes clear both developers and operations teams have different requirements and different responsibilities. Developers want speed, the operations teams want stability, both are trying to respond to the demands of the business which wants faster time to market. Gross oversimplification I know, but since devops is getting a lot of attention lately you’ll have no problem digging up articles and blogposts on that.


Now as VMware’s customers rightfully expect, the idea is to try and marry both worlds and make these new methods enterprise grade without loosing the original benefits. I personally notice a lot of resistance in that enterprise customers understand the benefits, and some internal teams are pushing hard to incorporate these, but still a lot of people seem to adopt this “let’s wait and see / it’s just another fad” attitude.


VMware has been doing a lot of work in the past around making it’s products more “DevOps” friendly with things like vRealize CloudClient a command-line utility that provides verb-based access with a unified interface across vCloud Automation Center APIs, and vRealize Code Stream which provides release automation and continuous delivery to enable frequent, reliable software releases, while reducing operational risks.

You don’t have meetings with other teams, you talk to their API instead.
-Adrian Cockcroft

And now at the recent DockerCon in San Francisco VMware announced the tech-preview of AppCatalyst and Project Bonneville.

VMware AppCatalyst is an API and Command Line Interface (CLI)-driven MacOS type 2 hypervisor (based on VMware Fusion but without the GUI, 3D graphics support, virtual USB support, and Windows guest support) that is purpose-built for developers, with the goal of bringing the datacenter environment to the desktop. Currently a technology preview, VMware AppCatalyst offers developers a fast and easy way to replicate a private cloud locally on their desktop for building and testing containerized and microservices-based applications. The tool includes Project Photon (already announced in April), an open source minimal Linux container host, Docker Machine and integration with Vagrant. Panamax and Kitematic support are planned in the near future. AppCatalyst uses MacOS as its host operating system (i.e., the user must use MacOS 10.9.4 or later as their host operating system to use AppCatalyst).

You can download the Tech Preview of AppCatalyst here, it comes with an installer so it pretty easy to get up and running. Once it is installed AppCatalyst does not appear under your Applications folder, instead you can use your Terminal to navigate to /opt/vmware/appcatalyst

Screen Shot 2015-06-27 at 10.11.41

As mentioned above AppCatalyst comes pre-bundled with Project Photon – VMware’s compact container host Linux distribution. When you download AppCatalyst, you can point docker-machine at it, start up a Photon instance almost instantly (since there’s no Linux ISO to download), and start using Docker.

Another common use of the desktop hypervisor is with Vagrant. Developers build Vagrant files and then Vagrant up their deployment. Vagrant creates and configures virtual development environments, it can be seen as a higher-level wrapper to AppCatalyst. You can find the plugin for Vagrant here. (git clone https://github.com/vmware/vagrant-vmware-appcatalyst.git)

Since Project Photon is included in AppCatalyst it’s pretty easy to get started with deploying a Photon Linux Container Host.

appcatalyst vm create photon1
Info: Cloned VM from ‘/opt/vmware/appcatalyst/photonvm/photon.vmx’ to ‘/Users/filipv/Documents/AppCatalyst/photon1/photon1.vmx’

appcatalyst vm list
Info: VMs found in ‘/Users/filipv/Documents/AppCatalyst’

appcatalyst vmpower on photon1
2015-06-27T10:18:38.530| ServiceImpl_Opener: PID 2949
Info: Completed power op ‘on’ for VM at ‘/Users/filipv/Documents/AppCatalyst/photon1/photon1.vmx’

appcatalyst guest getip photon1

I can now SSH into the VM:

Screen Shot 2015-06-27 at 10.22.28

And securely launch a Docker container via Project Photon:

Screen Shot 2015-06-27 at 10.26.17

Screen Shot 2015-06-27 at 10.28.07

As mentioned before the idea is to interface with AppCatalyst via REST API calls, you can enable this by first starting the app catalyst-deamon and then going to port 8080 on your localhost.

Screen Shot 2015-06-27 at 10.30.31once the deamon is running we can start to make REST API calls, for example retrieve the IP address of the Docker VM we previously created:

Screen Shot 2015-06-27 at 10.34.09

Since the last VMworld VMware has been talking about this concept of containers and VMs being better together, this kinda led to a lot of discussion about overhead of the hypervisor and each container needing it’s own OS, potential lock-in, etc. But again, this is where VMware is trying to marry Dev with Ops and make the use of containers feasible in the enterprise environment. Project Bonneville takes another step in this direction by making containers first class citizens on the vSphere hypervisor.

Bonneville orchestrates all the back-end systems: VM template (with Photon), storage, network, Docker image cache, etc. It can manage and configure native ESX storage and network primitives automatically as part of a container deploy.

Bonneville is a Docker daemon with custom VMware graph, execution and network drivers that delivers a fully-compatible API to vanilla Docker clients. The pure approach Bonneville takes is that the container is a VM, and the VM is a container. There is no distinction, no encapsulation, and no in-guest virtualization. All of the necessary container infrastructure is outside of the VM in the container host. The container is an x86 hardware virtualized VM – nothing more, nothing less.

Screen Shot 2015-06-27 at 15.22.23

Bonneville uses VMFork (Instant Clone / Project Fargo) to spin up a new VM every time a container is launched, by doing this the operations team now sees VM instances in it’s environment that it can treat, i.e. “operationlize”, just like regular Virtual Machines (Bonneville updates VM names and metadata fields for the container VMs it creates for full transparency in vCenter and any vSphere ecosystem products), and the obvious added benefit is that each container might be a VM but each container is not using a full blown linux host os to run. Instant Cloned VMs are powered on and fully booted in under a second and use no physical memory initially.

You can see a demo of Project Bonneville below:

New Year, New Job.

New Year, New Job.

I’m super excited to be taking on a new role in the NSBU at VMware, as of the 1st of January I’ll officially be joining the team as a Sr. Systems Engineer for the Benelux. I’ll be focused mainly on VMware NSX, including it’s integrations with other solutions (Like vRA and OpenStack for example).

Unofficially I’ve been combining this function with my “real” job for a couple of months now ever since a dear and well respected colleague decided to leave VMware. Recently I was fortunate enough to get the opportunity to attend a 2 week training at our Palo Alto campus on NSX-v, NSX-MH, OpenStack, VIO, OVS,…


The experience was face-meltingly good, I definitely learned a lot and got the opportunity to meet many wonderful people. One conclusion is that the NSX team certainly is a very interesting and exciting place to be in the company.

In the last few months I got my feet wet by training some of our partner community on NSX (most are very excited about the possibilities, even the die-hard hardware fanatics), staffing the NSX booth at VMworld Europe, and by having some speaking engagements like my NSX session at the Belgian VMUG.


So why NSX?

In the past I’ve been working on a wide variety of technologies (being in a very small country and working for small system integrators you need to be flexible, and I guess it’s also just the way my mind works #squirrel!) but networking and virtualisation are my two main fields of interest so how convenient that both are colliding!
I’ve been a pure networking consultant in the past, mainly working with Cisco and Foundry/HP ProCurve and then moved more into application networking at Citrix ANG and Riverbed.

The whole network virtualisation and SDN (let’s hold of the discussion of what’s what for another day) field are on fire at the moment and are making the rather tedious and boring (actually I’ve never really felt that, but I’m a bit of a geek) field of networking exciting again. The possibilities and promise of SDN have lot’s of potential to be disruptive and change an industry, and I’d like to wholeheartedly and passionately contribute and be a part of that.

As NSX is an enabling technology for a lot of other technologies it needs to integrate with a wide variety of solutions. 2 solutions from VMware that will have NSX integrated for example are EVO:RACK and VIO. I look forward to also work on those and hopefully find some time to blog about it as wel.

Other fields are also looking to the promise of SDN to enable some new ways of getting things done, like SocketPlane for example, trying to bring together Open vSwitch and Docker to provide pragmatic Software-Defined Networking for container-based clouds. As VMware is taking on a bigger and bigger role in the Cloud Native Apps space it certainly will be interesting to help support all these efforts.

“if you don’t cannibalise yourself, someone else will”
-Steve Jobs

I’m enjoying a few days off with my family and look forward to returning in 2015 to support the network virtualisation revolution!