Containers and persistent storage

Containers and persistent storage

Containers are a method of operating system virtualization that allow you to run an application and its dependencies in resource-isolated processes. Containers allow you to easily package an application’s code, configurations, and dependencies into easy to use building blocks that deliver environmental consistency, operational efficiency, developer productivity, and version control. Containers are immutable, meaning they help you deploy applications in a reliable and consistent way independent of the the deployment environment.

As containers continue to rise in popularity beyond the developer populous the way these constructs are being used becomes increasingly varied, especially (but not exclusively) in light of enterprise applications the questions of persistent storage comes up more and more. It is a fallacy to think only stateless application can or should be containerized. If you take a look at you’ll see that about half of the most popular applications on Docker Hub are stateful, like databases for example. (Post hoc ergo propter hoc?). If you think about monolithic applications versus micro-services, a monolithic application typically requires state, if you pull this application out into micro-services, some of these services can be stateless containers but others will require state.

I’ll mainly use Docker as the example for this post but many other container technologies exist like LXD, rkt, OpenVZ, even Microsoft offers containers with Windows Server Containers, Hyper-V isolation, and Azure Container Service.

Running a stateless container using Docker is quite straightforward;

$ docker run --name demo-mysql mysql

When you execute docker run, the container process that runs is isolated in that it has its own file system, its own networking, and its own isolated process tree separate from the (local or remote) host.

The docker container is created from a readonly template called docker image. The “mysql” part in the command relates to this image, i.e. containerized application, that you want to run by pulling it from the registry. The data you create inside a container is stored on a thin writable layer, called the container layer, that sits on top of the stack of read-only layers, called the image layers, present in the base docker image. When the container is deleted the writable layer is also deleted so your data does not persist , in docker the docker storage driver is responsible for enabling and managing both the read-only image layers and the writable layer, both read and write speeds are generally considered slow.

Image result for docker layers

Assuming you want persistent data for your containers there are several methods to go about this. You can add a storage directory to a container’s virtual filesystem and map that directory to a directory on the host server. The data you create inside that directory on the container will be saved on the host, allowing it to persist after the container shuts down. This directory can also be shared between containers. In docker this is made possible by using volumes, you can also use bind mounts but these are dependent on the directory structure of the host machine whereas volumes are completely managed by Docker itself. Keep in mind though that these volumes don’t move with container workloads as they are local to the host. Alternatively you can use volume drives (Docker Engine volume plugins) to store data on remote systems instead of the Docker host itself. If you are only interested in storing data in the container writeable layer (i.e. on the docker host itself) you can use Docker storage drivers which then determine which filesystem is supported.

Typically you would create a volume using the storage driver of your choice in the following manner;

$ docker volume create -—driver=pure -o size=32GB testvol1

And then start a container and attach the volume to it;

$ docker run -ti -v testvol1:/data mysql

Storage Vendors and Persistent Container Storage

Storage vendors have an incentive to make consuming their particular storage as easy as possible for these types of workloads so many of them are providing plug-ins to do just that.

One example is Pure Storage who provide a Docker Volume Plugin for their FlashArray and FlashBlade systems. Current they support Docker, Swarm, and Mesos. Most other big name storage vendors also have plugins available.

Then there are things like REX-Ray which is an open source, storage management solution, it was born out of the now defunct {code} by Dell EMC team. It allows you to use multiple different storage backends and serve those up as persistent storage for your container workloads.

On the virtualization front VMware has something called the vSphere Docker Volume Service which consists of two parts, the Docker Volume Plugin and a vSphere Installation Bundle (VIB) to install on the ESXi hosts. This allows you to serve up vSphere Datastores (be it Virtual SAN, VMFS, NFS based) as persistent storage to your container workloads.

Then there are newer companies that have been solely focusing on providing persistent storage for container workloads, one of them is Portworx. Portworx want to provide another abstraction layer between the storage pool and the container workload. The idea is that they provide a “storage ” container that can then be integrated with the “application” containers. You can do this manually or you can integrate with a container scheduler like Docker Swarm using Docker Compose for example (Portworx provides a volume driver).

Docker itself has built specific plugins as well, Cloudstor is such a volume plugin. It comes pre-installed and pre-configured in Docker swarms deployed through Docker for AWS. Data volumes can either be backed by EBS or EFS. Workloads running in a Docker service that require access to low latency/high IOPs persistent storage, such as a database engine, can use a “relocatable” Cloudstor volume backed by EBS. When multiple swarm service tasks need to share data in a persistent storage volume, you can use a “shared” Cloudstor volume backed by EFS. Such a volume and its contents can be mounted by multiple swarm service tasks since EFS makes the data available to all swarm nodes over NFS.

Container Orchestration Systems and Persistent Storage

As most enterprise production container deployments will utilize some container orchestration system we should also determine how external persistent storage is managed at this level. If we look at Kubernetes for example, that supports a volume plugin system (FlexVolumes) that makes it relatively straightforward to consume different types of block and file storage. Additionally Kubernetes recently started supporting a implementation of the Container Storage Interface (CSI) which helps accelerate vendor support for these storage plug-ins as volume plugins are currently part of the core Kubernetes code and shipped with the core Kubernetes binaries meaning that vendors wanting to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) must align themselves with the Kubernetes release process. With the adoption of the Container Storage Interface, the Kubernetes volume layer becomes extensible. Third party storage developers can now write and deploy volume plugins exposing new storage systems in Kubernetes without having to touch the core Kubernetes code.

When using CSI with Docker, it relies on shared mounts (not docker volumes) to provide access to external storage. When using a mount the external storage is mounted into the container, when using volumes a new directory is created within Docker’s storage directory on the host machine, and Docker manages that directory’s contents.

To use CSI, you will need to deploy a CSI driver, a bunch of storage vendors have these available in various stages of development. For example there is a  Container Storage Interface (CSI) Storage Plug-in for VMware vSphere.

Pre-packaged container platforms

Another angle how vendors are trying to make it easier for enterprises to adopt these new platforms, including solving for persistence, is by providing packaged solutions (i.e. making it turnkey), this is not new of course, not too long ago we saw the same thing happening with OpenStack through the likes of VIO (VMware Integrated OpenStack), Platform9, Blue Box (acquired by IBM), etc. Looking at the Public Cloud providers these are moving more towards providing container as a service (CaaS) models with Azure Container Service, Google Container Engine, etc.

One example of packaged container platforms though is the Cisco Container Platform. This is provided as an OVA for VMware (meaning it is provisioning containers inside virtual machines, not on bare metal at the moment), initially this is supported on their HyperFlex Platform which will provide the persistent storage layer via a Kubernetes FlexVolume driver. It then can communicate externally via Contiv, including talking to other components on the HX platform like VMs that are running non containerized workloads. For the load-balancing piece (between k8s masters for example) they are bundling NGINX and for logging and monitoring they are bundling Prometheus (monitoring) and an ELK stack (logging and analytics).

Another example would be VMware PKS which I wrote about in my previous post.


Containers are ready for enterprise use today, however there are some areas that could do with a bit more maturity, one of them being storage. I fully expect to see continued innovation and tighter integrations as we figure out the validity of these use-cases. A lot progress has been made in the toolkits themselves, leading to the demise of earlier attempts like ClusterHQ/Flocker. As adoption continues so will the maturity of these frameworks and plugins.

VMware’s Cloud Nativeness

VMware’s Cloud Nativeness

At first glance it feels like VMware’s efforts related to cloud native applications are meant to cater to the traditional admins, people who are mainly running virtual machines today but want, or need, to support other types of workloads as well. People who secretly don’t “get” this new container hotness* and really want to manage their environment with the tools and knowledge that got them where they are today and are very comfortable with. Or to put it another way, VMware is ultimately trying to cater to both the infrastructure/operations person needing to support these new types of workloads and the developer building them.

Before looking at VMware’s portfolio related to cloud native applications let’s take a step back and examine both what CNA means and what drives it. According to Pivotal, Cloud-native is an approach to building and running applications that fully exploits the advantages of the cloud computing delivery model. Cloud-native is about how applications are created and deployed, not where. “Where” meaning the underlying infrastructure is not as important as it once was, granted, virtualisation already decreased our reliance on static infrastructure, but if we are honest with ourselves infrastructure was still the thing we got excited about the most as VMware admins. Depending on where you see yourself on the “Public Cloud” is the only thing that makes sense going forward scale, you’ll be more or less inclined to accept the idea that cloud computing can be considered a delivery model that is appropriate for both public and private cloud like environments alike. Core to the cloud computing model is the ability to offer seemingly unlimited computing power, on-demand, along with modern data and application services for developers. It is this last piece, I believe, where private cloud solutions fall down the most compared to public cloud vendors, the onslaught of services delivered at more and more rapidly increasing pace by companies like Amazon (AWS), Microsoft (Azure), and Google (GCP) are creating an understandable pull, attracting new workloads. Therefore it seems that when we discuss Cloud Native Applications from the lens of VMware, we again tend to mostly argue the needs of these applications related to the infrastructure piece. If we agree that products and services delivered through software are becoming major competitive differentiators of the business then it is precisely this what is driving the need of Cloud Native Applications. This is where Marc Andreesen’s software is eating the world quote usually comes in…

The way VMware describes it’s portfolio for CNA on their own website is;

VMware offers cloud-native solutions across container networking, security, storage, automation, operations and monitoring for enterprises and service providers.

Since VMware is trying hard to reach beyond it’s traditional enterprise base, including forays into multiple open-source type projects, I wanted to try and provide a somewhat comprehensive overview of everything it’s doing in those spaces and it’s potential use cases. Generally speaking VMware’s cloud native portfolio includes VMware vSphere Integrated Containers, for running containerised workloads with existing infrastructure. But please don’t interpret this as a silver bullet to “lift and shift” all your legacy applications to a more modern environment, start by asking why you can’t re-architecture your existing apps, and only then ask yourself if you will truly benefit from moving to this new approach, maybe for easier maintenance or improved scalability, but I’m willing to bet that mostly this wont be the best approach for your old and crusty stuff. For applications that naturally fit a containerised approach the story if different of-course. Secondly VMware Photon Platform is targeted to building new cloud-native infrastructure solutions from the ground up.

VMware VIC - Page 1 (2)

Very simplified view of components fitting together

vSphere Integrated Containers

VIC is meant for people who want to stick with the more traditional vSphere infrastructure but also provide Container based services to internal developers. It is available with vSphere Enterprise Plus 6.0 or later, or with vSphere with Operations Management Enterprise Plus. It consists of 3 main components, the vSphere Integrated Container Engine (VIC Engine), Admiral, the Container Management Portal, and Harbor, the Container Registry (based on Docker Distribution). The portal provides a user interface and an API for managing container repositories, images, hosts, and instances. The registry furnishes a user interface and an API so developers can make their own container repositories and images. The engine is a container runtime integrated with vSphere.

vSphere, as opposed to Linux, acts as the container host, containers are deployed as VMs (not in VMs), every container is fully isolated from the host and from the other containers.

 VIC Marketecture

With VIC the idea is that you can still leverage the existing management toolsets in your vSphere based environment, since these tools typically have an understanding about the virtual abstraction as a VM, the idea is that you run a Container as a VM (C-VM in the picture above). The goal is to run both traditional and containerised applications side-by-side on a common infrastructure, this, if you will, is it’s main use case, take what you know and extent it so you can support container based workloads in a way that is transparent for the developer in theory. It’s meant to avoid having to build a completely separate environment.

VMware Photon Platform 

VMware Photon Platform is a container-optimised cloud platform, at first it seems to have a somewhat similar goal to VIC, which is to enable developers to build, test, and run container based workloads while still allowing “traditional IT” to maintain security, control and performance of data center infrastructure. But the use cases have a different focus, namely;

  • Kubernetes as a Service: Developers can deploy, resize, and destroy Kubernetes clusters to develop, test, and automate containerised applications.
  • Platform as a Service: Photon Platform integrates with Pivotal Cloud Foundry to build and scale applications.
  • Continuous integration and delivery: Try to improve the CI/CD pipeline with uniformity and reusability, especially in environments with high container churn.
  • API-managed on-premises cloud: IT can deploy resources and automate their management through a RESTful API.
  • Security: The VMware Lightwave security service can protect applications, Kubernetes clusters, and Photon Platform components.

Screen Shot 2017-09-14 at 08.45.45.png

Photon Platform Marketecture

Note that the Photon Controller is no longer a part of the Photon Platform, VMware has made the decision to stop driving this project and instead lean on the open source community to continue this effort. It was originally envisioned to replace vCenter as the control plane for these new type of workloads since vCenter’s monolithic design was never expected to serve these types of applications. More recently of-course the community has moved to other control plane architectures and container orchestration systems.

NSX-T as the secret sauce.

With VIC you can select vSphere networks that appear in the Docker client as container networks. NSX-T intends to extent the networking capabilities and make them more enterprise appropriate. By default a container sits behind a NAT’d interface and as such form a operations point of view you struggle with visibility. An other challenge is consuming services outside the container realm, if you have services that need to be consumed from your traditional VMs or even bare metal systems you typically need to pass via bridging nodes potentially choking traffic. NSX-T tries to overcome these challenges by supporting native container networking and micro-segmentation for CaaS (Container-as-a-Service ) and PaaS (Platform-as-a-Service), providing tools with which network and security teams can operationalise containerised apps in a vSphere environment.

Statefulness and persistent storage

When discussing containers and container storage you cannot ignore the difference between stateless and stateful workloads and the impact that has on storage persistency. Containers are ephemeral, so if you want to run stateful applications that require persistent storage you need to take some additional steps to ensure your data is maintained through the lifecycle of the containers. VMware provides two persistent volume offerings for situations like these, one for Docker and one for Kubernetes. For Docker VMware provides the vSphere Docker Volume Service (vDVS), this plug-in abstracts the underlying storage of the vSphere environment and makes it available as Docker volumes, it also integrates with Docker Swarm. For stateful containers orchestrated by Kubernetes you can also leverage persistent vSphere Storage (vSAN, VMFS, and NFS) with Kubernetes persistent volume, dynamic provisioning and StatefulSet primitives. Both of these projects are now under the Project Hatchway umbrella.

Operationalising a Cloud Native environment with VMware

Again going back to using the tools you know as a VMware admin, VMware has tried to extend vRealize Automation to make it more relevant for these new types of workloads, it leverages the open source Project Admiral, a scalable and lightweight container management platform, to deploy and manage containers through virtual container hosts on VMware vSphere Integrated Containers. The idea is still to provide a separation between the operational folks and the developers. Developers can provision container hosts from the vRealize Automation service catalog as well as model containerised applications using unified service blueprints or Docker Compose.

Open Source components

Photon OS

Version 2.0 of Photon OS was announced at the beginning of November, Photon OS is a Container-Optimized Linux Operating System. It is an open source minimal Linux container host optimized for cloud-native applications, cloud platforms, and VMware infrastructure. Think of container Operating Systems in general as an OS that provides only the minimal functionality required for deploying applications inside software containers, but typically including built-in mechanisms for shared services consumption and orchestration. Photon OS even comes in 2 flavours, the minimal version is a lightweight host tailored to running containers when performance is paramount, whilst the full version includes additional packages to help deploy containerized applications. Other notable container OSes include a.o. CoreOS, RancherOS, Atomic, and to some extent Intel Clear Linux.

For easy consumption Photon OS is available as a pre-built image for the 3 main public cloud providers as well as an ISO and OVA image. Photon OS includes the open source version of the Docker daemon. After installing Photon OS and enabling Docker, you can immediately run a Docker container.

I’ve written about Photon (and AppCatalyst, and Bonneville) on this blog before if you want some more background.

vSphere Integrated Containers Engine

vSphere Integrated Containers Engine (VIC Engine) is the key element of vSphere Integrated Container, it is a container runtime for vSphere, allowing developers familiar with Docker to develop in containers and deploy them alongside traditional VM-based workloads on vSphere clusters, and allowing for these workloads to be managed through the vSphere UI in a way familiar to existing vSphere admins. The VIC Engine provides lifecycle operations, vCenter support, logs, basic client authentication, volume, and basic networking support.


Harbor is an enterprise-class container registry server based on Docker Distribution, it is embedded in both vSphere Integrated Containers and Photon Platform, it provides advanced security, identity, role based access control, auditing, and management services for Docker images. With Harbor, you can deploy a private registry to maintain your data behind the company firewall. In addition, Harbor supports AD/LDAP integration and the setup of multiple registries with images replicated between registries for high availability.


Admiral is a key component of vSphere Integrated Containers solution. It is a container management solution with an accent on modelling containerised applications and providing placement based on dynamic policy allocation. It manages Docker hosts, policies, multi-container templates, and applications to simplify and automate resource utilisation and application delivery. Developers can use Docker Compose, Admiral Templates, or the Admiral UI to compose their app and deploy it using the Admiral provisioning and orchestration engine. VMware admins can manage container host infrastructure and apply governance to its usage, including resource grouping, policy based placement, quotas and reservations, and elastic placement zones.

Photon controller

Photon controller is a distributed, multi-tenant host controller optimised for containers. The Photon Controller exposes RESTful APIs, SDKs, and CLI tooling to automate infrastructure resources. It is built to support open container orchestration frameworks such as Kubernetes and Pivotal Cloud Foundry as well virtualised environments. Photon Controller functions as the brain of Photon Platform.

NOTE:  Photon Controller is no longer actively maintained by VMware.


Xenon is a decentralised Control Plane framework for writing small REST-based services (Microservices). The photon controller project makes heavy use of Xenon to build a scalable and highly available Infrastructure-as-a-Service fabric, composed of stateful services, responsible for configuration (desired state), work flows (finite state machine tasks), grooming and scheduling logic.

Project Lightwave 

Project Lightwave offers enterprise-grade, identity and access management services such as single sign-on, authentication, authorization, and certificate authority, as well as certificate key management for container based workloads. It is designed for environments that require multi-tenant, multi-master, scalable LDAP v3 directory services. Lightwave authentication services support Kerberos, OAuth 2.0/OpenID Connect, SAML and WSTrust which enable interoperability with other standards-based technologies in the datacenter.

Photon OS + Project Lightwave

Lightwave is an open source security platform from VMware that authenticates and authorises users and groups with AD or LDAP. Through integration between Photon OS and Project Lightwave, organisations can enforce security and governance on container workloads, for example, by ensuring only authorised containers are run on authorised hosts, by authorised users.

Evolution of the Software Defined Data Center

One of the core tenets of the SDDC is the speed at which you can adapt your IT infrastructure to the needs of the business, so what if those needs are not foreseen in the original plans for the SDDC, in other words what if they go beyond the capabilities of the software that is powering your SDDC environment. To this end VMware is partnering with Pivotal to deliver something called a Developer-Ready Infrastructure. Integrating Pivotal Cloud Foundry (PCF) with VMware SDDC infrastructure solutions. PCF will provide a PaaS to the developer community so they don’t have to worry (or care) about the underlying infrastructure at all, and at the same time the VMware admin is still empowered by his VMware toolset to maintain and provide services to make this PaaS run.

VMware PKS

At VMworld 2017 VMware announced their collaboration with Pivotal to deliver Kubernetes-based container services for multi-cloud enterprises and service providers. It pulls in many of the technologies mentioned before to make a Kubernetes environment more “enterprise grade”, like NSX-T for the networking piece and Project Hatchway for persistent storage. Additionally PKS will include a service broker that provides out-of-the-box access to GCP services. It will enable a VMware admin to expose certain GCP services so that development teams can provision and consume GCP services by creating and managing “service instances” with the kubectl CLI or API. The GCP service broker supports offering GCP subscription services such as Google Cloud Storage, Google BigQuery, and Google Stackdriver. These services will be able to be consumed by applications running on-premises or from within GCP.

In summary, VMware is trying to provide a bridge between multiple different parties, a bridge between developers and VMware admins, a bridge between traditional VMware infrastructure and Public Cloud based services (incidentally, this was also the premise of VMware’s former vCloud Air platform, now sold to French cloud hosting provider OVH.) The question you should pose yourself is, I think, whether it makes sense to incorporate these new architectures, to what extend are your applications unique snowflakes that do not fit this new world of micro-services, have you tended to pets all your life or can some of them be considered cattle. Furthermore it remains to be seen if the pendulum will swing back at some point. Legacy will be here for a long time yet.

*That was tongue in cheek, just in case that wasn’t clear

Rubrik’s Masterless Cluster approach

Rubrik’s Masterless Cluster approach

I’ve written a blog post about Atlas, our distributed filesystem before, where I tried to differentiate between pre-packaged solutions, ultimately using existing legacy architectures with inherited build-in single points of failures, and a truly distributed system where each node in the cluster can perform all of the tasks that are required for successful operation of said cluster.

Atlas is just one piece of the puzzle, we need to further differentiate between using a partially distributed system design, i.e. using a distributed storage layer, in combination with a legacy management architecture, and using a distributed storage layer as part of a modern masterless architecture. In the case of Rubrik, all components operate in a distributed, masterless design. Each 2U appliance contains 4 individual nodes (individual servers), each individual server runs our Cloud Data Management solution as part of the bigger overall cluster. If a node (or an individual component inside a node) fails it has no impact on the overall correct operation of the cluster. The only component that is shared is the redundant power supply.

Can the same be said of running a legacy software stack on top of a distributed storage layer? Are all components running completely distributed? If the management components fail, does a distributed storage layer help you?

Let’s start by looking at a legacy stack example;

If you think about this from the pov of taking a backup, most of these components have a role to play. If one of the components is not operational no backups are taken, and the RPO keeps increasing until the backup solution is repaired.

Now let’s focus on distributing part of the legacy stack;

Again thinking about this from the pov of taking a backup, assuming the system/management components are operational you now have a scale-out, redundant storage layer which brings additional flexibility but it does not lessen the burden on the rest of the stack. If something in the system/management layer fails having a redundant storage layer does not save you, you are again increasing your RPO until the infrastructure is restored.

Now contrast this with Rubrik’s masterless approach;

Seeing that all processes (we don’t have the traditional demarcations of media server, search server, proxy/data mover, etc, but we do run processes/applications in a distributed manner in the cluster that provide all the CDM functions) are running on all nodes, we can similarly lose hardware components (because everything fails eventually) but this will not bring down the master server, simply because we don’t have a master server. Let’s say a node is working on ingesting backup data from your primary system and the node fails, the task is not marked as complete in our distributed system so another node in the cluster will pick up the job and make sure the backup is taken.

Rubrik has opted to build a truly scalable, masterless cluster, neatly leapfrogging legacy approaches of the past by taking lessons from massive-scale modern environments found at Google, Facebook and the like. To be fair, when legacy backup system where designed these giants’ shoulders were not available…

The bigger picture

Even though the principles and technology behind Rubrik are fascinating, it is important not to lose sight of the bigger picture. The idea is to deliver a greatly simplified approach to data management, that the architecture is a converged, masterless, scale-out, self-healing system, etc is nice but the real benefit lies in the operational use.

An additional benefit of using this design is that because we have a unified system, we can present this via a unified RESTful API. Integrating/automating the storage target, or integrating with the management components is not separate, simply because the system is one. Now contrast that with running multiple different components, each with their own, or non-existing integration points, the practical experience with be night and day. A better (much better) experience is ultimately what we are after, after all.

And finally, I wanted to point out that the pictures above are sort of looking at a traditional type of on-premises only environment, but what about running in the public cloud, multi-cloud, hybrid environment, branch offices, etc? Again the model we use delivers the flexibility to run anywhere since you don’t have to drag around these legacy interlinked pieces.


You snooze, you lose.

You snooze, you lose.

At the end of October 2016 I created a short blog post called “Data on tape is no longer useful” it’s premise was that if your historical data is being stored offline you can’t readily use it for other value-add services.

More recently Rubrik has delivered the ability to use archived backup copies of your data and spin those up on-demand as public cloud resources, called CloudOn. Initially taking VMware based virtual machines, archive them to Amazon S3 and when the time comes automatically migrate those workloads from VMware to Amazon and spin them up as AMI based EC2 instances.

Now with the release of version 4.1 we added the ability to support a similar scenario but spin up the workloads on Microsoft Azure instead of AWS. Additionally we added archiving support for Google Cloud Storage enabling more and more multi-cloud capabilities. (CloudOn for Google Cloud Platform is currently not available).

Multi-Cloud er… Multi-Pass

Since I’ve just written a post about all the things Rubrik is (was) doing with Microsoft I now need to add Microsoft Azure CloudOn to the list (I snoozed).

The idea is you add Microsoft Blob Storage as an archive location to one or more of your SLAs and once the data has been archived off you can now use that archive copy to instantiate your workloads, which where originally running on VMware (VMware VMDK based), on Microsoft Azure, as Azure based VMs (VHD based).

Screen Shot 2017-10-01 at 16.15.45

You can opt to do this conversion completely on-demand or choose to auto-convert the latest backup copy to save time during instantiation. The inner-workings a bit different depending on which scenario you choose but generally speaking Rubrik takes the VMDK file and converts it to a VHD file, uploads the VHD file to Azure Blob storage as page blobs (vs block blobs which are typically used for discrete storage objects like jpg’s, log files, etc. that you’d typically view as a file in your local OS.)

Screen Shot 2017-10-01 at 16.22.29

We also added support for Azure Stack in 4.1, in this initial release we provide the same type of functionality as we do with Azure (public cloud), meaning we support Windows, Linux, and SQL (customer installed) based workloads via the Rubrik backup service.

For a grand discussion of Azure Stack (a.o.) I suggest listening to this wonderful episode of the Datanauts podcast with Jeffrey Snover, Technical Fellow at Microsoft and Chief Architect for Azure Infrastructure and the creator of Powershell.

If you want to get more of the details about the 4.1 release please check out one (or all) of these fine posts;

Rubrik and Microsoft, rest assured.

Rubrik and Microsoft, rest assured.

Rubrik has been working with Microsoft’s solutions in various ways since version 2.3 with the initial support of Microsoft Azure as an archive location for long-term retention data. You can even argue the relationship started earlier than that with support for application consistent backups for Microsoft applications through our own VSS provider and requester (for Windows OS’s on Virtual Machines) since the beginning.

But for this overview, I’ll show the Microsoft specifics from version 2.3 till version 4.0, which is currently GA.

Azure Blob storage as an archive location

At the heart of Rubrik sit our SLA policies that govern how we handle your data end-to-end, as part of that SLA policy you can define archival settings, these will determine when we need to start moving data of the local Rubrik cluster and onto Azure Blob storage.

Azure Blob storage is a massively-scalable object storage location for unstructured data, the nice thing about using it as an archive with Rubrik is that because we index all data we can search for specific files/folders in this unstructured data, and only pull back those specific objects instead of needing to re-hydrate the entire fileset. Furthermore the data we sent to Azure Blob storage is encrypted.

Screen Shot 2017-09-03 at 21.22.03.png

Support for physical Windows and Microsoft SQL Server

In Rubrik CDM version 3.0 we added support for physical Windows and MS SQL server via the use of the backup service. The backup service is a very lightweight agent that you install on your physical Windows server, it runs in user-space and doesn’t require a reboot, once installed it is managed throughout it’s lifecycle via the Rubrik cluster itself. In other words once the backup service is installed you don’t need to manage it manually, it is also valid for all applications, meaning that it provides filesystem backup but also SQL backup capabilities.

Screen Shot 2017-09-05 at 15.53.05

Rubrik also provides protection and data management of SQL Server databases that are installed across nodes in a Windows Server Failover Clustering, additionally we support Always-On availability groups as wel, Rubrik detects that a database is part of an availability group. In the event of a failover, protection will be transferred to another availability database in the same availability group. When protection is transferred, the Rubrik cluster transfers the existing metadata for history and data protection to the replacement database.

Rubrik Cloud Cluster in Microsoft Azure

Since Rubrik CDM v3.2 we support running a 4-node (minimum) cluster in Microsoft Azure. We use the DSv2-series VM sizes, which gives us 4 vCPU, 14GiB RAM, 400 GiB SSD, and a max of 8 HDDs per VM.

Screen Shot 2017-09-06 at 12.00.56

Through the Rubrik backup service we are able to support native Azure workloads running either Windows or Linux, and Native SQL. Since Cloud Cluster in Azure has the same capabilities we can also archive to another Azure Blob storage location within Azure for long term retention data (i.e. move backup data of the the DSv2 based instances and unto more cost-effective Blob storage), and even replicate to another Rubrik Cluster, either in Azure or another Public Cloud provider, or even to a physical Rubrik cluster on-premises.

Microsoft SQL Server Live Mount

One of the coolest, in my humble opinion, new features in version 4.0 of Rubrik CDM is the ability to Live Mount SQL databases. Rubrik will use the SSD tier to rapidly materialize any point-in-time copy of the SQL database and then expose this, fully writetable DB to the SQL server as a new DB. In other words you are not consuming any space on your production storage system to do this.

Screen Shot 2017-09-06 at 12.13.16

As a SQL DBA you can now easily restore individual objects between the original and copy. The speed of recovery is greatly reduced, and the original backup copy is still maintained as an immutable object on Rubrik safeguarding it against ransomware.

Microsoft Hyper-V

Last, but not least, is our support for Microsoft Hyper-V based workloads which we achieve by integrating directly with the Hyper-V 2016 WMI based APIs, so this works independent of the underlying storage layer for the broadest support.

Screen Shot 2017-09-06 at 12.43.17

We leverage Hyper-V’s Resilient Change Tracking (RCT) to perform incremental forever backups. Older versions of Hyper-V are also supported through the use of the Rubrik backup service.

Independent of the source, be it virtual or physical, we can leverage the same SLA policy based system to avoid having to set individual, and manual backup policies.


VMworld US – Sunday

VMworld US – Sunday

This is my 7th VMworld event, but the first time attending VMworld in the US, I’ve been to Vegas before but arriving this Saturday evening it felt different somehow, maybe it was the scorching heat or the onslaught of UFC/Boxing fans at the MGM Grand, both where very “in your face”.


VMware signage welcoming you to Las Vegas

I decided to check out the OpeningActs event at the Beerhaus which was a pretty neat experience. The first panel session, moderated by John Troyer (aka John Trooper) was on VMware’s continued relevance in todays rapidly changing tech world which spurred a lot of Amazon AWS remarks. I think the overall consensus was that VMware might be in “sustainability” mode but that it is still an excellent foundation in a lot of customer environments. “Basic infrastructure” might be boring, but it is of great importance nevertheless. I think the other angle that might have been overlooked a bit here is that Amazon needs this relationship with VMware too, it needs a way into that “boring” enterprise market.


Second panel session was on “How failing made me better”, moderated by Jody Tyrus.
My favourite quote from the session was, and I’m paraphrasing here; “I sometimes still feel like I’m a 12 year old in a body of a 42 year old that is about to be exposed”. Rebecca Fitzhugh wrote and excellent blog on her experience with dealing with failure here.



Unfortunately I had to skip the 3rd session because I needed to get to the Expo for a Rubrik all-hands briefing before the Welcome Reception. After the reception I headed to the VMUG party at the House of Blues, which was pretty cool (because they served Stella Artois 😉 ), Michael Dell made an appearance and talked about his love for the VMUG and hinted at some cool Pivotal announcements coming this week.


After the party I needed to get jet-lagged self to bed. Looking forward to a great VMworld week.



A year at Rubrik, moving to marketing; et tu, Brute?

A year at Rubrik, moving to marketing; et tu, Brute?

I wasn’t really planning on writing a post on this initially but since I’m moving to a (technical) marketing role it seemed only fitting on second thought.
Switching from an SE role to technical marketing seems to typically bring about the “I’m moving to the dark side and betraying my technical roots” blog posts so here goes 😉

Actually I think my previous role as an SE for the EMEA region already had quite some tech marketing aspects to it. I traveled to a bunch of countries in Europe, the Middle East, and Africa spreading the Rubrik gospel, was a speaker at countless events, presented numerous webinars, and very occasionally wrote about some Rubrik related stuff on my personal blog. So when my bosses asked me what I wanted to do next I immediately blurted out “tech evangelism!”.

My role as an EMEA SE mainly revolved around covering the countries where we did not have a local presence yet (did you know EMEA consists of 128 countries? me neither!, but hello frequent flyer status) and I had a blast doing that, but since the company is growing like crazy, i.e. hiring local sales teams everywhere, I naturally started thinking about what could be next as well. I feel extremely lucky, and very humbled, to be joining the amazing Rubrik Tech Marketing team (Chris Wahl, Andrew MillerRebecca Fitzhugh) and only hope I can live up to their standard.

…googles “imposter syndrome”…

Andrew Miller already did a great job describing what the tech marketing function is/can be all about here. Or as Matthew Broberg jokingly (but not really though) described it in episode 94 of The Geek Whisperers

It’s this weird unicorn position that seems to fit in the middle of a Venn diagram between sales, marketing, engineering, and some other stuff that we can’t quite put our finger on.

June also marked my 1 year anniversary at Rubrik and with the risk of sounding insincere, it has been an amazing ride so far. The company has more than doubled in size since I joined, seen massive customer adoption, went through multiple successful funding rounds and delivered very significant features release after release.

Screen Shot 2017-06-21 at 15.37.59

To infinity and beyond!


Having worked at large incumbent companies before I find it quite amazing what you can achieve if you’re not bogged down by big company inertia and office politics. Sticking with the rocket ship analogy imagine building one at a start-up vs. building one at a big established company.


On the left you have the potential result after 6 months of trying to build a rocket ship at a large established company, at the right the potential result at a start-up (oops, time to iterate baby!)


After 1 year… you get the idea…

All very tongue-in-cheek of course, both have their merits and drawbacks, in the end it’s about finding where you fit best I think, and I certainly feel I’ve done that.

So in light of the new function, expect me to show up at more events, deliver more content, and be a bit more vocal on social media (but always trying hard to maintain technical integrity).

Rubrik Alta feature spotlight: AWS.

Rubrik Alta feature spotlight: AWS.

With the announcement of Rubrik CDM version 4, codenamed Alta, we have added tons of new features to the platform, but since most of the release announcements are focussed on providing an overview of all the goodies, I wanted to focus more deeply on one specific topic namely our integration with AWS.

Rubrik has had a long history of successfully working with Amazon Web Services, we’ve had integration with Amazon S3 since our first version. In our initial release you could already use Amazon S3 as an archive location for on-premises backups, meaning take local backups of VMware Virtual Machines and then keep them on the local Rubrik cluster for a certain period of time (short term retention) with the option to put longer term retention data into Amazon S3. The idea was to leverage cloud storage economics and resiliency for backup data and at the same time have an active archive for longer term retention data instead of an offline copy on tape. Additionally the way our metadata system works allows us to only retrieve the specific bits of data you need to restore instead of having to pull down the entire VMDK file and incurring egress costs potentially killing the cloud storage economics benefit.

Screen Shot 2017-06-12 at 18.40.13

Also note there is no need to put a gateway device between the Rubrik cluster and Amazon S3, instead Rubrik can natively leverage the S3 APIs.

The ability to archive to Amazon S3 is still here in version 4 of course but now all the supported sources besides VMware ESXi, like Microsoft Hyper-V, Nutanix AHV, Physical Windows/Linux, Native SQL, and so on can also leverage this capability.

Then in Rubrik CDM version 3.2 we added the ability to protect native AWS workloads by having a Rubrik cluster run inside AWS using EC2 for compute and EBS for storage.

Screen Shot 2017-06-13 at 10.40.43We’ll run a 4 node (protecting your data using Erasure Coding) Rubrik cluster in you preferred AWS location (the Rubrik AMI image is uploaded as a private image).
We use the m4.xlarge instance type, using 64GB RAM, 256GB SSD (GP SSD (GP2)) and 24TB Raw Capacity (Throughput Optimised HDD (ST1)), resulting in 15TB usable capacity before deduplication and compression.

Once the Cloud Cluster is running you can protect your native AWS workloads using the connector based approach, i.e. you can protect Windows and Linux filesets, and Native SQL workloads in the Public Cloud.

Additionally since potentially you can now have a Rubrik cluster on-premises and a Rubrik Cloud Cluster you can replicate from your in-house datacenter to your public cloud environment and vice versa. Or replicate from one AWS region to another.Screen Shot 2017-06-13 at 10.46.06

Since the Cloud Cluster has the same capabilities as the on-premises one it can also backup your AWS EC2 workloads and then archive the data to S3, essentially going from EBS storage to S3. (Christopher Nolan really likes this feature).

Screen Shot 2017-06-13 at 10.53.11

Version 4 of Rubrik CDM extends our AWS capabilities by delivering on-demand app mobility, called CloudOn. The idea is that now you can take an application that is running on-premises and move it on the fly to AWS for DR, or Dev/Test, or analytics scenarios.

Screen Shot 2017-06-13 at 17.31.23

The way it will work is that just as since v1 you archive your data to AWS S3, once you decide to instantiate that workload in the public cloud you select “Launch On Cloud” from the Rubrik interface and a temporary Rubrik node (spun up on-demand in AWS in the VPC of your choice) converts those VMs into cloud instances (i.e. going from VMware ESXi to AWS AMI images). Once complete the temporary Rubrik node powers down and is purged.

launch on cloud

Rubrik scans the configuration file of a VM to understand its characteristics (compute, memory, storage, etc.) and recommends a compatible cloud instance type so you are not left guessing what resources you need to consume on AWS.

Alternatively we can also auto-convert the latest snapshot going to S3 so you don’t have to wait for the conversion action.

Data has gravity, meaning that once you accumulate more and more data it starts to make more sense to move the application closer to the data since the other way around starts to become less performant, and more and more cost prohibitive in terms of transport costs. So what if your data sits on-premises but your applications are running in the public cloud? Screen Shot 2017-06-13 at 18.06.53
For instance let’s say you want to perform business analytics using Amazon QuickSight but your primary data sources are in your private data center, now you simply archive data to S3 as part of your regular Rubrik SLA (archive data older than x days) and pick the point-in-time dataset you are interested in, use Rubrik to “Launch on Cloud”, and point Amazon QuickSight (or any other AWS service) to your particular data source.

Together these AWS integrations allow you to make use of the public cloud on your terms, in an extremely flexible way.

Marrying consumer convenience with enterprise prowess

If you look at the consumer applications we interface with on a daily basis, things like Facebook, Google, Twitter,… these all tend to be very easy to use and understand, typically little to no explanation is needed on how to use them, you simply sign-up and you get going.

dc8370d5-9876-458d-8f97-4bfcce30bfd4But behind the covers of these very straightforward interactions lies a pretty complex world of intricate components, a lot of moving parts that make the application tick. For a little insight into the back-end architecture of Facebook for example, check out these videos;

The typical target audience of Facebook I would guess is completely unaware of this, and rightfully so.

Now think about your typical enterprise applications, probably a lot less straight forward to interact with, a lot of nerd knobs to turn and a lot of certifications to be attained in mastering how to make best use of them.

“Well of course it is a lot more complex, because I need it to be!”  I hear you say, but does it really though?

What if most of the heavy lifting is taken care of by the system itself, internal algorithms that govern the state of the solution and make the application perform how you need it to, minimizing the interaction with the end-user, and making that interaction as enjoyable as possible.

That is what the Rubrik Cloud Data Management solution is trying to achieve, under the hood it is a very, very capable piece of equipment but most of the complexity that comes with these enterprise capabilities has been automated away, and the little interaction that is left to the end-user is very straight forward, and dare I say enjoyable?


Matching enterprise data management capabilities with the simplicity of a consumer application is a lofty goal, but one worthy to pursue in my humble opinion. After all…

“Simplicity is the Ultimate Sophistication”



Atlas distributed filesystem, think outside the box.

Atlas distributed filesystem, think outside the box.

Rubrik recently presented at Tech Field Day 12 and one of the sessions focused on our distributed filesystem called Atlas. As one of the SE’s at Rubrik I’m in the field every day (proudly) representing my company but also competing with other, more traditional backup and recovery vendors. What is apparent more and more is that these traditional vendors are also going down the appliance route to sell their solution into the market, and as such I sometimes get the pushback from potential customers saying they can also get an appliance based offer from their current supplier, or not really immediately grasping why this model can be beneficial to them.
A couple of things I wanted to clarify first, when I say “also down the appliance route” I need make clear that this is purely a way to offer the solution to market for us, there is nothing special about the appliance as such, all of the intelligence in Rubrik’s case lies in the software, we even started to offer a software only version in the form of a virtual appliance for ROBO use cases recently.
Secondly, some traditional vendors can indeed deliver their solution in an appliance based model, be it their own branded one, or pre-packaged via a partnership with a traditional hardware vendor. I’m not saying there is something inherently bad about this, simplifying the acquisition of a backup solution via an appliance based model is great, but there the comparison stops, it will still be a legacy based architecture with disparate software components, typically these software components, think media server,  database server, search server, storage node, etc. need individual love and care, daily babysitting if you will, to keep them going.
Lastly from a component point of view our appliance consists of multiple independent (masterless) nodes that are each capable of running all tasks of the data management solution, in other words there is no need to protect, or indeed worry about, individual software and hardware components as everything is running distributed and able to sustain multiple failures while remaining operational.

nospoon There is no spoon (box)

So the difference lies in the software architecture, not the packaging, as such we need to look beyond the box itself and dive into why starting from a clustered distributed system as a base makes much more sense in todays information era.

The session at TFD12 was presented by Adam Gee and Roland Miller, Adam is the lead of Rubrik’s distributed filesystem called Atlas, it shares some architectural principles with a previous filesystem Adam worked on while he was at Google called Colossus, most items you store while using Google services end up on Colossus, it itself is the successor to the Google File System (GFS) bringing the concept of a masterless cluster to it and making it much more scalable. Not a lot is available on the web in terms of technical details around Colossus, but you can read a high level article on Wired about it here.

Atlas, which sits at the core of the Rubrik architecture, is a distributed filesystem, built from scratch with the Rubrik data management application in mind. It uses all distributed local storage (DAS) resources available on all nodes in the cluster and pools them together into a global namespace. As nodes are added to the cluster the global namespace grows automatically, increasing capacity in the cluster. The local storage resources on each node consist of both SSD and HDD’s, the metadata of Atlas (and the metadata of the data management application) is stored in the metadata store (Callisto) which is also running distributed on all nodes in the SSD layer. The nodes communicate internally using RPC which are presented to Atlas by the cluster management component (Forge) in a topology aware manner thus giving Atlas the capability to provide data locality. This is needed to ensure that data is spread correctly throughout the cluster for redundancy reasons. For example, assuming we are using triple mirror, we need to store data in 3 different nodes in the appliance, let’s now assume the cluster grows beyond 1 appliance, then it would make more sense from a failure domain point of view to move 1 copy of the data from one of the local nodes to the other appliance.


The system is self healing in the way that Forge publishes the disk and node health status and Atlas can react to that, again assuming triple mirror, if a node or entire brik (appliance) fails Atlas will create a new copy of the data on another node to make sure the requested failure tolerance is met. Additionally Atlas also runs a background task to check the CRC of each chunk of data to ensure what you have written to Rubrik is available in time of recovery. See the article How To Kill A Supercomputer: Dirty Power, Cosmic Rays, and Bad Solder on why that could be important.


The Atlas filesystem was designed with the data management application in mind, essentially the application takes backups and places them on Atlas, building snapshots chains (Rubrik performs an initial full backup and incremental forever after that). The benefit is that we can instantly materialize any point in time snapshot without the need to re-hydrate data.


In the example above you have the first full backup at t0, and then 4 incremental backups after that. Let’s assume you want to instantly recover data at point t3, this will simply be a metadata operation, pointers to the last time blocks making up t3 where mutated, there is no data movement involved.

Taking it a step further let’s now assume you want to use t3 as the basis and start writing new data to it from that point on. Any new data that you write to it (redirect-on-write) now goes to a log file, no content from the original snapshot is changed as this needs to be an immutable copy (compliancy). The use case here could be copy data management where you want to present a copy of a dataset to internal dev/test teams.


Atlas also dynamically provides performance for certain operations in the cluster, for example a backup ingest job will get a higher priority than a background maintenance task. Because each node also has a local SSD drive Atlas can use this to place critical files on a higher performance tier and is also capable of tiering all data and placing hot blocks on SSDs. It also understands that each node has 3 HDDs and will these to stripe the data of a file across all 3 on a single node to take advantage of the aggregate disk bandwidth resulting in a performance improvement on large sequential reads and writes, by utilizing read-ahead and write buffering respectively.

For more information on Atlas you can find the blogpost Adam Gee wrote on it here, or watch the TFD12 recording here.