Author: Filip Verloy

Rubrik Datos IO and MongoDB

Rubrik Datos IO and MongoDB

In februari of 2018 Rubrik announced the acquisition of Datos IO, a startup with a focus on data management for an underserved part of the market namely (a.o.) NoSQL databases like Cassandra and MongoDB. Before jumping into more specifics about what Rubrik Datos IO is enabling for NoSQL databases, let’s first talk about the concept of these types of databases using MongoDB as an example.

The name MongoDB is derived from the term huMONGOus, a nod to the capability of NoSQL DB’s to handle large amounts of information.

NoSQL gained popularity through the needs of large webscale players like Facebook and Google which could no longer be easily met by traditional RDBMS (NoSQL referring to “non-relational” as opposed to the R in RDBMS) systems like Oracle and MS SQL. They required more flexible data models and needed to be able to serve extremely large amounts of incoming data from application users around the world whilst maintaining consistent performance and reliability.

Note: besides relational databases there are others as well, including Columnar (HBase), Graph (Neo4j), Key-value (Redis), etc.

MongoDB uses yet another variant called a document-based data model, meaning that you can sort of put anything you want into MongoDB as long as you follow it’s JSON (as it’s standard object type) like document style. You don’t need to use the same schema across different documents, you can have different fields in every document, and there is also no need to have a single key value (this is for example different in Cassandra), it does however automatically apply a unique object id (“_id”) to each document. This translates to a lot of flexibility overall. Today the use of NoSQL has gone beyond the large webscale players and infiltrated start-ups (with ambitions of becoming the next Facebook) and indeed also enterprise environments (it could possibly sometimes be debatable if the RDBMS route wouldn’t suffice in some of these cases but that is another discussion).

NoSQL and Cap theorem and ACID transactions

CAP (Consistency, Availability, and Partition-tolerance) theorem determines how distributed database systems behave in the face of network instability. The database can be Consistent, meaning writes are atomic (transactions either execute in full or do not execute at all) and when you read data after the write the value is the last write. The database can be Available, meaning you will get a value back as long as a single node is operational. And/or the database can be Partition-tolerant, meaning operational even when communication is (temporarily) lost between the nodes. Cap theorem states however that at most two of these are applicable at once, and never all three.

Note: there are claims of potentially achieving all three at once, as in the case of this research paper on Google Spanner.

ACID (Atomicity, Consistency, Isolation, Durability) then, is a set of properties of database transactions intended to guarantee validity even in the event of errors, power failures, etc.

Many NoSQL stores compromise consistency (as used in CAP) in favor of availability, partition-tolerance, and speed. Most NoSQL stores also lack true ACID transactions. Instead, most NoSQL databases offer a concept of “eventual consistency” in which database changes are propagated to all nodes “eventually” (typically within milliseconds) so queries for data might not return updated data immediately or might result in reading data that is not accurate, a problem known as stale reads.

MongoDB falls in the consistency and partition-tolerance spectrum of CAP preferring consistency over availability by using a single master design, meaning that if the master is down the database is unavailable for whilst a new master is elected.

MongoDB Architecture

MongoDB uses the concepts of databases, collections (traditionally tables), and documents (traditionally rows).

As mentioned before MongoDB uses a single master architecture, preferring consistency over availability, but you can use secondaries which maintain backup copies of your master database instance. These are managed using replica sets, when writes come into the primary these are replicated using an operation log (Oplog) to the secondary nodes (which can be spread across datacenter locations). When the primary goes down the secondaries can elect a new primary within seconds (all the while data is available as read-only) with the understanding that you need a majority vote to make this happen. Replica sets are meant to address durability, not scalability (i.e. you shouldn’t really read from your secondary nodes). To address scalability we need to look at sharding whereby you have multiple replica sets (combination of a primary plus a bunch of secondary nodes) with each replica set being responsible for a range of indices. In other words for this to work you need to have a way to use an index in the first place so you need to build that (index based on some unique value) into your data model somehow. From the point of view of your application server you need to use a set of routers (mongos) that talk to 3 config servers and then point you to the correct replica set. MongoDB mongos instances route queries and write operations to shards in a sharded cluster. mongos provide the only interface to a sharded cluster from the perspective of applications. Applications never connect or communicate directly with the shards.

Next-Generation Backup and Recovery for MongoDB

Rubrik Datos IO is meant to facilitate application-consistent backup, point-in-time recovery, and deduplication for sharded and unsharded MongoDB clusters and geo-distributed MongoDB deployments.


The solution is called RecoverX which is a scale-out data protection software that acts as the control plane to move backup data between your production MongoDB environment and a secondary storage location. It performs continuous backups using the Oplog’s, it streams data in parallel from multiple MongoDB nodes. It also semantically deduplicates files across the MongoDB cluster in order to only store only one instance of the data if you are backing up to different directories but is simultaneously tracked in a metadata map for recovery.

RecoverX workflow

The workflow of backing up your MongoDB data is based on a policy in which you select the datasource (MongoDB or others), select the specific MongoDB cluster and version store (secondary storage location), which specific database(s) you want to protect and a schedule that defines RPO/RTO values.

Screen Shot 2018-06-18 at 17.49.06

Once the policy is added the first full backup is triggered, this happens without quiescing the database.

Recovering data works via the Time Travel tab which allows you to select the directories to recover and see it’s available timestamps. Orchestrated recovery then allows you to select the MongoDB destination cluster, database, and collection to restore to.

Screen Shot 2018-06-18 at 18.38.11

Collections can be recovered to the same MongoDB database or a different instance for test and dev.

RecoverX can be deployed on physical or virtual servers or run in the Public Cloud.

Everything old is new again.

My first job in IT, many moons ago, was at a company called MOTEC, it was a subcontractor of Telenet, one of the biggest ISPs in Belgium. We were responsible for building the cable based telephony systems that their customers used to make phone calls over the tv distribution network.

I got hired because I knew some things about Quadruple Amplitude Modulation (QAM) and Quadrature Phase-Shift Keying (QPSK), and a bit about Sun Solaris as well. The system we operated was developed by ECI-Telecom, a Israeli based manufacturer of telecommunications equipment, and was running mostly on a Sun Microsystems infrastructure. I spent many days, and nights (incl. New Year’s Eve 1999 in fear of the Y2K bug) in the customers Network Operations Center (NOC) looking after this system and making sure it ran smoothly.

It was there that I developed a love for Sun Microsystems (my workstation was a SUN Ultra 5) and it’s Solaris OS, things like IPS, SMF, Dtrace (which Oracle recently open-sourced under the GPL), and later ZFS (I run a FreeNAS system at home, just because), and it’s rock solid stability where a joy compared to some other OS’es at the time. Besides the technology I was also fascinated by some of the people at the company, not the least of which was it’s co-founder Andreas von Bechtolsheim (or Andy Bechtolsheim for you Americans) currently the co-founder of Arista Networks, and one of the original investors in Google.  To get a sense of his many accomplishments check out the video below.

He also recently presented at a Tech Field Day event (NFD16) on the future of networking, specifically the 400 Gigabit Landscape, also available on YouTube if you are inclined to watch it. One of the other co-founders of Sun Microsystems, Vinod Khosla, recently came onboard as an investor at Rubrik (where I work) through his VC firm Khosla Ventures which warmed my heart. I always felt a great respect for the company even though I later ended up competing with them in the market when I joined my first vendor company.

We all know what happened to Sun Microsystems in the end, but I’m still waiting for Jonathan Schwartz to write that Sun Microsystems book he tweeted about shortly after the acquisition by Oracle in 2010 🙂

There is also an interesting article from the IEEE spectrum called “After the Sun (Microsystems) Sets, the Real Stories Come Out” that presents some insights into what is was like at the time.

Notwithstanding the rise of Linux, there is a surprising amount of Sun Solaris (and other Unix systems, like AIX) still out there in the market running mission-critical workloads at banking, retail, telecom and other organisations worldwide. (I couldn’t find a recent market share percentage number that was freely shareable, if you do please feel free to let me know).

I would recon that most of these systems are treated as their own little (or very big) island and are handled by specialised personnel. But nevertheless there are certain aspects of IT operation that are a common requirement across all heterogeneous systems, backup and recovery being one of them. I distinctly remember the tape-driven horror show I had to deal with on a daily basis, I’ll refrain from naming the backup solution to protect the not-so innocent. To that end Rubrik recently announced support for Sun Solaris in it’s 4.2 release. (AIX was already released earlier).

Rubrik CDM 4.2 support for Sun Solaris

As we do for all backups we also use our SLA based approach, incremental-forever backups, archival to the Public Cloud, etc. Similar to the Linux and Windows support, we are offering a connector (RBS/Rubrik Backup Service) with low resource and operational overhead. The connector supports granular folder/file level backups via the filesets concept that integrates with SLA policies. Filesets are a logical construct based on Includes and Excludes. These are often around a direct path to a file or folder but can use wildcards for greater flexibility around what is included or excluded.

Screen Shot 2018-05-23 at 14.27.43

Filesets can be customized to meet specific needs including the ability to call Pre/ Post Scripts to a.o. provide application-consistent backups. Welcome to the future! 😉

Containers and persistent storage

Containers and persistent storage

Containers are a method of operating system virtualization that allow you to run an application and its dependencies in resource-isolated processes. Containers allow you to easily package an application’s code, configurations, and dependencies into easy to use building blocks that deliver environmental consistency, operational efficiency, developer productivity, and version control. Containers are immutable, meaning they help you deploy applications in a reliable and consistent way independent of the the deployment environment.

As containers continue to rise in popularity beyond the developer populous the way these constructs are being used becomes increasingly varied, especially (but not exclusively) in light of enterprise applications the questions of persistent storage comes up more and more. It is a fallacy to think only stateless application can or should be containerized. If you take a look at you’ll see that about half of the most popular applications on Docker Hub are stateful, like databases for example. (Post hoc ergo propter hoc?). If you think about monolithic applications versus micro-services, a monolithic application typically requires state, if you pull this application out into micro-services, some of these services can be stateless containers but others will require state.

I’ll mainly use Docker as the example for this post but many other container technologies exist like LXD, rkt, OpenVZ, even Microsoft offers containers with Windows Server Containers, Hyper-V isolation, and Azure Container Service.

Running a stateless container using Docker is quite straightforward;

$ docker run --name demo-mysql mysql

When you execute docker run, the container process that runs is isolated in that it has its own file system, its own networking, and its own isolated process tree separate from the (local or remote) host.

The docker container is created from a readonly template called docker image. The “mysql” part in the command relates to this image, i.e. containerized application, that you want to run by pulling it from the registry. The data you create inside a container is stored on a thin writable layer, called the container layer, that sits on top of the stack of read-only layers, called the image layers, present in the base docker image. When the container is deleted the writable layer is also deleted so your data does not persist , in docker the docker storage driver is responsible for enabling and managing both the read-only image layers and the writable layer, both read and write speeds are generally considered slow.

Image result for docker layers

Assuming you want persistent data for your containers there are several methods to go about this. You can add a storage directory to a container’s virtual filesystem and map that directory to a directory on the host server. The data you create inside that directory on the container will be saved on the host, allowing it to persist after the container shuts down. This directory can also be shared between containers. In docker this is made possible by using volumes, you can also use bind mounts but these are dependent on the directory structure of the host machine whereas volumes are completely managed by Docker itself. Keep in mind though that these volumes don’t move with container workloads as they are local to the host. Alternatively you can use volume drives (Docker Engine volume plugins) to store data on remote systems instead of the Docker host itself. If you are only interested in storing data in the container writeable layer (i.e. on the docker host itself) you can use Docker storage drivers which then determine which filesystem is supported.

Typically you would create a volume using the storage driver of your choice in the following manner;

$ docker volume create -—driver=pure -o size=32GB testvol1

And then start a container and attach the volume to it;

$ docker run -ti -v testvol1:/data mysql

Storage Vendors and Persistent Container Storage

Storage vendors have an incentive to make consuming their particular storage as easy as possible for these types of workloads so many of them are providing plug-ins to do just that.

One example is Pure Storage who provide a Docker Volume Plugin for their FlashArray and FlashBlade systems. Current they support Docker, Swarm, and Mesos. Most other big name storage vendors also have plugins available.

Then there are things like REX-Ray which is an open source, storage management solution, it was born out of the now defunct {code} by Dell EMC team. It allows you to use multiple different storage backends and serve those up as persistent storage for your container workloads.

On the virtualization front VMware has something called the vSphere Docker Volume Service which consists of two parts, the Docker Volume Plugin and a vSphere Installation Bundle (VIB) to install on the ESXi hosts. This allows you to serve up vSphere Datastores (be it Virtual SAN, VMFS, NFS based) as persistent storage to your container workloads.

Then there are newer companies that have been solely focusing on providing persistent storage for container workloads, one of them is Portworx. Portworx want to provide another abstraction layer between the storage pool and the container workload. The idea is that they provide a “storage ” container that can then be integrated with the “application” containers. You can do this manually or you can integrate with a container scheduler like Docker Swarm using Docker Compose for example (Portworx provides a volume driver).

Docker itself has built specific plugins as well, Cloudstor is such a volume plugin. It comes pre-installed and pre-configured in Docker swarms deployed through Docker for AWS. Data volumes can either be backed by EBS or EFS. Workloads running in a Docker service that require access to low latency/high IOPs persistent storage, such as a database engine, can use a “relocatable” Cloudstor volume backed by EBS. When multiple swarm service tasks need to share data in a persistent storage volume, you can use a “shared” Cloudstor volume backed by EFS. Such a volume and its contents can be mounted by multiple swarm service tasks since EFS makes the data available to all swarm nodes over NFS.

Container Orchestration Systems and Persistent Storage

As most enterprise production container deployments will utilize some container orchestration system we should also determine how external persistent storage is managed at this level. If we look at Kubernetes for example, that supports a volume plugin system (FlexVolumes) that makes it relatively straightforward to consume different types of block and file storage. Additionally Kubernetes recently started supporting a implementation of the Container Storage Interface (CSI) which helps accelerate vendor support for these storage plug-ins as volume plugins are currently part of the core Kubernetes code and shipped with the core Kubernetes binaries meaning that vendors wanting to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) must align themselves with the Kubernetes release process. With the adoption of the Container Storage Interface, the Kubernetes volume layer becomes extensible. Third party storage developers can now write and deploy volume plugins exposing new storage systems in Kubernetes without having to touch the core Kubernetes code.

When using CSI with Docker, it relies on shared mounts (not docker volumes) to provide access to external storage. When using a mount the external storage is mounted into the container, when using volumes a new directory is created within Docker’s storage directory on the host machine, and Docker manages that directory’s contents.

To use CSI, you will need to deploy a CSI driver, a bunch of storage vendors have these available in various stages of development. For example there is a  Container Storage Interface (CSI) Storage Plug-in for VMware vSphere.

Pre-packaged container platforms

Another angle how vendors are trying to make it easier for enterprises to adopt these new platforms, including solving for persistence, is by providing packaged solutions (i.e. making it turnkey), this is not new of course, not too long ago we saw the same thing happening with OpenStack through the likes of VIO (VMware Integrated OpenStack), Platform9, Blue Box (acquired by IBM), etc. Looking at the Public Cloud providers these are moving more towards providing container as a service (CaaS) models with Azure Container Service, Google Container Engine, etc.

One example of packaged container platforms though is the Cisco Container Platform. This is provided as an OVA for VMware (meaning it is provisioning containers inside virtual machines, not on bare metal at the moment), initially this is supported on their HyperFlex Platform which will provide the persistent storage layer via a Kubernetes FlexVolume driver. It then can communicate externally via Contiv, including talking to other components on the HX platform like VMs that are running non containerized workloads. For the load-balancing piece (between k8s masters for example) they are bundling NGINX and for logging and monitoring they are bundling Prometheus (monitoring) and an ELK stack (logging and analytics).

Another example would be VMware PKS which I wrote about in my previous post.


Containers are ready for enterprise use today, however there are some areas that could do with a bit more maturity, one of them being storage. I fully expect to see continued innovation and tighter integrations as we figure out the validity of these use-cases. A lot progress has been made in the toolkits themselves, leading to the demise of earlier attempts like ClusterHQ/Flocker. As adoption continues so will the maturity of these frameworks and plugins.

VMware’s Cloud Nativeness

VMware’s Cloud Nativeness

At first glance it feels like VMware’s efforts related to cloud native applications are meant to cater to the traditional admins, people who are mainly running virtual machines today but want, or need, to support other types of workloads as well. People who secretly don’t “get” this new container hotness* and really want to manage their environment with the tools and knowledge that got them where they are today and are very comfortable with. Or to put it another way, VMware is ultimately trying to cater to both the infrastructure/operations person needing to support these new types of workloads and the developer building them.

Before looking at VMware’s portfolio related to cloud native applications let’s take a step back and examine both what CNA means and what drives it. According to Pivotal, Cloud-native is an approach to building and running applications that fully exploits the advantages of the cloud computing delivery model. Cloud-native is about how applications are created and deployed, not where. “Where” meaning the underlying infrastructure is not as important as it once was, granted, virtualisation already decreased our reliance on static infrastructure, but if we are honest with ourselves infrastructure was still the thing we got excited about the most as VMware admins. Depending on where you see yourself on the “Public Cloud” is the only thing that makes sense going forward scale, you’ll be more or less inclined to accept the idea that cloud computing can be considered a delivery model that is appropriate for both public and private cloud like environments alike. Core to the cloud computing model is the ability to offer seemingly unlimited computing power, on-demand, along with modern data and application services for developers. It is this last piece, I believe, where private cloud solutions fall down the most compared to public cloud vendors, the onslaught of services delivered at more and more rapidly increasing pace by companies like Amazon (AWS), Microsoft (Azure), and Google (GCP) are creating an understandable pull, attracting new workloads. Therefore it seems that when we discuss Cloud Native Applications from the lens of VMware, we again tend to mostly argue the needs of these applications related to the infrastructure piece. If we agree that products and services delivered through software are becoming major competitive differentiators of the business then it is precisely this what is driving the need of Cloud Native Applications. This is where Marc Andreesen’s software is eating the world quote usually comes in…

The way VMware describes it’s portfolio for CNA on their own website is;

VMware offers cloud-native solutions across container networking, security, storage, automation, operations and monitoring for enterprises and service providers.

Since VMware is trying hard to reach beyond it’s traditional enterprise base, including forays into multiple open-source type projects, I wanted to try and provide a somewhat comprehensive overview of everything it’s doing in those spaces and it’s potential use cases. Generally speaking VMware’s cloud native portfolio includes VMware vSphere Integrated Containers, for running containerised workloads with existing infrastructure. But please don’t interpret this as a silver bullet to “lift and shift” all your legacy applications to a more modern environment, start by asking why you can’t re-architecture your existing apps, and only then ask yourself if you will truly benefit from moving to this new approach, maybe for easier maintenance or improved scalability, but I’m willing to bet that mostly this wont be the best approach for your old and crusty stuff. For applications that naturally fit a containerised approach the story if different of-course. Secondly VMware Photon Platform is targeted to building new cloud-native infrastructure solutions from the ground up.

VMware VIC - Page 1 (2)

Very simplified view of components fitting together

vSphere Integrated Containers

VIC is meant for people who want to stick with the more traditional vSphere infrastructure but also provide Container based services to internal developers. It is available with vSphere Enterprise Plus 6.0 or later, or with vSphere with Operations Management Enterprise Plus. It consists of 3 main components, the vSphere Integrated Container Engine (VIC Engine), Admiral, the Container Management Portal, and Harbor, the Container Registry (based on Docker Distribution). The portal provides a user interface and an API for managing container repositories, images, hosts, and instances. The registry furnishes a user interface and an API so developers can make their own container repositories and images. The engine is a container runtime integrated with vSphere.

vSphere, as opposed to Linux, acts as the container host, containers are deployed as VMs (not in VMs), every container is fully isolated from the host and from the other containers.

 VIC Marketecture

With VIC the idea is that you can still leverage the existing management toolsets in your vSphere based environment, since these tools typically have an understanding about the virtual abstraction as a VM, the idea is that you run a Container as a VM (C-VM in the picture above). The goal is to run both traditional and containerised applications side-by-side on a common infrastructure, this, if you will, is it’s main use case, take what you know and extent it so you can support container based workloads in a way that is transparent for the developer in theory. It’s meant to avoid having to build a completely separate environment.

VMware Photon Platform 

VMware Photon Platform is a container-optimised cloud platform, at first it seems to have a somewhat similar goal to VIC, which is to enable developers to build, test, and run container based workloads while still allowing “traditional IT” to maintain security, control and performance of data center infrastructure. But the use cases have a different focus, namely;

  • Kubernetes as a Service: Developers can deploy, resize, and destroy Kubernetes clusters to develop, test, and automate containerised applications.
  • Platform as a Service: Photon Platform integrates with Pivotal Cloud Foundry to build and scale applications.
  • Continuous integration and delivery: Try to improve the CI/CD pipeline with uniformity and reusability, especially in environments with high container churn.
  • API-managed on-premises cloud: IT can deploy resources and automate their management through a RESTful API.
  • Security: The VMware Lightwave security service can protect applications, Kubernetes clusters, and Photon Platform components.

Screen Shot 2017-09-14 at 08.45.45.png

Photon Platform Marketecture

Note that the Photon Controller is no longer a part of the Photon Platform, VMware has made the decision to stop driving this project and instead lean on the open source community to continue this effort. It was originally envisioned to replace vCenter as the control plane for these new type of workloads since vCenter’s monolithic design was never expected to serve these types of applications. More recently of-course the community has moved to other control plane architectures and container orchestration systems.

NSX-T as the secret sauce.

With VIC you can select vSphere networks that appear in the Docker client as container networks. NSX-T intends to extent the networking capabilities and make them more enterprise appropriate. By default a container sits behind a NAT’d interface and as such form a operations point of view you struggle with visibility. An other challenge is consuming services outside the container realm, if you have services that need to be consumed from your traditional VMs or even bare metal systems you typically need to pass via bridging nodes potentially choking traffic. NSX-T tries to overcome these challenges by supporting native container networking and micro-segmentation for CaaS (Container-as-a-Service ) and PaaS (Platform-as-a-Service), providing tools with which network and security teams can operationalise containerised apps in a vSphere environment.

Statefulness and persistent storage

When discussing containers and container storage you cannot ignore the difference between stateless and stateful workloads and the impact that has on storage persistency. Containers are ephemeral, so if you want to run stateful applications that require persistent storage you need to take some additional steps to ensure your data is maintained through the lifecycle of the containers. VMware provides two persistent volume offerings for situations like these, one for Docker and one for Kubernetes. For Docker VMware provides the vSphere Docker Volume Service (vDVS), this plug-in abstracts the underlying storage of the vSphere environment and makes it available as Docker volumes, it also integrates with Docker Swarm. For stateful containers orchestrated by Kubernetes you can also leverage persistent vSphere Storage (vSAN, VMFS, and NFS) with Kubernetes persistent volume, dynamic provisioning and StatefulSet primitives. Both of these projects are now under the Project Hatchway umbrella.

Operationalising a Cloud Native environment with VMware

Again going back to using the tools you know as a VMware admin, VMware has tried to extend vRealize Automation to make it more relevant for these new types of workloads, it leverages the open source Project Admiral, a scalable and lightweight container management platform, to deploy and manage containers through virtual container hosts on VMware vSphere Integrated Containers. The idea is still to provide a separation between the operational folks and the developers. Developers can provision container hosts from the vRealize Automation service catalog as well as model containerised applications using unified service blueprints or Docker Compose.

Open Source components

Photon OS

Version 2.0 of Photon OS was announced at the beginning of November, Photon OS is a Container-Optimized Linux Operating System. It is an open source minimal Linux container host optimized for cloud-native applications, cloud platforms, and VMware infrastructure. Think of container Operating Systems in general as an OS that provides only the minimal functionality required for deploying applications inside software containers, but typically including built-in mechanisms for shared services consumption and orchestration. Photon OS even comes in 2 flavours, the minimal version is a lightweight host tailored to running containers when performance is paramount, whilst the full version includes additional packages to help deploy containerized applications. Other notable container OSes include a.o. CoreOS, RancherOS, Atomic, and to some extent Intel Clear Linux.

For easy consumption Photon OS is available as a pre-built image for the 3 main public cloud providers as well as an ISO and OVA image. Photon OS includes the open source version of the Docker daemon. After installing Photon OS and enabling Docker, you can immediately run a Docker container.

I’ve written about Photon (and AppCatalyst, and Bonneville) on this blog before if you want some more background.

vSphere Integrated Containers Engine

vSphere Integrated Containers Engine (VIC Engine) is the key element of vSphere Integrated Container, it is a container runtime for vSphere, allowing developers familiar with Docker to develop in containers and deploy them alongside traditional VM-based workloads on vSphere clusters, and allowing for these workloads to be managed through the vSphere UI in a way familiar to existing vSphere admins. The VIC Engine provides lifecycle operations, vCenter support, logs, basic client authentication, volume, and basic networking support.


Harbor is an enterprise-class container registry server based on Docker Distribution, it is embedded in both vSphere Integrated Containers and Photon Platform, it provides advanced security, identity, role based access control, auditing, and management services for Docker images. With Harbor, you can deploy a private registry to maintain your data behind the company firewall. In addition, Harbor supports AD/LDAP integration and the setup of multiple registries with images replicated between registries for high availability.


Admiral is a key component of vSphere Integrated Containers solution. It is a container management solution with an accent on modelling containerised applications and providing placement based on dynamic policy allocation. It manages Docker hosts, policies, multi-container templates, and applications to simplify and automate resource utilisation and application delivery. Developers can use Docker Compose, Admiral Templates, or the Admiral UI to compose their app and deploy it using the Admiral provisioning and orchestration engine. VMware admins can manage container host infrastructure and apply governance to its usage, including resource grouping, policy based placement, quotas and reservations, and elastic placement zones.

Photon controller

Photon controller is a distributed, multi-tenant host controller optimised for containers. The Photon Controller exposes RESTful APIs, SDKs, and CLI tooling to automate infrastructure resources. It is built to support open container orchestration frameworks such as Kubernetes and Pivotal Cloud Foundry as well virtualised environments. Photon Controller functions as the brain of Photon Platform.

NOTE:  Photon Controller is no longer actively maintained by VMware.


Xenon is a decentralised Control Plane framework for writing small REST-based services (Microservices). The photon controller project makes heavy use of Xenon to build a scalable and highly available Infrastructure-as-a-Service fabric, composed of stateful services, responsible for configuration (desired state), work flows (finite state machine tasks), grooming and scheduling logic.

Project Lightwave 

Project Lightwave offers enterprise-grade, identity and access management services such as single sign-on, authentication, authorization, and certificate authority, as well as certificate key management for container based workloads. It is designed for environments that require multi-tenant, multi-master, scalable LDAP v3 directory services. Lightwave authentication services support Kerberos, OAuth 2.0/OpenID Connect, SAML and WSTrust which enable interoperability with other standards-based technologies in the datacenter.

Photon OS + Project Lightwave

Lightwave is an open source security platform from VMware that authenticates and authorises users and groups with AD or LDAP. Through integration between Photon OS and Project Lightwave, organisations can enforce security and governance on container workloads, for example, by ensuring only authorised containers are run on authorised hosts, by authorised users.

Evolution of the Software Defined Data Center

One of the core tenets of the SDDC is the speed at which you can adapt your IT infrastructure to the needs of the business, so what if those needs are not foreseen in the original plans for the SDDC, in other words what if they go beyond the capabilities of the software that is powering your SDDC environment. To this end VMware is partnering with Pivotal to deliver something called a Developer-Ready Infrastructure. Integrating Pivotal Cloud Foundry (PCF) with VMware SDDC infrastructure solutions. PCF will provide a PaaS to the developer community so they don’t have to worry (or care) about the underlying infrastructure at all, and at the same time the VMware admin is still empowered by his VMware toolset to maintain and provide services to make this PaaS run.

VMware PKS

At VMworld 2017 VMware announced their collaboration with Pivotal to deliver Kubernetes-based container services for multi-cloud enterprises and service providers. It pulls in many of the technologies mentioned before to make a Kubernetes environment more “enterprise grade”, like NSX-T for the networking piece and Project Hatchway for persistent storage. Additionally PKS will include a service broker that provides out-of-the-box access to GCP services. It will enable a VMware admin to expose certain GCP services so that development teams can provision and consume GCP services by creating and managing “service instances” with the kubectl CLI or API. The GCP service broker supports offering GCP subscription services such as Google Cloud Storage, Google BigQuery, and Google Stackdriver. These services will be able to be consumed by applications running on-premises or from within GCP.

In summary, VMware is trying to provide a bridge between multiple different parties, a bridge between developers and VMware admins, a bridge between traditional VMware infrastructure and Public Cloud based services (incidentally, this was also the premise of VMware’s former vCloud Air platform, now sold to French cloud hosting provider OVH.) The question you should pose yourself is, I think, whether it makes sense to incorporate these new architectures, to what extend are your applications unique snowflakes that do not fit this new world of micro-services, have you tended to pets all your life or can some of them be considered cattle. Furthermore it remains to be seen if the pendulum will swing back at some point. Legacy will be here for a long time yet.

*That was tongue in cheek, just in case that wasn’t clear

Rubrik’s Masterless Cluster approach

Rubrik’s Masterless Cluster approach

I’ve written a blog post about Atlas, our distributed filesystem before, where I tried to differentiate between pre-packaged solutions, ultimately using existing legacy architectures with inherited build-in single points of failures, and a truly distributed system where each node in the cluster can perform all of the tasks that are required for successful operation of said cluster.

Atlas is just one piece of the puzzle, we need to further differentiate between using a partially distributed system design, i.e. using a distributed storage layer, in combination with a legacy management architecture, and using a distributed storage layer as part of a modern masterless architecture. In the case of Rubrik, all components operate in a distributed, masterless design. Each 2U appliance contains 4 individual nodes (individual servers), each individual server runs our Cloud Data Management solution as part of the bigger overall cluster. If a node (or an individual component inside a node) fails it has no impact on the overall correct operation of the cluster. The only component that is shared is the redundant power supply.

Can the same be said of running a legacy software stack on top of a distributed storage layer? Are all components running completely distributed? If the management components fail, does a distributed storage layer help you?

Let’s start by looking at a legacy stack example;

If you think about this from the pov of taking a backup, most of these components have a role to play. If one of the components is not operational no backups are taken, and the RPO keeps increasing until the backup solution is repaired.

Now let’s focus on distributing part of the legacy stack;

Again thinking about this from the pov of taking a backup, assuming the system/management components are operational you now have a scale-out, redundant storage layer which brings additional flexibility but it does not lessen the burden on the rest of the stack. If something in the system/management layer fails having a redundant storage layer does not save you, you are again increasing your RPO until the infrastructure is restored.

Now contrast this with Rubrik’s masterless approach;

Seeing that all processes (we don’t have the traditional demarcations of media server, search server, proxy/data mover, etc, but we do run processes/applications in a distributed manner in the cluster that provide all the CDM functions) are running on all nodes, we can similarly lose hardware components (because everything fails eventually) but this will not bring down the master server, simply because we don’t have a master server. Let’s say a node is working on ingesting backup data from your primary system and the node fails, the task is not marked as complete in our distributed system so another node in the cluster will pick up the job and make sure the backup is taken.

Rubrik has opted to build a truly scalable, masterless cluster, neatly leapfrogging legacy approaches of the past by taking lessons from massive-scale modern environments found at Google, Facebook and the like. To be fair, when legacy backup system where designed these giants’ shoulders were not available…

The bigger picture

Even though the principles and technology behind Rubrik are fascinating, it is important not to lose sight of the bigger picture. The idea is to deliver a greatly simplified approach to data management, that the architecture is a converged, masterless, scale-out, self-healing system, etc is nice but the real benefit lies in the operational use.

An additional benefit of using this design is that because we have a unified system, we can present this via a unified RESTful API. Integrating/automating the storage target, or integrating with the management components is not separate, simply because the system is one. Now contrast that with running multiple different components, each with their own, or non-existing integration points, the practical experience with be night and day. A better (much better) experience is ultimately what we are after, after all.

And finally, I wanted to point out that the pictures above are sort of looking at a traditional type of on-premises only environment, but what about running in the public cloud, multi-cloud, hybrid environment, branch offices, etc? Again the model we use delivers the flexibility to run anywhere since you don’t have to drag around these legacy interlinked pieces.


You snooze, you lose.

You snooze, you lose.

At the end of October 2016 I created a short blog post called “Data on tape is no longer useful” it’s premise was that if your historical data is being stored offline you can’t readily use it for other value-add services.

More recently Rubrik has delivered the ability to use archived backup copies of your data and spin those up on-demand as public cloud resources, called CloudOn. Initially taking VMware based virtual machines, archive them to Amazon S3 and when the time comes automatically migrate those workloads from VMware to Amazon and spin them up as AMI based EC2 instances.

Now with the release of version 4.1 we added the ability to support a similar scenario but spin up the workloads on Microsoft Azure instead of AWS. Additionally we added archiving support for Google Cloud Storage enabling more and more multi-cloud capabilities. (CloudOn for Google Cloud Platform is currently not available).

Multi-Cloud er… Multi-Pass

Since I’ve just written a post about all the things Rubrik is (was) doing with Microsoft I now need to add Microsoft Azure CloudOn to the list (I snoozed).

The idea is you add Microsoft Blob Storage as an archive location to one or more of your SLAs and once the data has been archived off you can now use that archive copy to instantiate your workloads, which where originally running on VMware (VMware VMDK based), on Microsoft Azure, as Azure based VMs (VHD based).

Screen Shot 2017-10-01 at 16.15.45

You can opt to do this conversion completely on-demand or choose to auto-convert the latest backup copy to save time during instantiation. The inner-workings a bit different depending on which scenario you choose but generally speaking Rubrik takes the VMDK file and converts it to a VHD file, uploads the VHD file to Azure Blob storage as page blobs (vs block blobs which are typically used for discrete storage objects like jpg’s, log files, etc. that you’d typically view as a file in your local OS.)

Screen Shot 2017-10-01 at 16.22.29

We also added support for Azure Stack in 4.1, in this initial release we provide the same type of functionality as we do with Azure (public cloud), meaning we support Windows, Linux, and SQL (customer installed) based workloads via the Rubrik backup service.

For a grand discussion of Azure Stack (a.o.) I suggest listening to this wonderful episode of the Datanauts podcast with Jeffrey Snover, Technical Fellow at Microsoft and Chief Architect for Azure Infrastructure and the creator of Powershell.

If you want to get more of the details about the 4.1 release please check out one (or all) of these fine posts;

Rubrik and Microsoft, rest assured.

Rubrik and Microsoft, rest assured.

Rubrik has been working with Microsoft’s solutions in various ways since version 2.3 with the initial support of Microsoft Azure as an archive location for long-term retention data. You can even argue the relationship started earlier than that with support for application consistent backups for Microsoft applications through our own VSS provider and requester (for Windows OS’s on Virtual Machines) since the beginning.

But for this overview, I’ll show the Microsoft specifics from version 2.3 till version 4.0, which is currently GA.

Azure Blob storage as an archive location

At the heart of Rubrik sit our SLA policies that govern how we handle your data end-to-end, as part of that SLA policy you can define archival settings, these will determine when we need to start moving data of the local Rubrik cluster and onto Azure Blob storage.

Azure Blob storage is a massively-scalable object storage location for unstructured data, the nice thing about using it as an archive with Rubrik is that because we index all data we can search for specific files/folders in this unstructured data, and only pull back those specific objects instead of needing to re-hydrate the entire fileset. Furthermore the data we sent to Azure Blob storage is encrypted.

Screen Shot 2017-09-03 at 21.22.03.png

Support for physical Windows and Microsoft SQL Server

In Rubrik CDM version 3.0 we added support for physical Windows and MS SQL server via the use of the backup service. The backup service is a very lightweight agent that you install on your physical Windows server, it runs in user-space and doesn’t require a reboot, once installed it is managed throughout it’s lifecycle via the Rubrik cluster itself. In other words once the backup service is installed you don’t need to manage it manually, it is also valid for all applications, meaning that it provides filesystem backup but also SQL backup capabilities.

Screen Shot 2017-09-05 at 15.53.05

Rubrik also provides protection and data management of SQL Server databases that are installed across nodes in a Windows Server Failover Clustering, additionally we support Always-On availability groups as wel, Rubrik detects that a database is part of an availability group. In the event of a failover, protection will be transferred to another availability database in the same availability group. When protection is transferred, the Rubrik cluster transfers the existing metadata for history and data protection to the replacement database.

Rubrik Cloud Cluster in Microsoft Azure

Since Rubrik CDM v3.2 we support running a 4-node (minimum) cluster in Microsoft Azure. We use the DSv2-series VM sizes, which gives us 4 vCPU, 14GiB RAM, 400 GiB SSD, and a max of 8 HDDs per VM.

Screen Shot 2017-09-06 at 12.00.56

Through the Rubrik backup service we are able to support native Azure workloads running either Windows or Linux, and Native SQL. Since Cloud Cluster in Azure has the same capabilities we can also archive to another Azure Blob storage location within Azure for long term retention data (i.e. move backup data of the the DSv2 based instances and unto more cost-effective Blob storage), and even replicate to another Rubrik Cluster, either in Azure or another Public Cloud provider, or even to a physical Rubrik cluster on-premises.

Microsoft SQL Server Live Mount

One of the coolest, in my humble opinion, new features in version 4.0 of Rubrik CDM is the ability to Live Mount SQL databases. Rubrik will use the SSD tier to rapidly materialize any point-in-time copy of the SQL database and then expose this, fully writetable DB to the SQL server as a new DB. In other words you are not consuming any space on your production storage system to do this.

Screen Shot 2017-09-06 at 12.13.16

As a SQL DBA you can now easily restore individual objects between the original and copy. The speed of recovery is greatly reduced, and the original backup copy is still maintained as an immutable object on Rubrik safeguarding it against ransomware.

Microsoft Hyper-V

Last, but not least, is our support for Microsoft Hyper-V based workloads which we achieve by integrating directly with the Hyper-V 2016 WMI based APIs, so this works independent of the underlying storage layer for the broadest support.

Screen Shot 2017-09-06 at 12.43.17

We leverage Hyper-V’s Resilient Change Tracking (RCT) to perform incremental forever backups. Older versions of Hyper-V are also supported through the use of the Rubrik backup service.

Independent of the source, be it virtual or physical, we can leverage the same SLA policy based system to avoid having to set individual, and manual backup policies.


VMworld US – Sunday

VMworld US – Sunday

This is my 7th VMworld event, but the first time attending VMworld in the US, I’ve been to Vegas before but arriving this Saturday evening it felt different somehow, maybe it was the scorching heat or the onslaught of UFC/Boxing fans at the MGM Grand, both where very “in your face”.


VMware signage welcoming you to Las Vegas

I decided to check out the OpeningActs event at the Beerhaus which was a pretty neat experience. The first panel session, moderated by John Troyer (aka John Trooper) was on VMware’s continued relevance in todays rapidly changing tech world which spurred a lot of Amazon AWS remarks. I think the overall consensus was that VMware might be in “sustainability” mode but that it is still an excellent foundation in a lot of customer environments. “Basic infrastructure” might be boring, but it is of great importance nevertheless. I think the other angle that might have been overlooked a bit here is that Amazon needs this relationship with VMware too, it needs a way into that “boring” enterprise market.


Second panel session was on “How failing made me better”, moderated by Jody Tyrus.
My favourite quote from the session was, and I’m paraphrasing here; “I sometimes still feel like I’m a 12 year old in a body of a 42 year old that is about to be exposed”. Rebecca Fitzhugh wrote and excellent blog on her experience with dealing with failure here.



Unfortunately I had to skip the 3rd session because I needed to get to the Expo for a Rubrik all-hands briefing before the Welcome Reception. After the reception I headed to the VMUG party at the House of Blues, which was pretty cool (because they served Stella Artois 😉 ), Michael Dell made an appearance and talked about his love for the VMUG and hinted at some cool Pivotal announcements coming this week.


After the party I needed to get jet-lagged self to bed. Looking forward to a great VMworld week.



A year at Rubrik, moving to marketing; et tu, Brute?

A year at Rubrik, moving to marketing; et tu, Brute?

I wasn’t really planning on writing a post on this initially but since I’m moving to a (technical) marketing role it seemed only fitting on second thought.
Switching from an SE role to technical marketing seems to typically bring about the “I’m moving to the dark side and betraying my technical roots” blog posts so here goes 😉

Actually I think my previous role as an SE for the EMEA region already had quite some tech marketing aspects to it. I traveled to a bunch of countries in Europe, the Middle East, and Africa spreading the Rubrik gospel, was a speaker at countless events, presented numerous webinars, and very occasionally wrote about some Rubrik related stuff on my personal blog. So when my bosses asked me what I wanted to do next I immediately blurted out “tech evangelism!”.

My role as an EMEA SE mainly revolved around covering the countries where we did not have a local presence yet (did you know EMEA consists of 128 countries? me neither!, but hello frequent flyer status) and I had a blast doing that, but since the company is growing like crazy, i.e. hiring local sales teams everywhere, I naturally started thinking about what could be next as well. I feel extremely lucky, and very humbled, to be joining the amazing Rubrik Tech Marketing team (Chris Wahl, Andrew MillerRebecca Fitzhugh) and only hope I can live up to their standard.

…googles “imposter syndrome”…

Andrew Miller already did a great job describing what the tech marketing function is/can be all about here. Or as Matthew Broberg jokingly (but not really though) described it in episode 94 of The Geek Whisperers

It’s this weird unicorn position that seems to fit in the middle of a Venn diagram between sales, marketing, engineering, and some other stuff that we can’t quite put our finger on.

June also marked my 1 year anniversary at Rubrik and with the risk of sounding insincere, it has been an amazing ride so far. The company has more than doubled in size since I joined, seen massive customer adoption, went through multiple successful funding rounds and delivered very significant features release after release.

Screen Shot 2017-06-21 at 15.37.59

To infinity and beyond!


Having worked at large incumbent companies before I find it quite amazing what you can achieve if you’re not bogged down by big company inertia and office politics. Sticking with the rocket ship analogy imagine building one at a start-up vs. building one at a big established company.


On the left you have the potential result after 6 months of trying to build a rocket ship at a large established company, at the right the potential result at a start-up (oops, time to iterate baby!)


After 1 year… you get the idea…

All very tongue-in-cheek of course, both have their merits and drawbacks, in the end it’s about finding where you fit best I think, and I certainly feel I’ve done that.

So in light of the new function, expect me to show up at more events, deliver more content, and be a bit more vocal on social media (but always trying hard to maintain technical integrity).

Rubrik Alta feature spotlight: AWS.

Rubrik Alta feature spotlight: AWS.

With the announcement of Rubrik CDM version 4, codenamed Alta, we have added tons of new features to the platform, but since most of the release announcements are focussed on providing an overview of all the goodies, I wanted to focus more deeply on one specific topic namely our integration with AWS.

Rubrik has had a long history of successfully working with Amazon Web Services, we’ve had integration with Amazon S3 since our first version. In our initial release you could already use Amazon S3 as an archive location for on-premises backups, meaning take local backups of VMware Virtual Machines and then keep them on the local Rubrik cluster for a certain period of time (short term retention) with the option to put longer term retention data into Amazon S3. The idea was to leverage cloud storage economics and resiliency for backup data and at the same time have an active archive for longer term retention data instead of an offline copy on tape. Additionally the way our metadata system works allows us to only retrieve the specific bits of data you need to restore instead of having to pull down the entire VMDK file and incurring egress costs potentially killing the cloud storage economics benefit.

Screen Shot 2017-06-12 at 18.40.13

Also note there is no need to put a gateway device between the Rubrik cluster and Amazon S3, instead Rubrik can natively leverage the S3 APIs.

The ability to archive to Amazon S3 is still here in version 4 of course but now all the supported sources besides VMware ESXi, like Microsoft Hyper-V, Nutanix AHV, Physical Windows/Linux, Native SQL, and so on can also leverage this capability.

Then in Rubrik CDM version 3.2 we added the ability to protect native AWS workloads by having a Rubrik cluster run inside AWS using EC2 for compute and EBS for storage.

Screen Shot 2017-06-13 at 10.40.43We’ll run a 4 node (protecting your data using Erasure Coding) Rubrik cluster in you preferred AWS location (the Rubrik AMI image is uploaded as a private image).
We use the m4.xlarge instance type, using 64GB RAM, 256GB SSD (GP SSD (GP2)) and 24TB Raw Capacity (Throughput Optimised HDD (ST1)), resulting in 15TB usable capacity before deduplication and compression.

Once the Cloud Cluster is running you can protect your native AWS workloads using the connector based approach, i.e. you can protect Windows and Linux filesets, and Native SQL workloads in the Public Cloud.

Additionally since potentially you can now have a Rubrik cluster on-premises and a Rubrik Cloud Cluster you can replicate from your in-house datacenter to your public cloud environment and vice versa. Or replicate from one AWS region to another.Screen Shot 2017-06-13 at 10.46.06

Since the Cloud Cluster has the same capabilities as the on-premises one it can also backup your AWS EC2 workloads and then archive the data to S3, essentially going from EBS storage to S3. (Christopher Nolan really likes this feature).

Screen Shot 2017-06-13 at 10.53.11

Version 4 of Rubrik CDM extends our AWS capabilities by delivering on-demand app mobility, called CloudOn. The idea is that now you can take an application that is running on-premises and move it on the fly to AWS for DR, or Dev/Test, or analytics scenarios.

Screen Shot 2017-06-13 at 17.31.23

The way it will work is that just as since v1 you archive your data to AWS S3, once you decide to instantiate that workload in the public cloud you select “Launch On Cloud” from the Rubrik interface and a temporary Rubrik node (spun up on-demand in AWS in the VPC of your choice) converts those VMs into cloud instances (i.e. going from VMware ESXi to AWS AMI images). Once complete the temporary Rubrik node powers down and is purged.

launch on cloud

Rubrik scans the configuration file of a VM to understand its characteristics (compute, memory, storage, etc.) and recommends a compatible cloud instance type so you are not left guessing what resources you need to consume on AWS.

Alternatively we can also auto-convert the latest snapshot going to S3 so you don’t have to wait for the conversion action.

Data has gravity, meaning that once you accumulate more and more data it starts to make more sense to move the application closer to the data since the other way around starts to become less performant, and more and more cost prohibitive in terms of transport costs. So what if your data sits on-premises but your applications are running in the public cloud? Screen Shot 2017-06-13 at 18.06.53
For instance let’s say you want to perform business analytics using Amazon QuickSight but your primary data sources are in your private data center, now you simply archive data to S3 as part of your regular Rubrik SLA (archive data older than x days) and pick the point-in-time dataset you are interested in, use Rubrik to “Launch on Cloud”, and point Amazon QuickSight (or any other AWS service) to your particular data source.

Together these AWS integrations allow you to make use of the public cloud on your terms, in an extremely flexible way.