I've written a small PowerCLI commandline helper giving you a wizard driven workflow to quickly do Cross vCenter Migrations a.k.a. (E)xVC-vMotions. It's based on virtuallyGhetto's xMove-VM.ps1 ^[1] powershell script. ^[2][3]

There is also a cool VMware Fling, this cross-vcenter-workload-migration-utility ^[4] gives you an actually web gui for (E)xVC workload migration, very cool, but needs a machine to run on. Also a nice PowerShell module ^[5] for automating the fling on the commandline from virtuallyGhetto.

I wanted a simple script, easy to use for everybody and immediately ready to go after download (only needs a recent PowerCLI), that's why i created this helper. All have there use-cases imho.

How does it work?

The script shows a 7-step wizard asking you for a set of parameters needed to succesfully start the cross-vcenter-vmotion. Parameters like source and destination vCenter, destination Host, destination Datastore, destination Networks, etc.

It's very easy to use and was built to work on most environments, with a few checks here and there eliminating most frequent errors. You only need a recent PowerCLI, but the script tells you to if it cannot find the needed PowerCLI modules and directs you to the download website.

At the end of the wizard it shows a complete set of the selected workload to migrate, asks for confirmation and will start the vMotions in parallel showing a handy multi-progressbar.

Wishlist

Maybe i'll update the code later on giving the script Per-VM multi datastore support (it now only does single datastore for the entire workload selected). I have written this feature for the VM-Networks though.

I would also like to upgrade it with some intelligence regarding cross version migration; but in general it doesn't support migrating to a lower version. And also cross standard/distributed vSwitch support; it doesn't support vds to vss. Check this link ^[6] for more info on the matter.

And it lacks destination VM Folder support...

Code

Github - ExVC-vMotion-Helper.ps1 ^[7]

2nd in these series is setting up a virtual Cumulus VX Clos-fabric and connect it to our NSX setup from our previous article, including VTEP integration!

For this we need to expand our LAB environment:

• Freesco Router (for dns, ntp and internet)
• Management switch (gns3 builtin)
• VXLAN Router (Cisco 3640) (not really a vxlan router, but creates an L3 hop between the ESXi hosts, and so it routes the VXLAN packets)
• VMware vCenter Server Appliance 6.5.0d
• Two VMware ESXi 6.5.0d hosts
• NSX Manager 6.3.2
• NSX Controller 6.3.2 (3 for HA, but 1 works just fine in the lab)
• Two test vm's (we chose alpine linux for this occasion) running on esx1 & esx2
• Four Cumulus VX 3.5.0 switches consisting of two spines and two leafs.We started building the lab on Cumulus VX 3.3.2 though, and it's possible the configuration differs somewhat between the two. One big change between them is that Cumulus Linux 3.4 and later releases replaces Quagga with FRRouting.
• Two test "physical" servers (we chose damn small linux) connected to the leafs

Lab IP Address plan

SPINE & LEAF interface configuration

# The loopback network interface
auto lo
iface lo inet static
address 10.10.102.11
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 10.10.100.11
netmask 255.255.255.0
gateway 10.10.100.1

# Uplinks
auto swp1
iface swp1

auto swp2
iface swp2

# The loopback network interface
auto lo
iface lo inet static
address 10.10.102.12
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 10.10.100.12
netmask 255.255.255.0
gateway 10.10.100.1

# Uplinks
auto swp1
iface swp1

auto swp2
iface swp2

#spine01
net add loopback lo ip address 10.10.102.11/32
net add interface eth0 ip address 10.10.100.11/24
net add routing route 0.0.0.0/0 10.10.100.1
net commit

#spine02
net add loopback lo ip address 10.10.102.12/32
net add interface eth0 ip address 10.10.100.12/24
net add routing route 0.0.0.0/0 10.10.100.1
net commit

# The loopback network interface
auto lo
iface lo inet loopback
address 10.10.102.13
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 10.10.100.13
netmask 255.255.255.0
gateway 10.10.100.1

# Uplinks
auto swp1
iface swp1

auto swp2
iface swp2

# ESX1
auto swp3
iface swp3 inet static
address 10.10.101.2
netmask 255.255.255.252
mtu 1500

auto swp4
iface swp4

auto swp5
iface swp5

auto swp6
iface swp6

# The loopback network interface
auto lo
iface lo inet loopback
address 10.10.102.14
netmask 255.255.255.255

# The primary network interface
auto eth0
iface eth0 inet static
address 10.10.100.14
netmask 255.255.255.0
gateway 10.10.100.1

# Uplinks
auto swp1
iface swp1

auto swp2
iface swp2

# ESX2
auto swp3
iface swp3 inet static
address 10.10.101.10
netmask 255.255.255.252
mtu 1500

auto swp4
iface swp4

auto swp5
iface swp5

auto swp6
iface swp6

#leaf01
net add loopback lo ip address 10.10.102.13/32
net add interface eth0 ip address 10.10.100.13/24
net add routing route 0.0.0.0/0 10.10.100.1
net add interface swp3 ip address 10.10.101.2/30
net add interface swp3 mtu 1500

#leaf02
net add loopback lo ip address 10.10.102.14/32
net add interface eth0 ip address 10.10.100.14/24
net add routing route 0.0.0.0/0 10.10.100.1
net add interface swp3 ip address 10.10.101.10/30
net add interface swp3 mtu 1500

SPINE & LEAF routing configuration

To turn the Cumulus switches into an EVPN fabric, we enable BGP as the routing protocol.
We establish peering between all neighbors (leaf1 to spine1 and spine2, leaf2 to spine1 and spine2)

!
interface swp1
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
!
interface swp2
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
!
interface swp3
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
!
interface swp4
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
! enabling BGP
router bgp 65020
 bgp router-id 10.0.0.21
 bgp bestpath as-path multipath-relax
! we configure a peer-group to put all the fabric nodes in and configure their capabilities.
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric description Internal Fabric Network
 neighbor fabric capability extended-nexthop
! we configure the switchports to be in the peer-group we just created.
 neighbor swp1 interface peer-group fabric
 neighbor swp2 interface peer-group fabric
 neighbor swp3 interface peer-group fabric
 neighbor swp4 interface peer-group fabric
 ! we enable the default adress-families for L3 routing between VTEP endpoints
 address-family ipv4 unicast
  ! add loopback interface to BGP (this is the VTEP endpoint on this switch)
  network 10.0.0.21/32
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor fabric activate
 exit-address-family
 ! we enable the EVPN af this is the EVPN VXLAN control-plane protocol.
 address-family evpn
  neighbor fabric activate
 exit-address-family
!

!
interface swp1
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
!
interface swp2
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
!
interface swp3
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
!
interface swp4
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
! enabling BGP
router bgp 65020
 bgp router-id 10.0.0.22
 bgp bestpath as-path multipath-relax
! we configure a peer-group to put all the fabric nodes in and configure their capabilities.
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric description Internal Fabric Network
 neighbor fabric capability extended-nexthop
! we configure the switchports to be in the peer-group we just created.
 neighbor swp1 interface peer-group fabric
 neighbor swp2 interface peer-group fabric
 neighbor swp3 interface peer-group fabric
 neighbor swp4 interface peer-group fabric
 ! we enable the default adress-families for L3 routing between VTEP endpoints
 address-family ipv4 unicast
  ! add loopback interface to BGP (this is the VTEP endpoint on this switch)
  network 10.0.0.22/32
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor fabric activate
 exit-address-family
 ! we enable the EVPN af this is the EVPN VXLAN control-plane protocol.
 address-family evpn
  neighbor fabric activate
 exit-address-family
!

#spine01
net add bgp autonomous-system 65020
net add bgp router-id 10.0.0.21
net add bgp bestpath as-path multipath-relax
net add bgp neighbor fabric peer-group
net add bgp neighbor fabric remote-as external
net add bgp neighbor fabric description Internal Fabric Network
net add bgp neighbor fabric capability extended-nexthop
net add bgp neighbor swp1 interface peer-group fabric
net add bgp neighbor swp2 interface peer-group fabric
net add bgp ipv4 unicast network 10.0.0.21/32
net add bgp ipv4 unicast redistribute connected
net add bgp ipv6 unicast neighbor fabric activate
net add bgp evpn neighbor fabric activate

#spine02
net add bgp autonomous-system 65020
net add bgp router-id 10.0.0.22
net add bgp bestpath as-path multipath-relax
net add bgp neighbor fabric peer-group
net add bgp neighbor fabric remote-as external
net add bgp neighbor fabric description Internal Fabric Network
net add bgp neighbor fabric capability extended-nexthop
net add bgp neighbor swp1 interface peer-group fabric
net add bgp neighbor swp2 interface peer-group fabric
net add bgp ipv4 unicast network 10.0.0.22/32
net add bgp ipv4 unicast redistribute connected
net add bgp ipv6 unicast neighbor fabric activate
net add bgp evpn neighbor fabric activate

!
interface swp1
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
!
interface swp2
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
! enabling BGP
router bgp 65011
 bgp router-id 10.0.0.11
 bgp bestpath as-path multipath-relax
 ! we configure a peer-group to put all the fabric nodes in and configure their capabilities.
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric description Internal Fabric Network
 neighbor fabric capability extended-nexthop
! we configure the switchports to be in the peer-group we just created.
 neighbor swp1 interface peer-group fabric
 neighbor swp2 interface peer-group fabric
 ! we enable the default adress-families for L3 routing between VTEP endpoints
 address-family ipv4 unicast
  ! add loopback interface to BGP (this is the VTEP endpoint on this switch)
  network 10.0.0.11/32
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor fabric activate
 exit-address-family
 ! we enable the EVPN af this is the EVPN VXLAN control-plane protocol.
 address-family evpn
  neighbor fabric activate
  ! provision all locally configured VNIs to be advertised by the BGP control plane.
  advertise-all-vni
 exit-address-family
!

!
interface swp1
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
!
interface swp2
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
! enabling BGP
router bgp 65012
 bgp router-id 10.0.0.12
 bgp bestpath as-path multipath-relax
 ! we configure a peer-group to put all the fabric nodes in and configure their capabilities.
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor fabric description Internal Fabric Network
 neighbor fabric capability extended-nexthop
! we configure the switchports to be in the peer-group we just created.
 neighbor swp1 interface peer-group fabric
 neighbor swp2 interface peer-group fabric
 ! we enable the default adress-families for L3 routing between VTEP endpoints
 address-family ipv4 unicast
  ! add loopback interface to BGP (this is the VTEP endpoint on this switch)
  network 10.0.0.12/32
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor fabric activate
 exit-address-family
 ! we enable the EVPN af this is the EVPN VXLAN control-plane protocol.
 address-family evpn
  neighbor fabric activate
  ! provision all locally configured VNIs to be advertised by the BGP control plane.
  advertise-all-vni
 exit-address-family
!

#leaf01
net add bgp autonomous-system 65011
net add bgp router-id 10.0.0.11
net add bgp bestpath as-path multipath-relax
net add bgp neighbor fabric peer-group
net add bgp neighbor fabric remote-as external
net add bgp neighbor fabric description Internal Fabric Network
net add bgp neighbor fabric capability extended-nexthop
net add bgp neighbor swp1 interface peer-group fabric
net add bgp neighbor swp2 interface peer-group fabric
net add bgp ipv4 unicast network 10.0.0.11/32
net add bgp ipv4 unicast redistribute connected
net add bgp ipv6 unicast neighbor fabric activate
net add bgp evpn neighbor fabric activate
net add bgp evpn advertise-all-vni

#leaf02
net add bgp autonomous-system 65012
net add bgp router-id 10.0.0.12
net add bgp bestpath as-path multipath-relax
net add bgp neighbor fabric peer-group
net add bgp neighbor fabric remote-as external
net add bgp neighbor fabric description Internal Fabric Network
net add bgp neighbor fabric capability extended-nexthop
net add bgp neighbor swp1 interface peer-group fabric
net add bgp neighbor swp2 interface peer-group fabric
net add bgp ipv4 unicast network 10.0.0.12/32
net add bgp ipv4 unicast redistribute connected
net add bgp ipv6 unicast neighbor fabric activate
net add bgp evpn neighbor fabric activate
net add bgp evpn advertise-all-vni

Setup VTEP Integration

The next step is integrating the "hardware" VTEP with NSX.
Now hardware-VTEP-integration is cool because we can manage physical ports from NSX and add these into a NSX logical switch, bringing the physical world into the virtual world.

Also check Cumulus DOCS for a comprehensive step-by-step.

1. Configure the NSX Replication Cluster
Networking & Security > Service Definitions > Hardware Devices > Replication Cluster > Edit.
The replication cluster will be responsible for forwarding the broadcast traffic sent from a hardware VTEP.

2. Configure openvswitch-vtep on both Cumulus leaf switches
Run these command on both leaf switches:

sudo systemctl enable openvswitch-vtep.service
sudo systemctl start openvswitch-vtep.service

3. Bootstrap both Cumulus leaf switches
LEAF1:

sudo vtep-bootstrap --credentials-path /var/lib/openvswitch --controller_ip 10.10.100.23 leaf1 10.10.102.13 10.10.100.13

LEAF2:

sudo vtep-bootstrap --credentials-path /var/lib/openvswitch --controller_ip 10.10.100.23 leaf2 10.10.102.14 10.10.100.14

4. Configure the switch as a VTEP gateway
For LEAF1 grab (copy) the public key content from LEAF1:/var/lib/openvswitch/leaf1-cert.pem and add this BASE64 begin/end request to the Certificate field while adding the Hardware Device, also enable BFD.

Networking & Security > Service Definitions > Hardware Devices > Hardware Devices > Click +.

Do the same for LEAF2 : LEAF2:/var/lib/openvswitch/leaf2-cert.pem

5. Bring the physical server switch port into the virtual world
Goto Networking & Security > Logical Switches and click Manage Hardware Bindings from the Actions menu of your logical switch.

Add leaf1:swp2 (the physical port connected to DSL1) to the logical switch. Do this for both physical servers LEAF1:DSL_server1 and LEAF2:DSL_server2!

Verify our setup

And we're done! Let's look at our LAB and test it!
We've got two DSL VM's running on ESX1 and ESX2 and two DSL physical servers connected to LEAF1 and LEAF2. All four nodes are connected to the same VXLAN, this was possible due to the hardware VTEP integration between NSX and Cumulus, cool right?!?

Next up is adding redundancy to the network, how... wait and see!

Articles in these series
https://scict.nl/nsx-cumulus-part1-en/
https://scict.nl/nsx-cumulus-part2-en/
https://scict.nl/nsx-cumulus-part3-en/

In these series we're going to set up NSX 6.3.2 on vSphere 6.5 in combination with a clos fabric based on Cumulus VX switches including hardware VTEP integration. Me and my friend and collegae Wouter van der Vaart will guide you through setting up the different parts needed to get everything up and running including problems we've encountered, but also tips & tricks.

This doesn't include a write-up of both products, but what these products do you can find here for NSX end here for Cumulus VX. But they coexist and integrate nicely especially in an automated SDDC (Software-Defined DataCenter).

First thing we're gonna do today is setup NSX and test basic VXLAN functionality over L3.

To set this lab up in GNS3 we need a few things:

• Freesco Router (for dns, ntp and internet)
• Management switch (gns3 builtin)
• VXLAN Router (Cisco 3640) (not really a vxlan router, but creates an L3 hop between the ESXi hosts)
• VMware vCenter Server Appliance 6.5.0d
• Two VMware ESXi 6.5.0d hosts
• NSX Manager 6.3.2
• NSX Controller 6.3.2 (3 for HA, but 1 works just fine in the lab)
• Two test vm's (we chose alpine linux for this occasion) running on esx1 & esx2

We've already installed the NSX Manager, connected it to the vCenter and deployed a single NSX Controller.

Lab IP Address plan

1. Install NSX to ESXi hosts

The first step is to install NSX to the ESXi hosts, just click Install from the Action menu on the Host Preparation tab. This could take a few minutes, depending on your environment.

2. Configure VXLAN

The second step is to configure VXLAN and add a single VTEP VMKernel interface to each ESXi host. Click the "Not Configured" link on the Host Preparation tab and a popup will show.

In our case we're leaving the defaults, it already selected the correct Distributed Switch, "VLAN" zero and the Fail Over "VMKNic Teaming Policy".

The "MTU" is set to 1600, that's because the VXLAN encapsulation overhead, it will add 50 bytes to each packet:
Ethernet/MAC (14 byte) + IP (20 byte) + UDP (8 byte) + VXLAN (8 byte) + Original ethernet frame
In our lab environment though we are limited to 1500, therefore we will later on change the vmkernel ports to 1500 and the clients to 1400 to get everything up and running.

For "VMKNic IP Addressing" we're setting it to "Use IP Pool", because we haven't setup DHCP, but in our lab we're setting the VMKNics manually to get them in separate L2 domains, we'll show you how in the next few steps.

3. Logical network preparation

Goto Networking & Security > Installation > Logical Network Preparation

A. VXLAN is now configured on the cluster, check if all the settings are correct and all the check marks are green.

B. Next is to set the Segment ID range of VXLAN Segments, each VXLAN tunnel will have it's own segment ID. Don't create more than 10.000 ID's though, this is because vCenter limits you to 10.000 distributed port groups (recommended maximum). We created a thousand from 5000-5999.

C. Last thing we need to do is configure a Transport Zone, the VXLAN control plane handles the frame forwarding decision (which VTEP to send it to by VTEP discovery and MAC address lookup), there are three modes: multicast, unicast and hybrid, all three modes have their cons & pros. But because multicast and hybrid require special configuration on the physical networking layer (IGMP), we chose unicast in our LAB setup.


For a in-depth explanation of the three modes check the articles here and here.

4. Create a logical switch and add the VM's

Next up is creating a NSX logical switch, each logical switch gets it's own Segment ID and will create a virtual wire dvPortgroup.

A. Goto Networking & Security > Logical Switches and click + to add a new logical switch. We will call our switch "NSX Test Switch 10.10.200.0/24", add it to our previously created Transport Zone, select unicast and IP Discovery and press OK.

B. Add the two test VM's Alpine01 and Alpine02 to the newly created "NSX Test Switch 10.10.200.0/24".

Select the logical switch "NSX Test Switch 10.10.200.0/24" and click Add VM from the Actions menu.

Select which adapters to add to the logical switch.

5. IP Pool Multiple subnets + MTU fix

In our LAB setup using VMware Workstation and GNS3 we weren't able to set an higher MTU and because of that limitation we lowered the NSX VTEP interface MTU to 1500 (1600 is the default) and the MTU of our Alpine test VM's down to 1400.

We also wanted to build the LAB with both NSX VTEP's in different broadcast domains with a router in between, which we also needed for our integration with de Cumulus VX switches in the next article. But NSX doesn't allow you to configure this using the GUI, the only options through the NSX GUI are IP Pool and DHCP, the IP Pool option doesn't allow for multiple subnets per cluster and we didn't have a DHCP server setup in our LAB, so we configured it manually.

Check the ESXCLI below for setting both the VTEP IP addresses on the VMKernel interfaces and lowering the MTU.

ESX1

esxcli network ip interface ipv4 set -i vmk1 -t static -I 10.10.101.1 -N 255.255.255.252
esxcli network ip interface ipv4 get -i vmk1

esxcli network ip route ipv4 remove -g 10.10.101.254 -n default -N vxlan
esxcli network ip route ipv4 add -g 10.10.101.2 -n default -N vxlan
esxcli network ip route ipv4 list -N vxlan

esxcli network ip interface set -i vmk1 -m 1500
esxcli network ip interface list -N vxlan

ESX2

esxcli network ip interface ipv4 set -i vmk1 -t static -I 10.10.101.9 -N 255.255.255.252
esxcli network ip interface ipv4 get -i vmk1

esxcli network ip route ipv4 remove -g 10.10.101.254 -n default -N vxlan
esxcli network ip route ipv4 add -g 10.10.101.10 -n default -N vxlan
esxcli network ip route ipv4 list -N vxlan

esxcli network ip interface set -i vmk1 -m 1500
esxcli network ip interface list -N vxlan

6. Test: Connectivity between the both ESXi hosts

Ping test using the VTEP VMKernel ports.

7. Final test: VM connectivity over the virtual wire

The hard work has paid off, the ping test from test VM Alpine01 to VM Alpine02 worked as expected!

Next up building the Cumulus VX CLOS Fabric + VTEP Integration!

Articles in these series

/nsx-cumulus-part1-en/
/nsx-cumulus-part2-en/
/nsx-cumulus-part3-en/

Rubrik's 3rd release Firefly was announced a couple of days ago, but what is it? According to Chris' blog on rubrik.com it's a code upgrade for the current briks. Supporting new features like backup for physical MSSQL and linux servers, a virtual appliance offering and erasure coding. Besides the existing S3 and NFS options also Azure Blob Storage was added as an archival destination, Yeah!

Rubrik Edge

I especially like the Virtual Edge Appliance for use in remote and branch offices. Rubrik doesn't mention this, but I assume this could also be of use for hybrid cloud architectures for backup, replication and restores between and back and forth the local datacenter and public clouds like Amazon and MS Azure? I'm wondering though how the architecure differs from the physical one. Does it consist of a single virtual node in ROBO environments instead of multiple ones? And without the physical backing of the supermicro hardware and the SSD for intake in each node, did they've downsized on options like indexing, dedup/compression or erasure coding? Let's hope I can get my hands on a OVA soon! :)

Erasure Coding

Since the current brik's save each block three times you only get a third of available space from your R334, R344, R348 or R528. Global dedup and compression off course do a lot to save space, but erasure coding still was missing, until now! Erasure coding gives you 66% of total space (at least on the 4 node models). And therefore doubles the available space and stretches the 15TiB on the R344 up to 30TiB. But that's not all, Rubrik is also saying the performance will go up for writes and reads and scales as the cluster grows!

I really like Rubrik and the architecture and the vision behind their product, and i'm sure the extra 61M is in good hands and the Firefly release will do great for current customers' briks!

I always try to keep up with the news and stuff while on vacation (this year the journey went to sri lanka), but i guess not everybody does. So I though why not share the highlights?!?

• New hacking technique imperceptibly changes memory virtual servers
  
  A cool and nifty but disturbing Virtual Machine memory page attack:
  With this technique an attacker can crack the keys of secured virtual machines or install malware without it being noticed. It's a new deduplication-based attack in which data can not only be viewed and leaked, but also modified using a hardware glitch. By doing so the attacker can order the server to install malicious and unwanted software or allow logins by unauthorized persons.
  
  This attack manages to do the hack without the use of a software bug and therefore cannot be patched. It is uncertain but practically unlikely that ECC DRAM memory can reliably be attacked. ECC DRAM is normally used in enterprise environments.
  Also see my previous article about recommendations for inter- and intra TPS in VMware environments, a must read considering the risks!
• VEEAM v9 update 2
  
  This patch comes with over 300 enhancements. I personnally like this improvement the most:
  Backported a number of isolated Enterprise Scalability enhancements from 9.5 code branch to improve transaction log backup, tape backup and user interface performance.
  
  I encountered an issue where per-VM backup files showed performance issues with tape which, according to veeam support, was a known problem caused by the number of storages to be backed up and should be fixed in the next release only. But let's hope this fixes it before 9.5 (which is also announced)!
• Pernix bought by Nutanix
  
  OK... so I suppose the rumours were true. Also backed by Frank Denneman. My subconscious saw this coming; after hearing about the rumours of an AFA, i thought by myself why not stay a software company and improve the FVP & Architect features, platform support and stability. Or maybe even build a VSAN competitor with better pricing and/or more or other options? I imagine... what would this mean for the sales for competitors like Infinio?
• ESXi 6.0 patch fixed CBT (again)
  
  VMware has released a major patch on the 4th of August ESXi600-201608001 which has buildnumber 4192238. It fixes another CBT issue introduced in Express Patch 6 (only during VM quiescence with VMware Tools VSS though). The only ESXi6 images free from CBT issues and production ready are 6 Update 1b and Express Patch 5 I think (Update 2 has the vmxnet3 psod). Let's hope ESXi600-201608001 will become a stable one!

Other news:

• Cisco fires 5500
• Nimble AF1000, affordable AFA?
  
  Dutch article
• PowerShell is open sourced and available on Linux

As many of you may know VMware changed the default Transpararent Page Sharing (TPS from now on) setting in the latest versions/updates of ESXi. Specifically the behaviour for Inter-VM TPS. But what about Intra-VM TPS?

What is TPS?

TPS is a technique which, when enabled, lets the ESXi host reclaim used memory pages by searching for duplicate small pages (4k) and elimate them. Which results in a potentially higher VM density. It's an asynchronous (so not in-line/realtime) proces running in the VMkernel and deduplicates memory within each NUMA node. There is also a second process which only kicks in when the physical host is under memory pressure due to overcommitment or fragmentation, this second process works by breaking large pages (2MB) up into small pages (4K) to enable page sharing. If the work of TPS is insufficient or disabled, ballooning kicks in followed by compression and eventually swapping to disk.

Inter-VM TPS disabled

So, as mentioned before, primarily based on research VMware disabled Inter-VM TPS as of the latest updates of ESXi, and is disabled in ESXi 6.0 altogether. Check this VMware KB to see in which versions the default and additional TPS management features changed for your environment.

Why did VMware changed this?
Well VMware says

The Risk:

Published academic papers have demonstrated that by forcing a flush and reload of cache memory, it is possible to measure memory timings to try and determine an AES encryption key in use on another virtual machine running on the same physical processor of the host server if Transparent Page Sharing is enabled between the two virtual machines. This technique works only in a highly controlled system configured in a non-standard way that VMware believes would not be recreated in a production environment.

Mitigation:

Even though VMware believes information being disclosed in real world conditions is unrealistic, out of an abundance of caution upcoming ESXi Update releases will no longer enable TPS between Virtual Machines by default (TPS will still be utilized within individual VMs).

Changed from default or not, Inter-VM TPS could still be very useful in memory overcommited scenario's. Especially in on-premise, single organization server and VDI environments, how useful depends...

Howto enable Inter-VM TPS?
By setting the host value Mem.ShareForceSalting to "0". Or for VM groups (like per cloud customer) by setting the host value Mem.ShareForceSalting to "1" and the Per-VM setting Sched.Mem.Pshare.Salt to the same salt for each customer.

The table below probably shows best howto achieve the required behavior.

source

What about Intra-VM TPS?

So while Inter-VM TPS is disabled by default, Intra-VM TPS is still enabled. Intra-VM means from within the VM. So a running VM can share pages allocated within the same NUMA node with itself. Still keeping some of the benefits TPS gives you when running memory overcommitment, just not ESXi host wide (actually NUMA node wide) anymore.

When looking at cloud/shared hosting environments, where VM's are shared between customers, like for example in:

• Shared Docker/container VMs
• Shared Apache/nginx/webserver VMs
• Shared Database server VMs

Should you disable Intra-VM TPS in these scenario's? VMware states the risks are low. But though it maybe harder, perhaps even impossible, to exploit TPS in an Intra-VM scenario. Shouldn't you disable TPS? Just to be safe, so a different customer or hacker (a hacker could rent a Web Virtual Host or Docker container) isn't able to possibly exploit TPS?

I guess it depends on the environment and type of customer. And if you run on-premise there probably are few companies who have the security guidelines in place which demand disabling TPS. But if you're a cloud container operator or webhoster, it's my opinion you should at least inform the customer of these risks and/or give them the option to run without TPS enabled (without the memory sharing benefits) or just disable TPS altogether, cause it could also be difficult to explain the risks to a (potential) client.

Howto disable Intra-VM tps?
Well, looking at the table above showing the configuration options for TPS, there doesn't appear to be an option to disable Intra-VM TPS, which means it's always on! So that's tricky, cause there may not be an official way to do this. Is there? NO? Maybe there should be? :) But it is possible to disable TPS altogether, though the steps are a bit tedious!

source

So, IMHO, when to (not) use TPS?

For most workloads and organizations I think the table below shows when to use TPS, call it a best practice. But maybe some organizations do not want to take any risk at all, like banks, federal governement and healthcare. They can just disable TPS altogether (and buy sufficient amounts of RAM).

More info on TPS and the changes VMware made:

1. http://blogs.vmware.com/apps/2014/10/disabling-tps-vsphere-impact-critical-applications.html
2. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2080735
3. http://frankdenneman.nl/2015/02/02/new-tps-management-capabilities/
4. http://www.yellow-bricks.com/2014/10/27/tps-disabled-default/
5. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021095
6. http://blogs.vmware.com/virtualreality/2011/02/hypervisor-memory-management-done-right.html
7. https://www.vmware.com/files/pdf/mem_mgmt_perf_vsphere5.pdf pages 7-9

While prepping for my VCP6-DCV delta exam, I stumbled upon something called VSAN Observer and though i'm running VSAN for quite awhile in a lab environment and played around with it quite a lot, but I haven't heard about or seen this VSAN Observer. What is it? Time to check it out!

Ruby vSphere Console

So first of all you need the Ruby vSphere Console (rvc from now on) to start the webserver for VSAN Observer, which is available in both the Windows version of vCenter Server and the vCenter Server Appliance (VCSA). For those of you who are unfamiliar with it, there is a nice 3 part series on VMware Blogs here[^1], here[^2] and here[^3].

I'm using the VCSA in the lab, and the first VMware KB I found about using VSAN Observer probably relates to older versions of vCenter, because i couldn't get it running based on this KB. So i came accross some other blogs, but didn't get any further than the first rvc step. After some googling and trying i noticed that this "rvc administrator@vsphere.local@localhost" worked! and others didn't, it probably changed in version 6?

The Steps

So let's layout the steps needed to get VSAN Observer running :)

Also check out this short video, showing the steps above!

What is it?

So what is it? VSAN Observer gives you in-depth VSAN en VM-level metrics, and while the vSphere Web Client already gives you a lote of information... If you feel the need to drill down on performance and verify VSAN or VM behavior for example, VSAN Observer it the place to go!

Let's check out the gui and some useful tabs within VSAN Observer!

About

VSAN Client (overview of per host stats)

VSAN Disks (deep-dive)

VSAN VMs (drill down into per VM Virtual Disks stats)

Both Pernixdata and Infinio released a new version of their Server Side Caching products yesterday!

Having used previous versions of them both in production environments and compared them very recently here, I'm thrilled to see the evolution here! Both still only support VMware, so no change there.

Pernixdata FVP 3.5

What are the new features FVP 3.5 brings to the table?

• Pernixdata 3.5 Management Appliance
• Support for RDM LUNs (with some limitations)
• Some nifty GUI enhancements, like a search box and performance views.

Also check out [this](nice video video, a nice short video/discussion explaining all new features!

Pernixdata Appliance

The appliance is probably the biggest change in this release, it is used for both FVP and Architect and comes in 4 "flavours" or "sizes" if you will. These are almost the same as the vCenter appliance sizes. The appliance comes in OVA form-factor, is linux-based (CentOS 7) and can only be used in fresh installations (no upgrade from windows management server possible). But the Windows Management Server is also still available and the only viable upgrade path from 3.0 up to 3.5.

The appliance is really easy to install, just deploy the OVF, put in some network configuration and power-up the VM. After which you have to setup the vCenter with it's credentials on the appliance configuration page which can be found at https://<applianceFQDN/IP>/config/. When that's finished the Pernixdata Portal will be up and running at https://<applianceFQDN/IP>:60002/, just like in 3.0. I created a short video showing the appliance first setup, check it out!

Infinio Accelerator 3.0

Now on to Infinio, the biggest news I think is the use of VAIO in their 3.0 product. Very cool!

What are the changes according to the Infinio Blog?:

• Support for block devices (SSDs and Flash) in the caching layer in a tiered fashion combined with RAM; a “memory-first” architecture.
• VM-level acceleration. This is something we did miss in the past. Some workloads just don't cache well and/or polute the cache by consuming a lot due to their high change rate and/or random read rate..
• And Infinio we'll be certified as VMware Ready!
• Support for many datastores types; NFS, VMFS, VSAN, and VVOLs are supported!
• VAIO Support, last but not least! "vSphere APIs for IO Filtering" integrates with VMware Storage Policy-Based Management and has the potential to lower in-kernel/software latency. Infinio states they can deliver 1,000,000 IOPS and 20GB/sec throughput per host, and response times under 100 microseconds.

Also check out the white paper!

Maybe the hype has passed. But as a user of SSC solutions for the last 2 years a quick comparising and experience sharing is in place I think. The trigger was a quick conversation about this topic i had this week with someone from a company we're doing a PoC with right now (hint: check an earlier post :)

What is server side caching?

I guess most of you will know, but a quick explanation doesn't hurt anybody (or just skip it :)).

Server side caching or short SSC (at work we called it "SuperSnelMaakSoftware") in a virtualized environment caches I/O requests requested by a VM from the storage array in a local cache within the hypervisor or computer layer usually in RAM or flash. And when that same data is accessed for the second time the I/O is dealt with by the caching layer and the request doesn't have to go all the way down the array.

In generally only for reads, but some solutions also support write caching where the write is being ack'ed at the cache layer in the host and is later destaged to the storage array. A solution with only read caching could have positive effect on writes though, because it gives the array and the spindles more headroom to do other stuff like storing data.

Server side caching works because the most recently used data or hot data is close to the VM. Less hops and no san/ethernet bottlenecks. Which results in lower latency. And the media types used for acceleration (ram and any type of flash (ssd, nvme, flashondimm) all have extremely low latency nanoseconds for RAM and microseconds for flash coupled with extreme high iops; 100k+ per esxi host is no exeption. Also the I/O blender effect gets almost nullified by the speed of flash, it's still there... but flash is just way better in random IOPS and has more available than spinning disk.

A. What happens on a write with only write through is enabled?

A write with a write-through caching solution remains the same besides the fact that it's being cached for a future read.
A write is cached and simultaniously written to the array. The acknowledgement has to return from the array before the VM receives the ack and can continue.

B. What happens on a write with write-back enabled?

The difference with write-through is that the write to the array is asynchronously and the VM receives the ack when the write hits the cache.
Potentially with some form of fault tolerance; the write that's being cached is also written to 1 or 2 other cache hosts (a distributed cache) synchronously. And only ack'ed back to the client when written in the cache on these hosts.

C. what happens on a cache-hit?

When data is in the cache, there is no need to fetch it from the array and the VM get's the data quickly. Data could end up two ways in cache:

1. by a write (check A)
2. by a previous read

D. What happens on a cache-miss?

When data is accessed for the first time it has to come from the array. Reason's for this could be that:

1. The acceleration by the SSC software was turned on after the data had already been written to disk we call this a first read.
2. Data hasn't been used in a long while and has been evicted from the cache

2013

Back in 2013 we struggled a lot with application performance, clearly the Netapp we bought in 2009 couldn't handle the load of the ongrowing virtual infrastructure anymore and it was getting worse everyday, all optimization efforts like block alignment and ramdisks in VM's just weren't enough anymore. The Netapp just couldn't deliver anymore and latency was high (full NVRAM and 1gig networking used for NFS). The environment grew, started with a few douzen VM's in 2009, now easily going over 200VM's. But we had enough storage capacity and therefore didn't really feel the need to invest earlier on in expansion or upgrading the filer just to get more performance, and also wasn't on the budget.

2014

We started looking into various options, we tried out various products back in 2014 like VMware Flash Read Cache (vFRC), Infinio Accelerator, Proximal Data AutoCache (now part of samsung) and PernixData FVP.

We quickly wrote off VMware's own flash read cache, the management overhead and the insights in every VM's data usage needed needed to configure this effectively didn't fit our environment. vFRC is configured on a per VMDK base and the optimal block size for the VM's VMDK needs to be set.

We contacted pernix, infinio and proximal in March 2014. We got the bits and started testing from april to june and finally went with Infinio for a year! Infinio had an easy to deploy solution (matter of minutes without any downtime) an supported NFS, which we needed for our Netapp files. Back then Pernix didn't have support for NFS datastores.

Finally overall performance was at an exceptable rate, SSC fixed the problem we we're then having and "saved" our asses! :)

2015

We felt the need to drive a bit more permanent solution into our datacenter. I liked the way Pernix was going (NFS support, Fault domains, etc). We saw good read io offloading with infinio, but Infinio in our situation added a bit of latency for writes and unaccelerated reads. Probably due to the extra hop of the infinio accelerator VM; Pernix doing write acceleration and using an in-kernel module should eliminate that problem completely! We also got ourselves a Netapp CDoT Metrocluster running NFS so fault domains with Pernix would be ideal!

2016

With our stretched HA vSphere Metro Storage Cluster with NetApp CDoT running NFS, PernixData FVP doing read/write acceleration using the sites as fault domains and corresponding vSphere config with regard to HA/DRS and vMSC site affinity. Now everything is going strong in our datacenter. End-users are getting the application performance they deserve, and we can much easier buy capacity when we need capacity and performance when we need performance.

General use cases, and experience

It sounds cool, but it's not for everybody. It really depends on your needs and on your architecture. If your applications don't need the faster response or higher throughput or if you already have flash in your array you probably won't see big benefits (or any at all), though accelerating with RAM within FVP on a hybrid or AFA could drive the performance up, especially for bigger block sizes. It all depends! And some newer vSphere 6 features are often not supported, like VVOLs and NFS4.1 (session trunking & kerberos). Also check pNFS is not Session Trunking

Adding VSAN support for SSC solutions doesn't make any sense I think. Being a hyperconverged solution, data already is close to vm. writes already go to flash and reads depends; them being hot or cold, and also if your using a normal hybrid setup for VSAN or all-flash. The speed of ram could potentially fasten up VSAN. With an option for a big(ger) read cache to start with and writes going to ram (especially future non-volatile solutions like flashondimm, 3d crosspoint) as second.

Possible use cases?

Off course it depends but probably works best for sas/sata users and for VDI on normal datastores, without things like Citrix PVS and there often is smart integration for things like linked clones. Or off-course when struggling with slower/older storage array performance. Especially when higher latencies are seen. It can extend the lifetime of your existing array probably by one or two years.

Also write acceleration isn't always needed. And complexes the architecture. Also for consistent backup in case of storage level integration for example. And when used in a fault tolerance mode it adds networklatency for the network writes. Which could or could not be higher than your existing array (nvram over 10g ethernet also acks quickly)

Today AutoCache is part of samsung, haven't heard of them since. Infinio news is also lacking. Pernix is still going strong, not only with FVP but also with their last year announced Architect!

Quick feature comparison:

Contact me if you have any questions or have an interest in detailed insights in our test results for example for some of these solutions!

A bit of a sidestep... But i felt the urge to share something which never gets the attention I think it deserves.

MyPanel

This is a mini-control panel for settings I found were missing in Windows and Citrix for use in a SBC (XenApp/TS) or VDI (View/XenDesktop) environment, but users have to deal with every day, especially in flex space scenario's. And is also useful when using different type of managed clients like a mix a multivendor zero/thin clients and fat clients. We call it MyPanel!

MyPanel is simple, is has settings for screen resolution, mouse, browser of choice and switching between different environments (like prod & test). Settings are set a single time on the first working day of an end-user. An at logon settings are automatically deployed to all different tipe of clients. It uses a simple 2 table MSSQL database for storing the user settings. Backend client integration consists of several batch & powershell scripts and are different for each client. The settings within the SBC or VDI session are configuratied at logon by MyPanel itself (-hidden parameter).

Display tab

Sorry for the screenshots, they are in Dutch, but translating them shouldn't be that hard.

The first tab handles the screen resolution. It supports setting a specific resolution, native/ddc is also an option. And it has twin display support.

In our setup we set the resolution on a per user/per device base, but this is a choice.
You clould also do per use only or per device only.

Mouse tabs

The mouse tabs let's you set the three most often used settings for your mouse. right or leftie, doubleclick speed and pointer speed. Pretty straightforward!

Browser tab

I think the browser tab is cool, this is something we added just last year to MyPanel.
And obviously it let's the user choose it's preferred browser.

We then use a wrapper called MyBrowser, and it checks the user's preference in the MSSQL database and starts the browser.
MyBrowser.exe becomes the default browser in a user's session. and when used with a parameter Mybrowser.exe "url" it opens the given url.
We are used to set links to websites internal and external with RES ONE Workspace, exceptions can be made by just using IE, FF or Chrome in the configured shortcut in RES instead of MyBrowser.

Environment tab

This tab is used for lettings the user choose a specific default environment, like prod or dev&test.
It gives the user the option to choose a specific desktop, but also makes automatic logon to full desktops on Citrix, TS or View possible.
Which is a way cleaner user experience I think than storefront web gives for example. Just enter credentials, click OK and the desktop starts!

This is something I created back in 2007, there have been some revisions since, but the idea remained the same. And due to the fact it was built back in 2007 with visual studio 2005 it’s also not that pretty, I’m sorry :)

If you're interested in the code or detailed explained just give me a ring! I'd like to share! I want happy end-user all over the world! :)

Veeam added tape support in version 7, we immediately started testing it, but never found the performance and stability to completely trust on the tape support for all our backups. We just weren't sure the daily tape copies were put to tape before the end of the day and in some cases weren't sure they'd be put to tape at all! So we also used third-party solutions to get the back-up files safeley on tape. We eleminated this somewhat after v8 which did improve a lot, but still...

After struggling with tape support in Veeam since version 7, tests with version 9 update 1 (1491) finally showed promosing results! And after using tape in v9 for about a week now.... waking up, going to work en checking the veeam tape results from last night isn't a controlling factor of cortisol-stress level anymore!
All jobs (a copy of incrementals) succeeded easiliy within our timeframe with good performance!

The changes I like most

Global Media Pools

Global Media Pools make it possible to share tapes in pools accross several libraries and tape proxies, this adds a separate layer of configuration for the pool and its tapes as they are now decoupled from the library.

You can see we currently do not have multiple libraries, but Veeam does have a preview here.

Parallel Processing

Together with Global Media Pools parallel processing, especially when using the per-VM backup file chains, gives an amazing performance boost and adds flexibility and ease for your tape and backup-to-tape job management.

Still some things to wish for

Veeam also adds GFS (Grandfather-Father-Son) retetion for tape jobs. And though GFS sounds cool, it's a seperate job type/media pool type which does not save the incrementals, but always creates (synthetic/virtual) fulls with a GFS retention scheme (choices are weekly, monthly, quarterly and yearly). So the incrementals aren't included!

A single retention scheme for hourly/daily incrementals, weekly fulls, monthly fulls and yearly fulls is still not possible. But sort of doable when combining the normal media pool with GFS and seperate tape jobs for each.

But when doing daily incrementals and weekly fulls with the normal mode job and the monlthies and yearlies with the GFS... BUT in the weekend the monthlies and yearlies are created. Veeam also creates the full for the normal job, which seems a bit redundant... (if that even exist in backup space :)). Maybe it's possible, but i haven't found a wat to do so. anyone from veeam reading this? :)

Also GFS doesn't support paralell processing, check here.

Our solution to this?

To get the GFS like retention we want, including the incrementals, we manually protect (automated through powershell) the monthlies en yearlies and add "YYYY-MM" to the description field of each tape and keep the protected tapes off our daily printed tapelist, you just have to make sure the tapes contain (synthetic/virtual) fulls. With this solution it's possible to keep the tapes in the same media pool, have no redundant tape copies of the same set and unprotect the tapes based on the description field after the retention period.

Our retention scheme and a simple powershell used for our own GFS solution:

Github - VEEAM Tape Protect-Retire GFS.ps1