Hyperflex With ESXi thick to thin conversion

[etmarcof]

Hi need to copy one CUCM Virtual Machine (thick disk ) from one Hyperflex 4.5.2b cluster to another Cluster Managed by the same Vcenter 7.0   So  i went to one datastore a copy all the files to the other datastore location.  But at destination side when i register VM again i noticed that  vmdk file are Thin now.  Then I tried to inflate disk but vmdk size not increase to the original size one the inital Cluster.

Is this a  Vmware  related or Hyperflex related?  Any workaround?

Thanks.

Best Regards

MC

[RedNectar]

Hi @etmarcof ,

This would be a LONG explanation to give the full story, but the SIMPLE story is:

---

NEVER use thick disks on a HyperFlex data store

---

To understand why, you'd need to understand

1. How HyperFlex stores data using a pointer-based Log Structured file system, where THICK and THIN have NO meaning except
2. IF you use THICK provisioning, you can force the system to run out of space WAY before it actually runs out of space.

So be glad that it is thin provisioned. Even if you'd have managed to make it thick, you would NOT be reserving space in the traditional sense of "THICK". Instead, you'd be reducing the amount of available space to ensure you run out of space more quickly than if you'd left it as THIN - mainly by negating the savings that would normally be made from deduplicaiton and compression.

Sorry I don't have time for a more detailed explanation. 

[etmarcof]

Hi @RedNectar,

Thank you for your  explanation. I'm using this Hyperflex Clusters to deploy Cisco Unified Communications Manager (CUCM) using OVA file available from Cisco.  Regarding vSphere Thin provisioning  Cisco documentation says that:

HX filesystem is inherently thin on the backend, regardless of what VMs are set to. Collab apps don't support Thin on TRCs (see, caveats). To avoid HX resource waste and increased deployment time, VMs should set to Thick Provision (Lazy Zeroed), but NOT Thick (Eager Zeroed).

Virtualized Collaboration Storage Design Requirements (cisco.com)

[RedNectar]

Hi @etmarcof ,

I've written to the feedback link on the Virtualized Collaboration Storage Design Requirements (cisco.com) document you referenced.  Here is my feedback - it may help you (and the document writers) to understand

---

Hi there Cisco Document Writers,

I'd like to highlight a point of confusion in the Features of Cisco HXDP and VMware vSphere ESXi leveraged by Cisco HXDP section of your document Virtualized Collaboration Storage Design Requirements

In particular, the table with the title Features of Cisco HXDP and VMware vSphere ESXi leveraged by Cisco HXDP

Feature	HX uses for ...	Collab apps ...
vSphere Thin provisioning	HX filesystem is inherently thin on the backend, regardless of what VMs are set to.	Collab apps don't support Thin on TRCs (see, caveats). To avoid HX resource waste and increased deployment time, VMs should set to Thick Provision (Lazy Zeroed), but NOT Thick (Eager Zeroed).

Firstly, the link in caveats leads to a dead page - with a reference to a new page that says NOTHING about THICK or THIN

Secondly, and MOST IMPORTANTLY, the statement ...

"To avoid HX resource waste and increased deployment time, VMs should set to Thick Provision (Lazy Zeroed), but NOT Thick (Eager Zeroed)." 

...is

1. confusing, and has already led to one customer to have grief about this - see this post in the Cisco Community board
2. is actual WRONG - you should NOT set VMs to Thick Provision (Lazy Zeroed) or Thick (Eager Zeroed).  Please read the Capacity Management in Cisco HyperFlex White Paper
   

Can I respectfully suggest that the wording be updated to avoid confusion - I have NO IDEA why any TRC would not support Thin provisioning on HyperFlex (perhaps that's why there is no reference to "thick" or "thin" in the caveat that was references)

HyperFlex uses a pointer-based log structured file system which does not support space reservation (i.e. thick provisioning) in the traditional sense - it just has no meaning in this context.

I can (and have done) easily show how provisioning disks as "Thick" on a HXDP can cause the system to "reserve" space that never gets used, and in fact can cause the HXDP to "appear" to run out of space way before it physically runs out - to the point of VMs not being able to start even with only 5% of the actual disk space consumed

So I tell people to NEVER thick provision VMs on HyperFlex

Here's a simple example

Let's say there is a VM with a 1TB "thick" provisioned disk, but only 100MB used (at the moment). And to keep the maths simple, let's say the HXDP is only 3TB total (9TB raw if RF3 is used)

On the HXDP, because it is "thick" it will appear to use 1TB

Now clone that VM - because of how the pointer based log structured file system works, you've still only consumed 100MB of space.

BUT VMware (and the HXDP) has reserved 2TB of space

Now clone it again - still only 100MB of actual data on the disk - but now your system is hosed because you've used all 3TB. Your VMs will not even be able to power on.

[RedNectar]

[Edit 2022.09.21 - added Cisco's final reply at the end]

Hi @etmarcof ,

Here is the reply I got from the docs team. It doesn't excuse the fact that there is a document out there that has misleading information.

---

Looks like you’re referencing an older link (e.g. this references “TRCs” vs. contemporary support policies no longer use concepts of “Cisco TRC” vs. “Cisco Specs-based” vs. “3^rd-party Specs-based”).

Latest storage layer requirements for hardware and VMware (HyperFlex or otherwise) are located on the latest page here:
https://www.cisco.com/c/dam/en/us/td/docs/voice_ip_comm/uc_system/virtualization/cisco-collaboration-infrastructure.html#Specs_Storage 

(accessed from home page www.cisco.com/go/virtualized-collaboration -> pick an app -> click link for Collaboration Infrastructure requirements; we moved to this new approach in CSR 14 on esxi 6.7+ and CSR 12.7 on esxi 7.0+;  those older pages date back to circa CY16+/- when support policies were still heavily configuration-based for CSR 12.0/11.6 & older, vs. what is done today)

Hope that helps. 

---

And my reply is:

---

Thanks for getting back to me.

But really, all you have done is put me in a circle.  I wrote about document A which had a link to document B which redirected me to the document you've referred me to.

So I'm back where I started.

Unfortunately the "newer" document does nothing to resolve the question of "thick" vs "thin" provisioning.  The only reference I can find to either way of provisioning is:

• ESXi datastores may be provisioned with eager zero or lazy zero. Non-appliance hardware may also use thin provisioning, with caveat that disk space must be available to the VM as needed; running out of disk space due to thin provisioning will cause application instability and data corruption, including prevention of restore from backup.

Which does NOT really help in relation to the HXDP because this really does not spell out that thin provisioning should be used when using the pointer-based log structured file system that characterises the HXDP

Can I suggest that if the original document - Virtualized Collaboration Storage Design Requirements be either

• removed if it is no longer relevant - although it does contain a lot of information that doesn't appear in the updated version, such as the section on IOPS Capacity and Performance Planning
  OR
• updated to
• remove the WRONG information - specifically "To avoid HX resource waste and increased deployment time, VMs should set to Thick Provision (Lazy Zeroed), but NOT Thick (Eager Zeroed)" - this statement is causing customers to DEPLOY THE SYSTEM WRONGLY
• remove references to the TRC and other out-of-date references, including the link to the now dead caveats document which lead to the the document you suggested was the replacement

  ---

[Edit: 2022.09.21]

And Cisco's reply:

---

The old page is eventually going to be taken down.  It’s “deprecated” right now, and the material on that page was only intended to be examples and guidelines, not prescriptive designs.

The intent of the original statement dating before HyperFlex was if BE6K/7K appliance, then thick-provision-only.

---

So I guess that's where it ends. I've given up chasing - hope the documentation is made clearer in the future.