When you fall in love with Spanned Volumes

In our company, our major productive servers are hosted in vSphere as Windows-based virtual machines. Some of them need very big disks. Through the largest disk space supported in vSphere is 16TB (or maybe more, i dunno), I’m not interested to create such a big file in VMFS. It will be really a pain your *** when you have to move such a big file among datastores, even among data facilities by using portable hard drives.

Creating smaller disks instead of a big one for every Windows-based VM, IMHO.

Fortunately, Windows support to merge small disks as a big one, using dynamic disk function. There are 4 kinds of volume can be supported in such a condition: Spanned, Striped, Mirrored and RAID-5. You can consider them like JBOD, RAID-0, RAID-1 and RAID-5 functions in a simple legacy storage device.

Because data protection is existed in the SAN, Mirrored and RAID-5 volumes are not considered at all. Striped will make it a little faster when lots of virtual disks fight for limited IOPS resources within one datastore. But there is no way to extend any stripped volume (https://technet.microsoft.com/en-us/library/cc770938.aspx).

So, the useless volume format in a traditional world, spanned volume, is the best solution, maybe the only one. Next time when you need to create big virtual disks for Windows in any virtualization environment, don’t forget to consider using Spanned Volumes.

Strange problem: VMware Inventory Service cannot start due to Microsoft Update (July 2016)

In one datacenter of my company, there was a strange problem last month of VMware vSphere.

Here is our structure:
SqlServer instance for vSphere is installed dedicated on cluster server. SqlServer version is 2012 Enterprise and Windows is 2012R2 Standard
vCenter is installed on another server.

After we applied July patches from Microsoft and restart the vCenter server, nearly all services from VMware cannot start. In event viewer, one event said:

The VMware Inventory Service service terminated with the following service-specific error:
Incorrect function.

As the request from VMware support, I uninstalled all updates related and reboot but it won’t help. I thought there may be some error related to the database connection. I ran the connection test from ODBC setup window, test finished succeeded. So I collected and submitted the log file generated from vCenter server command line. You know it’s really huge.

2 weeks later, I got an email from the high level engineers of VMware. By digging the log, they found that the vCenter cannot connect to SqlServer and the tcp service port is specified. At that moment, thanks to documentation I wrote :), I found that tcp service port is different than it should be. When I checked the SqlServer, I found the reason really weird: the TCP port of this instances is modified, automatically and silently. Due to I really used vSphere client just before upgrading Windows, I’m quite sure this issue is related to at least one of the patch which applied on the database cluster, launched by Microsoft in July 2016. And after a search, there is another instance of SqlServer which has the port changed. Due to the port changed and our firewall policy is set based tcp port, the client, vCenter in this issue, cannot connect to this SqlServer instance. After the port setting changed back and instance restarted, vCenter is back to normal.

Don’t ask me why ODBC test passed without any problem. If you know the answer, I’m listening as well.

Best Practice: Move Windows Server VM from vSphere to SCVMM (2012 R2)

Due to the removal of P2V wizard in SCVMM 2012 R2, we need to to this migration manually.

In this scenario, we will move a Windows Server VM hosted in vSphere/ESXi 6 to SCVMM/Hyper-V 2012 R2.
We need:

  • Administrator account with password of the VM to be moved.
  • A working SCVMM 2012 R2 with Hyper-V, and the client tool for management.
  • A working vSphere 6 with ESXi, and the client tool for management.
  • A network shared folder which can be accessed with read-write permission by VM, Hyper-V (optional) and the computer which the client tool of SCVMM located.

Here are the steps:

  1. Download the tool disk2vhd from https://technet.microsoft.com/en-us/sysinternals/ee656415.aspx and decompress it into the network shared folder.
  2. Log into the VM, check these settings:
    IP Address. If it’s set to automatic, make announce to your network admin to either set the VM to fixed IP address, or rebinding the MAC address later.

    • System CPU and memory.
    • OS Release name and bit wide, e.g. Windows Server 2008 Standard 32bit.
    • Hard disks and partitions structure.
  3. Run the disk2vhd at the network shared folder from VM. I suggest you not to copy it to the VM. All partitions which belong to the same hard disk should be exported to the same file at the SAME time. And all partitions from different disk should NOT be in the same exported file.
    For the VM running Windows Server 2003 (R2 included, x86 x64 included), do NOT check the option for virtual-pc. It will prevent the success from installing Hyper-V integration services.
    This step may take a while.
  4. Create a new VM in SCVMM by using the correct name, CPU, memory, OS release name. For my test, I cannot create a VM without hard drive. So just delete it after the creating finished.
  5. Copy the file(s) created from step 3 into the folder of the new VM.
    Due to the Windows bug of calculating free space of the host drive instead of the cluster volume disk, you may be prompted with an error said no enough disk space left for the VHD files.
    In this case, you need to execute the copy from Hyper-V, not form another computer. If your Hyper-V is installed without the full OS, aka Windows Hyper-V Server, you may have to do this by using cmd. The command for mapping a network shared folder as a driver is:

    net use x: \\path\to\your\shared\folder

    If you need to specify the account for accessing that folder:

    net use x: \\path\to\your\shared\folder password /User:domain\account

    After VHD files copying, use this to unmap the folder:

    net use x: /d

  6. Adding the VHD files to the new created VM by correct sequence.
  7. Shutdown the old VM and boot the new one.
  8. Log into the new VM, uninstall the VMware tools and shutdown the OS. If your VM is using Windows Server 2003 and you are using Remote Desktop to access SCVMM, now you have to use keyboard to finish this job.
  9. Install Hyper-V integration services. For Windows Server 2012 and above, it’s optional but recommended. For Windows 2008 R2 and previous, it’s necessary.
  10. Start the VM again and set the IP addresses of network interface cards.
  11. Mark or remove the old VM for avoiding conflicts.

Hope this work-though helps.

Work through: Upgrade vSphere 5.5 to 6

This is a record for upgrading vSphere 5.5 to 6. In this scenario, database for vCenter is a powered by SqlServer on a dedicated server.

 

Phase 1: Prepare

  • Before upgrading, something need to be ready.
  • Password of vCenter domain account.
  • Password of vCenter SSO.
  • ISO files for vCenter and ESXi. These files should be tested by checking MD5 or SHA hash codes.
  • License Keys. vCenter and ESXi keys can be gotten from VMware site by upgrading the currents.
  • vCenter server and databases should be backed up.
  • vSphere Update Manager Client (not server side) and vSphere Client should be uninstalled.

 

Phase 2: Upgrade vCenter

Mount the ISO of vCenter and perform the upgrading. Passwords of SSO and account will be required by this process.

The current instance of vCenter will be detected by setup program and all settings will be preserved. Current settings will be exported and will be imported automatically but not be removed after.

This step will need more than 1 hour.

At the end of this process, the post upgrade steps will be prompted including entering new license key and deleting the exported setting folder. You have to wait several minutes for web client service initializing. This is a good time for upgrading Update Manager Server and PowerCLI.

 

Phase 3: Install vSphere Client

Install vSphere Client and Update Manager Extension. If database error is occurred in extension installing, you need to check the account for running the VMware vSphere Update Manager Service. It should be a domain account but will be replaced to system account by setup process incorrectly.

 

Phase 4: Upgrade VM tools

You have to upgrade all VM tools one by one.

Get ready for embracing vSphere 6

Although there are many official documents about vSphere 6 launched recently, the release date of this product is still not uncovered.

One sure thing is there will be a new product, vFabric Postgres database, to replace the SQL server express database. The external database is still supported as well.

I will post a workthough for upgrading after it launched.

Work through: Upgrade vSphere 5.1 to 5.1 Update 1

This document is about upgrading vSphere 5.1 to 5.1 Update 1.

 

Phase 1: Preparation

You need these listed below:

  1. vCenter Installation Disc (iso): VMware-VIMSetup-all-5.1.0-1065152.iso;
  2. ESXi update package (zip, optional, will be used in Update Manager): update-from-esxi5.1-5.1_update01.zip;
  3. SSO admin password;
  4. vCenter service account password;
  5. An account within local administrators group of vCenter server; and,
  6. If you have Update Manager Client Plug-in installed on the same server of vCenter, it should be uninstalled before upgrading.

 

Phase 2: Upgrading vCenter and additions

  1. Insert vCenter Installation disc or mount iso.
  2. You cannot perform an upgrading installation by VMware vCenter Simple Install. It is required to upgrade all components manually as the same sequence as items listed in VMware vCenter Installer: vCenter Single Sign On, VMware vCenter Inventory Service, VMware vCenter Server, VMware vSphere Client (optional), VMware vSphere Web Client (optional) and VMware vSphere Update Manager.
  3. While upgrading the SSO (vCenter Single Sign On), you will be notified to enter the SSO admin password. A restart is required after performing SSO upgrading.
  4. Nothing more information is required while upgrading Inventory Service.
  5. Password of vCenter service account is required while upgrading vCenter Server. It is possible but highly not recommended to change service address or port while upgrading.
  6. All other 3 upgrading tasks are easy to perform. All you need is press Next button after starting the processes.
  7. (Optional) Check the server status by logging on from vSphere Client.
  8. (Optional) Install the Update Manager Client Plug-in now.

 

Phase 3: Upgrading ESXi

  1. Log on vSphere Client with Update Manager Client Plug-in.
  2. (Optional) Uploading ESXi upgrading package
    1. In the Update Manager Tab of  the top level of vCenter  Hosts and Clusters (top item from the left panel), select Admin View.
    2. In the Patch Repository tab, click Import Patches and upload update-from-esxi5.1-5.1_update01.zip. It will save downloading time of ESXi upgrading process.
    3. Go back to Compliance View.
  3. Scan your ESXi server.
  4. (Optional) Stage. You may want to stage one server first to test your patch repository before staging all servers.
  5. Remediate. You may need to set the servers in maintenance mode but not necessary. ESXi servers will be rebooted automatically after remediation.

 

Phase 4: VMtool upgrading

Upgrade VMtool on each virtual machine.

 

Phase 5: vSphere Client upgrading

Upgrade vSphere Client on all installed computer.

 

Phase 6: Troubleshooting

If there are some templates in your vSphere, you may need to convert them to virtual machine to upgrading VMtool. If this function is grayed out, you can re-enabled it by removing the related template from inventory and adding it back.

[Fixed] vSphere 5.1 cannot login with domain account after changing domain controller

We changed our domain controllers of our company these days. Of course, we did that by SOP:

  1. use dcpromo to promote new servers as domain controllers;
  2. migrate PDC and other critical roles to the new domain controllers;
  3. use dcpromo to depromote the old domain controllers.

After these steps, all Windows servers works great.

But we can use neither vSphere Client nor vSphere Web Client to login vCenter with a domain account any more.

 

By asking VMware support, we realize we have to adjust Single Sign On for this change.

Steps:

  1. Open vSphere Web Client, using admin@system-Domain to login;
  2. Select Sign-On and Discovery – Configuration from the left panel;
  3. You will the a record related to your domain on the right;
  4. Select the record and make a screen capture of all information;
    (You cannot edit it directly, cos it won’t save successfully.)
  5. Delete that record and create a new one, with the same setting and new LDAP address;
    (For Authentication Type, the existed one may set to Reuse Session, but you have to change it Password and leave an account with password below. I don’t know the reason.)
  6. Save it.

All done.