Strange problem: VMware Inventory Service cannot start due to Microsoft Update (July 2016)

In one datacenter of my company, there was a strange problem last month of VMware vSphere.

Here is our structure:
SqlServer instance for vSphere is installed dedicated on cluster server. SqlServer version is 2012 Enterprise and Windows is 2012R2 Standard
vCenter is installed on another server.

After we applied July patches from Microsoft and restart the vCenter server, nearly all services from VMware cannot start. In event viewer, one event said:

The VMware Inventory Service service terminated with the following service-specific error:
Incorrect function.

As the request from VMware support, I uninstalled all updates related and reboot but it won’t help. I thought there may be some error related to the database connection. I ran the connection test from ODBC setup window, test finished succeeded. So I collected and submitted the log file generated from vCenter server command line. You know it’s really huge.

2 weeks later, I got an email from the high level engineers of VMware. By digging the log, they found that the vCenter cannot connect to SqlServer and the tcp service port is specified. At that moment, thanks to documentation I wrote :), I found that tcp service port is different than it should be. When I checked the SqlServer, I found the reason really weird: the TCP port of this instances is modified, automatically and silently. Due to I really used vSphere client just before upgrading Windows, I’m quite sure this issue is related to at least one of the patch which applied on the database cluster, launched by Microsoft in July 2016. And after a search, there is another instance of SqlServer which has the port changed. Due to the port changed and our firewall policy is set based tcp port, the client, vCenter in this issue, cannot connect to this SqlServer instance. After the port setting changed back and instance restarted, vCenter is back to normal.

Don’t ask me why ODBC test passed without any problem. If you know the answer, I’m listening as well.

Work through: Upgrade vSphere 5.5 to 6

This is a record for upgrading vSphere 5.5 to 6. In this scenario, database for vCenter is a powered by SqlServer on a dedicated server.

 

Phase 1: Prepare

  • Before upgrading, something need to be ready.
  • Password of vCenter domain account.
  • Password of vCenter SSO.
  • ISO files for vCenter and ESXi. These files should be tested by checking MD5 or SHA hash codes.
  • License Keys. vCenter and ESXi keys can be gotten from VMware site by upgrading the currents.
  • vCenter server and databases should be backed up.
  • vSphere Update Manager Client (not server side) and vSphere Client should be uninstalled.

 

Phase 2: Upgrade vCenter

Mount the ISO of vCenter and perform the upgrading. Passwords of SSO and account will be required by this process.

The current instance of vCenter will be detected by setup program and all settings will be preserved. Current settings will be exported and will be imported automatically but not be removed after.

This step will need more than 1 hour.

At the end of this process, the post upgrade steps will be prompted including entering new license key and deleting the exported setting folder. You have to wait several minutes for web client service initializing. This is a good time for upgrading Update Manager Server and PowerCLI.

 

Phase 3: Install vSphere Client

Install vSphere Client and Update Manager Extension. If database error is occurred in extension installing, you need to check the account for running the VMware vSphere Update Manager Service. It should be a domain account but will be replaced to system account by setup process incorrectly.

 

Phase 4: Upgrade VM tools

You have to upgrade all VM tools one by one.

Get ready for embracing vSphere 6

Although there are many official documents about vSphere 6 launched recently, the release date of this product is still not uncovered.

One sure thing is there will be a new product, vFabric Postgres database, to replace the SQL server express database. The external database is still supported as well.

I will post a workthough for upgrading after it launched.

Change the host mode again

I’ve just change this site from VM based Windows Azure into a Web site based mode to mitigate the jobs for maintenance.

It still slow and not stable as predictable.

Azure service is really not good, especially for PHP+MySQL.

Enable Remote Management for Hyper-V Server

After a clean installation process, firewall of Hyper-V Server is set to deny any remote connection by default.

If you need to management other than using console, you may want to enable inbound rules for Remote Management and Remote Desktop by running these commands in the console.

To enable the Remote Management:

cscript C:\windows\system32\scregedit.wsf /im 1

To enable the Remote Desktop:

cscript C:\windows\system32\scregedit.wsf /ar 0

And, you need to enable the remote desktop from the sconfig (the blue console window) also.

AVMA: An easier way to activate Windows Server 2012 R2 in Hyper-V

Microsoft provides a new way to activate VMs hosted in Hyper-V, named Automatic Virtual Machine Activation (AVMA). It makes the activation step of VMs much easier.

Requirement:

  • Hypervisor: Windows Server 2012 R2 Datacenter with Hyper-V role. Other versions are not supported. Dedicated Hyper-V Server is not supported.
  • VM: Windows Server 2012 R2 Datacenter, Standard and Essentials.

Steps:

  1. Prepare and activate your Hypervisor;
  2. Install supported OS as VM with the key listed below, or change the key of an installed VM with command “slmgr /ipk <key>”.

Keys:

  • Windows Server 2012 R2 Datacenter: Y4TGP-NPTV9-HTC2H-7MGQ3-DV4TW
  • Windows Server 2012 R2 Standard: DBGBW-NPF86-BJVTX-K3WKJ-MTB6V
  • Windows Server 2012 R2 Essentials: K2XGM-NMBT3-2R6Q8-WF2FK-P36R2

These keys are supported to being used in any unattend.exe setup file also.

 

Source: http://technet.microsoft.com/en-us/library/dn303421.aspx

Work through: Upgrade vSphere 5.1 to 5.1 Update 1

This document is about upgrading vSphere 5.1 to 5.1 Update 1.

 

Phase 1: Preparation

You need these listed below:

  1. vCenter Installation Disc (iso): VMware-VIMSetup-all-5.1.0-1065152.iso;
  2. ESXi update package (zip, optional, will be used in Update Manager): update-from-esxi5.1-5.1_update01.zip;
  3. SSO admin password;
  4. vCenter service account password;
  5. An account within local administrators group of vCenter server; and,
  6. If you have Update Manager Client Plug-in installed on the same server of vCenter, it should be uninstalled before upgrading.

 

Phase 2: Upgrading vCenter and additions

  1. Insert vCenter Installation disc or mount iso.
  2. You cannot perform an upgrading installation by VMware vCenter Simple Install. It is required to upgrade all components manually as the same sequence as items listed in VMware vCenter Installer: vCenter Single Sign On, VMware vCenter Inventory Service, VMware vCenter Server, VMware vSphere Client (optional), VMware vSphere Web Client (optional) and VMware vSphere Update Manager.
  3. While upgrading the SSO (vCenter Single Sign On), you will be notified to enter the SSO admin password. A restart is required after performing SSO upgrading.
  4. Nothing more information is required while upgrading Inventory Service.
  5. Password of vCenter service account is required while upgrading vCenter Server. It is possible but highly not recommended to change service address or port while upgrading.
  6. All other 3 upgrading tasks are easy to perform. All you need is press Next button after starting the processes.
  7. (Optional) Check the server status by logging on from vSphere Client.
  8. (Optional) Install the Update Manager Client Plug-in now.

 

Phase 3: Upgrading ESXi

  1. Log on vSphere Client with Update Manager Client Plug-in.
  2. (Optional) Uploading ESXi upgrading package
    1. In the Update Manager Tab of  the top level of vCenter  Hosts and Clusters (top item from the left panel), select Admin View.
    2. In the Patch Repository tab, click Import Patches and upload update-from-esxi5.1-5.1_update01.zip. It will save downloading time of ESXi upgrading process.
    3. Go back to Compliance View.
  3. Scan your ESXi server.
  4. (Optional) Stage. You may want to stage one server first to test your patch repository before staging all servers.
  5. Remediate. You may need to set the servers in maintenance mode but not necessary. ESXi servers will be rebooted automatically after remediation.

 

Phase 4: VMtool upgrading

Upgrade VMtool on each virtual machine.

 

Phase 5: vSphere Client upgrading

Upgrade vSphere Client on all installed computer.

 

Phase 6: Troubleshooting

If there are some templates in your vSphere, you may need to convert them to virtual machine to upgrading VMtool. If this function is grayed out, you can re-enabled it by removing the related template from inventory and adding it back.