In one of our labs we are using iSCSI to boot our ESXi5.1 hosts from a VNXe. If you lose connectivity to the NIC that runs the boot LUN (switch reboot, cable disconnect, etc.), you will see the following error: Lost connectivity to the device backing the boot filesystem. As a result, host configuration changes will not be saved to persistent storage.
This error is being displayed because connectivity is lost and the iSCSI boot does not support Multi pathing, which means that if connectivity is lost between the Storage Processor (SP) on the VNXe and the NIC on the host, the host can no longer access its boot lun and cannot write logs etc.
The good news is that the whole ESXi OS is loaded into memory so there is no outage for the VMs. Once the connectivity is restored the host can access the storage again.
The bad news is that the error does not clear automatically. As no one likes to see errors/warning in their production environment I needed to find a solution to this issue.
The simplest solution is to put the host into maintenance mode, reboot it and the problem is solved. This however takes time, depending on the number of VMs on the host and how busy the environment is.
The next option is to restart the Management Agents on the host. Here are the steps to complete this.
The KVM Java applet is loaded and the KVM will be displayed.
A Confirmation dialog window opens as shown below
Please read the warning, the host will automatically disconnect and reconnect from vCenter, however the VMs will continue to work without any outage. Click F11 to continue
Back on the KVM console, once the restart is completed we can hit Enter to close the window and we can logout of the console by hitting the ESC key twice
How about a faster way?
Click on the Host, Click on the Configuration Tab, Click on Security Profile under the Software section.
Click on Properties, scroll down to SSH and click on Options. To start the SSH service click on Start and click on Ok until all properties windows are closed.
After you enabled SSH, use your favorite SSH client to connect to the ESXi host.
When you are logged in as root run the following command:
/etc/init.d/hostd restart && /etc/init.d/vpxa restart
This will restart the hostd and vpxa agents right after each other.
Once completed, close the SSH session. Done!
NOTE: If you want to take it one more step further, you can suppress the SSH enabled warning within vCenter (not recommended!!!) and use kitty.exe to automatically login to the ESXi host and run the command listed above. In that way you only have to open Kitty, double click the host entry, and it all happens automatically.