PowerVC detach and attaching volumes hanging

I recently encountered issues with PowerVC 2.1.1 when attempting to communicate to HMC managed hosts. When attempting to deallocate or allocate storage that was allocated that were allocated using vSCSI the state of the task would remain in attaching/detaching state indefinitely within PowerVC.

To troubleshoot the first step I always take when I encounter issues like this is to investigate the nova logs on the PowerVC server, you can do this by looking in /var/log/nova as root for the nova log related to the managed system you are having issues on. The log fill have a name in the format openstack-nova-compute-_<MODEL-TYPE>-<SERIAL>.log.

In this case, when tailing the log, I saw the following error:

Error occurred while querying for VirtualSCSIMapping from VIOS vio2 with ID 2 in System <redacted> - Error while retrieving data from VIOS. It could because VIOS response has invalid information or containing special characters. Contact your service representative for help

Logging into the HMC (via UI) and viewing the virtual storage tab for the same partition resulted in the same error message. To understand this message, you need to understand what services provide the information to the HMC. In this case, it is a service on the VIO servers called vio_daemon. This service provides an interface to the internal CMDB for the HMC to query so that it can see device mappings on a VIO server. When a change is made on a VIO server, it is reflected in the CMDB internally and gets updated by the vio_chgmgmt process.

In my case, the issue was the vio_daemon failed to start after a change where the VIO server was rebooted, after the daemon was started using the command (as root via oem_setup_env): startsrc -s vio_daemon -a '-d 4' the error persisted until I forced the vio_chgmgmt daemon to refresh the database by running kill -1 <process ID of vio_chgmgmt process>.

After waiting a short period (approximately one minute) I was able to verify the error message had disappeared from the HMC’s virtual storage management page for the partition, the next step was cleaning up PowerVC as openstack still showed the state of the volumes as detaching.

PowerVC showed the state of the volumes as ‘detaching’ and had remained hung in this state for approximately an hour. To resolve this it required updating the volume state via the openstack CLI. After logging into the PowerVC server and running the following:

. /opt/ibm/powervc/powervcrc
export OS_USERNAME=${USER}
read OS_PASSWORD # enter password afterwards
export OS_PASSWORD
export OS_PROJECT_NAME=<NAME OF PROJECT>

After configuring your environment variables, you can then run a cinder volume list to validate/identify the volume you had issues with and see the state and the attachment status.

If the attachment status is not ‘attached’ you can reset the state by running – assuming you were attempting to detach the lun when the issue occurred:
cinder reset-state --attach-status=attached

If the volume state is not ‘in-use’ then you can reset the state by running:
cinder reset-state –state=in-use

After this, you can re-attempt to detach the volume.