Upgrading the UCS Express to v1.1

Cisco released version 1.1 of their SRE-V software (more commonly referred to as UCS Express) earlier this month. We had a customer waiting for RAID1 support before they would build any VMs on their UCS Expresses. This post entails the steps involved in upgrading the software and setting up RAID1. Read on…

Upgrading the Software

First things first, download the software from cisco.com. I downloaded the sre-v-k9-upgrade.smv.1.1.1.zip file and unzipped it to my FTP server. Next, I SSH’d to the 3945 router and connected to a session on the SRE:

router#service-module sm2/0 session
ucse-1# software install package url ftp://192.168.0.191/sre-v-k9.smv.1.1.1.pkg username varrow password huh-riiiight

This will kick off the software upgrade process. We were copying our installation files over a WAN to the branch office with the UCS Express, so it took some time (45 minutes) to copy all of the files up to the SRE. The system will reboot, loading the new software, and eventually reach this point:

STARTED: superthread_startup.sh
Waiting 15 ...
SYSTEM ONLINE
ucse-1#

Congratulations, the software is installed. Pretty simple…

Configuring RAID 1 – The Easy Way

Now, we’re ready to configure RAID 1 on the UCS Express. There are two methods to converting the JBOD disks to RAID 1: keeping the VMs or blowing them away. Since our customer has two UCS Expresses, I tried both methods. First up (and my favorite), blowing it all away. For this method, you will need to backup any of your VMs to shared storage such as NFS. I would recommend this anyway just in case something goes wrong. At the hypervisor prompt, move the system into disk maintenance mode:

ucse-1# hypervisor set disk maintenance

This will reboot the SRE. Once it comes back up, go ahead and move the VMs to an NFS share if you haven’t already done so:

Inventory > Configuration > Storage > Browse Datastore
Right-click the VM folder and select Move To…
Move the VM to shared storage not on the UCS Express.

Now you can delete datastore1 and datastore2:

Inventory > Configuration > Storage > datastore1 > Delete
Inventory > Configuration > Storage > datastore2 > Delete

Now is the fun part, going into RAID setup to remove all of the drives and create a new logical RAID 1 volume. At the hypervisor prompt, enter RAID setup mode:

ucse-1# raid setup
You are about to enter RAID management CLI console.
!!! CAUTION !!!
We recommend that you shutdown all the virtual machines, backup datastores, and move the system into disk maintenance mode (command 'hypervisor set disk maintenance') before changing any RAID configuration. Otherwise, it might damage the system.
We also recommend that you first remove the datastore, and then remove the disk volume.

After modifying the RAID configuration, use the ‘Rescan All’ function in the vSphere Client GUI. Go to Inventory > Configuration > Storage. The rescan process might take several minutes. Next, display the logical drives:


raid-cli>logdrv
ID RAID Disks Stripe Size(MB) DiskID Name
0 JBOD 1 N/A 476940.02 (1) JBOD on port 01 (00)
1 JBOD 1 N/A 476940.02 (2) JBOD on port 02 (00)


Delete all of the logical drives in the SRE (two drives):


raid-cli>logdrv -a clear
raid-cli>logdrv
Array with ID 0 in controller 0 does not exist.


Now that the logical drives have been deleted, we can create a new logical volume that is RAID 1:


raid-cli>logdrv -a add -i 0 -p 1,2 -e 0,0 -z "name=RAID1,raid=RAID1,init=quick"
raid-cli>logdrv -v
*******************************
Array ID : 0
Array name : RAID1
Array size : 476837.12 MB
Array sector size : 512
Array raid mode : RAID1
Array write cache mode : Write Through
Number of disks in array : 2
Disk members with ID in array : (1,2)
Array activity status : Idle
Array functional status : Online
Driver Cache Mode: Write Thru
Driver Lookahead threshold: 0
Driver Consolidation: enabled


Now that RAID 1 has been enabled, exit out of RAID setup mode and remove the system from disk maintenance mode:


raid-cli>exit
ucse-1# hypervisor unset disk maintenance
This command will move system out of disk maintenance mode.
The system reboots after the scratch storage is switched to use a local datastore.
We recommend that you shutdown all the virtual machines before you continue.
Are you sure you want to continue?[confirm]
No system reboot is required.


With the new RAID 1 logical volume, we need to rescan the storage. Open up your vSphere client to your UCS Express and navigate to Inventory > Configuration > Storage > Rescan All… Before, you probably saw two drives. Now we see only one, mirrored drive!

Follow through the steps to add the new storage. Finally, you’ll reach this screen where you can Finish.

Now that was easy!

Configuring RAID 1 – The “I Don’t Want to Move My VMs Off” Way

This second method assumes you don’t want to, or are unable to, move your VMs off of your local storage. There are many more hoops to jump through in this method since you have to convert the disks to RAID 0 and then RAID 1, but we’ll get you there. Go ahead and place the SRE into disk maintenance mode and once it reboots, move any VMs from datastore1 to datastore2:

Inventory > Configuration > Storage > Browse Datastore
Right-click the VM folder and select Move To…
Move the VMs to datastore2.

Next, we need to determine which JBOD volume you want to migrate to RAID 0:
Inventory > Configuration > Storage > datastore1 > Properties > Manage Paths…
The Runtime Name column provides information about the location of datastore1:
vmhba0:C0:T0:L0
vmhba0:C0:T1:L0
Looking at T0, T is the iSCSI target and 0 is the ID number of the JBOD volume. You’re now ready to delete datastore1 now that your VMs have been moved to datastore2. I had an issue with datastore1 being in use, so I was unable to delete it; I had to reboot the UCS Express to delete it.

Now, with only datastore2 remaining, enter RAID setup mode and delete the disk associated with datastore1:
ucse-2# raid setup
raid-cli>logdrv
ID RAID Disks Stripe Size(MB) DiskID Name
0 JBOD 1 N/A 476940.02 (1) JBOD on port 01 (00)
1 JBOD 1 N/A 476940.02 (2) JBOD on port 02 (00)

From the output above, take note of the value in the ID column. You will delete the logical disk with that ID. In our case, the disk that datastore1 was on was vmhba0:C0:T0:L0. That 0 in T0 matches the 0 in the ID column.

raid-cli>logdrv -a del -l 0

Rescan the storage in your vSphere client and then create the RAID 0 logical volume.

raid-cli>logdrv -a add -p 1 -e 0 -z "raid=raid0,name=RAID1,init=quick"
raid-cli>logdrv
ID RAID Disks Stripe Size(MB) DiskID Name
0 RAID0 1 128 476837.12 (1) RAID0
1 JBOD 1 N/A 476940.02 (2) JBOD on port 02 (00)

Once the RAID 0 logical volume is created, perform another storage Rescan in your vSphere client. Click Add Storage… and select the new volume. Navigate through the menu and finally click FInish. Now you’re ready to migrate your VMs from datastore2 to your newly created datastore. Once you’ve completed this, delete datastore2. Back in RAID setup mode, delete the disk associated with datastore2:

raid-cli>logdrv -a del -l 1
raid-cli>logdrv
ID RAID Disks Stripe Size(MB) DiskID Name
0 RAID0 1 128 476837.12 (1) RAID0

Now you’re ready to merge the unused SATA drive to the RAID 0 volume. Refer to the output above for the disk ID (0 in our case) of our RAID 0 volume and refer to the earlier output above for the physical disk ID (2 in our case) of the disk we deleted.

raid-cli>migrate -a start -i 0 -l 0 -p 2 -s "raid=raid1"
Migration is started on Array 0.
raid-cli>migrate
Migration is in progress 1% on logical drive with ID 0 in controller #0!

raid-cli>logdrv -v
*******************************
Array ID : 0
Array name : RAID1
Array size : 476837.12 MB
Array sector size : 512
Array raid mode : RAID1
Array write cache mode : Write Through
Number of disks in array : 2
Disk members with ID in array : (1,2)
Array activity status : Idle
Array functional status : Online
Driver Cache Mode: Write Thru
Driver Lookahead threshold: 0
Driver Consolidation: enabled


This process was crazy slow, but once it’s complete you’ll be ready to exit RAID setup mode and move the SRE out of disk maintenance mode.

raid-cli>exit
ucse-2# hyper unset disk maintenance
This command will move system out of disk maintenance mode.
The system reboots after the scratch storage is switched to use a local datastore.
We recommend that you shutdown all the virtual machines before you continue.
Are you sure you want to continue?[confirm]

System is rebooting.


After exiting disk maintenance and rebooting, open up your vSphere Client and check the datastore name. I had to change mine as it was “snap…”. I opted for something more readable. I then browsed the datastore and added my VM(s) back to inventory.

Conclusion

We successfully upgraded two UCS Expresses today and setup RAID1. Moving the VMs around and creating the RAID 0 volume took a lot longer and I’m not sure it was worth the effort of not moving the VMs off of the UCS Express onto some form of shared storage. However, if you don’t have shared storage available, at least you can configure RAID 1 without losing your VMs. I’d like to hear how your experiences have been. Please post your results in the comments below. Thanks for reading.

- Andrew