Guide to Installing Cisco UCS Express

I just completed my first installation of Cisco’s UCS Express, a server implanted in a ISR router. If you’re looking to get more information on what the UCS Express actually is, please check out my older post. Not surprisingly on the UCS Express is the lack of Cisco documentation available to configure and deploy it. After an exhaustive search, plenty of trial and error, and a ticket with Cisco TAC, I’ve written up the guide below.

Overview

The Cisco UCS Express is basically a hypervisor running on a Services-Ready Engine (SRE). The SRE is really just a server that fits into the new ISR Generation 2 routers and will run a purpose-built operating system. In this engagement, I worked with the SM-SRE-900-K9 (1.86GHz, 2x500GB SATA, 4GB RAM). The hypervisor is really called a SRE-V. Cisco and VMware partnered to create this hypervisor. Our customer purchased the Cisco 3945 ISR G2, which supports up to four SREs. In the remote router, we have two UCS Express SREs and one WAAS Express SRE (future post coming on the WAAS configuration). The SRE has one GigabitEthernet port on the front of the module and two GigabitEthernet ports on the backplane to the ISR chassis.

Design

Once I wrapped my head around the plethora of terms for this new technology, I began designing… This page, from a Cisco Lab, is hands-down the best resource for configuring UCS Express at present. When I called into TAC, they mentioned they used the same lab to prepare them for supporting UCS Express. When you execute a “show ip interface brief”, you’ll see the SRE interfaces listed as SMx/x:

router#sh ip int bri
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 unassigned YES NVRAM administratively down down
GigabitEthernet0/1 192.168.254.1 YES NVRAM up up
GigabitEthernet0/2 unassigned YES NVRAM administratively down down
Serial0/0/0 unassigned YES NVRAM up up
Serial0/0/0.100 x.x.x.x.x YES NVRAM up up
Serial0/1/0 unassigned YES NVRAM down down
SM1/0 unassigned YES NVRAM administratively down down
SM1/1 unassigned YES unset administratively down down
SM2/0 192.168.13.1 YES TFTP up up
SM2/1 unassigned YES unset up up
SM3/0 unassigned YES NVRAM administratively down down
SM3/1 unassigned YES unset up up
Vlan1 192.168.13.1 YES NVRAM up up

Thought must be placed into what IP address to assign the UCS Express:

In my case, we were simply going to use an IP address on the server subnet at the remote site. The gateway for this subnet resided on a Cisco Catalyst 4500 switch connected to the 3945 router. The problem with this is that the default gateway for the SRE (that you will see configured below) has to be an IP address on the router or nothing outside of the router will be able to reach the UCS Express. So, with that in mind, I set the UCS Express default gateway to the inside (LAN) interface of the router. This didn’t work either. I still couldn’t reach the UCS Express from anything that wasn’t connected directly into the router. Another option was to move the SVIs from the 4500 switch to the 3945 router. The problem with this is that all inter-Vlan routing must go across a 1GigabitEthernet link up to the router. This wasn’t ideal, so I opted to convert the link between the 4500 and the 3945 to a Layer 3 link, assigning IP addresses to each interface. This meant I needed to create a new subnet for the UCS Express servers on the 3945, using a new Vlan. I created a SVI for Vlan 13 (to match the third octet of my server subnet), but since no interfaces were configured for Vlan 13 the Vlan 13 SVI stayed down. Ultimately I had to set the default gateway for my UCS Express to the Vlan 1 SVI (which always stays up), using the subnet I created.

Configuration

interface Vlan1
description Cisco SRE Servers
ip address 192.168.13.1 255.255.255.0
!
interface SM2/0
description UCS Express
ip address 1.1.1.1 255.255.255.252
service-module ip address 1.1.1.2 255.255.255.252
!Application: SRE-V Running on SMV
service-module ip default-gateway 1.1.1.1
service-module mgf ip address 192.168.13.10 255.255.255.0
!
interface SM2/1
description Internal switch interface connected to Service Module
switchport mode trunk
!
ip route 1.1.1.2 255.255.255.255 SM2/0

Cisco TAC didn’t really have much advice on why it had to be configured this way, but through trial and error I found that it worked (and this is the only way I got it to work). Confused on why I have so many IP addresses on the SM2/0 interface? Me too. But the 1.1.1.0 /30 addresses are used for communication on the ISR backplane to the SRE. The 192.168.13.0 /24 addresses are routable and used for our UCS Express servers. The next step is to install the SRE-V software on the SRE. Go to cisco.com and download the latest SRE software. At time of writing, I downloaded sre-v-k9.smv.1.0.2.zip and unzipped it to my FTP server. Installing SRE-V requires a FTP or HTTP server serving up the files. I unzipped the files to my FTP server:

ftp> ls
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
-rw-rw—- 1 538 538 141975 Feb 17 14:45 sre-v-installer-k9.smv.1.0.2.prt1
-rw-rw—- 1 538 538 14162632 Feb 17 14:45 sre-v-installer.smv.1.0.2
-rw-rw—- 1 538 538 916 Feb 17 14:45 sre-v-k9.smv.1.0.2.key
-rw-rw—- 1 538 538 250717 Feb 17 14:45 sre-v-k9.smv.1.0.2.pkg
-rw-rw—- 1 538 538 1692 Feb 17 14:45 sre-v-k9.smv.1.0.2.pkg.install.sre
-rw-rw—- 1 538 538 698 Feb 17 14:45 sre-v-k9.smv.1.0.2.pkg.install.sre.header
-rw-rw—- 1 538 538 65453522 Feb 17 14:47 sre-v-mgmt-k9.smv.1.0.2.prt1
-rw-rw—- 1 538 538 299803259 Feb 17 14:51 visor.smv.1.0.2.prt1
226 Directory send OK.

Next, I executed the following command on the 3945:

service-module sm 2/0 install url ftp://user:password@192.168.0.191/sre-v-k9.smv.1.0.2.pkg
Proceed with installation? [no]: yes
Loading sre-v-k9.smv.1.0.2.pkg.install.sre !
[OK - 1692/4096 bytes]

Service module installation
ios_version 15.1(3)T,
ios_image c3900-universalk9-mz
pkg_name sre-v-k9.smv.1.0.2.pkg
key_file sre-v-k9.smv.1.0.2.key
helper_file sre-v-installer.smv.1.0.2

Check target platform capabilities
cpu 1864
Resource check completed successfully. Proceeding to Install….

router#
*Feb 2 20:04:48.526: %SM_INSTALL-6-INST_RESET: SM2/0 is reset for software installation.

router#service-module sm2/0 status
Service Module is Cisco SM2/0
Service Module supports session via TTY line 131
Service Module is trying to recover from error
Service Module heartbeat-reset is enabled
Service Module is in fail open
Service Module status is not available

Module resource information:
CPU Frequency: 1864 MHz
Memory Size: 2530 MB
Disk 0 Size: 500107 MB
Disk 1 Size: 500107 MB
Disk 2 Size: 2055 MB

Install of ftp://*****:*****@192.168.0.191/sre-v-k9.smv.1.0.2.pkg in progress
Install status : File sre-v-k9.smv.1.0.2.key requested

Local Partition Info – (0 apps)
=====================
Retrieving partition information

*Feb 2 20:28:28.302: %SM_INSTALL-6-INST_PROG: SM2/0 PROGRESSING: Validating package signature ….
*Feb 2 20:28:28.426: %SM_INSTALL-6-INST_PROG: SM2/0 PROGRESSING: Parsing package manifest files ….
*Feb 2 20:28:30.338: %SM_INSTALL-6-INST_PROG: SM2/0 PROGRESSING: Starting payload download.
*Feb 2 20:28:34.226: %SM_INSTALL-6-INST_PROG: SM2/0 PROGRESSING: Starting payload download.
*Feb 2 20:28:45.614: %SM_INSTALL-6-INST_PROG: SM2/0 PROGRESSING: Performing Hot install ….

Install successful on SM2/0

*Feb 2 20:31:33.734: %SM_INSTALL-6-INST_SUCC: SM2/0 SUCCESS: install-completed.

Now the hypervisor is installed on the SRE. I can log into the module, verify the IP address, define the gateway, and activate the hypervisor license:

router# #service-module sm2/0 session
Trying 1.1.1.9, 2131 … Open
se-1-1-1-10#
se-1-1-1-10# show hypervisor ip
Hostname: localhost
Domain Name: None
IP Config: 192.168.13.100(Subnet Mask: 255.255.255.0)
169.254.1.1(Subnet Mask: 255.255.255.0)
Default Gateway: None
Preferred DNS Server: None
Alternative DNS Server: None
se-1-1-1-10# hypervisor set ip default-gateway 192.168.13.1
se-1-1-1-10# license activate sreVHost
Evaluation licenses are being activated in the device for the following feature(s):

Feature Name: SRE-V-HOST-LIC
……..
ACCEPT? [y/n]?y

License activation count saved for use at next reload

At this point you’ll want to reload the module for the license to take effect: router# service-module sm2/0 reload. Once the module has reloaded, you should be able to connect to it with your vSphere Client. But, just in case you can’t (and believe me, I couldn’t the first 4-5 times I tried setting this up) here are some useful commands to troubleshoot (taken from here):

se-192-168-13-10# sh ip route
Main Routing Table:
DEST GATE MASK IFACE
1.1.1.0 0.0.0.0 255.255.255.252 eth0
169.254.1.0 0.0.0.0 255.255.255.0 eth2
0.0.0.0 1.1.1.1 0.0.0.0 eth0

se-192-168-13-10# show hypervisor nics
Name PCI I/E Driver Link Speed Duplex MAC Address
vmnic0 0000:01:00.00 E e1000e Down 0Mbps Half c8:4c:75:2e:ce:41 (this is the physical port on the module front)
vmnic1 0000:02:00.00 I bnx2 Up 1000Mbps Full 50:3d:e5:0d:74:c1 (these are the logical ports on the module backplane)
vmnic2 0000:02:00.01 I bnx2 Up 1000Mbps Full 50:3d:e5:0d:74:c0

3 total nics (I – Internal nic, E – External nic)

se-192-168-13-10# show hypervisor vmknic
Intf. Portgroup/DVPort IP Address Netmask MAC
—————————————————————————
vmk0 Management Network 192.168.13.10 255.255.255.0 c8:4c:75:2e:ce:41
vmk1 CiscoReservedLocal 169.254.1.1 255.255.255.0 00:50:56:7f:2f:5c

2 total VMkernel nic(s)

se-192-168-13-10# show hypervisor vswitch
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 128 3 128 1500 vmnic2

PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 0 vmnic2
Management Network 0 1 vmnic2

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
ciscoSwitchLocal 8 3 8 1500

PortGroup Name VLAN ID Used Ports Uplinks
CiscoReservedLocal 0 2

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
ciscoSwitch 8 3 8 1500 vmnic1

PortGroup Name VLAN ID Used Ports Uplinks
CiscoReserved 0 1 vmnic1

When you’re ready to connect to the UCS Express, the default username is “esx-admin” and the default password is “change_it”:

Hopefully you’ll be able to connect. After I logged in and confirmed that I could in fact login, I went ahead and changed the default password. To do this, login to the SRE (service-module sm2/0 session) and enter “user update esx-admin password __________”. You can go wild creating additional accounts and groups (check this page), but I stuck to the one account for this simple deployment.

When you’ve logged in, you’ll see this familiar screen:

If you want to create an additional vSwitch, it must use the physical GigabitEthernet interface on the front of the module (that’s the only free interface left). You can create additional Virtual Machine Port Groups on vSwitch0. Remember how we defined sm2/1 to be a trunk? Now we can create multiple Port Groups in multiple Vlans on vSwitch0. For us, this is a simple deployment so we left the VM Network Port Group using Vlan 1 (since our VMs will be in the same subnet as the Service Console). Cisco has a pretty stern warning on modifying the ciscoSwitchLocal and ciscoSwitch vSwitches — don’t do it! At this point you can begin deploying Virtual Machines. Oh and don’t forget to write the SRE configuration:
se-192-168-13-10# write memory

Anyone else have experience (good or bad) deploying the UCS Express? Please share in the comments below.

– Andrew

About these ads

About this entry