Skip to content

CUCM Upgrade 11.5 to 12.5.x

Published: February 5, 2021
Read time: 12 minutes

This checklist gives you a quick overview of all the steps you need to do for a CUCM upgrade from 11.5.x to 12.5.x. I will list existing resources available on the Internet and add a few things from my experience. Before making any changes, please consult with Cisco TAC or do your own research/testing in a lab environment.

Disclaimer

Please don't consider this as a step-by-step guide and be open to the possibility that I may have missed a few points:

  • I performed this upgrade a few months ago, so I might have forgotten a few things. I am human 🙂
  • You may not have an identical environment.
  • You may even hit different issues which I didn't come across.
  • In short, I am not responsible for any disaster 🙂

Look for BRKUCC-2011 on the Cisco Live library—there are 2 presentations/sessions containing all the screenshots and details. The Cisco Live on-demand library is very useful, informative, and free. Just log in with your CCO account or sign up.

Cisco Live On-Demand Library

Keep the official upgrade guide handy for an exhaustive list:

CUCM 12.5.x Upgrade Guide

Known Bug on CUCM 12.5 SU2

On CUCM 12.5 SU2, we are facing bug CSCth35648 (and a few others), causing ITL issues.

Run this command, and if you see the error on the last line, then you are facing the bug:

admin: show itl
"Failed or interrupted I/O operations.
Error parsing the ITL File."

Contact Cisco TAC as they need to fix the permission issue using the root account, along with a few other things. In version 12.x, we expect to have ITL and CTL signed by the ITLRecovery cert. If you don't see it, then it also needs to be fixed.

Rather than explain in detail, I'll let you contact Cisco TAC with the bug ID, and they will help you with actions based on your scenario/specific issue.

Pre-requisites

  • Figure out your target version. Know your WHY. Check release notes for open bugs and security vulnerabilities.
  • Check your platform/server hardware compatibility with the target version: CUCM 12.5 Compatibility Information
  • Check your ESXi version compatibility with the target version (check BRKUCC-2011 presentation).
  • Check if any OVA template changes are required (RAM, CPU increase changes recommended or required):
  • Check compatibility of other applications relying on CUCM/CUC/UCCX with the target version (SBC, Voice GW, 3rd party apps, etc.).
  • Check the list of deprecated phone models listed in the document. If you have any, take care of that in advance (upgrade, replacement, etc.).
  • Migrate your ELM/PLM licensing to Smart licensing. Check Cisco Live BRKUCC-2725 for more details.
  • If you decide, deploy CSSM in advance: Cisco Smart Licensing

About Smart Licensing

For some servers/devices, only Smart licensing is an option after a specific version (e.g., CUCM 12.5.x); for some, it's still optional (UCCX 12.5.x).

In plain English: With Smart licensing, Cisco syncs your license status/inventory to the cloud rather than being PAK-based/tied to your device in offline mode (so no pain when you change general settings and the license MAC changes). You need to decide whether you want to sync your infra devices to the Cisco licensing portal directly (device having internet access), via HTTP proxy, or you can deploy CSSM on-prem (a lightweight VM) which can sync licenses from Cisco and with your infra (man in the middle 🙂).

Smart licensing goes beyond UC servers—you can deploy one CSSM and use it for your entire organization. You will need to organize and clarify the Smart Account structure in advance. Check Cisco Live BRKUCC-2725 for more details.

Example Smart Account Structure:

  • Top Level: Smart Account = ABC.com
    • Sub Level: Virtual Account = AP Collab
    • Sub Level: Virtual Account = EU Collab

CSSM is simply a router/proxy that talks to internal devices with an identifier and syncs available licenses for your Smart accounts.

If you decide to use CSSM with Smart licensing, I suggest creating a Smart account with Cisco and deploying/configuring CSSM in advance, then focusing on license migration.

BRKUCC-2725 has all the details on how to do license migration. If you get an error while doing the migration, open a case with licensing (a few times I got an error and the licensing team helped from the back end)—they will deposit the licenses to your Smart account. You'll probably need to provide: Smart account name, Smart account domain name, and Virtual account name.

Plan Your Upgrade

  • Plan your change window, open a change, etc.
  • Keep in mind that with RU upgrade, the server is down while the upgrade is taking place on the inactive partition. RU = Refresh Upgrade = Underlying Linux version is changing. Plan your downtime window accordingly.
  • Decide your upgrade method – Direct or PCD. PCD = Prime Collaboration Deployment, a VM you can deploy to orchestrate the migration/upgrades for you.
  • Download the software images and keep them on an SFTP server. Notice the difference between Restricted vs. Unrestricted versions—use the correct image: CUCM 12.5(1) SU3 Download
  • Keep an OVA file/bootable image of your existing version on SFTP just in case you need to rebuild the server. If you have an Enterprise Agreement with Cisco, you can download it yourself; otherwise, open a Cisco TAC case and ask for images.
    • Bootable images aren't available/same under the download page.
    • From 12.x, CUCM and Unity Connection ISOs are separate.
  • Either verify that you have rights to deploy a new VM, or keep your team ready who can create a new VM in the change window if you have to rebuild a server (worst case scenario).
  • If your server has an uptime of more than 365 days, it's nice to plan and give it a reboot in advance of your upgrade.
  • Verify you have rights to remove existing/old VM snapshots before upgrade.
  • Verify the free space. You will need ~43 GB of space. Some ways to clean space are:
    • Run a COP file that cleans up the space (may need to do in downtime). Ask Cisco TAC before you do it—it might delete your inactive partition software.
    • Reduce the log size: Procedure to Adjust WaterMark in RTMT
    • Delete unused load files from CUCM OS Admin → Load Management.
  • Open a Cisco TAC case for support in advance and inform them to requeue the case to an engineer who is available in your change window. Put basic details about what you are doing in that case—it will save your time in the change window.
    • Note that you may need to call TAC frontline and may or may not need to raise a Sev in the change window, as Cisco TAC could be busy with multiple cases.
  • Read the upgrade guide carefully.
  • Review the Recommended Sequence in advance from the upgrade doc. See some notes here:
    • Publisher node MUST be upgraded first.
    • Switchover options depend on the collaboration product:
      • CUCM Publisher can be switched over before Subscriber(s) Upgrade Startup.
      • DRU Only → Switchover IM&P Publisher after CUCM Cluster Database Replication Completes.
      • CUC Publisher must NOT be switched over before Subscriber Upgrade Completes.
  • Try the upgrade in lab/Pre-Prod to avoid any surprises.
  • Prepare your test plan in advance. What you are going to test and how you are going to test it.

1 Day Before or in the Same Change Window

  • Verify/take your fresh backup. Back up custom ringtones and background images.
  • Ensure SFTP is up. Please use SFTP in the same location as CUCM to minimize delay (upgrade ISO is ~4 GB).
  • Keep your OS/Cluster password and SFTP password handy.

In Change Window (Pre-upgrade Tasks)

  • Run the Pre-upgrade Check COP file. It checks many pre-requisites, but common issues you may see are NTP/Free space/Expired cert/Simple password—fix issues before upgrade.
    • If you are using Windows NTP or Stratum 4 or above, you may see errors. Fix your cert. Change the password. Check the upgrade guide for more.
  • Verify your backup. Take details/screenshots of registered device count, VG, and SIP trunks.
  • Make VM/OVA-related changes (RAM/CPU) if needed.
  • Change VM compatibility. CUCM 12.5 needs ESXi 6.5 (13). Note that once you change it, you might not be able to roll this back.
  • Change Guest OS type. See the doc—you will need to change it to CentOS 7 (64-bit). It's under VM general settings; if it's greyed out, maybe because you need to shut down the server first. Should be done before CUCM upgrade.
  • Change adapter type to VMXNet3 (if it's not already).
  • Update to the latest VMware Tools.
  • Remove old VM snapshots.

Upgrade

  • You can disable TFTP services and do a few other tweaks mentioned in the document to expedite/smooth the upgrade.
  • Do the upgrade from OS Admin. Follow the sequence mentioned in the doc (Upgrade Pub → Sub1 → Sub2…, Switch version Pub → Sub1 → Sub2…). Be aware of some known issues—with CUC, even if you select "do not reboot after upgrade," it might 🙂
  • For CUC, you may need to run a pre-req COP file as the ISO file is changing. Ask TAC.
  • When you do UCCX ES release, verify the procedure from the doc—it may be recommended to do it from CLI (as there may be a known issue with Tomcat).

Database Replication

In RU (Refresh Upgrade), DB replication may take some time. Be patient or ask TAC.

Post Upgrade

  • Run the Post-upgrade COP file.
  • Reset any parameters you changed to optimize the upgrade (TFTP, enterprise parameters). Check the upgrade doc for more details.
  • Upgrade your RTMT.
  • Check device count, trunk, and VG status.
  • Check all the services. Check all the integrations (CCX, CUC, etc.).
  • Optional: Switch from Native to Open VMware Tools via OS Admin CLI → utils vmtools switch open
  • Change Transport Settings for CUCM for Smart Licensing (you can find the URL from CSSM or from the doc or Google it).
    • Common issues you might face are DNS resolution for CSSM, so use the IP address and port 80. If your DNS and cert are all good, then it could be a different issue like reachability (IP/Port).
  • Create or copy an existing registration token from CSSM or Smart Software Manager satellite server to CUCM.
  • Do your usual testing.

About the Post-upgrade COP File

The post-upgrade COP file compares data taken from the pre-upgrade COP and tells you if any errors occurred post-upgrade. In short, it saves you the manual effort of verifying state pre-upgrade vs. post-upgrade. I still advise taking some screenshots from RTMT about registration count and existing errors just in case, but this COP file should give you a report of what's broken (if anything) after the upgrade.

This is just for CUCM, and I think running it on the Publisher should be enough as it runs the check for the entire cluster (I don't remember 100%).

Common issues could be network services, NTP, cluster or DB-related issues, registration/GW/trunk active count down, etc. Look for this section in the doc for the detailed list.

References:

Share Your Feedback

If you have any feedback or suggestions, please reach out via my Notion form or DM me on LinkedIn. Thank you for your time!