CUCM Migration from One DC to Another

Published: March 21, 2021
Read time: 15 minutes

CUCM is a key component in Cisco collab deployment, and the deployment/dependencies can be very different from organization to organization. So changing anything on CUCM needs a lot of prep and checks.

Recently we did a CUCM migration from one DC to another. I've put together this checklist that gives you a quick overview of all the steps you need to do for CUCM migration. I will list existing resources available on the Internet and add a few things from my experience.

Before making any changes, please consult with Cisco TAC or do your own research/testing in a lab environment.

Disclaimer

Please don't consider this as a step-by-step guide and be open to the possibility that I may have missed a few points:

• You may not have an identical environment.
• You may even hit different issues which I didn't come across.
• In short, I am not responsible for any disaster 🙂

Background

IP/Subnet is usually tied to location/DC. So when you change DC, you may have to change the IP address of the server too. Changing IP address vs. hostname (or both) has different effects. If possible, try to change fewer things at a time. If you can keep the hostname/FQDN, it's good (less work in one change window).

If for some reason you can keep the same IP address and system settings, then you have even less work to do. You could use DRS (Backup-Restore) or the method I am going to mention here (using PCD).

What Are the Options?

VMotion isn't supported for this task (by Cisco, or so I was told) and isn't easy—taking the whole VM over the WAN can take time, and it's unpredictable/scary when you have limited downtime.

3rd party tools – Zerto or Commvault may copy the entire VM in advance and sync the changes just before the cutover window, but can create other issues and can be complicated with more dependencies.

DRS (Backup – Restore) and then re-addressing – Could work with lots of nerd knobs, but can be complicated and needs lots of manual work. If you can have the same system settings at the time of backup restore, etc. And then change all the system/enterprise parameters, URLs in config wherever you have the CUCM IP.

Cisco PCD (Prime Collaboration Deployment) – In plain English, it's a free orchestration tool by Cisco, which you can deploy on a VM and it can do many things for UC (CUCM, IM&P, CUC, UCCX) servers like Migration, Installation, Upgrade, etc.

Have a quick look at the chapter: Cisco Prime Collaboration Deployment Features

PCD Features Chapter

The Admin guide covers all details on how to deploy, maintain, etc.:

PCD Admin Guide

I have used PCD in the past for CUCM P2V migration (from MCS to VM) as well. Note that the Migration task is only supported for CUCM and IM&P. So for other apps:

• CUC and UCCX – You will need to use the DRS method.
• Expressway – VMotion is recommended by Cisco.
• Cisco Meeting Server – You can use Backup and Restore.

In this blog, I will only focus on CUCM using PCD.

Why PCD?

• Supported by Cisco, tested, and super easy.
• Doesn't cost anything. Fewer dependencies on other teams.
• Can do migration (installation, readdress, upgrade... even updates enterprise parameters after the migration).

High-Level Plan

1. Deploy PCD server. Try to deploy at the new DC. Check this chapter on how to.
2. Add/Discover your existing CUCM cluster on PCD. It will need the OS admin password. It will fetch the server info. At the time of migration, it will take the backup, restore at destination, migrate all user-facing features, change all the parameters at new clusters, and shut down the existing cluster.
3. Deploy OVF (Create VM) on the destination (new DC) side.
4. Add an ESXi host server on PCD where these new VMs (New DC) are created.
   • To communicate with an ESXi host server, Cisco Prime Collaboration Deployment requires either root access to the ESXi software or a non-root user with Host (Configuration, Storage Partition Configuration) and Virtual Machine (Interaction, Configure CD Media, Configure Floppy Media, Device Connection, Power Off, and Power On) privileges enabled.
   • If this root account password is 16+ characters, be aware of this bug: CSCva95404
5. Download bootable image for installation of new CUCM.
6. Define your SFTP or use the local SFTP within PCD for copying the bootable image for installation. If using SFTP built-in to PCD, use WinSCP and copy the file at this location:
   • For Migration → ISO files should be copied into /fresh_install directory.
   • The Username for SFTP will be "adminsftp" and the password will be the PCD OS Administration Password.
7. Plan a change window and create a task in PCD to do the migration. If you have SMTP configured, it will send you an email when migration is done 🙂 Or you can monitor the progress from GUI itself.

Plan Your Migration

• Check all the compatibilities with new HW, ESXi, etc.
• Figure out all the dependencies. Discuss and work with your Network, System/DC/AD team. This is not an exhaustive list here, but I suggest that you put all the things in writing and keep updating/discussing it with all the stakeholders.
  • Get your new IP address, Def GW, System settings, OS admin password, etc., ready.
  • Figure out what the new DNS, NTP, LDAP server config will be at the new DC.
  • When you create a new VM at the new DC, ensure all the network connectivity/firewall rules within the new DC, from the new DC to all branch locations, and from New to OLD DC (as some servers still are at old DC). VLAN config on server side, etc.
  • If you are keeping some apps/servers in the old DC, figure out if it's meeting all the latency requirements.
  • Access/process to update the DNS A records of CUCM with the new IP address, if you are keeping the hostname.
  • Write down all the DHCP scopes where you need to update Option 150 (TFTP IP address for IP Phones). Discuss who has access, verify the access, and that you understand where the DHCP scope is for all branch locations and who will update it.
  • Discuss with your local IT support about manual updates of TFTP IP address if some phones have static IP addresses. If you have a phone remote tool, then it's easier to do it yourself.
• Check all your infra is using the FQDN for this CUCM cluster. If not, on the other infra side, use FQDN instead of IP address. Such as other CUCM having SIP Trunks, VC endpoints configuration having FQDN of CUCM server in provisioning config, Expressway neighbor zones, Voice GW, media resources, 3rd party servers, etc.
  • If any of these are using the IP address of your CUCM, plan a change window in advance and make it use FQDN so it will save you some time.
• Have your PCD ready and get yourself familiar with it. Deploy the VMs, add the existing cluster, and new ESXi to PCD.
• Try to create a migration task, figure out all the steps it shows, and all the details you need handy to put. See the option of "Re-addressing with new IP" with migration. I suggest avoiding patching/upgrade in combination with migration—if you hit some bug, it will be complicated to troubleshoot.
• Keep your SFTP server ready; get the bootable image ready.
• Plan about your licensing. If it's 12.5.x with Smart licensing, then it should be okay and may not need any additional effort.
• Read the documentation carefully and look for any existing bugs with your PCD/CUCM version related to migration. I faced this unexpected bug in between migration and had to push through the errors. It went all fine but scared/confused me and wasted some time. In this case, it's asking to delete those last few steps where PCD shuts down the old cluster: CSCut43030
• If your CUCM version is 12.0.x, PCD Migrations require a COP file: ciscocm-slm-migration.k3.cop.sgn. Clarify with Cisco TAC in advance or check the document.
• Plan your change/downtime window carefully.
  • For example, on the weekend there may be a backup job running and choking all the bandwidth at the same time you are doing the migration. It will add more time when PCD is trying to copy the backup from source to destination. Talk to your Network team in advance.
  • Another point is, initial tasks such as copying the backup and installation to the new location of a cluster (3 nodes) may take up to approximately 8 hours (depends on size, BW), so maybe you can even start on Friday evening. Schedule it and do something else (maybe sleep 🙂). Take this advice lightly—it depends on your situation, process, and lots of things.
• Open a change, fix the date, and have people ready to support.
• Open a TAC case; ask all the questions. Leave the case open as Sev3 and inform them about the change window to pass onto the Engineer who will be available that day. So if you need any help, you will need to call TAC frontline and raise the Sev to get the Engineer on Webex right away.
• Write down your rollback plan. In this case, you could simply turn on your old cluster and undo all the network/DNS changes.
• Write down your test plan thoroughly.
• Do a test of the entire change, if possible.

1 Day Before or in the Same Change Window

• Have all the credentials ready/handy.
• Verify all your access.
• Take a fresh backup.
• Give a heads-up to all people involved.

In Change Window

• Take a fresh backup. Take all the screenshots from RTMT and CUCM config.
  • For example, when an IP phone/MGCP GW/endpoint isn't registered, it doesn't show the IP address on the CUCM admin page. So with a new cluster, if you want to check any particular IP phone GUI, it's difficult to find the IP from the CUCM side. Having a screenshot or recording might help.

RTMT Screenshots

Screenshots from RTMT are super helpful—don't miss it.

• Before you start the migration, if your cluster is in mixed mode, I recommend using "Prepare Cluster for Rollback to Pre 8.0" in enterprise parameter. This effectively disables the ITL functions on the phone, so phones blindly trust the next CTL or ITL file.
  • Check with Cisco TAC to be 100% sure.
• Start the migration task in PCD, with re-addressing of CUCM. Wait for it to finish. Once it's done, note that it should have updated the IP address in all the parameters/URLs in CUCM config as well.
• Make other changes (DNS record update, DHCP scope update), remove the "Prepare Cluster for Rollback to Pre 8.0." Check the status/health. Shut down the old cluster if not done automatically.
• Check all the registrations and services. Note that when you update DNS records, it may take time to propagate everywhere. Updates between AD/DNS servers take some time; this may add some delay for other servers to update the SIP trunk with the new cluster. So perhaps you can update the DNS records when you start (or in the mid of) the PCD task itself. It depends on your downtime window planning.

DNS Cache

If you are checking from your Windows PC cmd, make sure to do ipconfig /flushdns or else it will show your cached entry and you will be confused.

• If any SIP trunk/registration is stuck, you may need to reset it from the other side. We had an issue from the Skype side and had to reset the trunk from there.
• Verify all the registrations, compare your RTMT screenshots, and verify everything. Test all the services thoroughly. In my case, Extension Mobility was broken as PCD messed up the URLs (maybe some bug). It added something at the end and had other issues, so I had to find the URL from CUCM doc and put it again.
• If all checks/testing goes good, make sure to check if all tests are passed in utils diagnose test.
  • If you see NTP stratum error, it could be either you are using Stratum 4 or above, or Windows Server as NTP. Cisco UC apps throw an error and recommend using Linux-based NTP.
• Do check all the usual things like services up, DB cluster is good, etc.
• If you suspect some registrations are missing, turn on the old cluster and see if you have registrations still hitting it. If so, you will know which devices still need some work. Make sure to shut the old cluster down when you are done.

If you didn't do the re-address, then it should be relatively easy. If you also changed the hostname, then there could be some difference in steps—instead of updating the DNS records, you will create new ones and point to the new FQDN. You will also need to do other steps such as Cert regeneration, and maybe more.

Post Migration

• Take a fresh backup and VM snapshot of the new cluster.
• Keep your old VM for a month or so.
• You can keep the PCD for usual tasks or get rid of it. From 12.5.x, CUCM has mini-PCD built in for its own cluster upgrade orchestration. But you can keep this PCD VM for other installation/upgrade tasks (CUC, UCCX, etc.).

Conclusion

Migration of a CUCM cluster isn't an easy task, and you may see some different challenges based on your deployment, dependencies, and process. So plan it well. Write things down. Communicate with all stakeholders. Have people on standby.

Share Your Feedback

If you have any feedback or suggestions, please reach out via my Notion form or DM me on LinkedIn. Thank you for your time!