BREAKING UPDATE: Intel microcode reboot issues - https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/.

Intel is advising customers to continue their operating system patching. Cisco is working with Intel and will update the PSIRT advisory once we have additional information.

 

As the UCS product manager for UCS and covering the Meltdown and Spectre vulnerability response from the Cisco UCS team, the past couple of weeks have been busy. I’ve had many conversations with our engineering and QA teams, Cisco field teams, customers, and partners. I want to take this opportunity to provide additional information and clarification for topics that have come up many times in the past few weeks.

 

First, there are 3 vulnerabilities - two Spectre variants (CVE-2017-5753 and CVE-2017-5715) and Meltdown (CVE-2017-5754). All three vulnerabilities require operating system patches, and Spectre variant 2 (CVE-2017-5715) requires operating system patching as well as update processor microcode. Cisco officially communicates vulnerabilities, fixes, and timelines through our PSIRT organization and the advisory is available at https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180104-cpusidechannel. It is important to note that there are no known exploits at this time which is reflected in the severity of the advisory, although there is always a chance that could change at any time.

 

The next natural question is how to mitigate these vulnerabilities. As mentioned, all three vulnerabilities require operating system patches while CVE-2017-5715 requires both a processor microcode and an operating system patch. Applying both is the best practice. However, several operating systems, including RedHat Linux, SuSE Linux, and VMware (and potentially others), have options to load new microcode during the operating system boot process. While that microcode wouldn’t persist across reboots the way a firmware patch would provide, it absolutely does provide mitigation of the vulnerability as long the next server boot does the same thing. Basically, as is always the case with vulnerabilities, the standard response of keeping systems up to date on firmware and operating system patches is the standard and applicable response.

 

Of course, as patches are tested and planned, people want to know the impact of them. Spectre variant 1 and Meltdown operating system patches reportedly have little to no performance impact. Spectre variant 2 - CVE-2017-5715, which requires both operating system as well as processor microcode updates, may have performance impacts. The Cisco UCS team’s testing indicates that CPU and memory constrained workloads don’t have any significant performance impacts. However, IO constrained workloads show a measurable and sometimes significant performance impact. In particular, we are seeing the biggest impact on IO tests to local and SAN-based storage with sequential reads or sequential writes and especially with 4k block size. I’ve heard similar findings from customers and partners this week as well. You can find more additional information from RedHat and Microsoft that also provide similar guidance. While it is unrealistic to comment on specific workloads, these general performance guidelines are a good starting place.

 

Finally, from a Cisco UCS perspective, we are planning to patch several versions of software which will be detailed in the Cisco Bug IDs referenced in the Cisco PSIRT advisory as release updated firmware. We will update UCS Manager host firmware bundles (B and C bundles) for UCS Manager 3.2(2), 3.1(3) and 2.2(8), our current patch points. Starting with UCS Manager 2.2(4), you can run versions of host firmware (B/C) bundles without upgrading the UCS Manager infrastructure (A) bundle to the same or later version. This is documented in the UCS Manager Release Notes for each version and will apply to these firmware updates as well. Cisco IMC for non-UCS Manager connected C-Series servers will include updated microcode in Cisco IMC 3.1(2) and 3.0(3) initially. UCS firmware updates are also applicable to Cisco Hyperflex servers. Finally, firmware updates will roll out initially for Cisco UCS and Hyperflex M5 and M4 servers, with M3 and M2 servers to follow as detailed in the PSIRT advisory.

 

There has been a lot of press and information about these vulnerabilities and their impacts. Hopefully this blog post provides some relevant information and guidance that will help Cisco UCS server users.

 

Jacob Van Ewyk

UCS Management Product Manager