US cloud service provider Rackspace says an ongoing outage affecting its hosted Microsoft Exchange environments and likely thousands of customers was caused by a security incident.

The list of affected services includes MAPI/RPC, POP, IMAP, SMTP, ActiveSync, and the Outlook Web Access (OWA) interface used to access the Hosted Exchange instance to manage online emails.

“We are investigating an issue affecting our Hosted Exchange environments. More details will be released as they become available,” Rackspace said Friday night at 2:49 a.m. EST when it acknowledged the outage.

More than 15 hours later and multiple updates with no information on what is causing what it describes as a “system disruption”, the company said it was “aware of an issue affecting” Hosted environments. Exchange and that its engineering teams continue to work “to come to a resolution.”

The affected customers were informed to check the status page for the latest updates, even if those are also missing details about the root cause of the failure.

In response, disgruntled Rackspace customers demanded the company social media to provide an ETA for when the issue causing this outage will be resolved and shared plans to move to another more transparent managed service provider (MSP).

Nearly twenty-four hours later, at 1:57 a.m. EST, Rackspace revealed the real cause of the outage, a security incident “isolated to a part of our Hosted Exchange platform” that forced the company to disconnect the Hosted Exchange environment.

“On Friday, December 2, 2022, we became aware of an issue affecting our Hosted Exchange environment. We proactively disabled and disconnected the Hosted Exchange environment while we triaged to understand the extent and severity of the impact “, said the company. said.

“After further analysis, we have determined that this is a security incident. The known impact is isolated to a portion of our Hosted Exchange platform.”

This confirms some of the concerns of its customers who, due to the limited information, said they feared the outage was the result of a malware or ransomware attack.

Natalie Silva, Rackspace’s global PR manager, told BleepingComputer in an email Friday evening that the MSP is now providing affected customers with Microsoft Exchange Plan 1 licenses and instructions on how to migrate their email to Microsoft 365 until the outage is resolved.

“While we continue to work on the root cause of the issue, we have provided an alternate solution that will re-enable our customers’ ability to send and receive email by providing access to an alternate email solution at no cost to them” , Silva said.

“This solution will allow our impacted customers to resume their regular activities as soon as possible.”

Detailed instructions on activating free licenses and migrating user mailboxes to Microsoft 365 are available at Rackspace crash report.

The ProxyNotShell vulnerability

While Rackspace shared very little information about the attack, the cybersecurity expert Kevin Beaumont shared a possible explanation.

Beaumont told BleepingComputer that Rackspace appears to have run a Microsoft Exchange server that was vulnerable to the ProxyNotShell vulnerability.

ProxyNotShell was a zero-day vulnerability discovered to be actively exploited in September 2022 to install web shells on Microsoft Exchange servers.

Microsoft fixed the vulnerability in November as part of their Patch Tuesday updates.

However, Beaumont discovered through Shodan than one of Rackspace’s servers,’mex06.emailsrvr.com,’ was running Microsoft Exchange build 15.0.1497.40associated with the August patch level.

“This Exchange build number is from August 2022, before ProxyNotShell patches were available,” Beaumont explained in a publication about the security incident.

Beaumont says that while long build numbers aren’t always reliable, that could be why Rackspace suffered the security incident.

BleepingComputer has contacted Rackspace with questions about the security incident, but has yet to receive a response.

Updated December 03, 08:31 EST: Revision of article and title after Rackspace linked its ongoing outage to a security incident.
Updated December 03, 12:38 p.m. EST: Added information from Kevin Beaumont.