Ubiquiti All But Confirms Breach Response Iniquity

For four days this past week, Internet-of-Things giant Ubiquiti failed to respond to requests for comment on a whistleblower’s allegations the company had massively downplayed a “catastrophic” two-month breach ending in January to save its stock price, and that Ubiquiti’s insinuation that a third-party was to blame was a fabrication. I was happy to add their eventual public response to the top of Tuesday’s story on the whistleblower’s claims, but their statement deserves a post of its own because it actually confirms and reinforces those claims.

Ubiquiti’s IoT gear includes things like WiFi routers, security cameras, and network video recorders. Their products have long been popular with security nerds and DIY types because they make it easy for users to build their own internal IoT networks without spending many thousands of dollars.

But some of that shine started to come off recently for Ubiquiti’s more security-conscious customers after the company began pushing everyone to use a unified authentication and access solution that makes it difficult to administer these devices without first authenticating to Ubiquiti’s cloud infrastructure.

All of a sudden, local-only networks were being connected to Ubiquiti’s cloud, giving rise to countless discussion threads on Ubiquiti’s user forums from customers upset over the potential for introducing new security risks.

And on Jan. 11, Ubiquiti gave weight to that angst: It told customers to reset their passwords and enable multifactor authentication, saying a breach involving a third-party cloud provider might have exposed user account data. Ubiquiti told customers they were “not currently aware of evidence of access to any databases that host user data, but we cannot be certain that user data has not been exposed.”

Ubiquiti’s notice on Jan. 12, 2021.

On Tuesday, KrebsOnSecurity reported that a source who participated in the response to the breach said Ubiquiti should have immediately invalidated all credentials because all of the company’s key administrator passwords had been compromised as well. The whistleblower also said Ubiquiti never kept any logs of who was accessing its databases.

The whistleblower, “Adam,” spoke on condition of anonymity for fear of reprisals from Ubiquiti. Adam said the place where those key administrator credentials were compromised — Ubiquiti’s presence on Amazon’s Web Services (AWS) cloud services — was in fact the “third party” blamed for the hack.

From Tuesday’s piece:

“In reality, Adam said, the attackers had gained administrative access to Ubiquiti’s servers at Amazon’s cloud service, which secures the underlying server hardware and software but requires the cloud tenant (client) to secure access to any data stored there.

“They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration,” Adam said.

Adam says the attacker(s) had access to privileged credentials that were previously stored in the LastPass account of a Ubiquiti IT employee, and gained root administrator access to all Ubiquiti AWS accounts, including all S3 data buckets, all application logs, all databases, all user database credentials, and secrets required to forge single sign-on (SSO) cookies.

Such access could have allowed the intruders to remotely authenticate to countless Ubiquiti cloud-based devices around the world. According to its website, Ubiquiti has shipped more than 85 million devices that play a key role in networking infrastructure in over 200 countries and territories worldwide.

Ubiquiti finally responded on Mar. 31, in a post signed “Team UI” on the company’s community forum online.

“Nothing has changed with respect to our analysis of customer data and the security of our products since our notification on January 11. In response to this incident, we leveraged external incident response experts to conduct a thorough investigation to ensure the attacker was locked out of our systems.”

“These experts identified no evidence that customer information was accessed, or even targeted. The attacker, who unsuccessfully attempted to extort the company by threatening to release stolen source code and specific IT credentials, never claimed to have accessed any customer information. This, along with other evidence, is why we believe that customer data was not the target of, or otherwise accessed in connection with, the incident.”

Ubiquiti’s response this week on its user forum.

Ubiquiti also hinted it had an idea of who was behind the attack, saying it has “well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. As we are cooperating with law enforcement in an ongoing investigation, we cannot comment further.”

Ubiquiti’s statement largely confirmed the reporting here by not disputing any of the facts raised in the piece. And while it may seem that Ubiquiti is quibbling over whether data was in fact stolen, Adam said Ubiquiti can say there is no evidence that customer information was accessed because Ubiquiti failed to keep logs of who was accessing its databases.

“Ubiquiti had negligent logging (no access logging on databases) so it was unable to prove or disprove what they accessed, but the attacker targeted the credentials to the databases, and created Linux instances with networking connectivity to said databases,” Adam wrote in a whistleblower letter to European privacy regulators last month. “Legal overrode the repeated requests to force rotation of all customer credentials, and to revert any device access permission changes within the relevant period.”

It appears investors noticed the incongruity as well. Ubiquiti’s share price hardly blinked at the January breach disclosure. On the contrary, from Jan. 13 to Tuesday’s story its stock had soared from $243 to $370. By the end of trading day Mar. 30, UI had slipped to $349. By close of trading on Thursday (markets were closed Friday) the stock had fallen to $289.