Want to secure workload access to LLMs like ChatGPT? Join Our Webinar | 1 pm. PT on June 18

Blog

Real-Life Examples of Workload Identity Breaches and Leaked Secrets – and What to Do About Them (Updated Regularly)

Breaches involve non-human identities more than ever, and that trend should continue. Here is a catalog of those incidents, with advice for mitigating the risk.
Real-Life Examples of Workload Identity Breaches and Leaked Secrets – and What to Do About Them (Updated Regularly) header image

For the vast majority of IT history, identity has been thought of in the context of humans. Usernames, passwords and other authentication mechanisms dominate our daily lives on the web. However, as technology continues to advance, the concept of identity and access management (IAM) is expanding beyond the human realm to include workloads.

In the age of digital transformation, artificial intelligence and the internet of things, all of which has been propelled forward by the rise of remote work and its heavy reliance on cloud-based resources, workload – aka machine-to-machine – identities have surged alongside. According to Microsoft’s 2023 State of Cloud Permissions Risk report, workload identities are outnumbering human identities 10 to one, and just 1% of permissions are actively used.

workload is any program or application that utilizes computing, data, networking, and storage to perform one or more tasks. Some examples of workloads include:

  • Custom applications you run in the cloud or on-premises environment (including in Kubernetes).
  • HTTP-based APIs from third-party SaaS providers or API gateways.
  • Databases 
  • Application services provided by hyper-scale cloud vendors.

Identity-based attacks thus go beyond the familiar threat vectors that have dominated headlines and lined the pockets of your foes, such as credential stuffing and phishing. While human credentials will remain firmly in the cross-hairs of malicious hackers and at the epicenter of accidental data leakage incidents, the tide is changing. 

Today’s applications are composed of services across multiple clouds, with APIs connecting to third-parties and sensitive databases that require different levels of access. In addition to being distributed, modern apps act autonomously, meaning if in the past they acted on behalf of the user using their credentials, nowadays they operate with their own identity. This dynamic has created a broader attack surface – because in order to achieve their mission of interconnectedness, scalability and user-friendliness, they need to interact and exchange data. 

Adversaries have caught on. They are quickly realizing that the pickings are far greater on the proverbial back-end, yet many organizations are still in the early stages of securing these permissions and getting a true handle on workload identity and access management. Most businesses lack adequate best practices in this area—and often the job of managing workload identities falls to DevOps engineers who view the practice as cumbersome and inconvenient.

All this introduces greater risk for organizations, and we are starting to see it manifest with real-world instances of identity-based workload credential loss incidents affecting organizations, including high-profile brands. We’ll continue to update the below list as more cases emerge.

1) Source Code Leak: The New York Times

Disclosed: June 2024

The New York Times confirmed a breach involving the leak of 270GB of source code, reportedly stolen from the company’s GitHub repositories and posted on the 4chan bulletin board. The leaked data includes source code for various projects, including the popular game Wordle, and a WordPress database containing information on roughly 1,500 users, such as names, email addresses, and password hashes. Additionally, the exposed data includes API keys and secret keys for other systems. The breach was traced back to January 2024, when a credential for a cloud-based third-party code platform (GitHub) was inadvertently exposed, allowing unauthorized access to the repositories. The New York Times assured that no unauthorized access to its internal systems or operational impact occurred. In response, the company identified and mitigated the exposure, implemented additional security measures, and continues to monitor for anomalous activities. This incident underscores the importance of securing development environments and maintaining strict access controls to protect both user and workload identities.

2) Exposed Access Tokens via Unathorized Access: Hugging Face

Disclosed: June 2024

Hugging Face, a platform for AI and ML models, reported a breach involving unauthorized access to its Spaces platform, potentially exposing access authentication secrets. The breach raised concerns about unauthorized access to a subset of Spaces’ secrets. Hugging Face advised users to refresh access keys and tokens or switch to “fine-grained” access tokens, now the default. The company has revoked compromised tokens, notifying affected users via email. To enhance security, Hugging Face has removed organizational tokens, implemented a key management service (KMS) for Spaces secrets, and boosted detection of leaked tokens. External cybersecurity forensic specialists are involved in the ongoing investigation. With over 10,000 organizations and 1.2 million users, this incident highlights the need for stringent security measures to protect AI and ML model platforms.

3) Service Account Compromise: Dropbox 

Disclosed: May 2024 

Dropbox reported a breach affecting its Dropbox Sign service, where an unauthorized actor accessed the production environment via a compromised service account, detected on April 24. The attacker exploited an automated system configuration tool used by Dropbox Sign, gaining access to a service account with broad privileges in the production environment. This enabled access to the customer database, exposing emails, usernames, phone numbers, hashed passwords, API keys, OAuth tokens, and MFA details. Dropbox confirmed no unauthorized access to documents or payment information, and other Dropbox services were unaffected. The company has reset passwords, logged out users, and initiated API key rotations. A forensic investigation is ongoing, and Dropbox is actively reaching out to affected users with protective measures and further instructions. This incident underscores the critical need for stringent access controls and regular credential rotations to safeguard both user and workload identities.

4) GitLab Repository Compromise: Sisense

Disclosed: April 2024

Sisense disclosed a significant breach impacting its Fusion Managed Cloud product, discovered on April 9. Attackers reportedly accessed the company’s GitLab repository, obtaining credentials for Amazon S3 buckets and exfiltrating terabytes of customer data, including access tokens, email passwords, and SSL certificates. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) is involved due to potential impacts on critical infrastructure. Sisense’s response included rotating all authentication credentials, enhancing security monitoring, and restricting firewall ports. Only Fusion Managed Cloud customers were affected, with on-premises and Periscope customers remaining secure. Sisense continues to support customers with password resets, API key rotations, and SSO updates, reinforcing its security posture. This breach highlights the need for robust security and vigilant credential management to protect both user and workload identities.

5) Authentication Token Theft: Cloudflare

Disclosed: February 2024

Cloudflare reported a breach of its internal Atlassian server by a nation-state attacker, leveraging authentication tokens and service account credentials stolen during a prior Okta breach. The intrusion, initiated on November 14, targeted Cloudflare’s Confluence wiki, Jira bug database, and Bitbucket source code management system. The attackers used one access token and three service account credentials not rotated post-Okta compromise, establishing persistent access and attempting to breach a non-production data center in São Paulo, Brazil. In response, Cloudflare cut off access, rotated over 5,000 production credentials, and conducted a comprehensive forensic analysis and system hardening to prevent further intrusions. This incident highlights the ongoing challenges of defending against sophisticated cyber threats and underscores the importance of robust credential management and security protocols to protect corporate infrastructures.

6) OAuth Application Abuse: Microsoft (Midnight Blizzard Attack)

Disclosed: January 2024

Microsoft said it countered a sophisticated nation-state attack by Midnight Blizzard, leveraging malicious OAuth applications to exploit user accounts. This Russian state-sponsored group utilized password spray tactics to breach accounts without multifactor authentication (MFA), revealing a targeted approach towards user identities. Significantly, the attack also illuminated the exploitation of service accounts, a strategy that indirectly compromised workload identities by manipulating these highly privileged entities within Microsoft’s ecosystem. This method highlights the attackers’ nuanced understanding of leveraging elevated privileges, which could enable unauthorized access to critical systems and data, affecting the security of both human and machine aspects of Microsoft’s digital infrastructure. In response, Microsoft conducted a thorough audit of identity privileges, tightened conditional access controls, and enhanced anomaly detection to identify and mitigate malicious OAuth applications.

7) Compromised AWS Credentials: Sumo Logic

Disclosed: November 2023

Sumo Logic, a prominent data analytics firm, experienced a security breach when an attacker infiltrated its AWS account using stolen credentials. Although the exact method of how these credentials were obtained by the attacker has not been disclosed, it typically involves tactics like phishing, exploitation of weak or reused passwords, or perhaps previous data leaks. Despite the breach, the company asserts that its systems and customer data, which remains encrypted, were not affected. As a precaution, Sumo Logic promptly secured the compromised infrastructure, rotated potentially exposed credentials, and advised customers to reset their API keys, collector credentials, and any other shared credentials. The company is conducting a thorough investigation to determine the breach’s origin and extent and has enhanced its monitoring systems and security measures. 

8) SMS-Based Phishing Attack: Retool

Disclosed: August 2023

Retool, a software development company, disclosed that a sophisticated SMS phishing attack compromised 27 of its cloud customers, all in the cryptocurrency sector. The breach started when an employee received an SMS, allegedly from Retool’s IT team, tricking them into logging into a fraudulent site and sharing multi-factor authentication (MFA) codes from Google Authenticator. A subsequent voice call, seemingly from an IT member who used detailed knowledge of Retool’s internal operations, secured an additional MFA code. This allowed attackers to sync their device with the employee’s Google and Okta accounts, gaining access to internal systems and customer accounts. Retool blames the severity of the breach on Google Authenticator’s new cloud sync feature, which they claim converted their MFA into less secure single-factor authentication. The breach led to almost $15 million in cryptocurrency losses for one customer, Fortress Trust. 

9) API Vulnerability: T-Mobile

Disclosed: January 2023

The mobile giant announced in a regulatory filing that 37 million current and former customers were impacted by a breach in which attackers accessed sensitive personal information through one of its API interfaces. The company didn’t share how the malicious actors were able to compromise the API, which are able to access data by interfacing with databases via permissions, but said the attack had been going on for roughly two months before it was identified and halted.

10) Stolen Application Keys: CircleCI

Disclosed: December 2022

In December 2022, attackers gained access to CircleCI’s GitHub and Bitbucket application keys, which are used to interact with customer repositories, as well as certain customer environment variables and API tokens. According to the company: “An unauthorized third party leveraged malware deployed to a CircleCI engineer’s laptop in order to steal a valid, 2FA-backed SSO session. This machine was compromised on Dec. 16, 2022. The malware was not detected by our antivirus software. Our investigation indicates that the malware was able to execute session cookie theft, enabling them to impersonate the targeted employee in a remote location and then escalate access to a subset of our production systems.” `As a result of the attack, intruders potentially had access to customers’ source code, environment variables, and other sensitive information contained within their CircleCI workflows. CircleCI took immediate action to address the incident, including revoking the compromised credentials, conducting a thorough investigation, and implementing additional security measures.

11) Access Key Exposure: Toyota

Disclosed: October 2022

Toyota discovered recently that some of its T-Connect app connectivity source code was left exposed for roughly five years on GitHub and contained an access key to the data server that stored customer email addresses and management numbers. Roughly 300,000 customer records were exposed, according to a notice from Toyota (translated from Japanese).

12) Hardcoded Credentials: Uber

Disclosed: September 2022

An 18-year-old attacker managed to breach Uber’s defenses by first conducting a social engineering attack on an employee, thereby overcoming the barrier of multi-factor authentication through an “MFA fatigue” strategy. This method involved bombarding the employee with multifactor requests, coupled with a deceitful impersonation of Uber’s IT support, leading the victim to inadvertently grant the attacker access. Once inside, the hacker’s activities highlighted the vulnerabilities inherent in the management of workload identities and access within modern IT ecosystems. The attacker navigated through Uber’s internal network to discover a PowerShell script on a network share. This script contained hardcoded credentials for an admin account within Uber’s Thycotic privileged access management (PAM) platform, a system designed to safeguard and manage sensitive access credentials for both human and machine identities. With these credentials, the attacker gained unprecedented access to a broad spectrum of Uber’s critical internal services and systems, including its Amazon Web Services console, VMware vSphere/ESXi virtual machines, and the Google Workspace admin dashboard.

13) Hardcoded Credentials: GitHub

When: Multiple incidents

GitHub effectively serves as ground zero for workload identity-based attacks. The enormously popular online software development platform allows users to create public repositories where they can store and share their code openly. While this encourages collaboration and knowledge sharing, it also means that sensitive information, such as access keys, passwords, and API tokens, can accidentally be included in the codebase, which is contributing to hardcoded secrets sprawl. If developers and DevOps teams are not diligent, these secrets can be exposed to the public, leading to unauthorized access and breaches. In fact, attackers are already pouncing on the opportunity with several groups emerging who specialize in writing code to scan public-facing applications for things that look like keys. GitHub, for its part, publicly acknowledged in April 2022 at least one breach. It has responded by offering dev teams free secrets scanning

Best Practices for Managing and Securing Access Between Secrets

So what options exist for organizations wanting to mitigate the risk of workload-based credential loss? Here are some recommendations:

Regularly rotate secrets 

As you likely already have in place for user credentials, the practice of frequently changing workload credentials makes life more difficult for attackers–and helps limit the likelihood that the credentials will be usable if they are accidentally exposed to the public. Regular rotation also enhances security by increasing the likelihood of discovering unnoticed compromises and extending your organization’s adherence to the principle of least privilege.

Monitor and analyze workload activity

Keep a watchful eye on your workload identities by implementing robust monitoring and logging to track and analyze activity. This includes monitoring for suspicious or anomalous behavior, analyzing access logs, and setting up alerts for potential security incidents. Regularly reviewing your these logs will also help you uncover over-privileged or inactive identities contributing to credential sprawl.

Treat DevOps and security as a shared responsibility

Security education isn’t just an end-user priority. Security must be ingrained into the culture of everything an organization does, and that includes the relationship between developer, DevOps and security teams. By emphasizing the importance of security at every stage of the development lifecycle, organizations can create a collaborative environment that prioritizes secure coding practices, threat modeling, secure configurations, and vulnerability management. This vision is being realized with the DevSecOps movement.

Throttle down your need to manage secrets

Consider deploying a solution like Aembit, which lends visibility and centralization by acting as the control plane for workload IAM. When a workload wants to access a service, the workload reaches out to Aembit, which authenticates the workload, then checks the access policy and issues a short-lived credential instead of a long-lived secret. Your application traffic doesn’t travel over Aembit’s network. Aembit also logs all access and access attempts as events for analytics, audit, and alerting. It’s a closed-loop system that takes care of acquiring and provisioning credentials, when and where they are needed. 

Want to improve how your workloads securely access the services they depend on? For more information or to try Aembit for free forever, visit aembit.io.OO 

You might also like

Discover how these different approaches can work together to protect your organization's sensitive data and ensure seamless operations.
Stolen identity data remains part of a large percentage of breaches, according to the annual landmark report.
The updated framework addresses the need to secure non-human identities. Here's how that can extend across the guidance's five key functions.