Share This Article
On 2023-07-12, we alerted customers to a security incident that occurred starting on 2023-06-20. Now that our investigation has concluded, we want to share some additional information around what happened, what we’ve learned, and what our plans are to continuously improve our security posture for the future.
We would like to thank our customers for their patience and understanding for the mandatory key rotation on 2023-07-05. We want to also thank our customers and our community for your patience while we investigated and remediated this incident. We aim for full transparency and disclosure, while maintaining the integrity of the investigation.
What happened?
2023-06-20 a sophisticated North Korean threat actor successfully spear-phished a JumpCloud software engineer, causing them to download malicious code to their JumpCloud-issued device, which gave the threat actor developer-level access to JumpCloud environments.
2023-06-22 the threat actor used developer-level access gained on the engineer’s endpoint to pivot to other JumpCloud systems. From there, they were able to launch workloads to run at a later date in our container orchestration system.
2023-06-23 02:21 JumpCloud security tools alerted on anomalous activity taking place related to the compromised employee account. System access was revoked and known affected credentials were rotated. A number of mitigating actions that are detailed below were initiated at this time.
2023-06-27 15:13 JumpCloud Security noticed that developer access was used to run a workload that activated in our container orchestration system. We did not see evidence of customer impact at that time.
Containment, eradication, and remediation efforts continued at this time. Credentials were rotated, infrastructure was rebuilt, code deployment was frozen, and a number of other actions were taken to further secure our network and perimeter. Additionally, our prepared Incident Response (IR) plan was activated. Our IR partner was engaged to analyze all systems and logs for potential activity. As part of our IR plan, we contacted and engaged law enforcement in our investigation. Forensics and investigation work continued to ensure the full scope and impact of the malicious activity was understood.
2023-07-04 JumpCloud identified and rebuilt the last impacted system. No further indicators of compromise have presented themselves on JumpCloud systems since this date.
2023-07-05 JumpCloud discovered an anomaly in our database records ultimately identifying the intent and impact of the attack. We discovered database injection that occurred on the 27th to instruct target devices to download malware. This occurred on fewer than 10 devices total across fewer than 5 organizations. We immediately contacted the organizations to notify them of the impact and ensure that there was no further exposure to them. We also took the proactive measure to force rotate all API keys once we had evidence of customer impact.
Our audit of the entire database through objective analysis of this anomaly leads us to have extremely high confidence that we know the exact impact of the incident and have a comprehensive list of impacted devices.
Further investigation, containment, and remediation actions continued to secure our environment. These actions are detailed below.
2023-07-12 JumpCloud published a public statement advising the public of a security incident that took place and was the reason for the mandatory API key rotation. JumpCloud Security detected the compromise and responded accordingly to investigate, contain, and remediate the attack.
How do we know this attack vector is closed?
Indicators of compromise were thoroughly investigated, and a variety of containment and remediation strategies have been employed to secure the JumpCloud environment.
Access Revocation and Granular Restoration
Upon discovery of compromise and an active phishing campaign of engineering and development employees, access to the JumpCloud application infrastructure was revoked for a large portion of users and roles. This was implemented to prevent further exposure by other potentially compromised employee endpoints until full scoping of incident impact could be completed.
All IAM permissions were reviewed, rearchitected, and restored based on necessity and job function relevant to various roles and users. The hardening of users and roles will limit the potential impact of compromised accounts and endpoints. Elevated access now requires manual authorization by multiple parties to prevent unwanted privilege escalation. Robust monitoring and alerting provides visibility to review and audit elevated access activity.
API Key Rotation
Upon discovery of anomalous activity, action was taken to rotate all JumpCloud customer API keys. Although no exposure was found at that time, we acted to secure customer environments from the possibility of compromised API keys. A notification was sent to customers informing them of the forced rotation with instructions to resume normal operation in their environment. An article was also published on the JumpCloud support site with the same contents as the email sent to customers.
Infrastructure Destruction and Rebuilding
During the investigation, all infrastructure affected by the threat actor was identified and completely rebuilt from scratch to further ensure that all persistence mechanisms of the threat actor were removed. Every credential and key in the JumpCloud environment was rotated to ensure no lingering access existed for the threat actor to take advantage of.
Source Code and Binary Validation
To prevent any potentially compromised source code from being deployed into the production environment, a deployment freeze was implemented early in the incident. We verified that no source code or binary releases were compromised in this incident.
User Credential Rotation and Endpoint Verification
All JumpCloud internal users and administrators were forced to rotate their credentials. All user endpoints were audited to verify that security tools are present and functioning correctly. No evidence of further employee compromise was found.
Enhanced Monitoring
A number of measures were taken to expand monitoring capabilities. Indicators of compromise were added to security tools as they became available. Routine monitoring and system checks were expanded to include new indicators in order to provide visibility into new attempts by the threat actor. Monitoring of JumpCloud employee user access and API key activity improved visibility to ensure all activity is expected and appropriate for business processes.
Third-party Incident Response Services and Law Enforcement
JumpCloud engaged third-party incident response services to assist in the investigation, containment, and remediation of the incident. Forensics was conducted in all affected environments, and detailed reports were provided to contain and remediate the impacted systems. Contact with the appropriate law enforcement agencies has been established.
Communication and support for customers
The investigation revealed that the threat actor injected agent commands to run on fewer than 10 devices across fewer than 5 total organizations. At that point JumpCloud promptly contacted the affected customers to inform them of the attack and offer assistance.
At appropriate milestones, JumpCloud provided communications to customers and the general public. This was accomplished through customer email notifications, JumpCloud support site articles, as well as interactions between customers and JumpCloud support staff. Internal communication channels were created to bridge the gap between customer inquiries and various members of security and engineering teams.
How do I know if I was impacted?
Fewer than 5 organizations and fewer than 10 total devices were successfully targeted by the threat actor. JumpCloud made contact with all affected customers prior to public announcement. If your organization was not contacted and informed of impact, it was not impacted by this incident.
What can I do?
A list of indicators of compromise was published on the JumpCloud support site to enable customers to conduct forensics and inspect logs between June 20 and July 5 for any suspicious activity.
With any security incident involving a cloud provider, the best practice is to rotate all static credentials you have provided them. If you have not already done so when earlier security blog postings were published, we still recommend as a good practice that you rotate all static credentials you have with JumpCloud including SAML certificates, user passwords, and all secrets used for integrations.
We also recommend that all customers review the Admin Guide to Supporting Work From Home article found on the JumpCloud Knowledge Base to harden their own environments.
What we learned from this incident and the path ahead
IAM Roles and Permissions
Given the attack vector, we are rearchitecting IAM for granular permissions to secure the environment by making it more difficult to gain inappropriate elevated access. While we have RBAC policies in place, we are continuing to expand on those policies to reduce access by default.
Developers will need to elevate their permissions to perform some tasks. A number of safeguards outlined above (including multi-party authorization) were added to more safely allow these tasks to be performed.
All access to data that could affect customer devices or security directly or indirectly is now multi-party authorized. Continued development effort is in place to ensure such access is tied to a request initiated by the customer.
Phishing Awareness
Phishing attacks are a threat to any organization, and attacks continue to get more sophisticated with time. As a response to this incident, we are taking increased measures to enhance our communications and training to improve phishing awareness. We are continually improving our monitoring and controls to detect and prevent phishing attacks.
Infrastructure Isolation and Segregation
We identified improvements to our application environment that will further isolate production systems from others. This will allow for more granular access permissions, as well as reduced chances of unwanted access and connectivity between functionally separate areas.
Keys and Secrets
Developer access to static credentials needed to perform job functions was a factor in this incident. Ongoing efforts to replace these with ephemeral credentials is in place.
Visibility
Enhanced logging and monitoring have been put into place and more will be added in the future.
Final Thoughts
In closing, we want to express our sincere commitment to maintaining the utmost security and privacy for our valued customers. We understand the concerns that arise from incidents like these and want to assure you that we have taken swift and decisive actions to keep our customers safe. Our dedicated teams are working diligently to strengthen our systems, enhance our safeguards, and prevent any recurrence. We greatly appreciate the trust you’ve placed in us and remain steadfast in our dedication to providing a secure environment for all.
