SFDC Access Tokens Root Cause and Remediation

A timeline of the incident and response. For all questions, schedule time with Crossbeam CISO Chris Castaldo.

Last Modified:

Note: All times listed are in Eastern Standard Time


October 31, 2023

  • 1030 – A Crossbeam engineer discovered short lived access tokens used for customers’ Salesforce Push integration were erroneously being logged to our Datadog infrastructure. This was discovered during the normal course of work that was related to this codepath.
  • 1059: A fix was developed and deployed to prevent any further logging of access tokens.

November 1, 2023

  • 1630 An Engineering Manager informs Crossbeam’s cybersecurity team of the incident and that the logging of tokens has been stopped. Based on Salesforce documentation, Crossbeam expected these to be short lived tokens with 2 hrs expiry times.

November 3, 2023

  • 1737 – After additional research and testing of tokens Crossbeam discovered these specific access tokens were still valid. Crossbeam engineers attempted to revoke them via Salesforce’s API, but were not able to. Crossbeam engineers raised this new information to Crossbeam’s cybersecurity team.

November 4, 2023 – With this additional information Crossbeam’s cybersecurity team raised a security incident and enacted its internal Security Incident Response Policy to ensure proper triaging, alerting and staffing to resolve the incident as soon as possible.

  • 0738: Per our internal Security Incident Response Policy Crossbeam’s cybersecurity team simultaneously alerted Crossbeam’s Security Disaster Recovery Committee which consists of key members of our executive team.
  • 0854: Crossbeam’s cybersecurity team created an incident channel in Slack with key members of the response team to include principal engineers, engineering managers, CTO, CISO and GC.
  • 1031: A Zoom call is stood up to assist in communicating key findings real time.
  • 1040: Crossbeam engineers discover an issue with Salesforce API and we are unable to revoke tokens as described in their documentation.
  • 1149: Crossbeam submits a ticket to Salesforce with the highest severity allowed, SEV2, to request support on the revoke token API endpoint that is not functioning as described.


November 6, 2023

  • 0820 – Engineers continue to test our ability to revoke tokens and find the Salesforce API is now responsive and operating as expected when attempting to revoke an access token.
  • 0840: Crossbeam receives a response from Salesforce that our ticket is being routed to a technician for triaging.
  • 1414: Crossbeam engineers ran a script to revoke all Salesforce Push API tokens that were stored in Datadog.
  • 1643: Crossbeam engineers wrote a script to search our S3 archive which stored Datadog logs for long term storage. This bucket exists within Crossbeams AWS account.
  • 1700: Crossbeam engineers tested the script and calculated the time to finish would be just over 24 hours.


November 7, 2023

  • 0930 – S3 search script still running.


November 8, 2023

  • 0923 – The S3 search script completed.
  • 1504: The revocation script completed based on the S3 search results. No active tokens were found
  • 1630: Data remaining in Datadog has been removed from all indexes and searches. This data will age off in 15 days.

November 9, 2023

  • 0930 Delete script was started on S3 data.
  • 1130: Delete script on S3 data completed.
  • 1200: Event closed.

Remediation

Crossbeam has:

  • Implemented a code change fix to prevent access tokens from being logged to Datadog.
  • Refreshed all impacted access tokens.
  • Destroyed the logs which stored the access tokens.
  • Scheduled training with our engineering team on logging best practices.
  • Scheduled full code review of all logging infrastructure.