Postmortem

2024/11/13 Failed Tasks Not Retried / Pages Not Initialized

On October 31st, 2024, an issue introduced in our service caused a small percentage (< 0.05%) of sessions to not process a portion of data within them. This data was effectively “stuck” in our system. The issue lasted until November 13th, when it was fixed for all future sessions.

Customer Impact

The issue caused the affected sessions to be missing data, causing gaps/disruptions when loading playback, webhooks not firing, and events not being exported via the Data Direct feature.

Affected sessions captured after Nov 6th were identified and frozen so that we could re-process the data. Starting the week of Dec 9th, we ran all the frozen data back through our pipeline and the majority of it was recovered successfully for sessions with enough data to process.

Affected sessions before Nov 6th will remain in this incomplete state, as the data that had been unprocessed got deleted as it was misidentified as something that our system needed to clean up and remove, due to being stuck for some time.

We apologize for any impact this may have had on your business, and are working to better understand how to identify issues of this nature more quickly, and resolve them with even less disruption.

Root Cause

Due to a code change that altered the behavior of an uncommon failure condition, part of our system marked the data in question to be “completed” when it had actually failed. Normally in this case, it automatically retries this operation and the data gets processed, but did not do so during the incident period.

Due to this being a rare circumstance, the issue was not immediately noticed by our typical monitoring systems.

Resolution

After noticing some odd system behavior and finding that we had unprocessed data that should have been successful, we quickly traced it back to the code change, and deployed a fix immediately that would ensure no future data gets into this state.

Process Changes and Prevention

Actions Taken:

• Fix the issue for future sessions.
• Freeze and reprocess all affected sessions still in our system.
• Added more automated testing around this failure case.
• Adjusted our monitoring and alerting to better identify “stuck” data that’s legitimate, even when it’s a very small percentage of overall processing.

‌

Ongoing Improvements:

• Applying lessons learned to other parts of the system, including better detection systems and testing for any other components that process data in a similar way.
• Better automated observability triaging so that rare-circumstance issues don’t blend in with general service metric variability. 

‌

We deeply regret this incident and invite any FullStory customer who was materially affected to contact support@fullstory.com. We stand by ready to fully address all of your concerns.