Some connectors are not syncing
Incident Report for Fivetran
Postmortem

Incident Timeline

[2022-08-18 00:49 UTC]: Fivetran internal alert was generated due to higher than normal connector failures.

[2022-08-18 03:04 UTC]: The problem code was found and the fix was deployed. Monitoring saw that failed connectors started to decrease.

[2022-08-18 03:09 UTC]: The incident was marked as resolved.

Root Cause

Sync jobs were not starting due to a bug we found in an internal webhook service. This service received values from our production database and an invalid JSON object lead to invalid database records.

Future Improvements

Existing alerting caught the problem quickly. Improved internal documentation will resolve the problem faster.

Posted Sep 07, 2022 - 19:37 UTC

Resolved
We have resolved this incident. Connectors are syncing.
Posted Aug 18, 2022 - 04:02 UTC
Monitoring
We identified that the issue was caused by a recent change. After the change, syncs failed to start between 12:46 UTC to 03:07 UTC.

A fix has been implemented. Syncs should begin to recover, we're monitoring the results.
Posted Aug 18, 2022 - 03:19 UTC
Update
Due to this outage, we're observing the following.
- Sync History Chart is not displaying the current syncs.
- Syncs are failing with the error "No value present".
- Manual sync triggers are not working.
Posted Aug 18, 2022 - 03:08 UTC
Identified
The issue has been identified and we are working to resolve it.
Posted Aug 18, 2022 - 02:33 UTC