Hi All,

On the morning of 03/25/2020, our teams identified slow sync completion times for some of our customers. We saw customer tickets coming in that morning complaining of slow syncs as well.

Key issues surfaced during the initial analysis were:

• A large number of connections were running long.
• The majority of the connections that faced this issue saw too many concurrent queries for a single user on the destination warehouse.
• Multiple select statements in the destination schema were taking longer than normal.

Based on our root cause analysis, we uncovered the following reasons for the issue:

• Fivetran had reworked a logic on 03/23/2020 that would add foreign keys to the schema in the destination as needed.
• Based on the logic, select queries are run on the customer’s destination warehouse (information schema) to identify the need for adding foreign keys post sync completion.
• The above logic had a negative impact when queries started becoming a bottleneck and absorbing resources on both Snowflake and Fivetran infrastructure

Steps taken to resolve the issue:

• Reverted the code that was causing the issue back on 03/25/2020 16:05 UTC
• Paused some of the connectors and enabled them to ensure that code change took effect immediately
• Monitored a sample set of the connectors to completion.

We understand and apologize for the disruption caused and to ensure this does not happen again, we have taken certain mitigation steps to avoid the issue in future:

Steps to prevent/mitigate these risks in the future:

• Use wider test cases along with better load testing capabilities
• Test code changes thoroughly within the staging environment
• Mark these code changes to be released via Feature Flag with a slow rollout process

We appreciate your patience and help through the issue.

Regards,

Fivetran Team