Connector syncs delayed for Databricks destination
Incident
Report for Fivetran
Resolved
This incident has been resolved. We set the JDBC socketTimeout to 10 minutes, and this is successfully handling the stuck sync issue. Syncs that are stuck now fail in 10 minutes instead of being stuck for 6 hours.
We are in the process of enabling this workaround for all Databricks accounts, and the enablement will complete by the 14th of June.
We are in the process of enabling this workaround for all Databricks accounts, and the enablement will complete by the 14th of June.
Monitoring
We have created a workaround for this issue to set JDBC socketTimeout to 10 minutes. With this, the stuck syncs which were taking 6 hours to fail, would fail in 10 mins. After the sync fails in 10 mins, the next sync will start after the connector's sync frequency time. In case the frequency is more than 1 hour, the next sync will start after an hour. We are working on deploying this to affected connectors in batches based on the impact.
Update
We shared the trace log for one of the stuck connectors with Databricks Support for further investigation. We will continue to provide updates as more information becomes available.
Update
We have added tracing in the logs and now waiting for any connector to get stuck to get the trace logs for that sync. This information will be shared with Databricks Support for further investigation/action.
Identified
The previously deployed fix was unsuccessful. Our engineering team is currently investigating further with Databricks to resolve the issue.
Monitoring
A fix has been deployed. We are currently monitoring the connectors.
Update
We are actively working with the Databricks team to find a solution. We will continue to provide updates as more information becomes available.
Update
We had a call with the Databricks team and provided the code snippet we use to connect to the warehouse. Databricks Engineering team are validating further on the issue where the query is successful in the SQL warehouse but it is getting stuck on the client (Databricks) side. We will continue to provide updates as more information becomes available.
Update
We are continuing to work with the Databrick engineering team. We will continue to provide updates as more information becomes available.
Update
The issue seems to be on the Databricks end and we are working with them to find more details.
Identified
The issue has been identified and we are working to resolve it.
This incident affected: Systems (Destinations).