This incident has been resolved. We set the JDBC socketTimeout to 10 minutes, and this is successfully handling the stuck sync issue. Syncs that are stuck now fail in 10 minutes instead of being stuck for 6 hours.
We are in the process of enabling this workaround for all Databricks accounts, and the enablement will complete by the 14th of June.
Posted Jun 13, 2023 - 16:40 UTC
We have created a workaround for this issue to set JDBC socketTimeout to 10 minutes. With this, the stuck syncs which were taking 6 hours to fail, would fail in 10 mins. After the sync fails in 10 mins, the next sync will start after the connector's sync frequency time. In case the frequency is more than 1 hour, the next sync will start after an hour. We are working on deploying this to affected connectors in batches based on the impact.
Posted Jun 07, 2023 - 07:21 UTC
We shared the trace log for one of the stuck connectors with Databricks Support for further investigation. We will continue to provide updates as more information becomes available.
Posted Jun 01, 2023 - 16:52 UTC
We have added tracing in the logs and now waiting for any connector to get stuck to get the trace logs for that sync. This information will be shared with Databricks Support for further investigation/action.
Posted May 31, 2023 - 07:29 UTC
The previously deployed fix was unsuccessful. Our engineering team is currently investigating further with Databricks to resolve the issue.
Posted May 29, 2023 - 09:32 UTC
A fix has been deployed. We are currently monitoring the connectors.
Posted May 26, 2023 - 06:15 UTC
We are actively working with the Databricks team to find a solution. We will continue to provide updates as more information becomes available.
Posted May 25, 2023 - 17:28 UTC
We had a call with the Databricks team and provided the code snippet we use to connect to the warehouse. Databricks Engineering team are validating further on the issue where the query is successful in the SQL warehouse but it is getting stuck on the client (Databricks) side. We will continue to provide updates as more information becomes available.
Posted May 25, 2023 - 05:29 UTC
We are continuing to work with the Databrick engineering team. We will continue to provide updates as more information becomes available.
Posted May 23, 2023 - 19:30 UTC
The issue seems to be on the Databricks end and we are working with them to find more details.
Posted May 23, 2023 - 07:32 UTC
The issue has been identified and we are working to resolve it.