Fivetran: Connectors and Transformations Running in the Region ‘GCP-US-East4’ Are Delayed in Scheduling
Incident
Report for Fivetran
Resolved
We have resolved this incident.
Monitoring
The fix has been successfully deployed. We are monitoring connectors that previously experienced scheduling delays to confirm no further issues.
Update
We are continuing to work on a fix for this issue.
Update
Service Status Update: GCP-Europe-West3 and GCP-Australia-Southeast1 Regions
- Delays in scheduling were caused in the regions ’GCP-Europe-West3’ and ‘GCP-Australia-Southeast1’ due to an unsuccessful fix attempt. This problematic fix has now been reverted, and connectors in these regions are scheduled without issue.
Service Status Update: GCP-US-East4 Region
- We're currently addressing issues in the GCP-US-East4 region. To expedite resolution, we're deploying an additional fix. This update aims to improve the consumption speed of the Pub/Sub queue, which holds events and commands for the Integrated Scheduler.
- Delays in scheduling were caused in the regions ’GCP-Europe-West3’ and ‘GCP-Australia-Southeast1’ due to an unsuccessful fix attempt. This problematic fix has now been reverted, and connectors in these regions are scheduled without issue.
Service Status Update: GCP-US-East4 Region
- We're currently addressing issues in the GCP-US-East4 region. To expedite resolution, we're deploying an additional fix. This update aims to improve the consumption speed of the Pub/Sub queue, which holds events and commands for the Integrated Scheduler.
Update
Two additional regions have been affected by scheduling delays.
Connectors and Transformations have been affected in the regions ‘GCP-US-East4′, ’GCP_Europe_West3’, and ‘GCP_Australia_Southeast1’.
We have deployed a fix to rollback to a previously operational version. So far, we have reduced delays in the ’GCP_EUROPE_WEST3’ and ‘GCP_AUSTRALIA_SOUTHEAST1’ regions.
We are continuing to investigate to resolve any ongoing delays and will provide an update within 2 hours.
Connectors and Transformations have been affected in the regions ‘GCP-US-East4′, ’GCP_Europe_West3’, and ‘GCP_Australia_Southeast1’.
We have deployed a fix to rollback to a previously operational version. So far, we have reduced delays in the ’GCP_EUROPE_WEST3’ and ‘GCP_AUSTRALIA_SOUTHEAST1’ regions.
We are continuing to investigate to resolve any ongoing delays and will provide an update within 2 hours.
Identified
We are currently seeing an additional large increase in Connector And Transformations delayed in scheduling. We are investigating this and will provide an update within the next 2 hours.
Update
We are still seeing some spikes in job scheduling, which can result in short scheduling delays. To help prevent this, we are adding an additional fix to increase queue buffering to allow for the handling of spikes in scheduled jobs without causing delays.
We will continue to monitor once this fix has been deployed.
We will continue to monitor once this fix has been deployed.
Update
We are continuing to monitor for potential delays in scheduling. We are so far only seeing minor delays, which are quickly recovered from.
We are working on further infrastructure improvements, including increases in thread pool size and updates to memory management. This is expected to prevent further minor delays in scheduling.
We are working on further infrastructure improvements, including increases in thread pool size and updates to memory management. This is expected to prevent further minor delays in scheduling.
Update
We are continuing to monitor the cluster status to ensure Connectors and Transformations are scheduled at a normal rate.
Update
There are no delayed connectors and transformations currently, but from time to time some connectors and transformations might experience short delays in their start, we are monitoring the cluster status to adjust the service configuration.
Update
We are continuing to monitor to ensure the success of the fix.
Monitoring
We have deployed a fix to help prevent this issue from reoccurring.
The issue was caused by an increased load on the scheduling service and its multithreading internals. The high load produced too many threads to process requests. We have tuned the thread pool to handle this case.
Connectors and Transformations are now scheduling without any issues. We are continuing to monitor to ensure success of the fix.
The issue was caused by an increased load on the scheduling service and its multithreading internals. The high load produced too many threads to process requests. We have tuned the thread pool to handle this case.
Connectors and Transformations are now scheduling without any issues. We are continuing to monitor to ensure success of the fix.
Update
Scheduling delays have now been reduced. Connectors and Transformations are being scheduled again at a normal rate.
We are applying a fix to limit the thread pool size to help prevent this spike in schedule delays from occurring again. We will continue to monitor and we will apply further fixes as needed.
We are applying a fix to limit the thread pool size to help prevent this spike in schedule delays from occurring again. We will continue to monitor and we will apply further fixes as needed.
Update
We have identified a memory issue within our scheduling infrastructure which is causing delayed scheduling of Connectors and Transformations running in the region GCP-US-East4.
Identified
The issue has been identified and we are working to resolve it.
This incident affected: Systems (Transformations).