Fivetran Status

[Follow Up] Stripe Connector Data Integrity Issues

Monitoring - The fix has been deployed and we are monitoring the results.
Oct 23, 09:06 UTC

Update - We are in the process of deploying the fix and we will follow up to confirm once it has been deployed.

Following the deployment, we will be contacting all affected customers regarding re-syncs to backfill the missing data.

Thank you for your continued patience.
Oct 22, 14:41 UTC

Identified - We have identified what we believe to be the root cause of the issue and we are currently working on a potential fix.

We will update again once we have confirmation that it will work, and once confirmed, we will be deploying it as soon as possible.
Oct 21, 11:11 UTC

Update - After working with the Stripe Support Team, we are gathering the information they requested. We expect to provide their team with the requested resources by tomorrow.
Oct 20, 20:20 UTC

Monitoring - Fivetran Engineering has been in contact with Stripe Support and we are still working together to identify the root cause of these data integrity issues.

We will continue to provide updates as soon as they are available from Stripe Support.
Oct 19, 19:59 UTC

Uptime over the past 90 days. View historical uptime.

Connectors Operational

90 days ago

99.38 % uptime

Today

Logs Operational

90 days ago

99.89 % uptime

Today

Files Operational

90 days ago

99.76 % uptime

Today

Sales Operational

90 days ago

99.07 % uptime

Today

BI Tools (Managed BigQuery) Operational

90 days ago

99.89 % uptime

Today

Events Operational

90 days ago

99.76 % uptime

Today

Finance Operational

90 days ago

97.1 % uptime

Today

Support Operational

90 days ago

99.84 % uptime

Today

Databases Operational

90 days ago

99.53 % uptime

Today

Functions Operational

90 days ago

99.89 % uptime

Today

Marketing Operational

90 days ago

97.72 % uptime

Today

Engineering Operational

90 days ago

99.89 % uptime

Today

Productivity Operational

90 days ago

99.89 % uptime

Today

Human Resources Operational

90 days ago

99.7 % uptime

Today

Destinations Operational

90 days ago

100.0 % uptime

Today

Dashboard UI Operational

90 days ago

99.81 % uptime

Today

Transformations Operational

90 days ago

99.95 % uptime

Today

Fivetran API Operational

90 days ago

99.96 % uptime

Today

Replication Servers Operational

90 days ago

100.0 % uptime

Today

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Past Incidents

Oct 23, 2021

Workday connectors failing with HTTP 500/503 Internal Server Error

Resolved - We have resolved this incident.
Oct 23, 10:36 UTC

Update - Some Workday connectors are failing due to an Outage on the Workday side, and we are waiting for their services to be restored.

Once this happens, connectors will resume working as normal.
Oct 23, 09:15 UTC

Monitoring - A fix has been implemented and we are monitoring the results.
Oct 23, 08:51 UTC

Identified - The issue has been identified and we are working to resolve it.
Oct 23, 08:51 UTC

Oct 22, 2021

MySQL connectors failing with java.lang.AssertionError

Resolved - All connectors have now recovered.
Oct 22, 22:52 UTC

Monitoring - The fix is deployed and we're seeing connectors recover for syncs starting later than 12:30pm PDT.

We will continue to monitor for any unexpected failures with this error.
Oct 22, 19:41 UTC

Update - The fix has passed the QA process and we are starting the deploy to production. Current ETA for having this fix deployed is 1:30 PM PDT.
Oct 22, 17:18 UTC

Update - The fix is developed and is going through our QA process now. Once all tests pass we'll begin deploying to production.
Oct 22, 16:25 UTC

Update - We've identified that the source of the issue is related to the parsing of binlog events.

We are currently working on a fix to address it.
Oct 22, 12:03 UTC

Identified - The issue has been identified and we are working to resolve it.
Oct 22, 10:57 UTC

Oct 21, 2021

Oct 20, 2021

Oct 19, 2021

Postgres syncs fail with "Unable to initialize table buffers"

Postmortem - Read details
Oct 19, 02:09 UTC

Resolved - All affected connectors have returned to normal.
Oct 19, 02:08 UTC

Monitoring - Engineering has the workaround in place and we are seeing the affected connectors begin to recover.

You can click the sync now button in the upper right corner of the connector dashboard to manually start an incremental sync instead of waiting for the next automatically scheduled sync.
Oct 18, 21:43 UTC

Update - We expect the workaround to be fully deployed in the next two hours.
Oct 18, 19:19 UTC

Update - We have deployed a workaround and are monitoring the results.
Oct 18, 18:40 UTC

Identified - The issue has been identified and we are working to resolve it.
Oct 18, 17:45 UTC

Oct 18, 2021

Email connectors not syncing any data from the collected attachments

Resolved - We have resolved this incident.
Oct 18, 19:27 UTC

Update - We restored the service and are currently working on ingesting the attachments that were not previously ingested.
Oct 18, 16:39 UTC

Monitoring - A fix has been implemented and we are monitoring the results.
Oct 18, 16:30 UTC

Update - We are continuing to work on a fix for this issue.
Oct 18, 11:34 UTC

Update - The files sent after 3:30 PM UTC on 15 Oct, have been collected by the connector but the data has not been synced to the warehouse.

We are working on a code fix to restore the service as well as ingest the previously sent attachments.
Oct 18, 07:56 UTC

Identified - The issue has been identified and we are working to resolve it.
Oct 18, 07:54 UTC

Oct 17, 2021

No incidents reported.

Oct 16, 2021

CSV Uploader is failing with error “Error uploading file”

Resolved - We have resolved this incident.
Oct 16, 00:18 UTC

Monitoring - A fix has been implemented and we are monitoring the results.
Oct 15, 21:21 UTC

Identified - The issue has been identified and we are working to resolve it.
Oct 15, 18:36 UTC

Oct 15, 2021

Tiktok Ads connector failing with the error “System maintenance”

Resolved - We have resolved this incident.
Oct 15, 02:36 UTC

Monitoring - A fix has been implemented and we are monitoring the results.
Oct 15, 02:36 UTC

Update - We are continuing to see TikTok Ads connectors failing with 'System maintenance' error messages.

This has been identified as a limitation on the time ranges that Fivetran can query from the TikTok API. We are working with our TikTok contacts to restore access and reduce the size of queries.
Oct 14, 20:50 UTC

Update - We are continuing to see TikTok Ads connectors failing with 'System maintenance.' error messages when attempting to retrieve data from the TikTok Ads API. We received a report from TikTok that our API calls are failing when requesting larger data sets, and Fivetran has been investigating the size of the data that we request.

We have an open ticket #346090 with Tiktok Support, and this is currently being investigated by the TikTok team. We are requesting your help to escalate this issue with Tiktok to gain traction towards a quicker resolution.
Oct 14, 16:42 UTC

Update - We are continuing to see TikTok Ads connectors failing with 'System maintenance.' error messages when attempting to retrieve data from the TikTok Ads API.

The initial response from Tiktok states that API call is failing because of a big data load while requesting more than 1-day data. Fivetran team investigated and found the API is also failing for 1-day report data.

We have an open ticket #346090 with Tiktok Support and it is currently being investigated by the Tiktok team.

We are requesting your help to escalate this issue with Tiktok to gain traction towards a quicker resolution.
Oct 14, 09:53 UTC

Update - We are continuing to see TikTok Ads connectors failing with "java.lang.RuntimeException: 'System maintenance.'" error messages when attempting to retrieve data from the TikTok Ads API.

We are continuing to investigate this issue, and we have reached out to TikTok for more information on this issue. We will have more information as soon as we have a response.
Oct 13, 23:35 UTC

Update - TikTok Ads connectors are failing with a "java.lang.RuntimeException: 'System maintenance.'" error message after attempting to retrieve data from the TikTok Ads API.

We are continuing to investigate this issue and we have reached out to TikTok for more information on this system maintenance.
Oct 13, 20:29 UTC

Update - Connectors are still failing while attempting to access the TikTok Ads endpoint with an error message indicating system maintenance on the TikTok side.

We're still investigating this issue, and we've reached out to TikTok for more information.
Oct 13, 18:34 UTC

Update - Connectors are failing while accessing the API endpoint https://business-api.tiktok.com/open_api/v1.2/reports/integrated/get/.

We are still investigating the issue and reached out to the Tiktok support to gain some insights into the error.
Oct 13, 10:27 UTC

Identified - The issue has been identified and we are working to resolve it.
Oct 13, 06:57 UTC

Oct 14, 2021

Stripe Connector Data Integrity Issues due to a source side bug with the Stripe API

Resolved - This issue is still ongoing and Stripe Support has assured us that this has been prioritized. However, it is still going to take some time to resolve and an ETA is unfortunately unknown at this time.

We are requesting your help to escalate this issue with Stripe Support to gain traction towards a quicker resolution.

As there is nothing we can do from our side to work around this issue, we will be closing this on our Status Page. We will follow up with a communication in its stead.
Oct 14, 16:01 UTC

Update - The Stripe team has prioritzed this issue on their end and we are continuing to work with them on getting this resolved ASAP.
Oct 11, 09:29 UTC

Update - For the Stripe API endpoints, updates to records are received from the events endpoint. Currently we are not receiving all the updates from this endpoint and hence, customers will notice some data discrepancies and missing data in their destination warehouse.

We have an open ticket #19683007 with Stripe Support and it is currently being investigated by Stripe’s Engineering Team.

We are requesting your help to escalate this issue with Stripe Support to gain traction towards a quicker resolution.
Oct 6, 19:30 UTC

Identified - The issue has been identified and we are working to resolve it.
Oct 6, 19:12 UTC

Oct 13, 2021

WPS Service is down, affecting all connectors that use Webhooks

Postmortem - Read details
Oct 13, 19:49 UTC

Resolved - The root cause of this was an incident within Google Cloud which caused the API calls we use to read from the webhook storage buckets to suddenly return empty responses:

https://status.cloud.google.com/incidents/rXqQALuw55aCKd2QHfM3

We do not expect any data loss as all events exist within our storage buckets, and the failure was in the API calls to read from those buckets. Events which came in will update upon the next successful webhook sync.
Oct 13, 19:46 UTC

Monitoring - A fix has been implemented and we are monitoring the results.
Oct 13, 17:24 UTC

Update - The fix for this is in the deployment process and will be in production within the next two hours.
Oct 13, 15:46 UTC

Update - We have a fix being developed which is currently going through the testing and review stage.

Here is a list of the affected connectors currently:

- AppsFlyer
- Branch
- Eloqua
- GitHub
- Greenhouse
- Helpscout
- HubSpot
- Intercom
- Iterable
- Jira
- Mandrill
- Pipedrive
- Recharge
- Segment
- Sendgrid
- Shopify
- Snowplow
- Webhook
Oct 13, 14:56 UTC

Update - While syncing our Webhook based connectors, we use Google Cloud metrics to fetch the oldest unacknowledged message to allow us sync the latest Webhook data. However, the Google Cloud Monitoring API is returning an empty response.

We are able to capture events properly, so there is no data loss, but we cannot sync that data.

We are currently investigating the root cause for the empty response.
Oct 13, 14:29 UTC

Identified - The issue has been identified and we are working to resolve it.
Oct 13, 14:24 UTC

Some Apple Search Ads connectors are failing with "Unauthorized Connection " error

Resolved - This incident has been resolved.
Oct 13, 17:16 UTC

Monitoring - A fix has been implemented and we are monitoring the results.
Oct 13, 17:09 UTC

Update - Our engineers are working on a fix which will revert connectors using legacy (OAuth) authentication to using Apple Ads’s API version 3, rather version 4.
However, we highly recommend to our customers that they change their authentication method to OAuth2 to benefit from using Apple Ads's API version 4, which provides the most reliable data.
We expect the fix to current issue to be deployed later today.
Oct 13, 13:56 UTC

Update - All the connectors using the legacy auth are currently failing. We are continuing to investigate the issue.
Oct 13, 08:44 UTC

Identified - The issue has been identified and we are working to resolve it.
Oct 13, 08:06 UTC

Oct 12, 2021

No incidents reported.

Oct 11, 2021

Kubernetes Cluster Maintenance in GCP US (N. Virginia)

Completed - The scheduled maintenance has been completed.
Oct 11, 06:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 11, 03:00 UTC

Scheduled - We plan to migrate workload in GCP US (N. Virginia) region to new Kubernetes clusters 3-6AM UTC on 10/11/2021. The maintenance may affect customers with any connector / transformation in GCP US (N. Virginia) region. All other clouds and regions are not expected to be affected.

During the maintenance window, transformation, dbt and setup tests in GCP US (N. Virginia) region are expected to experience up to 5 min downtime. We expect zero downtime on all other Fivetran services, although they may or may not experience delayed scheduling or slow processing.
Oct 1, 05:45 UTC

Oct 10, 2021

No incidents reported.

Oct 9, 2021

No incidents reported.

Subscribe to Incident

Related