3rd Party: Multiple Customer.io connections are failing with 5xx error
Incident
Report for Fivetran
Resolved
This incident has been resolved. We have observed that instance rates are returning to normal levels and affected connectors are syncing successfully.
Incident Summary
Description:
We identified an issue for Customer.io, which resulted in syncs failing with a 502 HTTP error.
Timeline:
This issue began on 2026-06-13 20:26 UTC and was resolved on 2026-06-14 14:52 UTC.
Cause:
The issue was caused by intermittent upstream API failures on the Customer.io side. The root-cause behaviour fluctuated throughout the incident, shifting between generic server errors across multiple endpoints.
Resolution:
Fivetran implemented a skip-on-failure mechanism to bypass 502 HTTP errors and allow syncs to proceed. A permanent root-cause fix will be deployed by Customer.io.
Incident Summary
Description:
We identified an issue for Customer.io, which resulted in syncs failing with a 502 HTTP error.
Timeline:
This issue began on 2026-06-13 20:26 UTC and was resolved on 2026-06-14 14:52 UTC.
Cause:
The issue was caused by intermittent upstream API failures on the Customer.io side. The root-cause behaviour fluctuated throughout the incident, shifting between generic server errors across multiple endpoints.
Resolution:
Fivetran implemented a skip-on-failure mechanism to bypass 502 HTTP errors and allow syncs to proceed. A permanent root-cause fix will be deployed by Customer.io.
Update
We observed a shift in the issue behavior around 12:00 UTC for the Customer export api. While initial failures occurred during job creation, the connector is now failing during the report download phase.
- We have implemented a skip mechanism to mitigate these 502 errors on the affected endpoints, and are continuing to closely monitor all active syncs.
- Additionally, we reported the ongoing behavior to Customer.io, as their initial fix did not resolve the issue. We are currently awaiting their response.
Next Expected Update: < 3 hours >
- We have implemented a skip mechanism to mitigate these 502 errors on the affected endpoints, and are continuing to closely monitor all active syncs.
- Additionally, we reported the ongoing behavior to Customer.io, as their initial fix did not resolve the issue. We are currently awaiting their response.
Next Expected Update: < 3 hours >
Update
We noticed that the connections were failing again with "502 Server Error". We've raised a hotfix, and the failures are gradually reducing. We're monitoring the affected connections.
Next Expected Update: < 3 hours >
Next Expected Update: < 3 hours >
Update
We are continuing to monitor all affected connectors after the recent fix deployment.
Next Expected Update:3 hours
Next Expected Update:3 hours
Monitoring
A hotfix has been deployed to skip the failing endpoints until the source resolves the issue. We will continue monitoring the connections while we wait for a response from the source team.
Next Expected Update: 3 hours
Next Expected Update: 3 hours
Update
We are preparing a fix to skip the failing endpoints until the source resolves the issue.
Next Expected Update: 1 hour
Next Expected Update: 1 hour
Update
We identified that multiple REST API requests are failing with 502 Bad Gateway errors returned by the source API.
We have reached out to the source regarding this.
Next Expected Update: 1 hour
We have reached out to the source regarding this.
Next Expected Update: 1 hour
Identified
The issue has been identified and we are working to resolve it.
This incident affected: Systems (General Services).