3rd Party: Few Qualtrics connections were failing with HTTP Error Code 429
Incident
Report for Fivetran
Resolved
This incident has been resolved. We have observed that instance rates are returning to normal levels and affected connectors are syncing successfully.
Incident Summary
Description:
We identified an issue for Qualtrics which resulted in syncs failing with HTTP Error Code 429.
Timeline:
This issue began on 10th March at around 10PM UTC and was resolved on 14th March at 10:30PM UTC.
Cause:
This issues was caused by Third party 429s for distribution History endpoint which started spiking after 10th March at around 10PM UTC, we suspect if qualtrics had enforced additional rate limits at an app level in addition to the brand level api limit of 300 per minute
Resolution:
We increased the retries and also added exponential sleep between api calls to reduce the load on qualtrics server
Incident Summary
Description:
We identified an issue for Qualtrics which resulted in syncs failing with HTTP Error Code 429.
Timeline:
This issue began on 10th March at around 10PM UTC and was resolved on 14th March at 10:30PM UTC.
Cause:
This issues was caused by Third party 429s for distribution History endpoint which started spiking after 10th March at around 10PM UTC, we suspect if qualtrics had enforced additional rate limits at an app level in addition to the brand level api limit of 300 per minute
Resolution:
We increased the retries and also added exponential sleep between api calls to reduce the load on qualtrics server
Monitoring
A fix has been implemented and we are monitoring the results.
Update
We identified an issue where the Distribution Contact endpoint was retrying requests too frequently, which caused some syncs to fail with rate limit errors (429).
A hotfix has been raised to adjust the retry behavior so that retries are spaced out more effectively. We will continue to monitor the syncs after deployment.
A hotfix has been raised to adjust the retry behavior so that retries are spaced out more effectively. We will continue to monitor the syncs after deployment.
Identified
The issue has been identified and we are working to resolve it.
This incident affected: Support connectors (Qualtrics).