On April 15th 2020, approximately 19:00 UTC onwards multiple customers reported seeing the following on the Fivetran UI for their Postgres connector
Error: current transaction is aborted, commands ignored until end of transaction block.
There were changes committed to Fivetran code base that enables better data validation between source and destination. While rolling out the beta version to a few customers the feature instead got released to all Postgres connectors.
There were two issues identified in the code:
Steps taken to resolve the issue:
Action needed from customers:
There was a residual effect wherein certain Postgres connectors using XMIN for replication could have potentially lost the data during the affected period, in turn, causing data integrity issues.
If you do find a data integrity issue, contact Fivetran support team via support ticket to ensure that the issue is taken care of in a timely manner.
Fivetran has a script that would help set the state back for the Postgres connector to what it was before the faulty code was deployed. This would re-play changes that are already reflected in the warehouse, but guarantees that any potentially missing changes from the outage are also captured.
Steps to prevent/mitigate these risks in the future:
We appreciate your patience and help with this issue, and sincerely apologize for any inconvenience we caused.
The Fivetran Team