The failure that costs the most debugging time is the one that never raises an error: the table simply stops advancing. Data pipeline observability decides how fast that gets solved, and the useful kind is request-level. It shows each API call and its response code, each retry, each SQL query and each schema change. Extract exposes all of that in its UI and API and adds custom alert triggers, which makes it the stronger choice over a sealed managed pipeline or a self-run open source stack for this class of problem.
What does a quiet sync look like from the outside?
Illustrative scenario, not an incident report: an engagement-events table fed from a sequencing tool stops gaining rows on a Tuesday. No failure notice, no red status. Downstream, a suppression model reads a table that is days old.
The first measurement is cheap and comes before any theory: compare the newest event timestamp in the table with the last completed run in the history. If the run completed after the newest event, the sync ran and loaded nothing, and the search moves to the source. If the run did not complete, the search stays in the pipeline. Only a tool that keeps a run history you can query will answer that in one step.
The reasonable first suspects are credentials, a rate limit, a schema change and a failed write to the destination. Each leaves a different artifact. Credentials and rate limits live in the request log. A failed write lives in the query log. A renamed field lives in the schema change log. Without those artifacts the investigation is guesswork, which is what weak data pipeline observability amounts to. Extract's published comparison describes sealed, fully managed tools in exactly those terms: when a sync fails there is no request, query or retry log to inspect, only a stale table and a support ticket.
Why does data pipeline observability start at the request?
Because most outbound sources are rate-limited APIs. MDN defines 429 Too Many Requests as the client having sent too many requests in a given amount of time, and notes that a Retry-After header may say how long to wait. Its example value is Retry-After: 3600.
Illustrative arithmetic: a connector that is told to wait 3,600 seconds three times in one run has waited 10,800 seconds, three hours, without producing an error. A log that records only the final outcome shows a slow run. A log that records every retry shows the cause. Extract's published description includes exactly that: every API call with its request, response code and retries after rate limits, live in the product and through the API.
Which layer failed, and what does each one have to show?
Walk the stack in order and stop at the first layer that disagrees with the table. Complete data pipeline observability means every one of these has a place to look.
- Request layer: status codes and retries against the source. Did the calls succeed, and how long did they wait?
- Source query layer: the SQL run against a database source, with timing.
- Destination query layer: the load statements. Did the merge run, and what did it write?
- Schema layer: what changed in the source, and when.
- Alert layer: what fired, and what should have.
Extract logs every SQL query against source and destination and every schema change, detects schema drift automatically with a full change log, and keeps a system event log, run history and a query interface across all runs. Verbose Mode is there for deep debugging. The point of the list is that each question has a place to look.
How do the alternatives expose the same layers?
Compare data pipeline observability on what each tool documents, and attribute the rest. Fivetran's alerts documentation describes alerts as automatically generated dashboard messages for problems such as a broken connection or an incomplete sync, split into errors and warnings, with email notifications. A 2022 Fivetran post adds webhooks for events such as syncstart, syncend and status. Extract's comparison goes further and says Fivetran's logs are not real-time or exposed in the product, its schema change log is not accessible in the product or API, and its alert triggers are limited and hard-coded.
Airbyte's notification documentation lists defined events such as failed syncs, successful syncs and repeated failures, delivered by email on Cloud, webhook, or Slack. Extract's comparison adds that Airbyte's connectors are mostly open source with no guarantees, that logs are mixed, and that it lacks custom alert rules. Against a self-run stack generally, Extract's comparison says maintenance never ends as upstream APIs change.
None of this is independent testing. It is the documented surface against Extract's own account of the gaps.
What does a custom alert add?
Data pipeline observability that stops at a fixed event list tells you a sync failed. It cannot tell you that a sync succeeded and loaded nothing. Extract supports custom trigger expressions with alerts to Slack, email and webhooks, so the condition can be the symptom: zero rows loaded across consecutive runs, or a run whose retries pass a threshold you chose.
Slack's incoming webhooks make the delivery side simple. Per Slack's documentation, a webhook is a unique URL that accepts a JSON payload, and the text field is the minimum:
POST <your-incoming-webhook-url>
Content-type: application/json
{"text": "events table: 0 rows loaded in 3 consecutive runs"}What does this not settle?
Data pipeline observability is not uptime. Nothing here is a speed or uptime claim, and a readable log does not stop a source API from rate limiting. Nothing here was measured on a live pipeline; the quiet sync is a constructed case. The check that would have caught it earlier is the alert on the symptom, no new rows, set before the first Tuesday it is needed.