Most Oracle-to-lakehouse pipelines start the same way: a nightly job that pulls whole tables out of Oracle and rewrites them into Snowflake, Databricks, or an Iceberg table. It’s easy to stand up, it works on day one — and it quietly costs you two things you’ll miss later: freshness and money.
This post is about why the nightly reload gets expensive, what “incremental” really means when the source is Oracle, and the correctness traps that decide whether the lakehouse copy is actually right.
What the nightly reload actually costs
It’s a day stale, by design. If the job runs at 2 a.m., every dashboard, model, and query is reading yesterday. For finance close, fraud signals, inventory, or anything operational, “as of last night” is the wrong answer for most of the day.
You pay to move rows that didn’t change. A table with 50 million rows and 40,000 daily changes still gets fully scanned, extracted, and rewritten every night. You’re renting compute — on both the Oracle side and the warehouse side — to re-copy 49.96 million rows that are identical to yesterday’s.
A snapshot is not a history. A full reload gives you the current state and throws away how it got there. If a row was inserted at 9 a.m. and deleted at 4 p.m., the nightly snapshot never saw it. For audit, slowly-changing dimensions, or any “what did this look like at time T” question, that history is gone.
Load windows don’t scale. As tables grow, the nightly job takes longer and creeps toward the start of the business day. Eventually the reload and the workday overlap, and you’re tuning batch windows instead of shipping.
What “incremental” means when the source is Oracle
The alternative is to move only what changed. Oracle already records every committed change in its redo log; change data capture (CDC) reads that record — through Oracle’s documented interface — and turns it into a compact feed of inserts, updates, and deletes. Instead of re-copying the table, you apply the day’s 40,000 changes as they happen, at near-real-time (seconds-class) latency.
That flips both problems at once: the lakehouse tracks the source continuously instead of once a night, and you move deltas instead of whole tables, so the compute bill drops with it.
But “just apply the changes” hides four traps that separate a correct lakehouse copy from a plausible-looking wrong one.
Trap 1 — updates and deletes, not just appends
Lakehouse tables love appends. CDC produces updates and deletes too, and if you append those naively you get duplicates and ghost rows — a record that was updated appears twice, a deleted record lingers forever. A correct pipeline applies merge/upsert semantics on the primary key: an update replaces the current version, a delete removes it. Modern table formats support exactly this — Iceberg v2 and Delta merge-on-read, a ReplacingMergeTree collapse in ClickHouse — but the CDC layer has to drive it, keyed on the primary key, in the right order.
And the subtle one: when an update changes the primary key itself, the old key must be deleted and the new one written — otherwise you’re left with a phantom row under the old key. It’s an easy case to miss and a nasty one to debug months later.
Trap 2 — type fidelity across two very different engines
Oracle types don’t map cleanly onto warehouse or columnar types. A NUMBER(38) can lose precision if it’s forced through a floating-point double. A TIMESTAMP WITH TIME ZONE can shift an hour if the zone is dropped. A CLOB, a RAW, a large NUMBER — each needs a deliberate, canonical representation on the target so that equal values stay equal. Get this wrong and nothing errors; the numbers are just subtly off, which is the worst way to find out.
Trap 3 — schema drift
Source schemas move. Someone runs ALTER TABLE ... ADD COLUMN on Oracle, and a pipeline that decodes columns positionally silently misaligns every row after it. A durable pipeline notices the DDL and propagates the change to the target — new columns, new tables — instead of quietly corrupting the feed from that point on.
Trap 4 — order
Changes have to land in the order they were committed. Apply an update before the insert it depends on, or a delete after a later re-insert, and the target diverges from the source even though every individual change was “applied.” Commit-order fidelity is not optional.
Freshness and a smaller bill
Handle those four and the economics invert. You move a day’s worth of deltas instead of a full table, so both the Oracle extract cost and the warehouse write cost fall. And the lakehouse is minutes fresh instead of a day stale. You stop trading freshness against cost — incremental CDC improves both at once.
Keep the data in your own network
There’s one more thing worth insisting on when the source is Oracle: the whole pipeline should run on your infrastructure. A lot of managed pipeline services route your data through their cloud to do the transformation. For regulated data — healthcare, insurance, financial, life-sciences — that’s an extra copy of your most sensitive tables sitting somewhere you don’t control. A self-hosted capture engine keeps the redo feed, the transforms, and the lakehouse writes entirely inside your network, up to and including fully air-gapped. Data sovereignty is easiest to defend when the data never left.
And prove it converged
Everything above tells you the pipeline should be right. Before you trust the lakehouse copy for anything that matters, prove it: a cryptographic, primary-key-ordered, cross-engine convergence check compares the actual content of the Oracle table and the lakehouse table and gives you a single yes/no. (That’s the subject of our first post — verify before you rely on it, don’t spot-check and hope.)
Where KinetiShift fits
We built KinetiShift to do exactly this: a self-hosted, near-real-time change-data-capture & replication engine that reads Oracle changes through Oracle’s documented interface and applies them incrementally to a lakehouse — Snowflake, Databricks, Iceberg, or Delta — as well as Postgres, Kafka, and more. It handles all four traps by design: merge/upsert on the primary key (including primary-key changes), type-faithful value handling, schema-drift (DDL) propagation, and commit-order apply. Capture is skip-loud — it never writes a value it can’t decode with certainty. It runs entirely on your own infrastructure, and it ships a built-in kinetishift verify so you can prove the lakehouse converged on the source.
If you’re feeding a lakehouse from Oracle — or still on a nightly reload and feeling the freshness and cost of it — we’re taking on a small group of founding design partners: a free, pre-release evaluation on one of your Oracle databases, run on your own systems. If that’s useful, start a conversation.
Move the deltas, not the table. Keep the data in your network. And verify the copy before you build on it.