Lesson 2 · Connect

Lineage

Lineage is the backbone of governance. You can't do impact analysis, root-cause a bad number, or enforce a contract without knowing where data comes from and where it goes.

2.1 — What lineage actually is

Lineage is a directed graph: nodes are entities (tables, columns, dashboards, pipelines), edges are "data flows from A to B". Two resolutions:

Table-level — in our shop, orders → v_daily_revenue. Good for coarse impact analysis: "what depends on orders?"
Column-level — orders.total_amount → v_daily_revenue.revenue. The powerful one: it tells you exactly which downstream fields a column change breaks.

OpenMetadata can also extend lineage past tables into dashboards and pipelines — not in our small lab, but that's how you'd trace a number on an executive dashboard all the way back to the source column. That end-to-end trace is the payoff.

DMBOK lens

Lineage lives in DMBOK's Data Integration & Interoperability knowledge area and enriches Metadata Management. It's what makes impact analysis and root-cause possible — and a prerequisite for trustworthy Data Quality and governance.

2.2 — How lineage gets created

Four ways, roughly in order of automation:

Method	How
Query-log / view parsing (auto)	The lineage & usage workflows read query history (or view DDL), parse the SQL, and infer `INSERT INTO target SELECT … FROM source`. How most table lineage appears with no manual work.
dbt (auto)	Ingesting a dbt project turns `ref()` relationships into lineage — including column-level, with tests and descriptions.
API / SDK (programmatic)	You can `PUT` lineage edges directly — handy for custom integrations or sources OpenMetadata can't auto-parse.
Manual (UI)	Drag-and-drop edges in the Lineage tab. Good for filling gaps the parser missed.

Why automatic lineage has gaps

The parser needs query history or view DDL it can read. Ad-hoc loads, external transforms, or sources it can't see produce gaps — which you fill manually or via API. Knowing why a gap exists is half of governance work.

Our example: orders → daily revenue

The shop database has a view v_daily_revenue built from the orders table. That's a perfect, readable lineage relationship: orders is the source, v_daily_revenue is downstream.

It's probably already there — automatically

Because v_daily_revenue is a view, OpenMetadata read its SELECT … FROM orders definition during the Metadata ingestion and drew the orders → v_daily_revenue edge for you — no extra workflow needed. Open v_daily_revenue → Lineage tab and you'll likely see it already. (Check the View Definition tab to see the exact SQL that produced it.)

Lineage tab showing orders connected to v_daily_revenue — Table-level lineage, derived automatically from the view: `orders → v_daily_revenue`.

Do thisRead the lineage you already have

Open v_daily_revenue → Lineage tab. Confirm the orders → v_daily_revenue edge.
Expand columns via the "5 Columns" / "3 Columns" dropdowns, then click Edit on the graph (top-right) — this reveals the auto-derived column-level links. (They're often hidden in normal view; Edit mode just displays them.)
You'll see order_date → order_date (passes through) and total_amount → revenue (because revenue = SUM(total_amount)). Notice orders_count has no incoming line — it's COUNT(*), an aggregate with no single source column. That's correct. Exit Edit without saving — you changed nothing.
Open the View Definition tab to read the SELECT … FROM orders that created this lineage.

Column-level lineage: total_amount maps to revenue, order_date to order_date — Column-level lineage (revealed in Edit mode): `total_amount → revenue` and `order_date → order_date`. `orders_count` has no source — it's `COUNT(*)`.

The drag-and-drop method below is how you'd add lineage that wasn't auto-detected (e.g. between two plain tables loaded by an external process). Good to know even if you didn't need it here.

Do thisCreate lineage by hand: orders → v_daily_revenue

In online_shop, open the orders table.
Lineage tab → Edit → drag an edge from orders to v_daily_revenue.
Save, then open v_daily_revenue's Lineage tab — the edge shows from both sides. You've asserted that revenue flows from orders.

2.3 — Why this is the governance backbone

Hold these three use cases in mind — they're what lineage is ultimately for:

Impact analysis — "if I change/drop this column, what breaks downstream?" Walk the graph forward.
Root cause — "this dashboard number looks wrong, where did it come from?" Walk the graph backward.
Contract enforcement — a contract on v_daily_revenue is only enforceable if you know its upstream dependency (orders) and can detect when it changes shape.

Checkpoint

1. Table-level vs column-level lineage — what's the difference, and why is column-level more valuable?

Table-level connects whole tables; column-level connects individual fields. Column-level is more valuable because it tells you exactly which downstream fields a specific column change will break — precise impact analysis.

2. Name two ways lineage is created automatically and two ways you create it yourself.

Automatic: query-log/view parsing, and dbt ingestion. Manual: drawing edges in the UI, and asserting them via the API/SDK.

3. Why does automatic lineage have gaps?

Because the parser can only infer flows from query history or view DDL it can actually read. Ad-hoc loads, external transforms, or invisible sources leave gaps you fill manually or via API.

Saves to this browser so you can pick up where you left off.

← Previous

Catalog & Ingestion

Governance