{
  "id": "dwd_cb2bgvae",
  "type": "pattern",
  "slug": "active-metadata-consumption",
  "title": "Serving metadata to people, systems and agents",
  "summary": "One canonical model, one governed write path, many read paths. How to turn catalog changes into subscribable domain events, and which channel serves which consumer.",
  "status": "draft",
  "classification": "public",
  "authors": [
    {
      "name": "Shan Umasankar",
      "url": "https://dealwithdata.com",
      "role": "author"
    }
  ],
  "assisted_by": [
    "Claude"
  ],
  "scales": [
    "enterprise"
  ],
  "tags": [
    "architecture",
    "events",
    "ai",
    "mcp",
    "kafka"
  ],
  "terms": [
    "dwd_mgglmnwq",
    "dwd_xbqpgs7h",
    "dwd_2agj6wti",
    "dwd_qywin2zi",
    "dwd_5k2ibrbv",
    "dwd_tthoszn7",
    "dwd_gdefyuzz"
  ],
  "related": [
    "dwd_q3gbjug4",
    "dwd_5hhpphgh",
    "dwd_wg5zzqg2"
  ],
  "sources": [
    {
      "title": "OpenLineage",
      "url": "https://openlineage.io/",
      "accessed": "2026-10-07"
    },
    {
      "title": "CloudEvents specification",
      "url": "https://cloudevents.io/",
      "accessed": "2026-10-07"
    },
    {
      "title": "AsyncAPI",
      "url": "https://www.asyncapi.com/",
      "accessed": "2026-10-07"
    },
    {
      "title": "Model Context Protocol",
      "url": "https://modelcontextprotocol.io/",
      "accessed": "2026-10-07"
    },
    {
      "title": "Open Data Contract Standard (Bitol)",
      "url": "https://bitol-io.github.io/open-data-contract-standard/",
      "accessed": "2026-10-07"
    }
  ],
  "created": "2026-10-07",
  "updated": "2026-10-07",
  "version": 1,
  "url": "https://dealwithdata.com/c/active-metadata-consumption/",
  "markdown_url": "https://dealwithdata.com/c/active-metadata-consumption.md",
  "body_markdown": "Once a catalog has good business metadata, everyone asks for it: \"Can we\nget technical metadata with business metadata?\", \"Can our AI agent use the\ndefinitions?\" The failure mode is ten teams building ten extracts. The fix\nis a shape, not a tool.\n\n## The shape\n\n**One canonical model, one governed write path, many read paths.**\n\nThe hard, valuable part isn't the streaming. It's the governed link:\n[[business-term]] \u2192 [[business-element]] \u2192 physical column \u2192 lineage.\nEverything below is delivery.\n\n## Getting changes out\n\nMost catalogs sit on a relational database and are poll or harvest based.\nChanges arrive two ways:\n\n- **UI edits** (a steward changes a definition): capture with database\n  [[change-data-capture]] such as GoldenGate or Debezium.\n- **Harvests** (scanners reload technical metadata): diff successive\n  snapshots and emit a change set. This gives cleaner events than the\n  database log.\n\n## Three layers, not one\n\n```\nCatalog DB \u2500\u2500CDC\u2500\u2500\u25b6 raw.* topics          (private: one team reads these)\nHarvests \u2500\u2500diff\u2500\u2500\u25b6        \u2502\n                          \u25bc\n                    Translator            row changes \u2192 domain events\n                          \u2502               schema registry, versioned\n                          \u25bc\n              public topics               BusinessElementDefinitionChanged\n                                          LineageEdgeAdded\n                                          CdeStewardReassigned\n```\n\nNever expose raw CDC from a vendor's internal schema. It is undocumented,\nchanges on upgrade, and one business edit fires a dozen row changes.\nSubscribers break and drown.\n\n## Pick the channel by consumer\n\n| Consumer | Channel | Standard |\n|---|---|---|\n| Stewards, analysts | Catalog UI | n/a |\n| Applications | REST or GraphQL over the metadata graph | OpenAPI |\n| Change subscribers | Domain-event topics | CloudEvents envelope, AsyncAPI docs |\n| Bulk analytics | Metadata snapshots as tables | SQL |\n| AI agents | MCP server plus a search index | MCP |\n| Pipelines | Emit lineage at run time; enforce contracts in CI | OpenLineage, ODCS |\n| BI and AI tools | Semantic model export | OSI (Apache Ossie) |\n\nAll of these are projections of the same model. When a team asks for \"a\nfeed\", they get one of these, not a custom extract.\n\n## Rules that keep it trustworthy\n\n1. **Stable global IDs** for every term, element and physical asset, or the\n   links can't survive across systems.\n2. **The link is a governed object.** A business-to-technical mapping carries\n   an owner, a confidence score, provenance (human, rule or AI) and a status.\n   AI-proposed mappings enter as *proposed*; stewards authorize.\n3. **The catalog stays the system of record for its domain.** Publish\n   outward; don't try to make it the hub for everything.\n4. **Agents read through the MCP layer, never the database.** That gives one\n   place for entitlements, audit, and \"approved definitions only\".\n\n## When it becomes a nervous system\n\n[[active-metadata]] only matters if something depends on it. Wire in two or\nthree consumers that break when metadata is wrong, such as change\nmanagement, a regulatory report pipeline, and one AI agent. Until then,\nit's documentation.\n\n## Using LLMs for long descriptions\n\nFor bulk drafting of element descriptions, a fast, cheap model is the right\ndefault. Quality comes from grounding, not model size: feed the physical\nname and type, profile statistics, lineage neighbors, the parent dataset\nand nearby glossary terms. Route critical elements and low-confidence drafts\nto a stronger model or a human reviewer.\n"
}