Skip to content

011. ADR: Fabric Medallion Architecture for OSDU Data

Date: 17-07-2026

State: In Progress/Proposed/Accepted/Deprecated/Superseded

Status: Approved — By Task Lead

Deciders: OSDU Data Platform Team

Context and Problem Statement

The current Fabric architecture relies on the OSDU connector and the Notification Relay to pull new and updated records into the Fabric Bronze layer. This approach has proven slow and capacity-intensive, and it requires significant ongoing maintenance due to frequent errors in the relay pipeline.

To address this, the architecture is being revised to source Bronze data directly from the Analytics Consumption Zone (ACZ). The ACZ provides near-real-time updates and exposes OSDU records as a Delta table that can be mounted as the Fabric Bronze layer — eliminating the relay dependency entirely.

The remaining challenge is to transform these raw ACZ records into analytics-ready Silver and Gold tables: flattening dynamic OSDU schemas, excluding soft-deleted records, and enforcing row-level security aligned with OSDU ACL viewer groups.

Decision Drivers

  • Downstream consumers require flattened, analytics-ready Delta tables rather than raw JSON blobs.
  • Access to data must be controlled at the row level, consistent with OSDU ACL viewer group membership in Microsoft Entra ID.
  • Soft-deleted records must be excluded from analytical layers but retained at the raw layer for auditability.

Decision Outcome

Chosen option: Fabric Medallion Architecture

A three-tier medallion architecture (Bronze → Silver → Gold) will be implemented in Microsoft Fabric Lakehouse using Delta tables sourced from the OSDU Analytics Consumption Zone (ACZ).

Architecture Overview

Bronze Layer — Raw Ingest (ADME → Fabric Lakehouse)

  • Source: OSDU Analytics Consumption Zone (ACZ)
  • Pattern: One-to-one data dump from ADME; no transformation applied
  • Structure: Records flattened by OSDU system properties (id, kind, data, acls, tags, etc.)
  • Scope: All kinds, all versions, including soft-deleted records
  • Rationale: Preserves full fidelity of source data for reprocessing and auditing

Silver Layer — Curated & Typed (Bronze → Silver)

  • Source: Bronze Delta table (acz_bronze.dbo.osducatalog)
  • Runtime: Microsoft Fabric Spark; notebook version 0.4.0
  • Transformations:
  • a. Read OSDU records from the ACZ Bronze Delta table; unwrap ACZ Storage record envelopes where present
  • b. Filter out soft-deleted records (isActive == true); configurable via INCLUDE_INACTIVE_RECORDS
  • c. Resolve OSDU kind schemas dynamically from the ADME Schema Service (/api/schema-service/v1/schema)
  • d. Infer Spark types and flatten scalar and object fields
  • e. Produce output using the normalized output mode — parent table + typed child tables per array property
  • f. Write Silver Layer Delta tables; supports full_refresh (overwrite) or upsert (Delta merge on id + version)
  • Schema source: Flattening is driven exclusively by OSDU schemas retrieved from the ADME Schema Service. Custom or non-OSDU schemas are not supported and will not be processed.
  • Schema versioning: versioned_tables strategy suffixes table names by schema version; merge unions schema versions into one table
  • Missing schema handling: configurable as skip (default), infer, or fail
  • Scope: All versions retained; soft-deleted records excluded by default
  • All OSDU kinds processed (initial scope)
  • Output: Analytics-ready Delta tables for downstream engineering and reporting workloads

Silver Layer — Authentication

The notebook supports three authentication methods for the ADME Schema Service:

Method Use Case
SP (Service Principal) Scheduled / automated runs — uses a client secret stored in Azure Key Vault
DC (Device Code) Interactive developer validation
MI (Managed Identity) System-assigned or user-assigned managed identity in Fabric

Secrets are never stored in the notebook. SP secrets are retrieved at runtime from Azure Key Vault by name (ADME_SP_SECRET_KV_NAME / ADME_SP_SECRET_NAME).

Silver Layer — Operational Metadata Tables

The pipeline writes the following metadata tables to the Fabric Lakehouse alongside Silver data tables:

Table Purpose
silver_schema_cache Persisted OSDU kind schema cache; reduces schema service calls on reruns
silver_run_manifest Per-kind output manifest: table names, record counts, schema versions, status
silver_run_status Overall pipeline run status (started, completed, failed)
silver_incremental_state Watermark values for incremental/upsert runs
silver_output_documentation Column-level documentation for all produced Silver tables
silver_data_quality_issues Data quality violations caught during transformation (up to 100 examples per run)

Gold Layer — Access-Controlled & Business-Ready (Silver → Gold)

  • Source: Silver Delta tables
  • Access Control: Row-Level Security (RLS) applied based on OSDU ACL viewer groups
  • In the Fabric Lakehouse, navigate to Manage OneLake Security
  • Create a new Role; select the target table and define the filter SQL predicate based on the ACL viewer value
  • Under Members in Role, assign the matching Microsoft Entra ID group that corresponds to the ACL viewer group
  • Output: Secure, business-ready Delta tables for reporting and self-service analytics

Data Lifecycle — Hard Deletion

When a record is hard-deleted from ADME, the deletion propagates automatically through the ACZ and is therefore reflected in the Bronze layer, since Bronze is a mount of ACZ storage rather than a copy. No additional pipeline step is required to remove the record from Bronze.

Hard-deleted records are not automatically removed from the Silver or Gold Delta tables. However, because Silver and Gold are written as Delta tables in the Fabric Lakehouse, deleted records can be excluded from future pipeline runs and, if needed, removed explicitly via a targeted delete operation. The Delta table transaction log also enables time travel: the flattened and typed version of a hard-deleted record remains recoverable by querying a snapshot prior to the deletion, providing an audit trail without requiring Bronze retention.

Pipeline Overview

flowchart TD A[("Bronze Delta Table\nacz_bronze.dbo.osducatalog")] B["Ingest\nUnwrap ACZ envelope · filter soft-deleted records"] C["For each OSDU kind"] D["Resolve schema\nFetch from ADME Schema Service · cache result"] E["Classify columns\nscalar · json_object · json_array · null"] F["Build parent table\nFlatten scalars & objects · add ingested_at"] G["Build child tables\nExplode arrays & tags · add id, version, ordinal"] H["Write to Lakehouse\nfull_refresh overwrite or upsert Delta merge"] I[("Silver Delta Tables\nParent + child tables per kind")] J[("Metadata Tables\nrun manifest · run status · schema cache · DQ issues")] K["Apply Row-Level Security\nOneLake Security role per table\nFilter predicate on ACL viewer value"] L["Assign Entra ID groups\nMap OSDU ACL viewer groups\nto Fabric RLS role members"] M[("Gold Delta Tables\nSecure · business-ready\nfor reporting & self-service analytics")] A --> B --> C --> D --> E E --> F & G F & G --> H H --> I H -.-> J I --> K K --> L L --> M

Consequences

Positive

  • Clean separation of concerns between raw, curated, and secured data layers
  • Schema changes in OSDU kinds are handled dynamically at the Silver transformation step, reducing maintenance overhead
  • RLS in the Gold layer ensures that Fabric consumers cannot access records beyond their OSDU ACL entitlements
  • Soft-deleted records are isolated to Bronze, preventing accidental use in analytics

Negative / Trade-offs

  • Dynamic schema resolution from the ADME Schema Service introduces a runtime dependency; schema service availability affects Silver pipeline runs
  • Managing per-kind Delta tables (Silver) increases the number of tables as OSDU kind count grows
  • Entra ID group membership must remain in sync with OSDU ACL viewer groups; misalignment will result in incorrect data access
  • The wide output mode (not selected) caps array cardinality at 20 columns to prevent schema explosion; the chosen normalized mode avoids this limitation

Risks

Risk Likelihood Mitigation
Schema service unavailability blocking Silver runs Medium Retry logic built-in (3 retries, 1 s backoff, HTTP 408/429/5xx); persistent schema cache in silver_schema_cache reduces live calls
Entra ID / ACL group drift Medium Periodic reconciliation job between OSDU ACLs and Entra ID group membership
Bronze layer growing unbounded with all versions High Define a data retention policy for Bronze; archive or TTL old versions
Duplicate merge keys causing upsert failures Low Pipeline asserts no duplicate (id, version) keys before executing Delta merge
SP secret expiry breaking scheduled runs Medium Rotate secrets in Key Vault; pipeline reads secrets at runtime so rotation requires no notebook change
Fabric capacity overload during large data ingestion making Fabric unavailable High See Fabric Capacity Constraints below; schedule heavy runs during off-peak hours; implement pipeline throttling; coordinate F-SKU scaling with the capacity-owning team in advance of large loads

Financial Considerations

Fabric Capacity Usage

The Bronze-to-Silver flattening process is the primary driver of Fabric capacity consumption. The initial run — which must process all existing OSDU records across all kinds — is expected to be high in CU usage due to the volume of data and the schema resolution overhead per kind. Once the initial flattening is complete, capacity usage is expected to taper significantly as subsequent runs only process new or updated records via upsert.

As part of this architecture change, the existing Fabric connector and Notification Relay pipeline can be decommissioned. Removing these components will reduce ongoing capacity usage and associated costs, partially offsetting the compute consumed by the Silver transformation pipeline.

Fabric Capacity Constraints

The Fabric capacity underpinning this architecture is owned and managed by a separate team. The OSDU Data Platform Team does not have direct control over F-SKU tier or burst limits, and any capacity scaling request must be submitted to and approved by the capacity-owning team. This creates a dependency that introduces both lead time and organisational friction.

At the same time, the platform has a cost objective to keep Fabric CU consumption as low as possible. These two pressures are in tension: scaling up capacity to absorb large data ingestion spikes increases cost, while keeping capacity low risks overloading the shared environment and causing Fabric to become throttled or fully unavailable — affecting all workloads running on that capacity, not just this pipeline.

This risk is most acute in two scenarios:

  1. Initial full load — Processing all OSDU records across all kinds for the first time will generate a sustained, high-CU workload. Without a temporary capacity increase, this run is likely to exhaust available CUs and trigger throttling.
  2. Bulk re-ingestion — If a full refresh is required (e.g. after a schema change or data correction), the same overload risk applies.

Mitigations to be agreed with the capacity-owning team:

  • Schedule the initial load and any full-refresh runs during off-peak hours to reduce contention with other workloads.
  • Process kinds in batches across multiple runs rather than a single job, spreading CU consumption over time.
  • Request a temporary, time-bound F-SKU upgrade for the duration of the initial load, then revert to the baseline tier.
  • Set pipeline-level CU budgets or runtime limits to prevent a single job from consuming all available capacity.

Open item: A formal agreement with the capacity-owning team on the process for requesting temporary capacity increases has not yet been established. This must be in place before the initial load is executed.

ACZ Costs

ACZ is currently a preview feature in ADME and is not yet charged. Microsoft has indicated that charges will be introduced once ACZ reaches General Availability, which is currently scheduled for Q4 2026. The exact pricing model has not been communicated, so the additional cost to the platform is unknown at this time and should be tracked as a financial risk.

Cost Summary

Component Direction Notes
Fabric capacity — initial Silver flattening Increase (one-off) High CU usage expected during first full run; tapers after
Fabric capacity — ongoing Silver pipeline Neutral / slight increase Incremental upsert runs are low-cost; replaces connector/relay workload
Fabric connector + Notification Relay Decrease Decommissioning these reduces ongoing CU consumption
OneLake storage Increase Bronze, Silver, and Gold Delta tables accumulate over time
ACZ (ADME add-on) Increase (future) No charge while in preview; pricing TBC when GA in Q4 2026
Gold layer RLS No change Built-in Fabric feature; no additional cost

Open Items

  • ACZ GA pricing has not been disclosed by Microsoft; costs should be revisited once pricing is available
  • A CU baseline for the initial Silver flattening run should be captured to inform F-SKU sizing

Security

ACZ Storage Account Access

Access to the Azure Storage account underlying the ACZ mount must use Azure AD (Entra ID) accounts only — shared access keys and SAS tokens must not be used. Permissions are granted via Azure RBAC, following least-privilege principles. Privileged roles (e.g. Storage Blob Data Reader/Contributor) must be assigned through PIM (Privileged Identity Management) with just-in-time activation, an approved justification, and a time-bound session. Standing privileged access is not permitted.

Fabric Workspace Isolation

The Fabric workspace hosting this architecture must be a dedicated workspace, separate from existing OSDU Fabric workspaces (e.g. OSDU-Fabric-Dev). This workspace must apply stricter access controls:

  • Workspace membership is restricted to a named set of approved users and service principals; access is not inherited from broader Fabric capacity or tenant groups.
  • All access grants must go through the Fabric Workspace Admin role; self-service access requests are not permitted.
  • Service principals used for scheduled notebook runs must be scoped to this workspace only and must not hold permissions in other OSDU Fabric workspaces.

Gold Layer Row-Level Security

As documented in the Gold Layer section, RLS is enforced via OneLake Security roles. RLS role membership must map to Entra ID groups that correspond directly to OSDU ACL viewer groups. Membership in those Entra ID groups must be managed through the standard Entra ID joiner/mover/leaver process and reviewed periodically.

File Access

The ACZ exposes OSDU records as Delta tables backed by Parquet files in the underlying ACZ storage account. In addition to accessing data through the Delta table layer, it is possible to grant direct access to these Parquet files using OneLake Security.

Direct Parquet file access will not be configured upfront. Access will be reviewed and granted on a case-by-case basis as requests are received. Each request must be assessed for business justification and approved before an OneLake Security rule is applied.

Secrets Management

No credentials, secrets, or connection strings are stored in notebooks or Lakehouse items. Service principal secrets are stored in Azure Key Vault and retrieved at runtime by name. Access to the Key Vault must follow the same RBAC and PIM controls as the storage account.

Roles and Responsibilities

Key: R = Responsible · A = Accountable · C = Consulted · I = Informed

Responsibility OSDU Data Platform Team Microsoft Fabric Workspace Admin
ACZ creation A/R C I
ACZ availability and operations I A/R I
Bronze layer mount / shortcut setup A/R C I
Transformations from Bronze to Silver A/R I I
Transformations from Silver to Gold A/R I I
Gold layer RLS configuration A/R I C
Entra ID group membership (ACL sync) A/R I C
Fabric capacity sizing & monitoring I I A/R

Clarifications Pending from Microsoft and Working Assumptions

The following items represent open questions raised during architecture design. Each entry documents our current working assumption and the verification action required.

ACZ Instance Lifecycle and Creator Credentials

Question: Once an ACZ instance is created, which credentials or identity govern its ongoing operation? Specifically, if the person who provisioned the ACZ subsequently leaves the organisation or has their permissions reduced, does that affect the availability or functioning of the ACZ?

Working assumption: Our understanding is that once an ACZ instance is provisioned, its ongoing operation is managed entirely by Microsoft on the backend. The creator's identity is only required at the point of provisioning; subsequent ACZ availability and data synchronisation are not tied to the creator's account, role, or employment status. A change in the creator's permissions or departure from the organisation should have no impact on the ACZ.

Status: Needs verification from Microsoft before this architecture is considered production-ready.

Automatic Inclusion of New OSDU Kinds in an Existing ACZ

Question: If an ACZ instance is configured with scope "all kinds, all versions" at the time of creation, will OSDU kinds that are introduced to ADME after the ACZ was provisioned be automatically included and kept up to date in the ACZ?

Working assumption: Yes — we assume that an ACZ configured for all kinds and all versions will automatically include new kinds as they are registered in ADME, without requiring the ACZ to be re-provisioned or reconfigured. New kind data would begin appearing in the ACZ Delta table once the kind is populated in ADME.

Status: Needs verification from Microsoft. If new kinds are not automatically included, a re-provisioning or reconfiguration process will need to be incorporated into the architecture's operational runbook.

Links

Last update: 2026-09-14