Add team_name tag to OpenLineage metrics#69109
Open
SameerMesiah97 wants to merge 1 commit into
Open
Conversation
SameerMesiah97
force-pushed
the
68996-OpenLineage-Team-Name-Metrics
branch
3 times, most recently
from
044f8d5 to
588db94
Compare
when multi-team is enabled on Airflow 3.1+. The team name is propagated through the Airflow run facet and included in ol.emit.attempts, ol.emit.failed, ol.extract, and ol.event.size. Tests are updated accordingly.
SameerMesiah97
force-pushed
the
68996-OpenLineage-Team-Name-Metrics
branch
from
588db94 to
e3f59ba
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This change adds a
team_nametag to OpenLineage metrics to improve observability in multi-team deployments.The following metrics now include the
team_nametag when available:ol.emit.attemptsol.emit.failedol.extractol.event.sizeA new
DagRunInfo.team_name()helper has been added to resolve the team associated with aDagRunfrom its DAG bundle.The helper caches team name lookups on a per-bundle basis, avoiding repeated database lookups when multiple task instances belong to the same DAG bundle.
For listener metrics (
ol.extractandol.event.size), theteam_nameis resolved directly from theDagRun. For adapter metrics (ol.emit.attemptsandol.emit.failed), the value is propagated through the Airflow run facet attached to the emitted OpenLineage event.Rationale
The OpenLineage integration emits metrics for lineage extraction and event publication, but these metrics could not previously be attributed to individual teams in multi-team deployments.
Adding the
team_nametag enables per-team dashboards, alerting, and troubleshooting while remaining backwards compatible for deployments that do not use multi-team scheduling.Tests
Added unit tests for the new
DagRunInfo.team_name()helper, including cache behaviour and multi-team configuration handling.Updated the existing OpenLineage listener and adapter unit tests to verify metric emission both with and without the
team_nametag. Test cases covering the presence ofteam_nameare skipped when running against Airflow versions earlier than 3.3.0.Backwards Compatibility
This change is additive only. Existing metric names remain unchanged, and the
team_nametag is emitted only when running on Airflow 3.3+ with multi-team enabled. Existing dashboards and integrations that do not use the new tag continue to function unchanged.Related: #68996
Was generative AI tooling used to co-author this PR?
Generated-by: [GPT 5.5] following the guidelines