Pipeline Inheritance

Create reusable base pipelines that can be extended and customized by child pipelines.

Pipeline inheritance allows you to create reusable base pipelines that can be extended and customized by child pipelines. This promotes code reuse and standardization across your organization.

Overview

Base Pipeline (abstract)
┌─────────┐     ┌─────────┐     ┌─────────┐
│validate │────▶│transform│────▶│ output  │
│         │     │(abstract)│     │         │
└─────────┘     └─────────┘     └─────────┘
                    │
                    │ extends
                    ▼
Child Pipeline
┌─────────┐     ┌─────────┐     ┌─────────┐
│validate │────▶│json_    │────▶│ output  │
│(inherited)│   │transform│     │(inherited)│
└─────────┘     └─────────┘     └─────────┘

Creating a Base Pipeline

Base pipelines define the structure that child pipelines will inherit. Mark a pipeline as abstract if it shouldn’t be executed directly:

{
  "id": "base-etl",
  "name": "Base ETL Pipeline",
  "abstract": true,
  "stages": [
    {
      "id": "validate",
      "component": "schema-validate",
      "config": { "strict": true }
    },
    {
      "id": "transform",
      "type": "abstract"
    },
    {
      "id": "output",
      "component": "logger",
      "depends_on": ["transform"]
    }
  ]
}

Extending a Pipeline

Use the extends field to inherit from a base pipeline and overrides to customize specific stages:

{
  "id": "customer-etl",
  "name": "Customer ETL Pipeline",
  "extends": "base-etl",
  "overrides": {
    "transform": {
      "component": "json-transform",
      "config": {
        "expression": "{customer_id: id, full_name: join(' ', [first_name, last_name])}"
      }
    }
  }
}

Multi-Environment Example

Create environment-specific pipelines from a shared base:

// base-etl.pipeline.json
{
  "id": "base-etl",
  "abstract": true,
  "stages": [
    { "id": "extract", "type": "abstract" },
    { "id": "validate", "component": "schema-validate" },
    { "id": "transform", "type": "abstract" },
    { "id": "load", "type": "abstract" }
  ]
}

// prod-etl.pipeline.json
{
  "id": "prod-etl",
  "extends": "base-etl",
  "overrides": {
    "extract": {
      "component": "http-request",
      "config": { "url": "https://api.prod.example.com" }
    },
    "transform": {
      "component": "json-transform",
      "config": { "expression": "..." }
    },
    "load": {
      "component": "database-write",
      "config": { "connection": "prod-db" }
    }
  }
}

// dev-etl.pipeline.json
{
  "id": "dev-etl",
  "extends": "base-etl",
  "overrides": {
    "extract": {
      "component": "file-read",
      "config": { "path": "./test-data.json" }
    },
    "transform": {
      "component": "json-transform",
      "config": { "expression": "..." }
    },
    "load": {
      "component": "logger",
      "config": { "level": "debug" }
    }
  }
}

Python API

InheritanceResolver

Resolves the complete pipeline configuration by merging parent and child configs:

from flowmason_core.inheritance import InheritanceResolver

resolver = InheritanceResolver()

# Load pipeline with inheritance resolved
pipeline = resolver.resolve("customer-etl", pipeline_registry)

# Check inheritance chain
chain = resolver.get_inheritance_chain("customer-etl", pipeline_registry)
print(chain)  # ['base-etl', 'customer-etl']

InheritanceValidator

Validates inheritance rules and detects issues:

from flowmason_core.inheritance import InheritanceValidator

validator = InheritanceValidator()
result = validator.validate(pipeline_config, pipeline_registry)

if not result.is_valid:
    for error in result.errors:
        print(f"Error: {error}")

Validation Rules

The inheritance system enforces:

1. No Circular Inheritance - A pipeline cannot extend itself or create a cycle
2. Abstract Stage Override - Abstract stages must be overridden in child pipelines
3. Type Compatibility - Overridden stages must be compatible with the parent
4. Dependency Preservation - Stage dependencies must remain valid after inheritance

CLI Commands

Validate Inheritance

fm validate --check-inheritance customer-etl.pipeline.json

Show Inheritance Chain

fm info --inheritance customer-etl.pipeline.json

Best Practices

1. Use Abstract Pipelines - Mark base pipelines as abstract to prevent accidental execution
2. Document Override Points - Clearly document which stages are meant to be overridden
3. Keep Inheritance Shallow - Avoid deep inheritance chains (max 3 levels recommended)
4. Version Base Pipelines - Use semantic versioning for base pipelines
5. Test Both Levels - Write tests for both base and child pipelines