Column Filtering for Change Retention

[leejustin]

Table Column Filtering

Adds column filtering capabilities to change retention pipelines, allowing users to exclude sensitive columns (like user data fields or large metadata columns) or include only specific columns they need. This provides fine-grained control over which column data is captured and stored in change retention tables.

Data schema changes

• Migration: Adds include_column_attnums and exclude_column_attnums array fields to the source_tables JSONB column in wal_pipelines. Existing pipelines are migrated with NULL values for these fields.
• SourceTable schema: Extends the SourceTable embedded schema with two new optional array fields storing column attribute numbers. The changeset validates that these fields are mutually exclusive—only one can be set at a time.
• Storage: Column selection is stored as arrays of PostgreSQL column attribute numbers (attnum), which are resolved from column names during YAML parsing and UI configuration.

UI changes

• ColumnSelectionForm component: New Svelte component that provides an interactive interface for column selection with three modes:
  • All columns: No filtering (default)
  • Exclude mode: Select columns to exclude from sync
  • Include mode: Select columns to include in sync
• Form integration: The component is integrated into the WAL pipeline form (Form.svelte), appearing after table selection. It displays available columns with checkboxes, automatically handles primary key constraints (PKs cannot be excluded and are always included), and shows selected columns as removable tags.
• Display: The pipeline show page displays configured column selection settings, showing excluded or included column names resolved from attribute numbers.

Setting Column Filtering at Change Retention Creation

Note that the PK is disabled and cannot be included/excluded

Viewing Change Retention Details

The details show the selected columns if filtering is enabled

Editing Change Retention

The user can modify the configuration to update the filtering. There is a note in this field indicating that the changes will not backfill historic data for those columns in those rows, which is out of scope.

Implementation details

• Filtering logic: New ColumnSelection module handles filtering at multiple points:
  • During WAL event creation: Filters fields in record and changes payloads
  • During backfills: Filters column attribute numbers when reading table data
  • In message transformation: Filters both old and new fields for update operations
• Integration points: Column filtering is applied in MessageHandler.wal_event/2 and Consumers.message_record/2/message_changes/2, ensuring filtered columns are excluded from all downstream consumers and sinks.
• YAML support: The YAML loader parses exclude_columns and include_columns (column name lists) and converts them to attribute numbers, with validation to ensure primary keys are never excluded and that both options aren't specified simultaneously.

Configuration

Configure column selection via:

• Web console: Use the column selection form in the WAL pipeline configuration UI
• YAML: Specify exclude_columns or include_columns arrays in change_retentions source table configuration

Primary key columns are automatically protected—they cannot be excluded and are always included, even when using include mode.

Tests

Added a few tests and ran mix test:

Finished in 46.9 seconds (17.0s async, 29.8s sync)
4 doctests, 1342 tests, 0 failures, 14 skipped