Skip to content

Transformer Function

The Transformer Function is responsible for mapping and transforming data between two schemas.

It takes two parameters:

  • source_schema: a list of dictionaries that define the source column names and types.
  • target_schema: a list of dictionaries that define the target column names and types.

The function: - Maps matching columns from source_schema to target_schema based on column name. - Casts each value to the data type defined in the target_schema. - Skips columns not present in target_schema.


Example Transformer Logic

```python from typing import List, Dict, Any

def transform_data(rows: List[Dict[str, Any]], source_schema: List[Dict[str, str]], target_schema: List[Dict[str, str]]) -> List[Dict[str, Any]]: """ Transforms a list of rows from source schema to target schema, casting values to the target type.

Parameters:
    rows: List of data rows (dictionaries).
    source_schema: List of source column definitions (e.g., [{'name': 'Quantity', 'type': 'str'}]).
    target_schema: List of target column definitions (e.g., [{'name': 'Quantity', 'type': 'int'}]).

Returns:
    List of transformed rows.
"""
type_casts = {
    'int': int,
    'float': float,
    'str': str,
    'bool': lambda x: str(x).lower() in ('true', '1', 'yes'),
}

# Build a lookup from target column name to type
target_map = {col['name']: col['type'] for col in target_schema}

transformed_rows = []

for row in rows:
    new_row = {}
    for col_name, col_type in target_map.items():
        if col_name in row:
            try:
                new_row[col_name] = type_casts[col_type](row[col_name])
            except Exception:
                new_row[col_name] = None  # or handle default/fallback
        else:
            new_row[col_name] = None  # column not found in input row
    transformed_rows.append(new_row)

return transformed_rows