vdk-postgres: schema inference for ingestion

[DeltaMichael]

Overview

The vdk-postgres plugin does not support schema inference when ingesting data.

from vdk.api.job_input import IJobInput

def run(job_input: IJobInput):
    payload = {
        "id": 1,
        "name": "Tony"
        "title": "Don't talk to strangers"
    }
    job_input.send_object_for_ingestion(
        payload=payload
    )

The above code, run as a single step in a data job against a postgres or redshift database should have the following result:

1. Create a table with columns corresponding to the payload keys (provided the table does not already exist)
2. If the table exists, the existing columns should be diffed with the payload keys and any new columns added
3. Ingest the data in the newly created table

Acceptance criteria

1. Above use case is supported