Export Contained Level Rollup Tables to Redshift via Zero ETL

[sureshc]

As @code-dot-org/red-team transitions from using tables imported via Database Migration Service to Materialized Views on tables imported via Zero ETL Integration, they've identified 3 tables that currently aren't available via the Zero ETL pipeline:

• contained_levels
• contained_level_answers
• pd_applications_status_logs

These tables are populated by a legacy data export cron job that flattens data in MySQL in advance of exporting it via Database Migration Service.

As a short term resolution, define ActiveRecord Model classes for each of these rollup tables and declare export_to_analytics on each of them.

A longer term solution would be to re-implement the rollup logic in dbt on the source tables, but there are a few hundred rows where the data exceeds the 64KB limit in the destination Redshift columns and JSON parsing in Redshift would fail. Until we have time to investigate solutions to that issue, we can export the pre-processed data.

Note that this Pull Request also changes the rollup cron job to start executing around the same time that the legacy export via Database Migration Service starts (23:00 UTC) so that when the legacy DMS job refreshes all the Zero ETL Materialized Views as its final step, the

Testing story

Deployment notes

1. Merge this change
2. Infrastructure engineer executes analytics_export:provision_materialized_views rake task
3. Wait for RED team to cutover from DMS table to the new Materialized Views.
4. Update aws/dms/tasks.yml to stop exporting these tables via DMS and deploy by updating the DATA-production CloudFormation Stack.

Privacy and security