VETL Payload Document
User documentation
2024-04-22
Validate jobs list, load from variables and should be a list.
Prepare the database driver and import necessary libraries to the runtime environment.
ETL Job group, each item identify a specific workflow on the current DAG schedule:
Extract tasks group:
Extract data from different sources.
(Optional) Extra operations may be involved.
Localization & Calculation. Prepare raw data for transforming.
Transform & Load tasks group:
Transform tasks group:
Operation 1
Operation 2
Operation 3
Operation 4
......
Load task
Cleanup all temporary files which have been generated by ETL tasks in a job.
Result reporting.
House clean.