Closed
Milestone
GRIS Pipeline Automation
Milestone ID: 23
Collection of issues related to running the GRIS data processing pipeline.
The goal of this Milestone is to deploy the pipeline to a containerized computing environment with some form of orchestration tool handling scheduling, triggering and ordering of the individual processing steps. Information will be documented in the SDC wiki, this means, closing any issue in this milestone requires an entry in wiki page.
Specific goals for this milestone are:
- Compare different containerization and orchestration schemes
- Pipeline steps need to be run in containers
- Containers need to be automatically distributed among available nodes
- Proof of concept of the selected solution
Requirements that need to be addressed:
- Automatic processing of new data
- Manually triggered reprocessing of old data
- e.g. if coordinate correction is applied retroactively.