Extend ship pipeline to abundance/biomass

[leewujung]

Right now the ship pipeline is run to producing NASC from the prediction mask. In the next stage we’ll turn that into abundance and biomass estimates.

Let’s say we call this stage “bio_estimate”. This stage will be run after each prediction flow run. We need to decide whether we want this as part of the prediction flow, or if we want this to be a separate flow that runs on schedule or triggered by prediction flow. Given the potential potential network problems it seems better to just have it as part of the prediction flow.

This stage will take the following input/output:

• input:
  • NASC: this comes from the prediction flow
  • a csv with length distribution of hake: this will be iteratively updated each time with each trawl — note the trawl runs are not synchronized with the NASC generation
  • a json with length-weight regression coefficients: this will be iteratively updated each time with each trawl — note the trawl runs are not synchronized with the NASC generation
  • a csv specifying the strata boundary: this would be set BEFORE the deployment, more like a config parameter
• output / action:
  • each bio_estimate run will produce a few rows of estimates with columns: lat, lon, abundance, biomass (the columns may change)
  • we “append” these new rows into some kind of database or parque files
  • the updated database/files will be synced to the cloud, so we can spin up a viz from that

@brandynlucca @Sohambutala : these are based on our discussion yesterday. Feel free to comment to add other things/changes. I’ll update this description accordingly.