Noob building a new catalog // advice on where to start?

[Thomas-Moore-Creative]

If one was looking to start, for the first time, building a new catalog for a large dataset at their local HPC centre and they:

• had a basic understanding of Pangeo workflows - jupyter, python, xarray, dask, etc
• had watched @andersy005 's screen cast here: https://www.youtube.com/watch?v=zjjpByZ0nOk
• had access via the HPC filesystem to the large dataset (reanalyses & hincasts across many realms of a coupled GCM)
• had installed intake-esm in their working environment
• had read the docs here https://intake-esm.readthedocs.io/en/latest/supplemental-guide/faq.html
• had a goal of making it easier for local HPC users to load data from this large dataset into xarray objects

what would be the most useful next steps and approach that would help "teach through doing" in terms of building a working catalog?

I do note this repo https://github.com/NCAR/esm-collection-spec and that there are examples ?

How would someone who understands the intake-esm landscape advise a noob to proceed actioning or planning a new catalog based off a local HPC filesystem?

Thanks!

[andersy005]

@Thomas-Moore-Creative, sorry for not getting back to you sooner.

what would be the most useful next steps and approach that would help "teach through doing" in terms of building a working catalog?

Could you take a look at this package: https://ecgtools.readthedocs.io/en/latest/ when you get a moment? Even though the ecgtools documentation is lacking, you may find the provided examples to be useful.

You may find these blog posts to be useful, too:

• https://ncar.github.io/esds/posts/2021/intake-obs-cesm2le-comparison/
• https://ncar.github.io/esds/posts/2021/ecgtools-history-files-example/

[Thomas-Moore-Creative]

Thanks @andersy005! The reply is appreciated whenever it arrives.