Altering the DRS

[znichollscr]

tl;dr - I wonder if we should consider altering the DRS. #17 provides a concrete use case of where it doesn't really work. I add one more below. This discussion is for considering such an alteration and its implications.

---

The case of solar data, see #17

Details

The solar datasets are published in (effectively) 3 forms: daily, monthly and pre-industrial control.

Under the current DRS, the filenames would be:

• multiple_input4MIPs_solar_CMIP_SOLARIS-HEPPA-CMIP-4-3_gn.nc
• multiple_input4MIPs_solar_CMIP_SOLARIS-HEPPA-CMIP-4-3_gn_185001-202312.nc
• multiple_input4MIPs_solar_CMIP_SOLARIS-HEPPA-CMIP-4-3_gn_18500101-20231231.nc

I think it's fair to say that only the eagle-eyed can spot the difference between these, and that the filenames don't really give much information.

For reference, the full filepaths add more information, but still aren't exactly obvious

• input4MIPs/CMIP6Plus/CMIP/SOLARIS-HEPPA/SOLARIS-HEPPA-CMIP-4-3/atmos/fx/multiple/gn/v20240729/multiple_input4MIPs_solar_CMIP_SOLARIS-HEPPA-CMIP-4-3_gn.nc
• input4MIPs/CMIP6Plus/CMIP/SOLARIS-HEPPA/SOLARIS-HEPPA-CMIP-4-3/atmos/mon/multiple/gn/v20240729/multiple_input4MIPs_solar_CMIP_SOLARIS-HEPPA-CMIP-4-3_gn_185001-202312.nc
• input4MIPs/CMIP6Plus/CMIP/SOLARIS-HEPPA/SOLARIS-HEPPA-CMIP-4-3/atmos/day/multiple/gn/v20240729/multiple_input4MIPs_solar_CMIP_SOLARIS-HEPPA-CMIP-4-3_gn_18500101-20231231.nc

The case of scenario data in CMIP6

Details

The key issue here is that the filename contains no information about what experiment is being target by the file. The only way to make a distinction is to supply the data under a different source ID. Using a difference source ID feels like the wrong solution to me (the same group is providing the data i.e. it has the same source hence the source ID should be the same) and I doubt it is an obvious convention for the modelling groups to parse either.

For example, in CMIP6, scenario data was submitted under a different source ID for each scenario. To me, this doesn't make much sense, the same groups were supplying the data i.e. the source was the same, hence the source ID should have been the same.

A potential solution

Details

One way out of this would be to alter the DRS, making it more like the model output DRS in many ways. The current DRS is:

{
    "directory_path_example":"input4MIPs/CMIP6Plus/CMIP/PCMDI/PCMDI-AMIP-1-1-9/ocean/mon/tos/gn/v20230512/",
    "directory_path_template":"<activity_id>/<mip_era>/<target_mip>/<institution_id>/<source_id>/<realm>/<frequency>/<variable_id>/<grid_label>/v<version>",
    "filename_example":"tos_input4MIPs_SSTsAndSeaIce_CMIP_PCMDI-AMIP-1-1-9_gn_187001-202212.nc",
    "filename_template":"<variable_id>_<activity_id>_<dataset_category>_<target_mip>_<source_id>_<grid_label>[_<time_range>].nc"
}

I would suggest to alter the DRS, so it becomes:

{
    "directory_path_example":"input4MIPs/CMIP6Plus/CMIP/historical/PCMDI/PCMDI-AMIP-1-1-9/ocean/mon/tos/gn/main/v20230512/",
    "directory_path_template":"<activity_id>/<mip_era>/<target_mip>/<target_experiment>/<institution_id>/<source_id>/<realm>/<frequency>/<variable_id>/<grid_label>/<variant_label>/v<version>",
    "filename_example":"tos_input4MIPs_SSTsAndSeaIce_CMIP_historical_PCMDI-AMIP-1-1-9_gn_main_187001-202212.nc",
    "filename_template":"<variable_id>_<activity_id>_<dataset_category>_<target_mip>_<target_experiment>_<source_id>_<grid_label>_<variant_label>[_<time_range>].nc"
}

So, we basically add two keys: "target_experiment" and "variant_label".

"target_experiment" would identify the experiment being targeted by the dataset, in the same way that "target_mip" identifies the target MIP. This would mostly be "historical", but would allow a single source ID to provide data for multiple experiments (helpful for the scenario data production discussed above). As with target MIP, the data could be used for other MIPs and experiments, but this would identify the experiment that the data producer had in mind when they produced the data.

"variant_label" would identify the variant of the data. All data providers would have to provide a "main" variant, and only "main" variants would be allowed in CMIP stuff (at least to start with while we don't have forcing uncertainty MIPs). However, this extra piece of information would allow us to clearly distinguish between variations coming from the same data source, something which will be increasingly needed as we move towards exploring uncertainty in forcings.

How would this solution work in our two cases?

Solar data

Details

The added fields make it easy to see a) which experiment is being targeted, which makes it way easier to pick out the pi-control file and b) the difference between the sensitivity case and the reference/main case.

Main files become:

• input4MIPs/CMIP6Plus/CMIP/piControl/SOLARIS-HEPPA/SOLARIS-HEPPA-CMIP-4-3/atmos/fx/multiple/gn/main/v20240729/multiple_input4MIPs_solar_CMIP_piControl_SOLARIS-HEPPA-CMIP-4-3_gn_main.nc
• input4MIPs/CMIP6Plus/CMIP/historical/SOLARIS-HEPPA/SOLARIS-HEPPA-CMIP-4-3/atmos/mon/multiple/gn/main/v20240729/multiple_input4MIPs_solar_CMIP_historical_SOLARIS-HEPPA-CMIP-4-3_gn_main_185001-202312.nc
• input4MIPs/CMIP6Plus/CMIP/historical/SOLARIS-HEPPA/SOLARIS-HEPPA-CMIP-4-3/atmos/day/multiple/gn/main/v20240729/multiple_input4MIPs_solar_CMIP_historical_SOLARIS-HEPPA-CMIP-4-3_gn_main_18500101-20231231.nc

Sensitivity files become:

• input4MIPs/CMIP6Plus/Prototype/piControl/SOLARIS-HEPPA/SOLARIS-HEPPA-CMIP-4-3/atmos/fx/multiple/gn/sensitivity/v20240729/multiple_input4MIPs_solar_Prototype_piControl_SOLARIS-HEPPA-CMIP-4-3_gn_sensitivity.nc
• input4MIPs/CMIP6Plus/Prototype/historical/SOLARIS-HEPPA/SOLARIS-HEPPA-CMIP-4-3/atmos/mon/multiple/gn/sensitivity/v20240729/multiple_input4MIPs_solar_Prototype_historical_SOLARIS-HEPPA-CMIP-4-3_gn_sensitivity_185001-202312.nc
• input4MIPs/CMIP6Plus/Prototype/historical/SOLARIS-HEPPA/SOLARIS-HEPPA-CMIP-4-3/atmos/day/multiple/gn/sensitivity/v20240729/multiple_input4MIPs_solar_Prototype_historical_SOLARIS-HEPPA-CMIP-4-3_gn_sensitivity_18500101-20231231.nc

I, at least, find this much easier to understand (I would probably also flip the order of some of the keys around so we don't have files starting with the confusing name 'multiple' and would add frequency information into the filename, perhaps removing some other stuff, but one thing at a time).

Scenario data

Details

The introduction of the target experiment makes it much easier to pick out what is actually varying in these files, while giving us the chance to use the same source ID for files which came from the same source.

• input4MIPs/CMIP6Plus/ScenarioMIP/ssp-new-1-15/CR/CR-1-1-3/atmos/day/mole-fraction-of-carbon-dioxide-in-air/gn/main/v20250201/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_ScenarioMIP_ssp-new-1-15_CR-1-1-_gn_main_20240101-22991231.nc
• input4MIPs/CMIP6Plus/ScenarioMIP/ssp-new-2-43/CR/CR-1-1-3/atmos/day/mole-fraction-of-carbon-dioxide-in-air/gn/main/v20250201/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_ScenarioMIP_ssp-new-2-43_CR-1-1-_gn_main_20240101-22991231.nc
• input4MIPs/CMIP6Plus/ScenarioMIP/ssp-new-1-15/CR/CR-1-1-3/atmos/day/mole-fraction-of-methane-in-air/gn/main/v20250201/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_ScenarioMIP_ssp-new-1-15_CR-1-1-_gn_main_20240101-22991231.nc
• input4MIPs/CMIP6Plus/ScenarioMIP/ssp-new-2-43/CR/CR-1-1-3/atmos/day/mole-fraction-of-methane-in-air/gn/main/v20250201/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_ScenarioMIP_ssp-new-2-43_CR-1-1-_gn_main_20240101-22991231.nc

etc. for other variables

Some thoughts on the practical realities

Details

There are clearly tradeoffs here. I, personally, find the argument, "It's how we did it before", incredibly weak. We've done lots of things before and found lots of problems, not fixing the problems doesn't make much sense to me.

Having said that, I am aware that we're on a fast-track, so we need to help modelling groups as much possible. Hence, in the spirit of compromise, a suggestion:

• at the drop-in session, we get as much feedback as possible from modelling groups about the extent to which file naming matters to them. The specific question would be, "Would a clean dataset be of interest to you, or do you literally want it identical to CMIP6, with all the confusion that format creates?"
• if a lot of groups want it the same as CMIP6, I am happy to commit to building a tool that will ensure the consistency with CMIP6.
  • I would note that this tool will be annoying. We're already seeing groups want to tinker with things (e.g. land-use want to change the name of their files), and this tool would forbid any of that (if it actually should be identical, then let's make it identical).
• if groups would prefer something sane and clean, then we can stop worrying about the CMIP6 consistency and just create a clean dataset
• if we get a split, then we are in the hardest spot. One option would be to distribute both variations through ESGF. The challenge here would be communication so that groups don't get confused. Having said that, if the data is the same, confusion doesn't matter that much, whichever dataset the groups use, they will get the correct data.

I'm sure I haven't thought through everything, but here is something to get the conversation started.

[znichollscr]

Copying some of #17 (comment) here to get things moving

...without standard naming and file formats, while modeling groups dealt with it, it was a mess - a manageable mess, but it could have been better. There is a balance between changing what works (and what modeling groups have already written code to read and use), and cleaning things up to make them far more interpretable and useable, as consistency is more uniform across the disparate data products. Having said that, it is a balance, and I'd be keen to hear from @vnaik60 on this one.

On the file naming and the DRS (directory structure), the key here is that we need uniquely named files, as having two identically named files that are downloaded via a web browser will only add a number to their name.

It would be good to push the file formatting as far as we can to conform to standards, and to simplify and standardize these as much as possible - we do this with the model data, so it would be ideal to follow this with forcing data too.

[vnaik60]

I like your suggestion of including a "variant_label" to identify whether the dataset is "main" or a "sensitivity". This may be a longer term solution for "prototype" datasets. Let see what @durack1 says after vacation.

Regarding "target_experiment", will this also handle the different IAMs that may have generated the different scenarios, provided this is indeed happening?

"I am aware that we're on a fast-track, so we need to help modelling groups as much possible. " and in the process we should try NOT to make the lives of data providers complicated :-). A win-win for both modelers and data providers should be our goal. I understand that this is easier said than done but lets try.

Regarding the importance of filename for modeling groups, my 2cents is that I dont care what the filename is - yes, it would probably require me to change my processing scripts but that is not a major issue. The major issue for me is what the file contains and that the name conveys easily what it contains. Given the advances since CMIP6, I cannot expect the variables to be written in exact the same way as in CMIP6, however, as long as the name makes it clear what the file contains, its all good to me. I would err towards keeping it all simple and minimizing file rewrites as much as possible. Of course, this is just one modeler speaking and others may have a totally opposite view. One common view of course is "get me the data as soon as possible" :-).

[znichollscr]

What is the reasoning for using "multiple" as a variable_id for solar forcing? Is it because it contains multiple variables?

Correct. That is what the variable ID is meant to be according to the DRS as it was defined in CMIP6 (https://docs.google.com/document/d/1pU9IiJvPJwRvIgVaSDdJ4O0Jeorv_2ekEtted34K9cA/edit?pli=1#bookmark=id.j35muvgap62e).

In the case when there are multiple variables in a file and the “multiple” variable_id is being used, to prevent filename clashes, an additional <variable_id>-[multiple_file_indentifier] can be appended.

The name as it was written was actually invalid according to the CMIP6 DRS. The "multiple" variable ID itself is a bit of a hack (in my opinion) because the entire ESGF architecture is built on the idea of, "one variable per file".

To me the CMIP6 filename provided more clarity on the dataset without looking at the metadata

Yes I agree with this. My true preference for filename template would actually be:

<dataset_category>_<activity_id>_<target_mip>_<target_experiment>_<variable_id>_<grid_label>_<source_id>_<variant_label>[_<time_range>].nc

which would result in the following filenames

• solar_input4MIPs_CMIP_piControl_multiple_gn_SOLARIS-HEPPA-CMIP-4-3_main.nc

• solar_input4MIPs_CMIP_historical_multiple_gn_SOLARIS-HEPPA-CMIP-4-3_main_185001-202312.nc

• solar_input4MIPs_CMIP_historical_multiple_gn_SOLARIS-HEPPA-CMIP-4-3_main_18500101-20231231.nc

• solar_input4MIPs_Prototype_piControl_multiple_gn_SOLARIS-HEPPA-CMIP-4-3_sensitivity.nc

• solar_input4MIPs_Prototype_historical_multiple_gn_SOLARIS-HEPPA-CMIP-4-3_sensitivity_185001-202312.nc

• solar_input4MIPs_Prototype_historical_multiple_gn_SOLARIS-HEPPA-CMIP-4-3_sensitivity_18500101-20231231.nc

• GHGConcentrations_input4MIPs_ScenarioMIP_ssp-new-1-15_mole-fraction-of-carbon-dioxide-in-air_gn_main_20240101-22991231.nc

• GHGConcentrations_input4MIPs_ScenarioMIP_ssp-new-2-43_mole-fraction-of-carbon-dioxide-in-air_gn_main_20240101-22991231.nc

• GHGConcentrations_input4MIPs_ScenarioMIP_ssp-new-1-15_mole-fraction-of-methane_gn_main_20240101-22991231.nc

• GHGConcentrations_input4MIPs_ScenarioMIP_ssp-new-2-43_mole-fraction-of-methane_gn_main_20240101-22991231.nc

To make this even clearer, we could use "multiple-variables" rather than just "multiple" in the filename. That would make it much more obvious what the file contains just by looking at the name. So we'd have e.g.

• solar_input4MIPs_CMIP_piControl_multiple-variables_gn_SOLARIS-HEPPA-CMIP-4-3_main.nc

rather than

• solar_input4MIPs_CMIP_piControl_multiple_gn_SOLARIS-HEPPA-CMIP-4-3_main.nc

"I am aware that we're on a fast-track, so we need to help modelling groups as much possible. " and in the process we should try NOT to make the lives of data providers complicated :-).

I think the data provider side is basically solved. With input4MIPs validation, Paul and I can receive a file with any name, and re-writing it with the correct one is trivial. So I think, whatever we decide here, it won't cause any extra work for data providers (if they want, they can still name files however they want and we can sort it out without any real work).

Regarding "target_experiment", will this also handle the different IAMs that may have generated the different scenarios, provided this is indeed happening?

I think so. I see two relatively simple solutions.

a) The IAM that generated the scenario goese in the "target_experiment". We/the ScenarioMIP team would pick this if we/they think the IAM is really a key part of defining the experiment and should be part of the name.
b) The IAM goes in the source ID of the emissions files associated with the scenario. This would mean that the IAM would be lost from downstream files e.g. GHG concentrations, ozone etc. I think that would also be ok though. The IAM doesn't produce the concentrations/ozone/etc., hence it isn't the source of those files and it's fine if the IAM is only recognised as the source of the emissions. (I would also not hate this, people might stop saying that "IAMs produced CMIP6 greenhouse gas concentrations".)

[vnaik60]

Thanks @znichollscr!

To make this even clearer, we could use "multiple-variables" rather than just "multiple" in the filename. That would make it much more obvious what the file contains just by looking at the name. So we'd have e.g.

• solar_input4MIPs_CMIP_piControl_multiple-variables_gn_SOLARIS-HEPPA-CMIP-4-3_main.nc
  rather than
• solar_input4MIPs_CMIP_piControl_multiple_gn_SOLARIS-HEPPA-CMIP-4-3_main.nc

I like this option - clean and clear!

So I think, whatever we decide here, it won't cause any extra work for data providers (if they want, they can still name files however they want and we can sort it out without any real work.

It would be better if data providers named the files the way ESGF wants as this would avoid any confusions if the same dataset is provided on other websites (like the solar forcing). See comment #17 (comment)

[znichollscr]

It would be better if data providers named the files the way ESGF wants as this would avoid any confusions if the same dataset is provided on other websites (like the solar forcing). See comment #17 (comment)

100%. I'm just wary of adding work to data provider's plates that they don't/won't/can't handle. Let's see, maybe they make it all happen without any headaches.

[vnaik60]

Totally and this goes against what I said about making the data providers' lives less complicated. Agree, let's see how the data providers take this.

[znichollscr]

As a note, this may also affect the volcanic data.

[vnaik60]

and possibly land-use data.

[znichollscr]

Yes, they both have slightly tricky use cases with their multiple variables. Looks like this "-[multiple_file_indentifier]" thing may be more important than I thought.

[durack1]

@znichollscr I like where you are going with this. As we're in prototype mode until January 2025, unlike with CMIP6 we have some time to work through this logic and thanks to your validator tool, make changes quickly and easily.

And a couple of comments (responding to various other threads too)

• We really should not produce data and host it in two places with subtle variations in the file contents/filenames/metadata - this is a recipe for very complex communications. Rather we stop publishing data outside of the ESGF publication, and make the publication process as quick and seamless as possible, so a dataset is created, and it's published within a day or two..
• I like the discussion above, but what projects and what MIPs/experiments are we serving? What is the registry for these and who manages it? In the case of RESCUE, we could register this information into this project, but will there be a RESCUE_CVs generated somewhere (using the CMIP6Plus_CVs template), and if yes, who is managing that info?

[vnaik60]

Is there a broader question - MIPs are not being endorsed by CMIP in this phase, though they are encouraged to register and follow best practices?

[znichollscr]

Yes that definitely plays in too. RESCUE might be one step further, as I don't even know if it's being encouraged to register (although I've only heard this through the grapevine and have never really understood the reasons against treating it like any other MIP).

[durack1]

Either way, I don't think it has any real implications for the discussion here

We need to calibrate. If we don't have a managed MIP:experiment list, then how will we validate our <target_mip>_<target_experiment> information? If we keep this simple (in this repo) then we are just serving the needs of the projects we are involved with CMIP6Plus:DECK in prep for CMIP7:DECK, rather than coming up with a solution that requires us to be custodians over a far less defined assortment of modeling activities. Trying to keep this as simple, manageable and deliverable to start is my immediate goal

[znichollscr]

We need to calibrate

Ok, trying again: I see two options:

a) RESCUE has a CVs attached to it that follows the template set by input4MIPs (whether that's me doing the work or someone else, doesn't mater for now). In this case, RESCUE is following the template, so if it works for input4MIPs, it will work for RESCUE. Implications for this discussion: none, because whatever we decide here, RESCUE will just follow.
b) RESCUE doesn't have CVs attached, in which case we can't validate the files. Implications for this discussion: none, because we won't be doing file validation anyway.

Maybe think about it the other way around. We should define something here that works for us. If RESCUE can use that, great. If they can't, ok, not the end of the world. In other words, let's make something work for CMIP6Plus and CMIP7, then see if we can apply it elsewhere, rather than trying to derive the general solution now without a clear use case.

Answering this question directly

If we don't have a managed MIP:experiment list, then how will we validate our <target_mip>_<target_experiment> information?

Correct, if you don't have a managed list, you obviously can't do validation. We will need a managed MIP: experiment list for CMIP6Plus/CMIP7. As I understand it, that is the point of https://github.com/WCRP-CMIP/CMIP6Plus_CVs and that will handle the CMIP6Plus/CMIP7 use case. (As I said above, I think that's all we should worry about right now. If RESCUE can fit this pattern, great. If it can't, we'll figure something else out.)

[vnaik60]

+1 for

In other words, let's make something work for CMIP6Plus and CMIP7, then see if we can apply it elsewhere, rather than trying to derive the general solution now without a clear use case.