Ensemble GDAS files

[MarcusGDaniels]

I want to run enkf runs using GFS v16 or UFS. As far as I can tell I cannot get my hands on the atmospheric data for each of the 80 runs. Does anyone know of how to get access to this data, or how I might reproduce these variant datasets?

My understanding is that two files are needed to create new initial conditions for GFS. A file of surface data of the form gdas.t18z.sfcf006.nc and also an atmospheric data file of the form gdas.t18z.atmf006.nc. However, I can only find one of them here:

https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/enkfgdas.20230321/06/atmos/mem001/

(These files turn over daily so the date 20230321 may need updating by the time this message is read.)

I understand the way these files are accessed in production is via HPSS, but I am outside.

Also I see here there were such files distributed at one time:

https://www.nco.ncep.noaa.gov/pmb/products/gfs/

..apparently no longer. Is there a way I could get these files? Even if it involves some administrative activity between organizations?
Another place I looked is https://noaa-gfs-bdp-pds.s3.amazonaws.com/index.html#enkfgdas.20230319/06/atmos/mem001/ but it has the same missing atmospheric files.

Thanks,

Marcus Daniels

[ShawnCebulaNOAA]

Hello Marcus,
Apologies for the delay in seeing this question, I didn't see there was a question here and didn't receive a notification. I'll take a look into this and check in with the team to see if they know, and we'll let you know!

Shawn Cebula, EPIC End User Support

[EdwardSnyder-NOAA]

Good afternoon Marcus,

From what I can tell the only way to access these atm files are via HPSS. According to the global-workflow docs which is used by the MRW App, the script can fetch the files for you if you set those variables in the config file assuming you have access to HPSS.

You mentioned you were outside of the production environment, but do you have access to any of the RDHPCS platforms (Jet, Hera, Orion, etc.)? I ask because you can have access to HPSS without having access to the production environment.

Lastly, is there a particular case you would like to run the MRW App with? Let us know the date(s), model run(s), and group(s) you are you interested in and perhaps we can provide the missing atm files for you.

My apologies again for the late response time.

Thanks,

Eddie Snyder, EPIC Advanced User Support (AUS) team

[MarcusGDaniels]

There isn't a particular date we want to study. Anything recent (preferably at C384) would be great. Our intention is to maintain/compute our own evolving set of files without having to download them all the time. If it isn't possible to match NOAA's operational set of files, we'd like to understand how to create something qualitatively similar. Is there a way to apply to get access to HPSS? (We don't have access.) We could send physical drives if that would help.

[EdwardSnyder-NOAA]

Thank you for that additional information! In order to get HPSS access, you or someone on your team will need to be a current user of a NOAA HPC compute resource like: RDHPC or WCOSS. It sounds like this isn't the case, so I'll reach out to some folks who are more familiar with the MRW App than myself for further guidance. And I'll get back to you shortly! We appreciate your patience!

Thanks,

Eddie Snyder, EPIC Advanced User Support (AUS) team

[MarcusGDaniels]

Btw, I can provide a login for rsync with 1 Gbit/sec bandwidth and the needed disk space. Then it would not be necessary to use S3 space/bandwidth and have the associated cost.

[EdwardSnyder-NOAA]

I'm not entirely sure if that can work. That question would be more suited for the HPSS helpdesk (rdhpcs.hpss.help@noaa.gov). According to our MRW Docs, David Huber (david.huber@noaa.gov) would be another good contact regarding getting initial conditions data if you don't have HPSS access.

The team got back to me and informed me that the atmfHHH.nc files were removed from storage because they were too large. In addition, since the application is going through rapid development, they cannot offer additional help desk support at this time. We apologize for the inconvenience.

Please let me know if you have any additional questions or comments.

Thanks,

Eddie Snyder, EPIC Advanced User Support (AUS) team

[MarcusGDaniels]

My guess is that these files are readily available on production parallel filesystems and could be rsync'ed offsite.