Timestamp issue

[mzelinka]

Question criteria

• I added a descriptive title here.
• I searched the xCDAT GitHub Discussions to find a similar question and didn't find it.
• I searched the xCDAT documentation and/or Xarray documentation.

Describe your question

I am using open_mfdataset() to read in many monthly files as a single Dataset. This usually works fine, but in the case below, the resulting Dataset has odd timestamps: Jan of year 1, followed by January of year 1, then a jump to April Year 1, April Year 1, and so on. I tried various options with open_mfdataset, but the time decoding does not change.

Are there are any possible answers you came across?

No, but I assume it is a time bounds issue when reading in a lot of individual monthly files.

Minimal Complete Verifiable Example (MVCE)

path='/p/css03/esgf_publish/CMIP6/CFMIP/NASA-GISS/GISS-E2-1-G/amip-p4K/r1i1p1f1/CFmon/clisccp/gn/v20200102/'
# Just load in a single year to get the point across:
gg=glob.glob(path+'clisccp_CFmon_GISS-E2-1-G_amip-p4K_r1i1p1f1_gn_2013*nc')
ds = xc.open_mfdataset(gg)
print(ds.time)

Relevant log output

array([cftime.DatetimeNoLeap(2013, 1, 15, 22, 30, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2013, 1, 17, 10, 30, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2013, 4, 16, 22, 30, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2013, 4, 17, 10, 30, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2013, 7, 16, 22, 30, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2013, 7, 17, 10, 30, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2013, 10, 16, 22, 30, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2013, 10, 17, 10, 30, 0, 0, has_year_zero=True)],
      dtype=object)

Environment

xCDAT version 0.6.1

INSTALLED VERSIONS

commit: None
python: 3.10.9 | packaged by conda-forge | (main, Feb 2 2023, 20:20:04) [GCC 11.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.102.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.8.1

xarray: 2023.4.2
pandas: 1.5.1
numpy: 1.23.4
scipy: 1.8.1
netCDF4: 1.6.1
pydap: None
h5netcdf: None
h5py: 3.8.0
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: 1.4.1
PseudoNetCDF: None
iris: None
bottleneck: 1.3.5
dask: 2022.10.2
distributed: 2022.10.2
matplotlib: 3.6.2
cartopy: 0.21.0
seaborn: 0.13.1
numbagg: None
fsspec: 2022.11.0
cupy: None
pint: 0.20.1
sparse: 0.13.0
flox: None
numpy_groupies: None
setuptools: 65.5.1
pip: 22.3.1
conda: 23.9.0
pytest: None
mypy: None
IPython: 8.6.0
sphinx: None

Anything else we need to know?

No response

[pochedls]

Dataset issue? ncdump gives me similarly weird answers:

ncdump -c -t clisccp_CFmon_GISS-E2-1-G_amip-p4K_r1i1p1f1_gn_197901.nc | grep "time ="

time = "1979-01-17 10:30" ;

ncdump -c -t clisccp_CFmon_GISS-E2-1-G_amip-p4K_r1i1p1f1_gn_197902.nc | grep "time ="

time = "1979-01-15 22:30" ;

ncdump -c -t clisccp_CFmon_GISS-E2-1-G_amip-p4K_r1i1p1f1_gn_197903.nc | grep "time ="

time = "1979-01-17 10:30" ;

I get similarly wrong-looking time stamps when I look at the un-decoded times (e.g., the 197901 and 197903 files say the units are days since 1979-01-01 and the time value is 16.4375).

[mzelinka]

Good catch. I should have examined the actual time stamps in the netcdfs rather than just looking at their filenames. My current solution is to manually read in the individual files and concatenate them rather than relying on open_mfdataset, which produces years with only 8 months due to the erroneous time stamps. It looks like they may have only screwed this up for clisccp, so I then just assign the time coordinates from the tas field to the clisccp field. Like this:

Load in tas data with correct metadata from same exp:

path='/p/css03/esgf_publish/CMIP6/CFMIP/NASA-GISS/GISS-E2-1-G/amip-p4K/r1i1p1f1/Amon/tas/gn/v20200102/'
f = xc.open_mfdataset(path+'tas_Amon_GISS-E2-1-G_amip-p4K_r1i1p1f1_gn_197901-201412.nc')

Manually load in each month's clisccp field

path='/p/css03/esgf_publish/CMIP6/CFMIP/NASA-GISS/GISS-E2-1-G/amip-p4K/r1i1p1f1/CFmon/clisccp/gn/v20200102/'
gg=glob.glob(path+'clisccp_CFmon_GISS-E2-1-G_amip-p4K_r1i1p1f1_gn_*nc')
gg.sort()
ds0 = []    
for g in gg:
    ds0.append(xc.open_dataset(g))

Concatenate all months together

ds=xr.concat(ds0, 'time')

Give it the correct time stamps from the tas field above

this = ds.assign_coords(time=f.time)
print(this.time) # <-- this looks right

[pochedls]

This makes sense (and I think I've had to do this kind of thing in the past). I think the only thing I'd want to verify is that when you call xr.concat it concatenates things in the order you've appended them together (and not in order of the faulty time axis)...I think it is in the order you append them to the list...