Current time settings not working for long duration (>600 years)

[OnnoEbbens]

I want to create a model which runs for over a thousand years but this does not seem possible at the moment. Currently we assume that ds.time is a pandas time range, see the nlmod.time.set_ds_time. Apparently pandas timestamps (or calculations with pandas timestamps) raise errors when using dates before the year 1678 or after the year 2262.

start = pd.Timestamp("1500-1-1")
perlen = [1000] * 10

ds = nlmod.time.set_ds_time(ds, start=start, perlen=perlen)

will raise this error:

OutOfBoundsDatetime: Cannot cast 1500-01-01 00:00:00 to unit='ns' without overflow.

This won't raise an error

start = pd.Timestamp("1700-1-1")
perlen = [1000] * 10

ds = nlmod.time.set_ds_time(ds, start=start, perlen=perlen)

The xarray documentation has a note on this:

When decoding/encoding datetimes for non-standard calendars or for dates before year 1678 or after year 2262, xarray uses the cftime library. It was previously packaged with the netcdf4-python package under the name netcdftime but is now distributed separately. cftime is an optional dependency of xarray.

and below that another note:

One unfortunate limitation of using datetime64[ns] is that it limits the native representation of dates to those that fall between the years 1678 and 2262. When a netCDF file contains dates outside of these bounds, dates will be returned as arrays of cftime.datetime objects and a CFTimeIndex will be used for indexing. CFTimeIndex enables a subset of the indexing functionality of a pandas.DatetimeIndex and is only fully compatible with the standalone version of cftime (not the version packaged with earlier versions netCDF4). See Non-standard calendars and dates outside the nanosecond-precision range for more information.

I will have a look into creating a cftime index.

Full error of the first code block:

---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\_libs\tslibs\timestamps.pyx in pandas._libs.tslibs.timestamps._Timestamp._as_creso()

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\_libs\tslibs\np_datetime.pyx in pandas._libs.tslibs.np_datetime.convert_reso()

OverflowError: value too large

The above exception was the direct cause of the following exception:

OutOfBoundsDatetime                       Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_5536\2908121607.py in <cell line: 5>()
      3 perlen = [1000] * 10
      4 
----> 5 ds = nlmod.time.set_ds_time(ds, start=start, perlen=perlen)
      6 ds.time

~\02_python\nlmod\nlmod\dims\time.py in set_ds_time(ds, start, time, steady, steady_start, time_units, perlen, nstp, tsmult)
    229     # convert time to Timestamps
    230     if isinstance(time[0], (int, np.integer, float)):
--> 231         time = pd.Timestamp(start) + pd.to_timedelta(time, time_units)
    232     elif isinstance(time[0], str):
    233         time = pd.to_datetime(time)

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\ops\common.py in new_method(self, other)
     79         other = item_from_zerodim(other)
     80 
---> 81         return method(self, other)
     82 
     83     return new_method

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\arraylike.py in __radd__(self, other)
    188     @unpack_zerodim_and_defer("__radd__")
    189     def __radd__(self, other):
--> 190         return self._arith_method(other, roperator.radd)
    191 
    192     @unpack_zerodim_and_defer("__sub__")

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\indexes\base.py in _arith_method(self, other, op)
   6815             return NotImplemented
   6816 
-> 6817         return super()._arith_method(other, op)
   6818 
   6819     @final

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\base.py in _arith_method(self, other, op)
   1346 
   1347         with np.errstate(all="ignore"):
-> 1348             result = ops.arithmetic_op(lvalues, rvalues, op)
   1349 
   1350         return self._construct_result(result, name=res_name)

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\ops\array_ops.py in arithmetic_op(left, right, op)
    222         # Timedelta/Timestamp and other custom scalars are included in the check
    223         # because numexpr will fail on it, see GH#31457
--> 224         res_values = op(left, right)
    225     else:
    226         # TODO we should handle EAs consistently and move this check before the if/else

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\roperator.py in radd(left, right)
      9 
     10 def radd(left, right):
---> 11     return right + left
     12 
     13 

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\arrays\datetimelike.py in __radd__(self, other)
   1367     def __radd__(self, other):
   1368         # alias for __add__
-> 1369         return self.__add__(other)
   1370 
   1371     @unpack_zerodim_and_defer("__sub__")

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\ops\common.py in new_method(self, other)
     79         other = item_from_zerodim(other)
     80 
---> 81         return method(self, other)
     82 
     83     return new_method

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\arrays\datetimelike.py in __add__(self, other)
   1325             result = self._add_offset(other)
   1326         elif isinstance(other, (datetime, np.datetime64)):
-> 1327             result = self._add_datetimelike_scalar(other)
   1328         elif isinstance(other, Period) and is_timedelta64_dtype(self.dtype):
   1329             result = self._add_period(other)

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\arrays\datetimelike.py in _add_datetimelike_scalar(self, other)
   1057 
   1058         other = Timestamp(other)
-> 1059         self, other = self._ensure_matching_resos(other)
   1060         self = cast("TimedeltaArray", self)
   1061 

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\core\arrays\datetimelike.py in _ensure_matching_resos(self, other)
   1977                 self = self.as_unit(other.unit)
   1978             else:
-> 1979                 other = other.as_unit(self.unit)
   1980         return self, other
   1981 

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\_libs\tslibs\timestamps.pyx in pandas._libs.tslibs.timestamps._Timestamp.as_unit()

c:\Users\oebbe\anaconda3\envs\nlmod\lib\site-packages\pandas\_libs\tslibs\timestamps.pyx in pandas._libs.tslibs.timestamps._Timestamp._as_creso()

OutOfBoundsDatetime: Cannot cast 1500-01-01 00:00:00 to unit='ns' without overflow.

[dbrakenhoff]

Perhaps you can take some inspiration from imod, from some random conversations at some point I know they've also dealt with this issue.

[OnnoEbbens]

Thanks, good idea. I had a look and it turns out they implemented the cftime option for this case.