Issue with reading JWST MRS cubes

[kykyelric]

Hi all, I ran into an issue with reading JWST MRS cubes. Is this a known issue or have others run into this problem before?

spec1d = Spectrum1D.read('file_s3d.fits')
ImportError: cannot import name 'convert_fitsrec_to_array_in_tree' from 'stdatamodels.util'

specutils version: 1.15.1.dev6+g28b0cdb
stdatamodels version: 1.10.2.dev5+g0326fd4
python version: 3.12.2

[rosteen]

Hmm, I haven't run into this, and I was unable to reproduce locally. What happens if you just try from stdatamodels.util import convert_fitsrec_to_array_in_tree in python? It seems like it might be a problem with your stdatamodels installation.

[kykyelric]

Hmm, I haven't run into this, and I was unable to reproduce locally. What happens if you just try from stdatamodels.util import convert_fitsrec_to_array_in_tree in python? It seems like it might be a problem with your stdatamodels installation.

This imported just fine when I run python in terminal. Strange.

[rosteen]

That is odd. I would probably try reinstalling specutils and see if that fixes the issue - I don't know if there's some way it could have ended up looking for stdatamodels in the wrong place, but you never know.

[kykyelric]

Unfortunately that didn't work. I found a way to accomplish what I needed to via a different package than specutils, but I hope this issue can get resolved later.

[rosteen]

What is the full traceback leading to that error? That would help track this down if it's something on our end. As-is, without being able to reproduce the error with the package versions you posted, I can't fix it.

[kykyelric]

ImportError                               Traceback (most recent call last)
Cell In[77], line 14
     10 q3di = q3din(name, 'test', datext=1, varext=2, dqext=3, 
     11              outdir='F20551-4250/s3d_files/test/', 
     12              logfile='F20551-4250/s3d_files/test/ch'+str(i+1)+'-fitlog.txt')
     13 wav = q3di.load_cube().wave
---> 14 spec1d = Spectrum1D.read(name)
     15 #wav = np.array(spec1d.spectral_axis.value)
     16 corr_interp = interp1d(corrx, corry)(wav)

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/astropy/nddata/mixins/ndio.py:59, in NDDataRead.__call__(self, *args, **kwargs)
     58 def __call__(self, *args, **kwargs):
---> 59     return self.registry.read(self._cls, *args, **kwargs)

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/astropy/io/registry/core.py:218, in UnifiedInputRegistry.read(self, cls, format, cache, *args, **kwargs)
    213     format = self._get_valid_format(
    214         "read", cls, path, fileobj, args, kwargs
    215     )
    217 reader = self.get_reader(format, cls)
--> 218 data = reader(*args, **kwargs)
    220 if not isinstance(data, cls):
    221     # User has read with a subclass where only the parent class is
    222     # registered.  This returns the parent class, so try coercing
    223     # to desired subclass.
    224     try:

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/specutils/io/default_loaders/jwst_reader.py:539, in jwst_s3d_single_loader(filename, **kwargs)
    521 @data_loader(
    522     "JWST s3d", identifier=identify_jwst_s3d_fits, dtype=Spectrum1D,
    523     extensions=['fits'], priority=10,
    524 )
    525 def jwst_s3d_single_loader(filename, **kwargs):
    526     """
    527     Loader for JWST s3d 3D rectified spectral data in FITS format.
    528 
   (...)
    537         The spectrum contained in the file.
    538     """
--> 539     spectrum_list = _jwst_s3d_loader(filename, **kwargs)
    540     if len(spectrum_list) == 1:
    541         return spectrum_list[0]

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/specutils/io/default_loaders/jwst_reader.py:563, in _jwst_s3d_loader(filename, **kwargs)
    549 def _jwst_s3d_loader(filename, **kwargs):
    550     """
    551     Loader for JWST s3d 3D rectified spectral data in FITS format.
    552 
   (...)
    561         The spectra contained in the file.
    562     """
--> 563     from stdatamodels import asdf_in_fits
    565     spectra = []
    567     # Get a list of GWCS objects from the slits

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/stdatamodels/__init__.py:1
----> 1 from .model_base import DataModel
      2 from . import _version
      5 __all__ = ['DataModel', '__version__']

File ~/anaconda3/envs/correct_cube/lib/python3.12/site-packages/stdatamodels/model_base.py:28
     26 from . import properties
     27 from . import schema as mschema
---> 28 from . import validate
     29 from .util import get_envar_as_boolean, remove_none_from_tree
     30 from . import s3_utils

ImportError: cannot import name 'convert_fitsrec_to_array_in_tree' from 'stdatamodels.util' (/home/kydan/anaconda3/envs/correct_cube/lib/python3.12/site-packages/stdatamodels/util.py)

[rosteen]

Hmm, looks like this could be ASDF-related. Could you tell me what the output of pip list | grep asdf is for you?

[kykyelric]

asdf 3.2.0
asdf-astropy 0.5.0
asdf-coordinates-schemas 0.2.0
asdf_standard 1.1.1
asdf_transform_schemas 0.5.0
asdf-unit-schemas 0.1.0
asdf-wcs-schemas 0.3.0

[rosteen]

Hm, some of those are out of date compared to what I have. Have you tried installing specutils in a fresh environment to ensure you have up-to-date/compatible package versions? I downgraded my asdf packages (and gwcs for compatibility) to what you have there and still couldn't reproduce, so I think new environment is probably the best way to solve this if you haven't tried it already. It seems to be an upstream package thing, not specutils, but I remain stumped on what package or why.

[kykyelric]

I am already using an isolated conda environment. I can export an environment file if you'd like to try it?

[rosteen]

I am already using an isolated conda environment. I can export an environment file if you'd like to try it?

Sure. I guess my question was more "when was the environment created", in case it's older and something didn't get updated that was needed for compatibility. But yes, the env export or pip list would help.

[kykyelric]

name: correct_cube
channels:

• anaconda
• conda-forge
• defaults
  dependencies:
• _libgcc_mutex=0.1=conda_forge
• _openmp_mutex=4.5=2_gnu
• affine=2.4.0=pyhd8ed1ab_0
• aom=3.8.1=h59595ed_0
• arrow=1.2.3=py312h06a4308_1
• asdf-astropy=0.5.0=pyhd8ed1ab_0
• asdf-coordinates-schemas=0.2.0=pyhd8ed1ab_1
• asdf-unit-schemas=0.1.0=pyhd8ed1ab_0
• asdf-wcs-schemas=0.3.0=pyhd8ed1ab_0
• astropy=5.3.4=py312ha883a20_0
• asttokens=2.0.5=pyhd3eb1b0_0
• attrs=23.2.0=pyh71513ae_0
• azure-core-cpp=1.10.3=h91d86a7_1
• azure-storage-blobs-cpp=12.10.0=h00ab1b0_0
• azure-storage-common-cpp=12.5.0=hb858b4b_2
• backcall=0.2.0=pyhd3eb1b0_0
• blas=1.0=mkl
• blosc=1.21.5=h0f2a231_0
• bottleneck=1.3.7=py312hc7c0aa3_1
• brotli=1.1.0=hd590300_1
• brotli-bin=1.1.0=hd590300_1
• brunsli=0.1=h9c3ff4c_0
• bzip2=1.0.8=h7b6447c_0
• c-ares=1.27.0=hd590300_0
• c-blosc2=2.13.2=hb4ffafa_0
• ca-certificates=2024.2.2=hbcca054_0
• cairo=1.18.0=h3faef2a_0
• certifi=2024.2.2=pyhd8ed1ab_0
• cfitsio=4.3.1=hbdc6101_0
• charls=2.4.2=h59595ed_0
• click=8.1.7=unix_pyh707e725_0
• click-plugins=1.1.1=py_0
• cligj=0.7.2=pyhd8ed1ab_1
• colorama=0.4.6=pyhd8ed1ab_0
• comm=0.1.2=py312h06a4308_0
• contourpy=1.2.0=py312h8572e83_0
• cycler=0.12.1=pyhd8ed1ab_0
• dav1d=1.2.1=hd590300_0
• debugpy=1.6.7=py312h6a678d5_0
• decorator=5.1.1=pyhd3eb1b0_0
• entrypoints=0.4=py312h06a4308_0
• executing=0.8.3=pyhd3eb1b0_0
• expat=2.5.0=hcb278e6_1
• font-ttf-dejavu-sans-mono=2.37=hab24e00_0
• font-ttf-inconsolata=3.000=h77eed37_0
• font-ttf-source-code-pro=2.038=h77eed37_0
• font-ttf-ubuntu=0.83=h77eed37_1
• fontconfig=2.14.2=h14ed4e7_0
• fonts-conda-ecosystem=1=0
• fonts-conda-forge=1=0
• fonttools=4.49.0=py312h98912ed_0
• freetype=2.12.1=h267a509_2
• freexl=2.0.0=h743c826_0
• geos=3.12.1=h59595ed_0
• geotiff=1.7.1=h6b2125f_15
• gettext=0.21.1=h27087fc_0
• giflib=5.2.1=h0b41bf4_3
• gwcs=0.20.0=pyhd8ed1ab_0
• hdf4=4.2.15=h2a13503_7
• hdf5=1.14.3=nompi_h4f84152_100
• icu=73.2=h59595ed_0
• idna=3.4=py312h06a4308_0
• imagecodecs=2024.1.1=py312ha6884dc_0
• imageio=2.34.0=pyh4b66e23_0
• importlib-metadata=7.0.1=pyha770c72_0
• importlib-resources=6.1.1=pyhd8ed1ab_0
• importlib_resources=6.1.1=pyhd8ed1ab_0
• intel-openmp=2023.1.0=hdb19cb5_46306
• ipykernel=6.28.0=py312h06a4308_0
• ipython=8.15.0=py312h06a4308_0
• isoduration=20.11.0=pyhd8ed1ab_0
• jedi=0.18.1=py312h06a4308_1
• jmespath=1.0.1=pyhd8ed1ab_0
• joblib=1.3.2=pyhd8ed1ab_0
• json-c=0.17=h7ab15ed_0
• jupyter_client=7.1.2=pyhd3eb1b0_0
• jupyter_core=5.5.0=py312h06a4308_0
• jxrlib=1.1=hd590300_3
• kealib=1.5.3=h2f55d51_0
• keyutils=1.6.1=h166bdaf_0
• kiwisolver=1.4.5=py312h8572e83_1
• krb5=1.21.2=h659d440_0
• lazy_loader=0.3=pyhd8ed1ab_0
• lcms2=2.16=hb7c19ff_0
• ld_impl_linux-64=2.38=h1181459_1
• lerc=4.0.0=h27087fc_0
• libabseil=20230802.1=cxx17_h59595ed_0
• libaec=1.1.2=h59595ed_1
• libarchive=3.7.2=h2aa1ff5_1
• libavif16=1.0.4=h1dcd450_0
• libblas=3.9.0=1_h86c2bf4_netlib
• libboost-headers=1.84.0=ha770c72_1
• libbrotlicommon=1.1.0=hd590300_1
• libbrotlidec=1.1.0=hd590300_1
• libbrotlienc=1.1.0=hd590300_1
• libcblas=3.9.0=5_h92ddd45_netlib
• libcrc32c=1.1.2=h9c3ff4c_0
• libcurl=8.5.0=hca28451_0
• libdeflate=1.19=hd590300_0
• libedit=3.1.20191231=he28a2e2_2
• libev=4.33=hd590300_2
• libexpat=2.5.0=hcb278e6_1
• libffi=3.4.4=h6a678d5_0
• libgcc-ng=13.2.0=h807b86a_5
• libgdal=3.8.4=h9323651_0
• libgfortran-ng=13.2.0=h69a702a_5
• libgfortran5=13.2.0=ha4646dd_5
• libglib=2.78.4=h783c2da_0
• libgomp=13.2.0=h807b86a_5
• libgoogle-cloud=2.17.0=h31df0ca_2
• libgoogle-cloud-storage=2.17.0=hc7a4891_2
• libgrpc=1.60.1=h74775cd_0
• libiconv=1.17=hd590300_2
• libjpeg-turbo=3.0.0=hd590300_1
• libkml=1.3.0=h01aab08_1018
• liblapack=3.9.0=5_h92ddd45_netlib
• libnetcdf=4.9.2=nompi_h9612171_113
• libnghttp2=1.58.0=h47da74e_1
• libnsl=2.0.1=hd590300_0
• libpng=1.6.42=h2797004_0
• libpq=16.2=h33b98f1_0
• libprotobuf=4.25.1=hf27288f_2
• libre2-11=2023.06.02=h7a70373_0
• librttopo=1.1.0=h8917695_15
• libsodium=1.0.18=h7b6447c_0
• libspatialite=5.1.0=h7bd4643_4
• libsqlite=3.45.1=h2797004_0
• libssh2=1.11.0=h0841786_0
• libstdcxx-ng=13.2.0=h7e041cc_5
• libtiff=4.6.0=ha9c0a0a_2
• libuuid=2.38.1=h0b41bf4_0
• libwebp-base=1.3.2=hd590300_0
• libxcb=1.15=h0b41bf4_0
• libxcrypt=4.4.36=hd590300_1
• libxml2=2.12.5=h232c23b_0
• libzip=1.10.1=h2629f0a_3
• libzlib=1.2.13=hd590300_5
• libzopfli=1.0.3=h9c3ff4c_0
• lz4-c=1.9.4=hcb278e6_0
• lzo=2.10=h516909a_1000
• matplotlib-base=3.8.2=py312he5832f3_0
• matplotlib-inline=0.1.6=py312h06a4308_0
• minizip=4.0.4=h0ab5242_0
• mkl=2023.1.0=h213fc3f_46344
• mkl-service=2.4.0=py312h5eee18b_1
• mkl_fft=1.3.8=py312h5eee18b_0
• mkl_random=1.2.4=py312hdb19cb5_0
• mpl_animators=1.1.1=pyhd8ed1ab_0
• munkres=1.1.4=pyh9f0ad1d_0
• ncurses=6.4=h6a678d5_0
• ndcube=2.2.0=pyhd8ed1ab_0
• nest-asyncio=1.5.6=py312h06a4308_0
• networkx=3.2.1=pyhd8ed1ab_0
• nspr=4.35=h27087fc_0
• nss=3.98=h1d7d5a4_0
• numpy=1.26.3=py312hc5e2394_0
• numpy-base=1.26.3=py312h0da6c21_0
• openjpeg=2.5.0=h488ebb8_3
• openssl=3.3.0=h4ab18f5_2
• packaging=23.1=py312h06a4308_0
• pandas=2.2.2=py312h1d6d2e6_1
• parso=0.8.3=pyhd3eb1b0_0
• patsy=0.5.6=pyhd8ed1ab_0
• pcre2=10.42=hcad00b1_0
• pexpect=4.8.0=pyhd3eb1b0_3
• photutils=1.10.0=py312hc7c0aa3_1
• pickleshare=0.7.5=pyhd3eb1b0_1003
• pillow=10.2.0=py312hf3581a9_0
• pip=23.3.1=py312h06a4308_0
• pixman=0.43.2=h59595ed_0
• platformdirs=3.10.0=py312h06a4308_0
• poppler=24.02.0=h590f24d_0
• poppler-data=0.4.12=hd8ed1ab_0
• postgresql=16.2=h7387d8b_0
• proj=9.3.1=h1d62c97_0
• prompt-toolkit=3.0.36=py312h06a4308_0
• psutil=5.9.0=py312h5eee18b_0
• pthread-stubs=0.4=h36c2ea0_1001
• ptyprocess=0.7.0=pyhd3eb1b0_2
• pure_eval=0.2.2=pyhd3eb1b0_0
• pyerfa=2.0.0=py312h5eee18b_0
• pygments=2.15.1=py312h06a4308_1
• pyparsing=3.1.1=pyhd8ed1ab_0
• python=3.12.2=hab00c5b_0_cpython
• python-dateutil=2.8.2=pyhd8ed1ab_0
• python-tzdata=2024.1=pyhd8ed1ab_0
• python_abi=3.12=4_cp312
• pytz=2024.1=pyhd8ed1ab_0
• pywavelets=1.4.1=py312hc7c0aa3_1
• pyyaml=6.0.1=py312h5eee18b_0
• pyzmq=25.1.0=py312h6a678d5_0
• rasterio=1.3.9=py312h26ef92c_2
• rav1e=0.6.6=he8a937b_2
• re2=2023.06.02=h2873b5e_0
• readline=8.2=h5eee18b_0
• scikit-image=0.22.0=py312hfb8ada1_2
• scikit-learn=1.4.0=py312h394d371_0
• scipy=1.12.0=py312heda63a1_2
• semantic_version=2.10.0=pyhd8ed1ab_0
• setuptools=68.2.2=py312h06a4308_0
• shapely=2.0.2=py312h9e6bd2c_1
• six=1.16.0=pyh6c4a22f_0
• snappy=1.1.10=h9fff704_0
• snuggs=1.4.7=py_0
• sqlite=3.41.2=h5eee18b_0
• stack_data=0.2.0=pyhd3eb1b0_0
• svt-av1=1.8.0=h59595ed_0
• tbb=2021.8.0=hdb19cb5_0
• threadpoolctl=3.3.0=pyhc1e730c_0
• tifffile=2024.2.12=pyhd8ed1ab_0
• tiledb=2.20.0=hd75ad12_0
• tk=8.6.13=noxft_h4845f30_101
• tornado=6.3.3=py312h5eee18b_0
• tqdm=4.66.2=pyhd8ed1ab_0
• traitlets=5.7.1=py312h06a4308_0
• tzcode=2024a=h3f72095_0
• tzdata=2023d=h04d1e81_0
• uriparser=0.9.7=hcb278e6_1
• wcwidth=0.2.5=pyhd3eb1b0_0
• wheel=0.41.2=py312h06a4308_0
• xerces-c=3.2.5=hac6953d_0
• xorg-kbproto=1.0.7=h7f98852_1002
• xorg-libice=1.1.1=hd590300_0
• xorg-libsm=1.2.4=h7391055_0
• xorg-libx11=1.8.7=h8ee46fc_0
• xorg-libxau=1.0.11=hd590300_0
• xorg-libxdmcp=1.1.3=h7f98852_0
• xorg-libxext=1.3.4=h0b41bf4_2
• xorg-libxrender=0.9.11=hd590300_0
• xorg-renderproto=0.11.1=h7f98852_1002
• xorg-xextproto=7.3.0=h0b41bf4_1003
• xorg-xproto=7.0.31=h7f98852_1007
• xz=5.4.5=h5eee18b_0
• yaml=0.2.5=h7b6447c_0
• zeromq=4.3.4=h2531618_0
• zfp=1.0.1=h59595ed_0
• zipp=3.17.0=pyhd8ed1ab_0
• zlib=1.2.13=hd590300_5
• zlib-ng=2.0.7=h0b41bf4_0
• zstd=1.5.5=hfc55251_0
• pip:
  • asdf==3.2.0
  • asdf-standard==1.1.1
  • asdf-transform-schemas==0.5.0
  • asteval==0.9.32
  • dill==0.3.8
  • future==1.0.0
  • jsonschema==4.22.0
  • jsonschema-specifications==2023.12.1
  • lmfit==1.3.1
  • plotbin==3.1.5
  • ppxf==9.2.2
  • q3dfit==1.1.4
  • referencing==0.35.1
  • rpds-py==0.18.1
  • specutils==1.15.1.dev6+g28b0cdb
  • statsmodels==0.15.0.dev280+gbc1899510
  • stdatamodels==1.10.2.dev5+g0326fd4
  • uncertainties==3.1.7
    prefix: /home/kydan/anaconda3/envs/correct_cube