AttributeError: 'LatentAttentionConfig' object has no attribute '_attn_implementation_internal'

[redshiva]

When I run the two-line program below, I get the error. Does anyone know how to solve this? Is there a specific list of versions of the dependencies I should be using?

Thank you!!

Python 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> from sentence_transformers import SentenceTransformer
>>> model = SentenceTransformer('nvidia/NV-Embed-v2', device="cuda:0", trust_remote_code=True)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/code/testing/chunking-testing/chunk-test-venv/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 294, in __init__
    modules, self.module_kwargs = self._load_sbert_model(
                                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/testing/chunking-testing/chunk-test-venv/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 1647, in _load_sbert_model
    module = module_class(model_name_or_path, cache_dir=cache_folder, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/testing/chunking-testing/chunk-test-venv/lib/python3.12/site-packages/sentence_transformers/models/Transformer.py", line 56, in __init__
    self._load_model(model_name_or_path, config, cache_dir, **model_args)
  File "/code/testing/chunking-testing/chunk-test-venv/lib/python3.12/site-packages/sentence_transformers/models/Transformer.py", line 87, in _load_model
    self.auto_model = AutoModel.from_pretrained(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/testing/chunking-testing/chunk-test-venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/testing/chunking-testing/chunk-test-venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 3832, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dave/.cache/huggingface/modules/transformers_modules/nvidia/NV-Embed-v2/26db444e203771ea231d32a795344655e1119d40/modeling_nvembed.py", line 323, in __init__
    self.latent_attention_model = AutoModel.from_config(config.latent_attention_config)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/testing/chunking-testing/chunk-test-venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 438, in from_config
    return model_class._from_config(config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/testing/chunking-testing/chunk-test-venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 1478, in _from_config
    if config._attn_implementation_internal is not None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/testing/chunking-testing/chunk-test-venv/lib/python3.12/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

pip list

Package                   Version
------------------------- --------------
aiohappyeyeballs          2.4.0
aiohttp                   3.10.5
aiosignal                 1.3.1
anyio                     4.6.0
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asttokens                 2.4.1
async-lru                 2.0.4
attrs                     24.2.0
babel                     2.16.0
beautifulsoup4            4.12.3
bleach                    6.1.0
certifi                   2024.8.30
cffi                      1.17.1
charset-normalizer        3.3.2
comm                      0.2.2
datasets                  3.0.0
debugpy                   1.8.5
decorator                 5.1.1
defusedxml                0.7.1
dill                      0.3.8
einops                    0.8.0
executing                 2.1.0
fastjsonschema            2.20.0
filelock                  3.13.1
fqdn                      1.5.1
frozenlist                1.4.1
fsspec                    2024.2.0
h11                       0.14.0
httpcore                  1.0.5
httpx                     0.27.2
huggingface-hub           0.25.0
idna                      3.10
ipykernel                 6.29.5
ipython                   8.27.0
isoduration               20.11.0
jedi                      0.19.1
Jinja2                    3.1.3
joblib                    1.4.2
json5                     0.9.25
jsonpointer               3.0.0
jsonschema                4.23.0
jsonschema-specifications 2023.12.1
jupyter_client            8.6.3
jupyter_core              5.7.2
jupyter-events            0.10.0
jupyter-lsp               2.2.5
jupyter_server            2.14.2
jupyter_server_terminals  0.5.3
jupyterlab                4.2.5
jupyterlab_pygments       0.3.0
jupyterlab_server         2.27.3
MarkupSafe                2.1.5
matplotlib-inline         0.1.7
mistune                   3.0.2
mpmath                    1.3.0
multidict                 6.1.0
multiprocess              0.70.16
nbclient                  0.10.0
nbconvert                 7.16.4
nbformat                  5.10.4
nest-asyncio              1.6.0
networkx                  3.2.1
notebook                  7.2.2
notebook_shim             0.2.4
numpy                     2.1.1
nvidia-cublas-cu12        12.4.2.65
nvidia-cuda-cupti-cu12    12.4.99
nvidia-cuda-nvrtc-cu12    12.4.99
nvidia-cuda-runtime-cu12  12.4.99
nvidia-cudnn-cu12         9.1.0.70
nvidia-cufft-cu12         11.2.0.44
nvidia-curand-cu12        10.3.5.119
nvidia-cusolver-cu12      11.6.0.99
nvidia-cusparse-cu12      12.3.0.142
nvidia-nccl-cu12          2.20.5
nvidia-nvjitlink-cu12     12.4.99
nvidia-nvtx-cu12          12.4.99
overrides                 7.7.0
packaging                 24.1
pandas                    2.2.3
pandocfilters             1.5.1
parso                     0.8.4
pexpect                   4.9.0
pillow                    10.4.0
pip                       24.0
platformdirs              4.3.6
prometheus_client         0.21.0
prompt_toolkit            3.0.47
psutil                    6.0.0
ptyprocess                0.7.0
pure_eval                 0.2.3
pyarrow                   17.0.0
pycparser                 2.22
Pygments                  2.18.0
python-dateutil           2.9.0.post0
python-json-logger        2.0.7
pytz                      2024.2
PyYAML                    6.0.2
pyzmq                     26.2.0
referencing               0.35.1
regex                     2024.9.11
requests                  2.32.3
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rpds-py                   0.20.0
safetensors               0.4.5
scikit-learn              1.5.2
scipy                     1.14.1
Send2Trash                1.8.3
sentence-transformers     3.1.1
setuptools                70.0.0
six                       1.16.0
sniffio                   1.3.1
soupsieve                 2.6
stack-data                0.6.3
sympy                     1.12
terminado                 0.18.1
threadpoolctl             3.5.0
tinycss2                  1.3.0
tokenizers                0.19.1
torch                     2.4.1+cu124
tornado                   6.4.1
tqdm                      4.66.5
traitlets                 5.14.3
transformers              4.44.2
triton                    3.0.0
types-python-dateutil     2.9.0.20240906
typing_extensions         4.9.0
tzdata                    2024.1
uri-template              1.3.0
urllib3                   2.2.3
wcwidth                   0.2.13
webcolors                 24.8.0
webencodings              0.5.1
websocket-client          1.8.0
xxhash                    3.5.0
yarl                      1.11.1

nvidia-smi

Sun Sep 22 04:05:04 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
|  0%   26C    P8             18W /  450W |       4MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:03:00.0 Off |                  Off |
|  0%   28C    P8             10W /  450W |       4MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[tomaarsen]

Hello!

This seems to be a model-specific issue. There's some more details in this discussion on the model itself: https://huggingface.co/nvidia/NV-Embed-v1/discussions/50

It points out some specific transformers versions for which everything should work again.

P.s. if that's your local machine, then consider me jealous

• Tom Aarsen