AttributeError: module 'dill._dill' has no attribute 'PY3'

[demongolem-biz2]

I am trying to learn how to fine-tune. As such, I am running to run preparation for end-to-end question generations as given:

python prepare_data.py
--task e2e_qg
--valid_for_qg_only \
--model_type t5
--dataset_path data/squad_multitask/
--qg_format highlight_qg_format
--max_source_length 512
--max_target_length 32
--train_file_name train_data_e2e_qg_t5.pt
--valid_file_name valid_data_e2e_qg_t5.pt \

I get the following stack trace:

File "/home/gregory.werner/question_generation/prepare_data.py", line 165, in main
train_dataset = train_dataset.filter(TASK_TO_FILTER_FN[data_args.task])
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/arrow_dataset.py", line 1053, in filter
return self.map(
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/arrow_dataset.py", line 902, in map
cache_file_name = self._get_cache_file_path(function, cache_kwargs)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/arrow_dataset.py", line 756, in _get_cache_file_path
function_bytes = dumps(function)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/utils/py_utils.py", line 278, in dumps
dump(obj, file)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/utils/py_utils.py", line 271, in dump
Pickler(file).dump(obj)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 418, in dump
StockPickler.dump(self, obj)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 487, in dump
self.save(obj)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 1965, in save_function
_save_with_postproc(pickler, (_create_function, (
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 1093, in _save_with_postproc
pickler.save_reduce(*reduction, obj=obj)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 692, in save_reduce
save(args)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 902, in save_tuple
save(element)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
StockPickler.save(self, obj, save_persistent_id)
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/gregory.werner/cpython-3.10.13/python/lib/python3.10/site-packages/nlp/utils/py_utils.py", line 304, in save_code
if dill._dill.PY3:
AttributeError: module 'dill._dill' has no attribute 'PY3'

Is this a versioning thing maybe where I have to old or new of a version of something?

[demongolem-biz2]

I don't really get nlp vs datasets. I guess I thought datasets was the same, but newer.

But anyhow, I took code from datasets file utils/py_utils.py in save_code method to hardcode the correct args in the nlp version and it worked. So there seems to be something about the tutorial which needs updating? I had initially tried replacing nlp with datasets in the prepare_data.py code but that did not work for some other reason.

[THANH-SSV]

AttributeError: module 'dill._dill' has no attribute 'PY3'
Have you solved this problem? Can you tell me? I really don't know what to deal with

[ZZZZkp]

I don't really get nlp vs datasets. I guess I thought datasets was the same, but newer.

But anyhow, I took code from datasets file utils/py_utils.py in save_code method to hardcode the correct args in the nlp version and it worked. So there seems to be something about the tutorial which needs updating? I had initially tried replacing nlp with datasets in the prepare_data.py code but that did not work for some other reason.

Don't forget the usage of nlp in squad_multitask.py.

[nitheeshk03]

AttributeError: module 'dill._dill' has no attribute 'PY3' Have you solved this problem? Can you tell me? I really don't know what to deal with

I faced the same issue and installing dill version 0.3.5 worked for me
pip install dill==0.3.5

[imkhoibui]

AttributeError: module 'dill._dill' has no attribute 'PY3' Have you solved this problem? Can you tell me? I really don't know what to deal with

You should either use a lower version of nlp (specifically 0.2.0) or just use datasets package and adapt everything else altogether.