Skip to content

Latest commit

 

History

History
 
 

data

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
__init__.py
__init__.py
 
 
classifier_data_lib.py
classifier_data_lib.py
 
 
classifier_data_lib_test.py
classifier_data_lib_test.py
 
 
create_finetuning_data.py
create_finetuning_data.py
 
 
create_pretraining_data.py
create_pretraining_data.py
 
 
create_pretraining_data_test.py
create_pretraining_data_test.py
 
 
create_xlnet_pretraining_data.py
create_xlnet_pretraining_data.py
 
 
create_xlnet_pretraining_data_test.py
create_xlnet_pretraining_data_test.py
 
 
data_loader.py
data_loader.py
 
 
data_loader_factory.py
data_loader_factory.py
 
 
data_loader_factory_test.py
data_loader_factory_test.py
 
 
dual_encoder_dataloader.py
dual_encoder_dataloader.py
 
 
dual_encoder_dataloader_test.py
dual_encoder_dataloader_test.py
 
 
pretrain_dataloader.py
pretrain_dataloader.py
 
 
pretrain_dataloader_test.py
pretrain_dataloader_test.py
 
 
pretrain_dynamic_dataloader.py
pretrain_dynamic_dataloader.py
 
 
pretrain_dynamic_dataloader_test.py
pretrain_dynamic_dataloader_test.py
 
 
pretrain_text_dataloader.py
pretrain_text_dataloader.py
 
 
question_answering_dataloader.py
question_answering_dataloader.py
 
 
question_answering_dataloader_test.py
question_answering_dataloader_test.py
 
 
sentence_prediction_dataloader.py
sentence_prediction_dataloader.py
 
 
sentence_prediction_dataloader_test.py
sentence_prediction_dataloader_test.py
 
 
sentence_retrieval_lib.py
sentence_retrieval_lib.py
 
 
squad_lib.py
squad_lib.py
 
 
squad_lib_sp.py
squad_lib_sp.py
 
 
tagging_data_lib.py
tagging_data_lib.py
 
 
tagging_data_lib_test.py
tagging_data_lib_test.py
 
 
tagging_dataloader.py
tagging_dataloader.py
 
 
tagging_dataloader_test.py
tagging_dataloader_test.py
 
 
train_sentencepiece.py
train_sentencepiece.py
 
 
wmt_dataloader.py
wmt_dataloader.py
 
 
wmt_dataloader_test.py
wmt_dataloader_test.py
 
 

README.md

This directory contains binaries and utils required for input preprocessing, tokenization, etc that can be used with model building blocks available in NLP modeling library nlp/modelling to train custom models and validate new research ideas.