Ch 24. 딥러닝 기반 질의응답.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Ch 24. 딥러닝 기반 질의응답.ipynb","provenance":[],"collapsed_sections":[],"authorship_tag":"ABX9TyPdUoNzJ+fMJ4NT80fCx7l3"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","metadata":{"id":"-x7n_t4F9FXa"},"source":["# 실습 [24-1]<br>\n","**실습명: BERT을 활용한 KorQuAD 데이터셋 학습&평가**<br>\n","- KorQuAD: QA(Question & Answering)의 데이터셋 중 하나 (한국어 Machine Reading 이해를 위해 만든 데이터셋)\n","- NAVER Clova에서 제공한 자연어 처리와 관련된 대표 모델들과 task들을 한번에 처리할 수 있는 패키지인 CLaF 사용"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ar43-W_2-ffK","executionInfo":{"status":"ok","timestamp":1624503924068,"user_tz":-540,"elapsed":24991,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"5341c53f-57cb-43d9-8257-d37f9f6955ce"},"source":["from google.colab import drive\n","drive.mount(\"/content/gdrive\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Mounted at /content/gdrive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"3DSJgfL2-tBJ","executionInfo":{"status":"ok","timestamp":1624503932677,"user_tz":-540,"elapsed":6,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"d7f200df-db62-407d-acb6-461acf59f469"},"source":["cd /root"],"execution_count":null,"outputs":[{"output_type":"stream","text":["/root\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"dATmhclEv-h6","executionInfo":{"status":"ok","timestamp":1624503985582,"user_tz":-540,"elapsed":2434,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"8fcf6217-e89b-406a-b76e-312f6d9264b1"},"source":["#1) 환경 세팅: 필요 패키지, 데이터셋 다운로드\n","!git clone https://github.com/naver/claf.git"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Cloning into 'claf'...\n","remote: Enumerating objects: 3932, done.\u001b[K\n","remote: Counting objects: 100% (107/107), done.\u001b[K\n","remote: Compressing objects: 100% (83/83), done.\u001b[K\n","remote: Total 3932 (delta 38), reused 53 (delta 24), pack-reused 3825\u001b[K\n","Receiving objects: 100% (3932/3932), 8.84 MiB | 14.87 MiB/s, done.\n","Resolving deltas: 100% (2861/2861), done.\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"TUQz6o6q9ifS","executionInfo":{"status":"ok","timestamp":1624503987541,"user_tz":-540,"elapsed":398,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"2a292ea9-45c0-41e6-b030-ca4691eb3cf5"},"source":["cd /root/claf"],"execution_count":null,"outputs":[{"output_type":"stream","text":["/root/claf\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"X2tvznA29prm"},"source":["pip install -r requirements.txt"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"hzZ7gnC3-82l"},"source":["!sh script/install_mecab.sh"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"eO76a_4u-_eP","executionInfo":{"status":"ok","timestamp":1624504272695,"user_tz":-540,"elapsed":2108,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"2687ce0f-1580-4883-d33d-9d23d1a56d0e"},"source":["import nltk\n","nltk.download('punkt')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["[nltk_data] Downloading package punkt to /root/nltk_data...\n","[nltk_data]   Unzipping tokenizers/punkt.zip.\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["True"]},"metadata":{"tags":[]},"execution_count":14}]},{"cell_type":"code","metadata":{"id":"_zIJp3By_LA3"},"source":["mkdir data"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"GTdut47j_LDJ","executionInfo":{"status":"ok","timestamp":1624504275650,"user_tz":-540,"elapsed":4,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"f9104f72-2a88-4616-a9d0-90d5c84fb0d2"},"source":["cd data"],"execution_count":null,"outputs":[{"output_type":"stream","text":["/root/claf/data\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"o_dBCZ-2_LFV"},"source":["mkdir korquad"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"YOluqEwb_LHB","executionInfo":{"status":"ok","timestamp":1624504277340,"user_tz":-540,"elapsed":3,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"19eb41c2-05b9-41de-d0d0-8f090d0e60e6"},"source":["cd korquad"],"execution_count":null,"outputs":[{"output_type":"stream","text":["/root/claf/data/korquad\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Ayi3afEj_LIp"},"source":["!wget https://korquad.github.io/dataset/KorQuAD_v1.0_train.json\n","!wget https://korquad.github.io/dataset/KorQuAD_v1.0_dev.json"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"gDn_jFYI_LKs","executionInfo":{"status":"ok","timestamp":1624504284134,"user_tz":-540,"elapsed":416,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"7aee43d3-943e-4044-b3aa-b3c931ebc6a1"},"source":["cd /root/claf"],"execution_count":null,"outputs":[{"output_type":"stream","text":["/root/claf\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"n6QzZFd__LMl"},"source":["#2) 모델 학습\n","#train에 있는 argument에 base_config 주고 cuda 설정\n","!python train.py --base_config korquad/bert_base_multilingual_uncased --cuda_devices '0' --batch_size 4"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"EgtnTPru_rFY"},"source":["#3) 모델 평가\n","#저장된 모델의 가중치 불러와서 dev로 모델 평가\n","!python eval.py /root/data/korquad/KorQuAD_v1.0_dev.json /root/claf/logs/squad_bert/checkpoint/model_48000.pkl  --cuda_devices '0'"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"IdVIl3smBcSH"},"source":["python predict.py <학습 웨이트 경로> --question \"질문 문장\" --context \"문맥 정보 입력\""]},{"cell_type":"code","metadata":{"id":"sbTqMzD2_rH7"},"source":["#4) 모델 테스트\n","!python predict.py /root/claf/logs/squad_bert/checkpoint/model_48000.pkl --question \"88올림픽 개최지는?\" --context \"1988년 9월 17일부터 10월 2일까지 서울특별시와 경기도 일부 수도권 지역에서 개최된 하계 올림픽이자, 대회의 정식 명칭은 제24회 서울 올림픽 경기대회(Games of the XXIV Olympiad Seoul 1988)이다.\" --cuda_devices '0'"],"execution_count":null,"outputs":[]}]}