Ch 10. 정보 추출.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Ch10. 정보 추출(Information Extraction).ipynb","provenance":[],"collapsed_sections":[],"authorship_tag":"ABX9TyMsOTRTOtpzB8VOjPpVtfGa"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","metadata":{"id":"x_k46_mQjI3M"},"source":["# 실습 [10-1]<br>\n","**실습명: 정보 추출에서 관계형 튜플 표현**<br>\n","예)<br>\n","- 기관 이름: 고려대학교\n","- 지역 이름: 서울\n","- 관계 표현: ('고려대학교', 'In', '서울')"]},{"cell_type":"code","metadata":{"id":"7EWHQac9XLM9","executionInfo":{"status":"ok","timestamp":1624010292474,"user_tz":-540,"elapsed":6,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}}},"source":["#구조화된 데이터로 표현\n","#(엔티티1, 관계, 엔티티2) 형식으로 표현\n","locs = [('고려대학교', 'In', '서울'),\n","        ('Naver', 'In', '성남'),\n","        ('KT 위즈', 'In', '수원'),\n","        ('삼성', 'In', '서울')]"],"execution_count":1,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"dkutSwWZjlHs","executionInfo":{"status":"ok","timestamp":1624010338773,"user_tz":-540,"elapsed":273,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"c52c2093-871e-453f-ff27-6d420545b352"},"source":["#서울에 있는 기관만 출력\n","query = []\n","for (e1, re1, e2) in locs:\n","  if e2 == '서울':\n","    query.append(e1)\n","\n","print(query)"],"execution_count":2,"outputs":[{"output_type":"stream","text":["['고려대학교', '삼성']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"i0-kIiMcj8FF"},"source":["# 실습 [10-2]<br>\n","**실습명: 정규식을 적용한 관계 추출 실습1**<br>\n",": (엔티티1, 관계, 엔티티2) 형식의 관계형 튜플 표현에서 관계는 엔터티1과 2 사이에 있는 단어의 문자열이다, 이를 기반으로 정규표현식을 사용해 관계의 인스턴스 추출 가능"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Nqhp2C2Yj7lx","executionInfo":{"status":"ok","timestamp":1624010480264,"user_tz":-540,"elapsed":1845,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"770d0ac4-d0b8-433d-b0fb-80f1694dbe19"},"source":["#관계 추출(relation extraction) 라이브러리 (ieer corpus 사용)\n","import re #정규표현식 라이브러리\n","import nltk\n","nltk.download('ieer')"],"execution_count":3,"outputs":[{"output_type":"stream","text":["[nltk_data] Downloading package ieer to /root/nltk_data...\n","[nltk_data]   Unzipping corpora/ieer.zip.\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["True"]},"metadata":{"tags":[]},"execution_count":3}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"b5-BTuoKkVHl","executionInfo":{"status":"ok","timestamp":1624010657656,"user_tz":-540,"elapsed":290,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"621e3968-1016-44e4-ae5f-888ae7e317e9"},"source":["#단어가 포함된 문자열 검색시 사용되는 관계 추출 정규식 = (?!\\b.+ing\\b)\n","IN = re.compile(r'.*\\bin\\b(?!\\b.+ing)')\n","\n","#ORG(기관이름), LOC(장소)를 관계형 튜플 방식으로 나타내기\n","for doc in nltk.corpus.ieer.parsed_docs('NYT_19980315'):\n","  print(\"doc : \", doc)\n","  for rel in nltk.sem.extract_rels('ORG', 'LOC', doc, corpus='ieer', pattern=IN):\n","    print(nltk.sem.rtuple(rel))"],"execution_count":6,"outputs":[{"output_type":"stream","text":["doc :  <IEERDocument NYT19980315.0063: 'PUBLIC RADIO HOSTS DROP IN AND MAYBE STAY TOO LONG'>\n","[ORG: 'WHYY'] 'in' [LOC: 'Philadelphia']\n","doc :  <IEERDocument NYT19980315.0064: 'IN CYBERSPACE, IS THERE LAW WHERE THERE IS NO LAND?'>\n","[ORG: 'McGlashan &AMP; Sarrail'] 'firm in' [LOC: 'San Mateo']\n","doc :  <IEERDocument NYT19980315.0067: 'THE SITES: TUNING INTO MUSIC ON THE WEB'>\n","[ORG: 'Freedom Forum'] 'in' [LOC: 'Arlington']\n","doc :  <IEERDocument NYT19980315.0069: 'ANALYSIS: TAXING INTERNET SALES _ GOVERNORS VS. TAX FREEDOM ACT'>\n","[ORG: 'Brookings Institution'] ', the research group in' [LOC: 'Washington']\n","doc :  <IEERDocument NYT19980315.0070: 'A SEARCH ENGINE THAT CHARGES FOR TOP BILLING'>\n","[ORG: 'Idealab'] ', a self-described business incubator based in' [LOC: 'Los Angeles']\n","[ORG: 'Open Text'] ', based in' [LOC: 'Waterloo']\n","doc :  <IEERDocument NYT19980315.0071: 'COMING SOON: TV DOCUMENTARIES TO A BOOKSTORE NEAR YOU'>\n","[ORG: 'WGBH'] 'in' [LOC: 'Boston']\n","doc :  <IEERDocument NYT19980315.0072: 'WEATHER'>\n","doc :  <IEERDocument NYT19980315.0073: 'DICAPRIO , CHARISMATIC STAR, BALKS AT TEEN IDOL IMAGE'>\n","doc :  <IEERDocument NYT19980315.0074: \"You've read the book? Now see the television program. Inspired by ``Angela's...\">\n","doc :  <IEERDocument NYT19980315.0084: 'A DIRECTOR WHO DARES, AND TAKES THE HEAT'>\n","[ORG: 'Bastille Opera'] 'in' [LOC: 'Paris']\n","doc :  <IEERDocument NYT19980315.0085: 'ADVERTISING: AFTER 32 YEARS , WELLS BDDP WILL CLOSE'>\n","[ORG: 'Omnicom'] 'in' [LOC: 'New York']\n","[ORG: 'DDB Needham'] 'in' [LOC: 'New York']\n","[ORG: 'Kaplan Thaler Group'] 'in' [LOC: 'New York']\n","[ORG: 'BBDO South'] 'in' [LOC: 'Atlanta']\n","[ORG: 'Georgia-Pacific'] 'in' [LOC: 'Atlanta']\n","doc :  <IEERDocument NYT19980315.0086: \"`THE GIFTS OF THE JEWS': DESERT NOMADS CHANGED THE WORLD WITH THE\">\n","doc :  <IEERDocument NYT19980315.0087: 'MUSICIANS ON BROADWAY TO VOTE ON A CONTRACT'>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"VGO4Nh-_lJG2"},"source":["# 실습 [10-3]<br>\n","**실습명: 정규식을 적용한 관계 추출 실습2**<br>\n","- conll2002 Dutch 코퍼스에는 명명된 엔티티 주석, 품사 태그 포함\n","- `cluase()` : 이진 관계 기호가 매개 변수 relsym 값으로 지정되는 관계를 clausal 형식으로 인쇄"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"U8djkL2tk556","executionInfo":{"status":"ok","timestamp":1624010862751,"user_tz":-540,"elapsed":404,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"d9914e3f-c1d9-42fa-9b18-0115cf38f403"},"source":["#라이브러리\n","import re\n","import nltk\n","from nltk.corpus import conll2002\n","nltk.download('conll2002')"],"execution_count":7,"outputs":[{"output_type":"stream","text":["[nltk_data] Downloading package conll2002 to /root/nltk_data...\n","[nltk_data]   Unzipping corpora/conll2002.zip.\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["True"]},"metadata":{"tags":[]},"execution_count":7}]},{"cell_type":"code","metadata":{"id":"2FldPd-Xlyzh","executionInfo":{"status":"ok","timestamp":1624010916193,"user_tz":-540,"elapsed":282,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}}},"source":["#여러 개의 정규식 추가\n","vnv = \"\"\"\n","(\n","  is/V|\n","  was/V|\n","  werd/V|\n","  wordt/V\n",")\n",".*\n","van/Prep\n","\"\"\""],"execution_count":8,"outputs":[]},{"cell_type":"code","metadata":{"id":"jJQ2qVs4l_4y","executionInfo":{"status":"ok","timestamp":1624010943929,"user_tz":-540,"elapsed":278,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}}},"source":["#정규식 기준으로 전처리\n","VAN = re.compile(vnv, re.VERBOSE)"],"execution_count":10,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"9grGcWlymFFi","executionInfo":{"status":"ok","timestamp":1624011180558,"user_tz":-540,"elapsed":1434,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"ba6a51c7-5e63-40f9-b94f-1d93197a7d97"},"source":["#conll2002 코퍼스에 대한 관계 추출 (네덜란드어)\n","for doc in conll2002.chunked_sents('ned.train'):\n","  #print(\"doc : \", doc)\n","  for rel in nltk.sem.extract_rels('PER', 'ORG', doc, \n","                                   corpus='conll2002', pattern=VAN):\n","    print(\"\", nltk.sem.clause(rel, relsym=\"VAN\"))\n","    #print(nltk.rtuple(rel, lcon=True, rcon=True))"],"execution_count":15,"outputs":[{"output_type":"stream","text":[" VAN(\"cornet_d'elzius\", 'buitenlandse_handel')\n"," VAN('johan_rottiers', 'kardinaal_van_roey_instituut')\n"," VAN('annie_lennox', 'eurythmics')\n"],"name":"stdout"}]}]}