한국 10대 일간지 크롤링 및 유사어 사전 제공 Python 라이브러리입니다.
기사에 대한 모든 저작권은 해당 언론사에 속해있습니다. 저희는 이에 대한 어떠한 법적인 책임을 지지 않으며 사용자는 이에 동의하는 것으로 간주합니다.
Open Source Project로 기여자, 참여자 상시 모집하고 있습니다. 연락주시면 감사하겠습니다.
This is Python library for crawling articles from Korean Top 10 Newspaper sites and providing synonym dictionary.
The copyright of articles are belong to original media company. We don't take any legal responsibility using of them. We assume that you have agreed to this.
We're greeting to join you as contibutors, collaborator. Thanks to give me contact.
- 조선일보(Chosun Ilbo)
- 동아일보(Dong-a Ilbo)
- 한국일보(Hankook Ilbo)
- 한겨레(Hankyeoreh)
- 중앙일보(JoongAng Ilbo)
- 국민일보(Kukmin Ilbo)
- 경향신문(Kyunghyang Shinmun)
- 문화일보(Munhwa Ilbo)
- 내일신문(Naeil News)
- 세계일보(Segye Ilbo)
- 서울신문(Seoul Shinmun)
Indigo_Coder |
pip install korean_news_crawler
BeautifulSoup, Selenium, Requests are required.
from korean_news_crawler import chosun
chosun = Chosun()
print(chosun.dynamic_crawl("https://www.chosun.com/..."))
chosun_url_list = list() #Chosun Ilbo url list
print(chosun.dynamic_crawl(chosun_url_list))
Chosun()
Donga()
Hankook()
Hankyoreh()
Joongang()
Kukmin()
Kyunghyang()
Munhwa()
Naeil()
Segye()
Seoul()
It provides crawling Chosun Ilbo.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Dong-a Ilbo.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Hankook Ilbo.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Hankyoreh.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Joongang Ilbo.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Kukmin Ilbo.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Kyunghyang Shinmun.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Munhwa Ilbo.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Naeil News.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Segye Ilbo.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
It provides crawling Seoul Shinmun.
Parameters | Type | Description |
---|---|---|
delay_time | float or tuple | - Optional, Defaults to None. - When 'delay_time=float', it will crawl sites with delay. - When 'delay_time=tuple', it will crawl sites with random delay. |
saving_html | bool | - Optional, Defaults to False. - When 'saving_html=False', it always requests url every function calling. - When 'saving_html=True', It will save requested html only first time. After that, it calls saved html. This will help to alleviate server load. |
Attributes | Type | Description |
---|---|---|
delay_time | float or tuple | |
saving_html | bool |
Methods | Description |
---|---|
dynamic_crawl(url) | Return article text using Selenium. |
static_crawl(url) | Return article text using BeautifulSoup. |
- Return article text using Selenium.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |
- Return article text using BeautifulSoup.
Parameters | Type | Description |
---|---|---|
url | str or list | - When 'url=str', it will only crawl given url. - When 'url=list', it will crawl with iterating url list. |
Returns Type | Description |
---|---|
list | Return article list. |