Features | 更新部分功能,详情看CHANGELOG

Cl0udG0d · Feb 26, 2023 · 8b71a96 · 8b71a96
1 parent 75dace0
commit 8b71a96
Show file tree

Hide file tree

Showing 10 changed files with 286 additions and 162 deletions.
diff --git a/OLD_README.md b/OLD_README.md
@@ -0,0 +1,182 @@
+# Fofa-hack
+
+### 简介
+
+非付费会员，fofa数据无限抓取版 , 配置FoFa普通用户账号密码即可使用
+
+截止至 `2023-2-25` 日 ，亲测可用，如果项目不行了欢迎联系我
+
+使用示例
+> python fofa.py --username fofa_hack_test --password Test123456 -k app="ThinkPHP" -t 5
+### 安装
+
+```shell
+git clone https://github.com/Cl0udG0d/Fofa-hack
+```
+
+安装运行所需的库文件，国内请加源 https://pypi.tuna.tsinghua.edu.cn/simple
+
+```shell
+pip install -r requirements.txt
+```
+
+### 配置
+有三种方式配置登录账号
+
+#### 1.运行传值
+传入`--username` 和 `--password` 参数
+> Fofa-hack>python fofa.py --username fofa_hack_test --password Test123
+
+#### 2.配置config.py
+配置`config.py`中的`fofa_account`，支持多账号
+```json
+fofa_account=[
+    {
+        "fofa_username" : "test@email.com",
+        "fofa_password" : "12345678"
+    },
+  {
+        "fofa_username" : "test1@email.com",
+        "fofa_password" : "12345678"
+    }
+]
+```
+
+也就是你的FOFA账号密码
+
+#### 3.配置fofa_cookie.txt文件
+将下图中的cookie直接复制到`fofa_cookie.txt`文件里,这样在启动的时候Fofa-hack就会识别到并且直接使用cookie爬取了
+
+注意不是`Authorization`
+
+`cookie`的位置如下
+  ![](https://github.com/Cl0udG0d/Fofa-script/blob/master/images/2.png)
+
+### 运行
+
+运行`fofa.py` , `-k`或`--keyword` 参数传入搜索关键字
+
+更多参数查看 `--help`
+
+> python3 fofa.py --help
+
+```shell
+Fofa-hack>python fofa.py --help                                         
+
+         ____  ____  ____  ____                 
+        | ===|/ () \| ===|/ () \                
+        |__|  \____/|__| /__/\__\               
+             _   _   ____   ____  __  __        
+            | |_| | / () \ / (__`|  |/  /       
+            |_| |_|/__/\__\\____)|__|\__\ V1.3.0
+                                                
+usage: fofa.py [-h] [--timesleep TIMESLEEP] --keyword KEYWORD                 
+               [--username USERNAME] [--password PASSWORD] [--endpage ENDPAGE]
+               [--level LEVEL]                                                
+
+Fofa-hack v1.3.0 使用说明
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --timesleep TIMESLEEP, -t TIMESLEEP
+                        爬取每一页等待秒数,防止IP被Ban,默认为3
+  --keyword KEYWORD, -k KEYWORD
+                        fofa搜索关键字,默认为test
+  --username USERNAME, -u USERNAME
+                        fofa用户名
+  --password PASSWORD, -p PASSWORD
+                        fofa密码
+  --endpage ENDPAGE, -e ENDPAGE
+                        爬取结束页码
+  --level LEVEL, -l LEVEL
+                        爬取等级: 1-3 ,数字越大内容越详细,默认为 1
+```
+
+爬取的结果会存储到`md5(搜索关键字)_运行时间戳.txt`文件中
+
+### 测试
+
+输入 搜索关键字 `app="ThinkPHP"`，等待秒数为5的情况下，下载1-50页数据经过测试无问题，经过自动去重之后剩余497条
+
+### 赞赏列表
+
+详情请见[SPONSOR](docs/SPONSOR.md)
+
+### 使用问题集合
+
+详情请见[QUESTIONS](docs/QUESTIONS.md)
+
++ [ERROR: Could not build wheels for opencv-python-headless, which is required to install pyproject.toml-based projects](docs/QUESTIONS.md#opencv-python错误)
++ [ddddocr错误解决](docs/QUESTIONS.md#ddddocr错误解决)
++ [FOFA综合语法使用](docs/QUESTIONS.md#FOFA综合语法使用)
+
+### 更新日志
+
+详情请见[CHANGELOG](docs/CHANGELOG.md)
+
+### TODO List
+<details>
+<summary>TODO</summary>
+<table >
+  <tr>
+    <td>名称</td>
+    <td>简介</td>
+  </tr>
+<tr>
+    <td>支持代理池</td>
+    <td>使用代理池的方式防止FOFA断开连接</td>
+  </tr>
+<tr>
+    <td>支持多种导出格式</td>
+    <td>支持json、txt、excel等方式导出结果</td>
+  </tr>
+<tr>
+    <td>编写图形化界面</td>
+    <td>生成可执行文件运行</td>
+  </tr>
+<tr>
+    <td>增加程序稳定性</td>
+    <td>防止程序因为各种情况运行失败或者被ban的情况</td>
+  </tr>
+<tr>
+    <td>内容去重</td>
+    <td>去除重复的url信息</td>
+  </tr>
+</table>
+</details>
+
+### 贡献者
+
+<!-- readme: collaborators,contributors -start -->
+<table>
+<tr>
+    <td align="center">
+        <a href="https://github.com/Cl0udG0d">
+            <img src="https://avatars.githubusercontent.com/u/45556496?v=4" width="100;" alt="Cl0udG0d"/>
+            <br />
+            <sub><b>潘一二三</b></sub>
+        </a>
+    </td>
+    <td align="center">
+        <a href="https://github.com/wanswu">
+            <img src="https://avatars.githubusercontent.com/u/49047734?v=4" width="100;" alt="wanswu"/>
+            <br />
+            <sub><b>Wans</b></sub>
+        </a>
+    </td></tr>
+</table>
+<!-- readme: collaborators,contributors -end -->
+
+### END 
+
+网络乞丐在线乞讨
+<div>
+    <img  alt="PNG" src="./images/sponsor.png"  width="280px" />
+</div>
+
+建了一个微信的安全交流群，欢迎添加我微信备注`进群`，一起来聊天吹水哇，以及一个会发布安全相关内容的公众号，欢迎关注 :)
+
+<div>
+    <img  alt="GIF" src="https://springbird.oss-cn-beijing.aliyuncs.com/img/mmqrcode1632325540724.png"  width="280px" />
+    <img  alt="GIF" src="https://springbird.oss-cn-beijing.aliyuncs.com/img/qrcode_for_gh_cead8e1080d6_344.jpg"  width="280px" />
+</div>
diff --git a/README.md b/README.md
@@ -2,101 +2,15 @@
 
 ### 简介
 
-非付费会员，fofa数据无限抓取版 , 配置FoFa普通用户账号密码即可使用
+`2023-2-26`因FOFA对于普通用户的限制,停止该项目的更新
+![fofa](./images/fofa.jpg)
 
-截止至 `2023-2-25` 日 ，亲测可用，如果项目不行了欢迎联系我
+结论是 `FOFA-Hack` 对于普通用户每个月最多能获取到3000条数据,在这个数据量内该项目是有效的
 
-使用示例
-> python fofa.py --username fofa_hack_test --password Test123456 -k app="ThinkPHP" -t 5
-### 安装
+不过对我来说这样 `FOFA-Hack` 的存在就没有意义了,所以在没有找到更好的方式之前,该项目不会再进行更新
+![result](./images/result.jpg)
 
-```shell
-git clone https://github.com/Cl0udG0d/Fofa-hack
-```
-
-安装运行所需的库文件，国内请加源 https://pypi.tuna.tsinghua.edu.cn/simple
-
-```shell
-pip install -r requirements.txt
-```
-
-### 配置
-有三种方式配置登录账号
-
-#### 1.运行传值
-传入`--username` 和 `--password` 参数
-> Fofa-hack>python fofa.py --username fofa_hack_test --password Test123
-
-#### 2.配置config.py
-配置`config.py`中的`fofa_account`，支持多账号
-```json
-fofa_account=[
-    {
-        "fofa_username" : "test@email.com",
-        "fofa_password" : "12345678"
-    },
-  {
-        "fofa_username" : "test1@email.com",
-        "fofa_password" : "12345678"
-    }
-]
-```
-
-也就是你的FOFA账号密码
-
-#### 3.配置fofa_cookie.txt文件
-将下图中的cookie直接复制到`fofa_cookie.txt`文件里,这样在启动的时候Fofa-hack就会识别到并且直接使用cookie爬取了
-
-注意不是`Authorization`
-
-`cookie`的位置如下
-  ![](https://github.com/Cl0udG0d/Fofa-script/blob/master/images/2.png)
-
-### 运行
-
-运行`fofa.py` , `-k`或`--keyword` 参数传入搜索关键字
-
-更多参数查看 `--help`
-
-> python3 fofa.py --help
-
-```shell
-Fofa-hack>python fofa.py --help                                         
-
-         ____  ____  ____  ____                 
-        | ===|/ () \| ===|/ () \                
-        |__|  \____/|__| /__/\__\               
-             _   _   ____   ____  __  __        
-            | |_| | / () \ / (__`|  |/  /       
-            |_| |_|/__/\__\\____)|__|\__\ V1.3.0
-                                                
-usage: fofa.py [-h] [--timesleep TIMESLEEP] --keyword KEYWORD                 
-               [--username USERNAME] [--password PASSWORD] [--endpage ENDPAGE]
-               [--level LEVEL]                                                
-
-Fofa-hack v1.3.0 使用说明
-
-optional arguments:
-  -h, --help            show this help message and exit
-  --timesleep TIMESLEEP, -t TIMESLEEP
-                        爬取每一页等待秒数,防止IP被Ban,默认为3
-  --keyword KEYWORD, -k KEYWORD
-                        fofa搜索关键字,默认为test
-  --username USERNAME, -u USERNAME
-                        fofa用户名
-  --password PASSWORD, -p PASSWORD
-                        fofa密码
-  --endpage ENDPAGE, -e ENDPAGE
-                        爬取结束页码
-  --level LEVEL, -l LEVEL
-                        爬取等级: 1-3 ,数字越大内容越详细,默认为 1
-```
-
-爬取的结果会存储到`md5(搜索关键字)_运行时间戳.txt`文件中
-
-### 测试
-
-输入 搜索关键字 `app="ThinkPHP"`，等待秒数为5的情况下，下载1-50页数据经过测试无问题，经过自动去重之后剩余497条
+原来的[README](./OLD_README.md)此处查看
 
 ### 赞赏列表
 
@@ -130,10 +44,6 @@ optional arguments:
     <td>支持多种导出格式</td>
     <td>支持json、txt、excel等方式导出结果</td>
   </tr>
-<tr>
-    <td>支持多种爬取内容</td>
-    <td>添加支持title、status、headers等内容</td>
-  </tr>
 <tr>
     <td>编写图形化界面</td>
     <td>生成可执行文件运行</td>
@@ -142,6 +52,10 @@ optional arguments:
     <td>增加程序稳定性</td>
     <td>防止程序因为各种情况运行失败或者被ban的情况</td>
   </tr>
+<tr>
+    <td>内容去重</td>
+    <td>去除重复的url信息</td>
+  </tr>
 </table>
 </details>
 

diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
@@ -1,5 +1,9 @@
 # CHANGELOG 代码变更记录
 
+### 1.3.1
+
++ 支持多种导出方式 txt,json,csv
++ 因FOFA对普通用户的限制停止该项目的更新
 ### 1.3.0
 
 + `README`添加示例运行参考

diff --git a/fofa.py b/fofa.py
@@ -16,6 +16,7 @@
 import argparse
 
 from tookit.levelData import LevelData
+from tookit.outputData import OutputData
 
 host_list = []
 timestamp_list = []
@@ -159,6 +160,7 @@ def init(self):
         parser.add_argument('--password', '-p', help='fofa密码')
         parser.add_argument('--endpage', '-e', help='爬取结束页码')
         parser.add_argument('--level', '-l', help='爬取等级: 1-3 ,数字越大内容越详细,默认为 1')
+        parser.add_argument('--output', '-o', help='输出格式:txt、json、csv,默认为txt')
         args = parser.parse_args()
         config.TimeSleep = int(args.timesleep)
         print("[*] 爬取延时: {}s".format(config.TimeSleep))
@@ -179,9 +181,16 @@ def init(self):
             print("[*] 爬取页码数: {}".format(self.want_page))
         self.level=args.level if args.level else "1"
         self.levelData=LevelData(self.level)
+
+
+        self.output = args.output if args.output else "txt"
+        print("[*] 输出格式为: {}".format(self.output))
+
+
         global filename
-        filename = "{}_{}.txt".format(unit.md5(config.SearchKEY), int(time.time()))
+        filename = "{}_{}.{}".format(unit.md5(config.SearchKEY), int(time.time()),self.output)
         print("[*] 存储文件名: {}".format(filename))
+        self.outputData = OutputData(filename, pattern=self.output)
         return
 
     def get_page_num(self, search_key,cookie):

diff --git a/fofa_cookie.txt b/fofa_cookie.txt
@@ -1 +0,0 @@
-befor_router=; fofa_token=eyJhbGciOiJIUzUxMiIsImtpZCI6Ik5XWTVZakF4TVRkalltSTJNRFZsWXpRM05EWXdaakF3TURVMlkyWTNZemd3TUdRd1pUTmpZUT09IiwidHlwIjoiSldUIn0.eyJpZCI6MjQ4MjI4LCJtaWQiOjEwMDE0MTU1OSwidXNlcm5hbWUiOiJmb2ZhX2hhY2tfdGVzdCIsImV4cCI6MTY3NzU3MjY3MH0.VBTqAd1imJfC9UsnlzW-n64ktBBxdY0FHI0uoB262wavOwzQaSFjdXlvL8MmqpDZ17-JNxTTdU-11NhQkJN_eg; is_flag_login=0; user=%7B%22id%22%3A248228%2C%22mid%22%3A100141559%2C%22is_admin%22%3Afalse%2C%22username%22%3A%22fofa_hack_test%22%2C%22nickname%22%3A%22fofa_hack_test%22%2C%22email%22%3A%22fofa_test%40163.com%22%2C%22avatar_medium%22%3A%22https%3A%2F%2Fnosec.org%2Fmissing.jpg%22%2C%22avatar_thumb%22%3A%22https%3A%2F%2Fnosec.org%2Fmissing.jpg%22%2C%22key%22%3A%2281d94208789ae707007ce80b5e637d7e%22%2C%22rank_name%22%3A%22%E6%B3%A8%E5%86%8C%E7%94%A8%E6%88%B7%22%2C%22rank_level%22%3A0%2C%22company_name%22%3A%22fofa_hack_test%22%2C%22coins%22%3A0%2C%22can_pay_coins%22%3A0%2C%22fofa_point%22%3A0%2C%22credits%22%3A1%2C%22expiration%22%3A%22-%22%2C%22login_at%22%3A1677313470%2C%22data_limit%22%3Anull%7D; _nosec_cas_session=c1NDeVE0WnFXK3R1VWVQMTJ6Tmtrdm5Vd3ZYUjI4ZHBYNDdTQU5lR040UFNjY3FPLzB0NHY1RUE2K2VtNDBGaGFKWGRSMDRaUFBscGpuRkE4Y3RuMXI4ZWhlTG1CNGsvS3JHc003czF6b2hldU1lY0RuY09NQUw1MWY3a1FMRUpqbjVNNEpva1g0ZURCOGJqVk5XNXZJdjdRaERRUTVOV2VldTdza2o0TXdibTkwMDZGS3AyR2M2QXRjSWxDSUdGcUdWbUZsSGNjczlIMmdmYkxCOWx1TkEvQ0g3YmtYdXhJM0pVeEVJVEJzND0tLTUrNVJvRExBTnU1VlF3Z2lydk82WHc9PQ%3D%3D--b113e6e0fc7e89b7f39834cbe7088eb21a5d496b; tgt=TGC-16773134699994-KZOHdbLgQQQUgT4AIs0qy9zkdEXyferuY4JZBPLJ; 

diff --git a/images/fofa.jpg b/images/fofa.jpg
diff --git a/images/result.jpg b/images/result.jpg
Original file line number	Diff line number	Diff line change
		@@ -1 +0,0 @@
		befor_router=; fofa_token=eyJhbGciOiJIUzUxMiIsImtpZCI6Ik5XWTVZakF4TVRkalltSTJNRFZsWXpRM05EWXdaakF3TURVMlkyWTNZemd3TUdRd1pUTmpZUT09IiwidHlwIjoiSldUIn0.eyJpZCI6MjQ4MjI4LCJtaWQiOjEwMDE0MTU1OSwidXNlcm5hbWUiOiJmb2ZhX2hhY2tfdGVzdCIsImV4cCI6MTY3NzU3MjY3MH0.VBTqAd1imJfC9UsnlzW-n64ktBBxdY0FHI0uoB262wavOwzQaSFjdXlvL8MmqpDZ17-JNxTTdU-11NhQkJN_eg; is_flag_login=0; user=%7B%22id%22%3A248228%2C%22mid%22%3A100141559%2C%22is_admin%22%3Afalse%2C%22username%22%3A%22fofa_hack_test%22%2C%22nickname%22%3A%22fofa_hack_test%22%2C%22email%22%3A%22fofa_test%40163.com%22%2C%22avatar_medium%22%3A%22https%3A%2F%2Fnosec.org%2Fmissing.jpg%22%2C%22avatar_thumb%22%3A%22https%3A%2F%2Fnosec.org%2Fmissing.jpg%22%2C%22key%22%3A%2281d94208789ae707007ce80b5e637d7e%22%2C%22rank_name%22%3A%22%E6%B3%A8%E5%86%8C%E7%94%A8%E6%88%B7%22%2C%22rank_level%22%3A0%2C%22company_name%22%3A%22fofa_hack_test%22%2C%22coins%22%3A0%2C%22can_pay_coins%22%3A0%2C%22fofa_point%22%3A0%2C%22credits%22%3A1%2C%22expiration%22%3A%22-%22%2C%22login_at%22%3A1677313470%2C%22data_limit%22%3Anull%7D; _nosec_cas_session=c1NDeVE0WnFXK3R1VWVQMTJ6Tmtrdm5Vd3ZYUjI4ZHBYNDdTQU5lR040UFNjY3FPLzB0NHY1RUE2K2VtNDBGaGFKWGRSMDRaUFBscGpuRkE4Y3RuMXI4ZWhlTG1CNGsvS3JHc003czF6b2hldU1lY0RuY09NQUw1MWY3a1FMRUpqbjVNNEpva1g0ZURCOGJqVk5XNXZJdjdRaERRUTVOV2VldTdza2o0TXdibTkwMDZGS3AyR2M2QXRjSWxDSUdGcUdWbUZsSGNjczlIMmdmYkxCOWx1TkEvQ0g3YmtYdXhJM0pVeEVJVEJzND0tLTUrNVJvRExBTnU1VlF3Z2lydk82WHc9PQ%3D%3D--b113e6e0fc7e89b7f39834cbe7088eb21a5d496b; tgt=TGC-16773134699994-KZOHdbLgQQQUgT4AIs0qy9zkdEXyferuY4JZBPLJ;