-
Notifications
You must be signed in to change notification settings - Fork 0
/
search.xml
137 lines (133 loc) · 73.6 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>接口自动化平台设计方案v1.0</title>
<url>/2021/03/06/api_automationtest_platform/design_v1/</url>
<content><![CDATA[<h2 id="1-技术框架"><a href="#1-技术框架" class="headerlink" title="1. 技术框架"></a>1. 技术框架</h2><p>从自己的技术栈、目前流行的框架、开发效率等方面综合考虑，选择了以下框架<br>前端：vue + element-ui<br>后端：python + Django + restframework<br>测试框架 HTTPrunner</p>
<h2 id="2-功能模块"><a href="#2-功能模块" class="headerlink" title="2. 功能模块"></a>2. 功能模块</h2><h3 id="2-1-环境管理"><a href="#2-1-环境管理" class="headerlink" title="2.1 环境管理"></a>2.1 环境管理</h3><p>新增环境、拷贝环境<br>编辑环境变量</p>
<h3 id="2-2-接口管理"><a href="#2-2-接口管理" class="headerlink" title="2.2 接口管理"></a>2.2 接口管理</h3><p>分模块、版本管理接口<br>接口文档，包含URI、请求方法、请求头、请求体、响应主体、demo等信息，支持在线调试<br>接口配置，配置请求的必要信息和响应断言、数据库校验等</p>
<span id="more"></span>
<h3 id="2-3-测试用例"><a href="#2-3-测试用例" class="headerlink" title="2.3 测试用例"></a>2.3 测试用例</h3><p>每个测试用例就是由一个或多个接口组成<br>包含关联模块、用例标题、用例步骤、创建人、创建时间、修改时间等信息</p>
<h3 id="2-4-测试计划"><a href="#2-4-测试计划" class="headerlink" title="2.4 测试计划"></a>2.4 测试计划</h3><p>每个测试计划就是由一个或多个用例组成<br>包含计划标题、用例集、负责人、创建人、创建时间等信息</p>
<h3 id="2-5-Mock管理"><a href="#2-5-Mock管理" class="headerlink" title="2.5 Mock管理"></a>2.5 Mock管理</h3><p>用于模拟响应</p>
<h3 id="2-6-告警管理"><a href="#2-6-告警管理" class="headerlink" title="2.6 告警管理"></a>2.6 告警管理</h3><p>邮件告警、短信告警、企微告警</p>
<h3 id="2-7-定时任务"><a href="#2-7-定时任务" class="headerlink" title="2.7 定时任务"></a>2.7 定时任务</h3><p>定时执行自动化测试用例</p>
<h3 id="2-8-函数管理"><a href="#2-8-函数管理" class="headerlink" title="2.8 函数管理"></a>2.8 函数管理</h3><p>提供已经封装好的函数，比如获取当前时间戳、加减乘除等<br>支持自定义函数</p>
<h3 id="2-9-用户管理（超级管理员）"><a href="#2-9-用户管理（超级管理员）" class="headerlink" title="2.9 用户管理（超级管理员）"></a>2.9 用户管理（超级管理员）</h3><p>添加用户，修改用户信息</p>
<h3 id="2-10-项目管理（超级管理员）"><a href="#2-10-项目管理（超级管理员）" class="headerlink" title="2.10 项目管理（超级管理员）"></a>2.10 项目管理（超级管理员）</h3><p>添加项目，编辑项目</p>
<h2 id="3-其他模块"><a href="#3-其他模块" class="headerlink" title="3. 其他模块"></a>3. 其他模块</h2><h3 id="3-1-分布式执行测试"><a href="#3-1-分布式执行测试" class="headerlink" title="3.1 分布式执行测试"></a>3.1 分布式执行测试</h3><p>使用celery进行任务调度</p>
<h3 id="3-2-权限管理"><a href="#3-2-权限管理" class="headerlink" title="3.2 权限管理"></a>3.2 权限管理</h3><p>用户进行分组，权限按照分组设计</p>
<h3 id="3-3-数据库校验"><a href="#3-3-数据库校验" class="headerlink" title="3.3 数据库校验"></a>3.3 数据库校验</h3><p>可以进行数据库校验</p>
<h3 id="3-4-postman生成测试用例"><a href="#3-4-postman生成测试用例" class="headerlink" title="3.4 postman生成测试用例"></a>3.4 postman生成测试用例</h3><p>可以将postman脚本一键导入生成用例</p>
]]></content>
<categories>
<category>1. 接口自动化平台</category>
<category>设计文档</category>
</categories>
<tags>
<tag>接口自动化</tag>
<tag>HttpRunner</tag>
<tag>Django</tag>
</tags>
</entry>
<entry>
<title>【Python】常用方法二次封装（二）</title>
<url>/2021/03/06/python/frequently_used_method/file-path/</url>
<content><![CDATA[<h2 id="文件及路径相关方法封装"><a href="#文件及路径相关方法封装" class="headerlink" title="文件及路径相关方法封装"></a>文件及路径相关方法封装</h2><h3 id="1-获取目录下的所有文件"><a href="#1-获取目录下的所有文件" class="headerlink" title="1. 获取目录下的所有文件"></a>1. 获取目录下的所有文件</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_all_files_path</span>(<span class="params">path</span>) -> list:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 获取某个目录下所有的文件路径（包括隐藏文件、子目录下的文件）</span></span><br><span class="line"><span class="string"> :param path: 绝对路径</span></span><br><span class="line"><span class="string"> :return: 返回每个文件的绝对路径</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> all_files = []</span><br><span class="line"> p = os.path.abspath(path)</span><br><span class="line"> <span class="keyword">for</span> path, dir_names, files <span class="keyword">in</span> os.walk(p):</span><br><span class="line"> <span class="keyword">for</span> file <span class="keyword">in</span> files:</span><br><span class="line"> all_files.append(os.path.join(path, file))</span><br><span class="line"> <span class="keyword">return</span> all_files</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_all_files</span>(<span class="params">path</span>) -> list:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 获取某个目录下所有的文件（包括隐藏文件、子目录下的文件）</span></span><br><span class="line"><span class="string"> :param path: 绝对路径</span></span><br><span class="line"><span class="string"> :return: 返回每个文件的文件名</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> all_files = []</span><br><span class="line"> p = os.path.abspath(path)</span><br><span class="line"> <span class="keyword">for</span> path, dir_names, files <span class="keyword">in</span> os.walk(p):</span><br><span class="line"> <span class="keyword">for</span> file <span class="keyword">in</span> files:</span><br><span class="line"> all_files.append(file)</span><br><span class="line"> <span class="keyword">return</span> all_files</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_files</span>(<span class="params">path</span>) -> list:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> :param path: 绝对路径</span></span><br><span class="line"><span class="string"> :return: 返回路径下的所有文件名(含隐藏文件)，不含子目录中的文件</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> all_files = []</span><br><span class="line"> <span class="keyword">for</span> p <span class="keyword">in</span> os.listdir(path):</span><br><span class="line"> <span class="keyword">if</span> os.path.isfile(p):</span><br><span class="line"> all_files.append(p)</span><br><span class="line"> <span class="keyword">return</span> all_files</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_current_path_files</span>() -> list:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> :return: 返回同级目录下所有文件名，不含子目录中的文件</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> all_files = []</span><br><span class="line"> <span class="keyword">for</span> p <span class="keyword">in</span> os.listdir(os.path.dirname(__file__)):</span><br><span class="line"> <span class="keyword">if</span> os.path.isfile(p):</span><br><span class="line"> all_files.append(p)</span><br><span class="line"> <span class="keyword">return</span> all_files</span><br></pre></td></tr></table></figure>
<span id="more"></span>
<h3 id="2-搜索目录下某个文件"><a href="#2-搜索目录下某个文件" class="headerlink" title="2. 搜索目录下某个文件"></a>2. 搜索目录下某个文件</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">search_file</span>(<span class="params">path, file</span>) -> list:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> :return: 搜索目录下是否存在某个文件，如果没有返回一个空列表，否则返回所有匹配的文件路径</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"></span><br><span class="line"> the_files = []</span><br><span class="line"> <span class="keyword">for</span> path, dir_names, files <span class="keyword">in</span> os.walk(os.path.abspath(path)):</span><br><span class="line"> <span class="keyword">for</span> f <span class="keyword">in</span> files:</span><br><span class="line"> <span class="keyword">if</span> f == file:</span><br><span class="line"> the_files.append(os.path.join(path, f))</span><br><span class="line"> <span class="keyword">return</span> the_files</span><br></pre></td></tr></table></figure>
<h3 id="3-os-path相关方法介绍"><a href="#3-os-path相关方法介绍" class="headerlink" title="3. os.path相关方法介绍"></a>3. os.path相关方法介绍</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">path = __file__ <span class="comment"># 当前文件路径（含文件名，有可能是相对路径，这取决于你在哪里运行该文件）</span></span><br><span class="line"></span><br><span class="line">os.path.abspath(path) </span><br><span class="line"><span class="comment"># 返回绝对路径</span></span><br><span class="line"></span><br><span class="line">os.path.basename(path) </span><br><span class="line"><span class="comment"># 返回文件名</span></span><br><span class="line"></span><br><span class="line">os.path.commonprefix(<span class="built_in">list</span>) </span><br><span class="line"><span class="comment"># 返回list(多个路径)中，所有path共有的最长的路径，该方法不建议使用，实际返回的是list中的path前面相同部分，当某些目录前缀相同时，它没有作区分，如 ['/a/b1/c1', '/a/b2/c2']，返回的是/a/b，而不是/a/</span></span><br><span class="line"></span><br><span class="line">os.path.dirname(path)</span><br><span class="line"><span class="comment"># 返回文件路径, 可能是相对路径，取决于你在哪里运行该文件，如python ./a/b/test.py，test.py里面调用以下方法，则返回的是./a/b</span></span><br><span class="line"></span><br><span class="line">os.path.exists(path)</span><br><span class="line"><span class="comment"># 如果路径 path 存在，返回 True；如果路径 path 不存在，返回 False。</span></span><br><span class="line"></span><br><span class="line">os.path.lexists(path)</span><br><span class="line"><span class="comment"># 和os.path.exists(path)的区别在于，当path是软连接时，不管链接的目标文件是否存在，都返回True</span></span><br><span class="line"></span><br><span class="line">os.path.expanduser(path)</span><br><span class="line"><span class="comment"># 把path中包含的"~"和"~user"转换成用户目录</span></span><br><span class="line"></span><br><span class="line">os.path.expandvars(path) </span><br><span class="line"><span class="comment"># 根据环境变量的值替换path中包含的"$var"和"${var}" </span></span><br><span class="line"></span><br><span class="line">os.path.getatime(path) </span><br><span class="line"><span class="comment"># 返回最近访问时间（浮点型秒数）</span></span><br><span class="line"></span><br><span class="line">os.path.getmtime(path) </span><br><span class="line"><span class="comment"># 返回最近文件修改时间</span></span><br><span class="line"></span><br><span class="line">os.path.getctime(path) </span><br><span class="line"><span class="comment"># 返回文件 path 创建时间</span></span><br><span class="line"></span><br><span class="line">os.path.getsize(path) </span><br><span class="line"><span class="comment"># 返回文件大小，如果文件不存在就返回错误</span></span><br><span class="line"></span><br><span class="line">os.path.isabs(path) </span><br><span class="line"><span class="comment"># 判断是否为绝对路径</span></span><br><span class="line"></span><br><span class="line">os.path.isfile(path) </span><br><span class="line"><span class="comment"># 判断路径是否为文件</span></span><br><span class="line"></span><br><span class="line">os.path.isdir(path) </span><br><span class="line"><span class="comment"># 判断路径是否为目录</span></span><br><span class="line"></span><br><span class="line">os.path.islink(path)</span><br><span class="line"><span class="comment"># 判断路径是否为链接</span></span><br><span class="line"></span><br><span class="line">os.path.ismount(path) </span><br><span class="line"><span class="comment"># 判断路径是否为挂载点</span></span><br><span class="line"></span><br><span class="line">os.path.join(path1[, path2[, ...]]) </span><br><span class="line"><span class="comment"># 把目录和文件名合成一个路径</span></span><br><span class="line"></span><br><span class="line">os.path.normcase(path) </span><br><span class="line"><span class="comment"># 转换path的大小写和斜杠</span></span><br><span class="line"></span><br><span class="line">os.path.normpath(path) </span><br><span class="line"><span class="comment"># 规范path字符串形式</span></span><br><span class="line"></span><br><span class="line">os.path.realpath(path) </span><br><span class="line"><span class="comment"># 返回path的真实路径</span></span><br><span class="line"></span><br><span class="line">os.path.relpath(path[, start]) </span><br><span class="line"><span class="comment"># 从start开始计算相对路径</span></span><br><span class="line"></span><br><span class="line">os.path.samefile(path1, path2) </span><br><span class="line"><span class="comment"># 判断目录或文件是否相同</span></span><br><span class="line"></span><br><span class="line">os.path.sameopenfile(fp1, fp2) </span><br><span class="line"><span class="comment"># 判断fp1和fp2是否指向同一文件</span></span><br><span class="line"></span><br><span class="line">os.path.samestat(stat1, stat2) </span><br><span class="line"><span class="comment"># 判断stat tuple stat1和stat2是否指向同一个文件</span></span><br><span class="line"></span><br><span class="line">os.path.split(path) </span><br><span class="line"><span class="comment"># 把路径分割成 dirname 和 basename，返回一个元组</span></span><br><span class="line"></span><br><span class="line">os.path.splitdrive(path) </span><br><span class="line"><span class="comment"># 一般用在 windows 下，返回驱动器名和路径组成的元组</span></span><br><span class="line"></span><br><span class="line">os.path.splitext(path) </span><br><span class="line"><span class="comment"># 分割路径，返回路径名和文件扩展名的元组</span></span><br><span class="line"></span><br><span class="line">os.path.splitunc(path) </span><br><span class="line"><span class="comment"># 把路径分割为加载点与文件</span></span><br><span class="line"></span><br><span class="line">os.path.walk(path, visit, arg) </span><br><span class="line"><span class="comment"># 遍历path，进入每个目录都调用visit函数，visit函数必须有3个参数(arg, dirname, names)，dirname表示当前目录的目录名，names代表当前目录下的所有文件名，args则为walk的第三个参数</span></span><br><span class="line"></span><br><span class="line">os.path.supports_unicode_filenames </span><br><span class="line"><span class="comment"># 设置是否支持unicode路径名</span></span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category>2. Python</category>
<category>常用方法封装</category>
</categories>
<tags>
<tag>Python</tag>
<tag>方法封装</tag>
</tags>
</entry>
<entry>
<title>【Python】常用方法二次封装（一）</title>
<url>/2021/03/06/python/frequently_used_method/time-md5-logger/</url>
<content><![CDATA[<h3 id="1-时间相关操作"><a href="#1-时间相关操作" class="headerlink" title="1. 时间相关操作"></a>1. 时间相关操作</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> datetime</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">now_datetime</span>(<span class="params">_format=<span class="string">"%Y-%m-%d %H:%M:%S"</span></span>) -> str:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 获取当前日期时间</span></span><br><span class="line"><span class="string"> :param _format: 日期时间格式，默认：年-月-日 时:分:秒</span></span><br><span class="line"><span class="string"> :return: 当前日期时间</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> <span class="keyword">return</span> time.strftime(_format)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">now_timestamp</span>(<span class="params">ms=<span class="literal">False</span></span>) -> int:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 获取当前时间戳</span></span><br><span class="line"><span class="string"> :param ms: 是否获取毫秒级时间戳，默认是秒级</span></span><br><span class="line"><span class="string"> :return: 时间戳</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> timestamp = time.time()</span><br><span class="line"> <span class="keyword">if</span> ms:</span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">round</span>(timestamp * <span class="number">1000</span>)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">int</span>(timestamp)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_the_datetime</span>(<span class="params">_type: <span class="built_in">str</span>, n: <span class="built_in">int</span>, _format=<span class="string">"%Y-%m-%d %H:%M:%S"</span></span>):</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 获取相对日期时间</span></span><br><span class="line"><span class="string"> :param _type: 类型</span></span><br><span class="line"><span class="string"> :param n: 数值</span></span><br><span class="line"><span class="string"> :param _format: 返回格式</span></span><br><span class="line"><span class="string"> :return: 返回前n天、后n天、前n个小时、后n个小时等</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> now = datetime.datetime.now()</span><br><span class="line"> <span class="keyword">if</span> _type == <span class="string">"day"</span>:</span><br><span class="line"> <span class="keyword">return</span> (now + datetime.timedelta(days=n)).strftime(_format)</span><br><span class="line"> <span class="keyword">elif</span> _type == <span class="string">"hour"</span>:</span><br><span class="line"> <span class="keyword">return</span> (now + datetime.timedelta(hours=n)).strftime(_format)</span><br><span class="line"> <span class="keyword">elif</span> _type == <span class="string">"minutes"</span>:</span><br><span class="line"> <span class="keyword">return</span> (now + datetime.timedelta(minutes=n)).strftime(_format)</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">None</span></span><br></pre></td></tr></table></figure>
<span id="more"></span>
<h3 id="2-计算md5值"><a href="#2-计算md5值" class="headerlink" title="2. 计算md5值"></a>2. 计算md5值</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> hashlib</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">string_md5</span>(<span class="params">_str</span>) -> str:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 计算字符串md5</span></span><br><span class="line"><span class="string"> :param _str: 待计算md5的字符串</span></span><br><span class="line"><span class="string"> :return: md5值</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">type</span>(_str) <span class="keyword">is</span> <span class="keyword">not</span> <span class="built_in">bytes</span>:</span><br><span class="line"> _str = _str.encode()</span><br><span class="line"> my_md5 = hashlib.md5()</span><br><span class="line"> my_md5.update(_str)</span><br><span class="line"> my_md5_digest = my_md5.hexdigest()</span><br><span class="line"> <span class="keyword">return</span> my_md5_digest</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">file_md5</span>(<span class="params">file_path</span>) -> str:</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 计算文件md5</span></span><br><span class="line"><span class="string"> :param file_path: 待计算md5值的文件</span></span><br><span class="line"><span class="string"> :return: md5值</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> <span class="keyword">with</span> <span class="built_in">open</span>(file_path, <span class="string">'rb'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> md5obj = hashlib.md5()</span><br><span class="line"> md5obj.update(f.read())</span><br><span class="line"> _hash = md5obj.hexdigest()</span><br><span class="line"> <span class="keyword">return</span> _hash</span><br></pre></td></tr></table></figure>
<h3 id="3-日志类二次封装"><a href="#3-日志类二次封装" class="headerlink" title="3. 日志类二次封装"></a>3. 日志类二次封装</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> logging</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_logger</span>():</span></span><br><span class="line"> <span class="comment"># 新建一个logger</span></span><br><span class="line"> _logger = logging.getLogger()</span><br><span class="line"> _logger.setLevel(logging.NOTSET) <span class="comment"># 等级总开关</span></span><br><span class="line"> rq = time.strftime(<span class="string">'%Y%m%d'</span>, time.localtime(time.time()))</span><br><span class="line"> log_path = os.path.dirname(__file__) + <span class="string">'/logs/'</span></span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> os.path.exists(log_path):</span><br><span class="line"> os.mkdir(log_path)</span><br><span class="line"> log_name = log_path + rq + <span class="string">'.log'</span></span><br><span class="line"> <span class="comment"># 创建一个FileHandler用于把日志写入文件</span></span><br><span class="line"> fh = logging.FileHandler(log_name, mode=<span class="string">'a'</span>)</span><br><span class="line"> fh.setLevel(logging.INFO) <span class="comment"># 输出到file的log等级的开关</span></span><br><span class="line"> <span class="comment"># 定义日志格式</span></span><br><span class="line"> formatter = logging.Formatter(<span class="string">"[%(asctime)s] [%(levelname)s] [%(filename)s: %(lineno)d]: %(message)s"</span>)</span><br><span class="line"> fh.setFormatter(formatter)</span><br><span class="line"> _logger.addHandler(fh)</span><br><span class="line"> <span class="comment"># 创建一个StreamHandler用于把日志输出到控制台</span></span><br><span class="line"> ch = logging.StreamHandler()</span><br><span class="line"> ch.setLevel(logging.WARNING) <span class="comment"># 输出控制台的log等级开关</span></span><br><span class="line"> ch.setFormatter(formatter)</span><br><span class="line"> _logger.addHandler(ch)</span><br><span class="line"> <span class="keyword">return</span> _logger</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 使用方法</span></span><br><span class="line">logger = get_logger()</span><br><span class="line">logger.warning(<span class="string">"haha"</span>)</span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category>2. Python</category>
<category>常用方法封装</category>
</categories>
<tags>
<tag>Python</tag>
<tag>方法封装</tag>
</tags>
</entry>
<entry>
<title>【Python】多进程入门</title>
<url>/2021/03/06/python/basic_knowledge/multiprocessing/</url>
<content><![CDATA[<p>参考博文：<br><a href="https://www.cnblogs.com/jiangfan95/p/11439207.html">https://www.cnblogs.com/jiangfan95/p/11439207.html</a></p>
<h2 id="一、多进程模块multiprocessing"><a href="#一、多进程模块multiprocessing" class="headerlink" title="一、多进程模块multiprocessing"></a>一、多进程模块multiprocessing</h2><p>python中的<a href="https://blog.csdn.net/PeakMoment/article/details/105563890">多线程</a>因为<a href="https://blog.csdn.net/PeakMoment/article/details/105090897">GIL</a>的原因，无法利用多核优势，如果想要充分地使用多核CPU的资源，在python中大部分情况需要使用多进程，这就要用到多进程模块multiprocessing。<br><strong>常用的类、方法有：</strong><br>multiprocessing.Process 用于创建子进程<br>multiprocessing.Queue 队列，用于进程间的数据同步、共享<br>multiprocessing.Pipe 管道，用于进程间的数据同步、共享<br>multiprocessing.Lock 互斥锁<br>multiprocessing.RLock 递归锁<br>multiprocessing.Value 用于进程间的数据同步、共享</p>
<h3 id="1-1-multiprocessing-Process"><a href="#1-1-multiprocessing-Process" class="headerlink" title="1.1 multiprocessing.Process()"></a>1.1 multiprocessing.Process()</h3><p>以下为Process的构造方法：</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Process</span>(<span class="params"><span class="built_in">object</span></span>):</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span>(<span class="params">self, group=<span class="literal">None</span>, target=<span class="literal">None</span>, name=<span class="literal">None</span>, args=(<span class="params"></span>), kwargs={}</span>):</span></span><br><span class="line"> self.name = <span class="string">''</span> </span><br><span class="line"> self.daemon = <span class="literal">False</span> </span><br><span class="line"> self.authkey = <span class="literal">None</span></span><br><span class="line"> self.exitcode = <span class="literal">None</span></span><br><span class="line"> self.ident = <span class="number">0</span></span><br><span class="line"> self.pid = <span class="number">0</span></span><br><span class="line"> self.sentinel = <span class="literal">None</span></span><br></pre></td></tr></table></figure>
<p>从构造方法中可以看出，我们实例化一个Process进程对象时，可以填写的参数有五个：<br>group：该参数一般不填写，实际也没有用到<br>target：目标函数名，就是你在子进程中需要执行的函数，必填<br>name：进程名字，是字符串类型，非必填<br>args：target指定的函数中需要用到的参数，是一个元祖，非必填<br>kwargs：target指定的函数中需要用到的key-value形式的参数，是一个字典，非必填<br>使用multiprocess.Process()创建启动子进程，有两种方法。</p>
<span id="more"></span>
<p><strong>方法一：直接调用Process()实例化一个进程对象</strong></p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 方法一 直接调用</span></span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">do_something</span>(<span class="params">name</span>):</span></span><br><span class="line"> print(<span class="string">'%s runing'</span> % name)</span><br><span class="line"> time.sleep(random.randint(<span class="number">2</span>, <span class="number">5</span>))</span><br><span class="line"> print(<span class="string">'%s running end'</span> % name)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">p1 = Process(target=do_something, args=(<span class="string">'process-1'</span>,)) </span><br><span class="line"><span class="comment"># 这里要注意的是args这个参数是一个元祖，当只有一个元素时，记得加逗号</span></span><br><span class="line">p2 = Process(target=do_something, args=(<span class="string">'process-2'</span>,))</span><br><span class="line">p3 = Process(target=do_something, args=(<span class="string">'process-3'</span>,))</span><br><span class="line">p4 = Process(target=do_something, args=(<span class="string">'process-4'</span>,))</span><br><span class="line"></span><br><span class="line">p1.start()</span><br><span class="line">p2.start()</span><br><span class="line">p3.start()</span><br><span class="line">p4.start()</span><br><span class="line">time.sleep(<span class="number">1</span>)</span><br><span class="line">print(<span class="string">'主进程结束'</span>)</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p><strong>方法二：继承Process类，重写run方法</strong></p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 方法二：重写run方法</span></span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">MyProcess</span>(<span class="params">Process</span>):</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span>(<span class="params">self, name</span>):</span></span><br><span class="line"> <span class="built_in">super</span>(MyProcess, self).__init__() <span class="comment"># 调用父类的构造函数</span></span><br><span class="line"> self.name = name</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run</span>(<span class="params">self</span>) -> <span class="keyword">None</span>:</span></span><br><span class="line"> print(<span class="string">'%s runing'</span> % self.name)</span><br><span class="line"> time.sleep(random.randint(<span class="number">2</span>, <span class="number">5</span>))</span><br><span class="line"> self.do_something()</span><br><span class="line"> print(<span class="string">'%s running end'</span> % self.name)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">do_something</span>(<span class="params">self</span>):</span></span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">p1 = MyProcess(<span class="string">'process-1'</span>)</span><br><span class="line"><span class="comment"># 这里要注意的是args这个参数是一个元祖，当只有一个元素时，记得加逗号</span></span><br><span class="line">p2 = MyProcess(<span class="string">'process-2'</span>)</span><br><span class="line">p3 = MyProcess(<span class="string">'process-3'</span>)</span><br><span class="line">p4 = MyProcess(<span class="string">'process-4'</span>)</span><br><span class="line"></span><br><span class="line">p1.start()</span><br><span class="line">p2.start()</span><br><span class="line">p3.start()</span><br><span class="line">p4.start()</span><br><span class="line">time.sleep(<span class="number">1</span>)</span><br><span class="line">print(<span class="string">'主进程结束'</span>)</span><br></pre></td></tr></table></figure>
<h3 id="1-2-join进程阻塞"><a href="#1-2-join进程阻塞" class="headerlink" title="1.2 join进程阻塞"></a>1.2 join进程阻塞</h3><p>有时我们需要等待创建的某个子进程执行完毕之后，再继续执行主进程的任务，这时候就需要用到join()这个方法。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">do_something</span>(<span class="params">name</span>):</span></span><br><span class="line"> print(<span class="string">'%s running'</span> % name)</span><br><span class="line"> <span class="keyword">if</span> name == <span class="string">"process-2"</span>:</span><br><span class="line"> time.sleep(<span class="number">4</span>)</span><br><span class="line"> time.sleep(<span class="number">1</span>)</span><br><span class="line"> print(<span class="string">'%s running end'</span> % name)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">p1 = Process(target=do_something, args=(<span class="string">'process-1'</span>,))</span><br><span class="line">p2 = Process(target=do_something, args=(<span class="string">'process-2'</span>,))</span><br><span class="line">p1.start()</span><br><span class="line">p2.start()</span><br><span class="line">p2.join() <span class="comment"># 这里设置让p2进程执行完毕之后再继续执行主进程，注意 join方法需要在进程start之后才能使用</span></span><br><span class="line">print(<span class="string">'主进程结束'</span>)</span><br></pre></td></tr></table></figure>
<h3 id="1-3-daemon守护进程"><a href="#1-3-daemon守护进程" class="headerlink" title="1.3 daemon守护进程"></a>1.3 daemon守护进程</h3><p>daemon()这个方法用于设置进程为守护进程，守护进程会随着主进程结束而结束，不管它有没有执行完，具体看如下例子：</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">do_something</span>(<span class="params">name</span>):</span></span><br><span class="line"> print(<span class="string">'%s running'</span> % name)</span><br><span class="line"> <span class="keyword">if</span> name == <span class="string">"process-2"</span>:</span><br><span class="line"> time.sleep(<span class="number">4</span>)</span><br><span class="line"> time.sleep(<span class="number">1</span>)</span><br><span class="line"> print(<span class="string">'%s running end'</span> % name)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">p1 = Process(target=do_something, args=(<span class="string">'process-1'</span>,))</span><br><span class="line">p2 = Process(target=do_something, args=(<span class="string">'process-2'</span>,))</span><br><span class="line">p2.daemon = <span class="literal">True</span> <span class="comment"># 设置p2进程为守护进程，注意设置守护进程需要在进程start之前设置</span></span><br><span class="line">p1.start()</span><br><span class="line">p2.start()</span><br><span class="line">time.sleep(<span class="number">1</span>)</span><br><span class="line">print(<span class="string">'主进程结束'</span>)</span><br></pre></td></tr></table></figure>
<p>output</p>
<blockquote>
<p>process-1 running<br>process-2 running<br>process-1 running end<br>主进程结束</p>
</blockquote>
<p>从输出的打印我们可以看出，p2进程还没有执行完毕，就随着主进程结束而结束了。</p>
<h2 id="二、进程间同步"><a href="#二、进程间同步" class="headerlink" title="二、进程间同步"></a>二、进程间同步</h2><p>进程之间数据不共享，但是共享同一套文件系统，所以访问同一个文件，或同一个打印终端，是没有问题的，而共享带来的是竞争，竞争带来的结果就是错乱，如何控制，就是加锁处理。<br>加锁可以保证多个进程修改同一块数据时，同一时间只能有一个任务可以进行修改，即串行的修改，没错，速度是慢了，但牺牲了速度却保证了数据安全。</p>
<h3 id="2-1-互斥锁Lock"><a href="#2-1-互斥锁Lock" class="headerlink" title="2.1 互斥锁Lock()"></a>2.1 互斥锁Lock()</h3><p>下面举一个模拟抢票的例子：</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 先在当前目录新建db.txt，文件的内容为：{"count":1}</span></span><br><span class="line"><span class="comment"># 注意一定要用双引号，不然json无法识别</span></span><br><span class="line"><span class="comment"># 购票行为由并发变成了串行，牺牲了运行效率，但保证了数据安全</span></span><br><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process,Lock</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> json</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">search</span>():</span></span><br><span class="line"> <span class="string">"""查询剩余票数"""</span></span><br><span class="line"> dic = json.load(<span class="built_in">open</span>(<span class="string">'db.txt'</span>))</span><br><span class="line"> print(<span class="string">'剩余票数%s'</span> % dic[<span class="string">'count'</span>])</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get</span>():</span></span><br><span class="line"> <span class="string">"""购票"""</span></span><br><span class="line"> dic = json.load(<span class="built_in">open</span>(<span class="string">'db.txt'</span>))</span><br><span class="line"> time.sleep(<span class="number">0.1</span>) <span class="comment"># 模拟读数据的网络延迟</span></span><br><span class="line"> <span class="keyword">if</span> dic[<span class="string">'count'</span>] > <span class="number">0</span>:</span><br><span class="line"> dic[<span class="string">'count'</span>] -= <span class="number">1</span></span><br><span class="line"> time.sleep(<span class="number">0.1</span>) <span class="comment"># 模拟写数据的网络延迟</span></span><br><span class="line"> json.dump(dic, <span class="built_in">open</span>(<span class="string">'db.txt'</span>, <span class="string">'w'</span>)) <span class="comment"># 更新剩余票数</span></span><br><span class="line"> print(<span class="string">'购票成功'</span> + <span class="string">"*"</span> * <span class="number">50</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">task</span>(<span class="params">_lock</span>):</span></span><br><span class="line"> search()</span><br><span class="line"> _lock.acquire() <span class="comment"># 获取锁</span></span><br><span class="line"> get()</span><br><span class="line"> _lock.release() <span class="comment"># 释放锁</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> lock = Lock() <span class="comment"># 互斥锁</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>): <span class="comment"># 模拟并发100个客户端抢票</span></span><br><span class="line"> p = Process(target=task, args=(lock,))</span><br><span class="line"> p.start()</span><br></pre></td></tr></table></figure>
<h3 id="2-2-递归锁RLock"><a href="#2-2-递归锁RLock" class="headerlink" title="2.2 递归锁RLock()"></a>2.2 递归锁RLock()</h3><p>递归锁，意思是一个进程如果已经获取了递归锁，那么它还可以再次获取，而不需要等待锁被释放。下面举一个简单的例子：</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process, RLock</span><br><span class="line"><span class="comment"># 初始化一个递归锁</span></span><br><span class="line">lock = RLock()</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test_RLock</span>(<span class="params">n</span>):</span></span><br><span class="line"> <span class="keyword">if</span> n == <span class="number">0</span>:</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> lock.acquire()</span><br><span class="line"> print(<span class="string">"获取锁"</span>)</span><br><span class="line"> test_RLock(n-<span class="number">1</span>) <span class="comment"># 进入递归，会再次调用lock.acquire()获取锁，如果这个锁是互斥锁，就会导致死锁。但是递归锁不会，它一旦被获取之后，可以在该进程内重复获取。</span></span><br><span class="line"> lock.release()</span><br><span class="line"> print(<span class="string">"释放锁"</span>)</span><br><span class="line"></span><br><span class="line">test_RLock(<span class="number">2</span>)</span><br></pre></td></tr></table></figure>
<h2 id="三、进程间通信"><a href="#三、进程间通信" class="headerlink" title="三、进程间通信"></a>三、进程间通信</h2><p>与线程不同的是，因为每个进程都有自己<strong>独立的内存空间</strong>，多进程间没有任何共享状态，没办法直接通信，必须通过一定的方法进行进程间通信，除了用文件进行共享数据来达到进程间通信的目的之外，进程间通信的常见的方法还有<strong>队列、管道、信号量、共享内存、事件</strong>等。<br>文件共享数据实现进程间通信的缺点：<br>1）效率低（共享数据基于文件，而文件是硬盘上的数据）<br>2）需要自己加锁处理</p>
<p>因此我们最好找寻一种解决方案能够兼顾：<br>1）效率高（多个进程共享一块内存的数据）<br>2）帮我们处理好锁问题。</p>
<p>mutiprocessing模块为我们提供的基于消息的IPC通信机制：队列和管道。</p>
<p>1 队列和管道都是将数据存放于内存中<br>2 队列又是基于（管道+锁）实现的，可以让我们从复杂的锁问题中解脱出来， 我们应该尽量避免使用共享数据，尽可能使用消息传递和队列，避免处理复杂的同步和锁问题，而且在进程数目增多时，往往可以获得更好的可获展性</p>
<h3 id="3-1-队列（推荐使用）"><a href="#3-1-队列（推荐使用）" class="headerlink" title="3.1 队列（推荐使用）"></a>3.1 队列（推荐使用）</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Queue</span><br><span class="line">q = Queue(<span class="number">3</span>) <span class="comment"># 定义一个容量为3的队列</span></span><br><span class="line"></span><br><span class="line">q.put(<span class="number">1</span>) </span><br><span class="line">q.put(<span class="number">2</span>)</span><br><span class="line">q.put(<span class="number">3</span>)</span><br><span class="line">print(q.full()) <span class="comment"># 满了会返回True</span></span><br><span class="line"></span><br><span class="line">print(q.get()) <span class="comment"># 队列是先进先出，所以首先拿到的是1</span></span><br><span class="line">print(q.get())</span><br><span class="line">print(q.get())</span><br><span class="line">print(q.empty()) <span class="comment"># 空了会返回True</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>output:</p>
<blockquote>
<p>True<br>1<br>2<br>3<br>True</p>
</blockquote>
<h3 id="3-2-队列的应用-生产者与消费者模式"><a href="#3-2-队列的应用-生产者与消费者模式" class="headerlink" title="3.2 队列的应用-生产者与消费者模式"></a>3.2 队列的应用-生产者与消费者模式</h3><p><strong>什么是生产者消费者模式</strong></p>
<p>生产者消费者模式是通过一个容器来解决生产者和消费者的强耦合问题。生产者和消费者彼此之间不直接通讯，而通过阻塞队列来进行通讯，所以生产者生产完数据之后不用等待消费者处理，直接扔给阻塞队列，消费者不找生产者要数据，而是直接从阻塞队列里取，阻塞队列就相当于一个缓冲区，平衡了生产者和消费者的处理能力。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process, Queue</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">consumer</span>(<span class="params">q</span>):</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> res = q.get() </span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>, <span class="number">2</span>))</span><br><span class="line"> print(<span class="string">'\033[45m%s 吃 %s\033[0m'</span> % (os.getpid(),res))</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">producer</span>(<span class="params">q</span>):</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>):</span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>, <span class="number">2</span>))</span><br><span class="line"> res = <span class="string">'包子%s'</span> % i</span><br><span class="line"> q.put(res)</span><br><span class="line"> print(<span class="string">'\033[44m%s 生产了 %s\033[0m'</span> % (os.getpid(), res))</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> q = Queue()</span><br><span class="line"> <span class="comment"># 生产者们:即厨师们</span></span><br><span class="line"> p1 = Process(target=producer, args=(q,))</span><br><span class="line"> <span class="comment"># 消费者们:即吃货们</span></span><br><span class="line"> c1 = Process(target=consumer, args=(q,))</span><br><span class="line"> <span class="comment"># 开始</span></span><br><span class="line"> p1.start()</span><br><span class="line"> c1.start()</span><br><span class="line"> print(<span class="string">'主进程结束'</span>)</span><br></pre></td></tr></table></figure>
<p>如果你执行了上述代码，你会发现主进程永远不会结束，原因是：生产者p在生产完后就结束了，但是消费者c没有在队列获取到值，就会一直卡在q.get()这一步。</p>
<p>解决方式很简单，让生产者在生产完毕后，往队列中再发一个结束信号，这样消费者在接收到结束信号后就可以break出死循环。<br>我们接着上面那个通俗的例子，假设厨房和餐厅大堂是完全隔离的，只能通过一个小窗口进行传递包子，如果厨师每天规定只能生产100个包子，生产完之后就下班了。但是在大堂消费者并不知道厨师下班了，就一直在等着这个包子递出来，这样就会导致消费者进程一直阻塞。那么解决方式就是，厨师生产完全部包子之后，就往传递包子的小窗口里面挂一个牌子，通知消费者我已经下班了，不继续生产了，你们可以不用等了。</p>
<p>具体看如下代码：</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process,Queue</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">consumer</span>(<span class="params">q</span>):</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> res = q.get()</span><br><span class="line"> <span class="comment"># 收到结束信号None则结束</span></span><br><span class="line"> <span class="keyword">if</span> res <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">break</span> </span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>,<span class="number">3</span>))</span><br><span class="line"> print(<span class="string">'\033[45m%s 吃 %s\033[0m'</span> % (os.getpid(), res))</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">producer</span>(<span class="params">q</span>):</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>):</span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>,<span class="number">3</span>))</span><br><span class="line"> res = <span class="string">'包子%s'</span> % i</span><br><span class="line"> q.put(res)</span><br><span class="line"> print(<span class="string">'\033[44m%s 生产了 %s\033[0m'</span> % (os.getpid(), res))</span><br><span class="line"> q.put(<span class="literal">None</span>) <span class="comment"># 发送结束信号None，或者其他约定的结束信号</span></span><br><span class="line"> </span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> q = Queue()</span><br><span class="line"> <span class="comment"># 生产者们:即厨师们</span></span><br><span class="line"> p1 = Process(target=producer,args=(q,))</span><br><span class="line"> <span class="comment"># 消费者们:即吃货们</span></span><br><span class="line"> c1 = Process(target=consumer,args=(q,))</span><br><span class="line"> <span class="comment"># 开始</span></span><br><span class="line"> p1.start()</span><br><span class="line"> c1.start()</span><br><span class="line"> print(<span class="string">'主'</span>)</span><br></pre></td></tr></table></figure>
<p>注意：结束信号None，不一定要由生产者发，主进程里同样可以发，但主进程需要等生产者结束后才应该发送该信号，我们可以给生产者进程加上join()方法。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process,Queue</span><br><span class="line"><span class="keyword">import</span> time,random,os</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">consumer</span>(<span class="params">q</span>):</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> res=q.get()</span><br><span class="line"> <span class="keyword">if</span> res <span class="keyword">is</span> <span class="literal">None</span>:<span class="keyword">break</span> <span class="comment">#收到结束信号则结束</span></span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>,<span class="number">3</span>))</span><br><span class="line"> print(<span class="string">'\033[45m%s 吃 %s\033[0m'</span> %(os.getpid(),res))</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">producer</span>(<span class="params">q</span>):</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">2</span>):</span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>,<span class="number">3</span>))</span><br><span class="line"> res=<span class="string">'包子%s'</span> %i</span><br><span class="line"> q.put(res)</span><br><span class="line"> print(<span class="string">'\033[44m%s 生产了 %s\033[0m'</span> %(os.getpid(),res))</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> q=Queue()</span><br><span class="line"> <span class="comment">#生产者们:即厨师们</span></span><br><span class="line"> p1=Process(target=producer,args=(q,))</span><br><span class="line"> <span class="comment">#消费者们:即吃货们</span></span><br><span class="line"> c1=Process(target=consumer,args=(q,))</span><br><span class="line"></span><br><span class="line"> <span class="comment">#开始</span></span><br><span class="line"> p1.start()</span><br><span class="line"> c1.start()</span><br><span class="line"></span><br><span class="line"> p1.join() <span class="comment"># 等待生产者进程结束</span></span><br><span class="line"> q.put(<span class="literal">None</span>) <span class="comment"># 发送结束信号</span></span><br><span class="line"> print(<span class="string">'主'</span>)</span><br></pre></td></tr></table></figure>
<p>但上述解决方式，在有多个生产者和多个消费者时，应该怎么做呢？有几个消费者就发几次信号？没错，我们要保证通知到每一个消费者，以免它们一直阻塞。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process,Queue</span><br><span class="line"><span class="keyword">import</span> time,random,os</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">consumer</span>(<span class="params">q</span>):</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> res=q.get()</span><br><span class="line"> <span class="keyword">if</span> res <span class="keyword">is</span> <span class="literal">None</span>:<span class="keyword">break</span> <span class="comment">#收到结束信号则结束</span></span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>,<span class="number">3</span>))</span><br><span class="line"> print(<span class="string">'\033[45m%s 吃 %s\033[0m'</span> %(os.getpid(),res))</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">producer</span>(<span class="params">name,q</span>):</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">2</span>):</span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>,<span class="number">3</span>))</span><br><span class="line"> res=<span class="string">'%s%s'</span> %(name,i)</span><br><span class="line"> q.put(res)</span><br><span class="line"> print(<span class="string">'\033[44m%s 生产了 %s\033[0m'</span> %(os.getpid(),res))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> q=Queue()</span><br><span class="line"> <span class="comment">#生产者们:即厨师们</span></span><br><span class="line"> p1=Process(target=producer,args=(<span class="string">'包子'</span>,q))</span><br><span class="line"> p2=Process(target=producer,args=(<span class="string">'骨头'</span>,q))</span><br><span class="line"> p3=Process(target=producer,args=(<span class="string">'泔水'</span>,q))</span><br><span class="line"></span><br><span class="line"> <span class="comment">#消费者们:即吃货们</span></span><br><span class="line"> c1=Process(target=consumer,args=(q,))</span><br><span class="line"> c2=Process(target=consumer,args=(q,))</span><br><span class="line"></span><br><span class="line"> <span class="comment">#开始</span></span><br><span class="line"> p1.start()</span><br><span class="line"> p2.start()</span><br><span class="line"> p3.start()</span><br><span class="line"> c1.start()</span><br><span class="line"></span><br><span class="line"> p1.join() <span class="comment">#必须保证生产者全部生产完毕,才应该发送结束信号</span></span><br><span class="line"> p2.join()</span><br><span class="line"> p3.join()</span><br><span class="line"> q.put(<span class="literal">None</span>) <span class="comment">#有几个消费者就应该发送几次结束信号None</span></span><br><span class="line"> q.put(<span class="literal">None</span>) <span class="comment">#发送结束信号</span></span><br><span class="line"> print(<span class="string">'主'</span>)</span><br></pre></td></tr></table></figure>
<p>有几个消费者就应该发送几次结束信号None。<br>其实我们的思路无非是发送结束信号而已，有另外一种队列提供了这种机制，JoinableQueue([maxsize])：这就像是一个Queue对象，但队列允许项目的使用者通知生成者项目已经被成功处理。通知进程是使用<strong>共享的信号和条件变量</strong>来实现的。</p>
<p>参数介绍：<br> maxsize是队列中允许最大项数，省略则无大小限制。 </p>
<p>方法介绍：<br>JoinableQueue的实例p除了与Queue对象相同的方法之外还具有：</p>
<p>q.task_done()：使用者使用此方法发出信号，表示q.get()的返回项目已经被处理。如果调用此方法的次数大于从队列中删除项目的数量，将引发ValueError异常。</p>
<p>q.join()：生产者调用此方法进行阻塞，直到队列中所有的项目均被处理。阻塞将持续到队列中的每个项目均调用q.task_done()方法为止。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process,JoinableQueue</span><br><span class="line"><span class="keyword">import</span> time,random,os</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">consumer</span>(<span class="params">q</span>):</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> res=q.get()</span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>,<span class="number">3</span>))</span><br><span class="line"> print(<span class="string">'\033[45m%s 吃 %s\033[0m'</span> %(os.getpid(),res))</span><br><span class="line"> q.task_done() <span class="comment">#向q.join()发送一次信号,证明一个数据已经被取走了</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">producer</span>(<span class="params">name,q</span>):</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>):</span><br><span class="line"> time.sleep(random.randint(<span class="number">1</span>,<span class="number">3</span>))</span><br><span class="line"> res=<span class="string">'%s%s'</span> %(name,i)</span><br><span class="line"> q.put(res)</span><br><span class="line"> print(<span class="string">'\033[44m%s 生产了 %s\033[0m'</span> %(os.getpid(),res))</span><br><span class="line"> q.join()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> q=JoinableQueue()</span><br><span class="line"> <span class="comment">#生产者们:即厨师们</span></span><br><span class="line"> p1=Process(target=producer,args=(<span class="string">'包子'</span>,q))</span><br><span class="line"> p2=Process(target=producer,args=(<span class="string">'骨头'</span>,q))</span><br><span class="line"> p3=Process(target=producer,args=(<span class="string">'泔水'</span>,q))</span><br><span class="line"></span><br><span class="line"> <span class="comment">#消费者们:即吃货们</span></span><br><span class="line"> c1=Process(target=consumer,args=(q,))</span><br><span class="line"> c2=Process(target=consumer,args=(q,))</span><br><span class="line"> c1.daemon=<span class="literal">True</span></span><br><span class="line"> c2.daemon=<span class="literal">True</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">#开始</span></span><br><span class="line"> p_l=[p1,p2,p3,c1,c2]</span><br><span class="line"> <span class="keyword">for</span> p <span class="keyword">in</span> p_l:</span><br><span class="line"> p.start()</span><br><span class="line"></span><br><span class="line"> p1.join()</span><br><span class="line"> p2.join()</span><br><span class="line"> p3.join()</span><br><span class="line"> print(<span class="string">'主'</span>) </span><br><span class="line"> </span><br><span class="line"> <span class="comment">#主进程等--->p1,p2,p3等---->c1,c2</span></span><br><span class="line"> <span class="comment">#p1,p2,p3结束了,证明c1,c2肯定全都收完了p1,p2,p3发到队列的数据</span></span><br><span class="line"> <span class="comment">#因而c1,c2也没有存在的价值了,应该随着主进程的结束而结束,所以设置成守护进程</span></span><br></pre></td></tr></table></figure>
<h3 id="3-3-管道"><a href="#3-3-管道" class="headerlink" title="3.3 管道"></a>3.3 管道</h3><p>创建管道的类：<br>Pipe([duplex]):在进程之间创建一条管道，并返回元组（conn1,conn2）,其中conn1，conn2表示管道两端的连接对象，强调一点：必须在产生Process对象之前产生管道</p>
<p>参数介绍：<br>dumplex:默认管道是全双工的，如果将duplex射成False，conn1只能用于接收，conn2只能用于发送。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process,Pipe</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> time,os</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">consumer</span>(<span class="params">p,name</span>):</span></span><br><span class="line"> left,right=p</span><br><span class="line"> left.close()</span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> baozi=right.recv()</span><br><span class="line"> print(<span class="string">'%s 收到包子:%s'</span> %(name,baozi))</span><br><span class="line"> <span class="keyword">except</span> EOFError:</span><br><span class="line"> right.close()</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">producer</span>(<span class="params">seq,p</span>):</span></span><br><span class="line"> left,right=p</span><br><span class="line"> right.close()</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> seq:</span><br><span class="line"> left.send(i)</span><br><span class="line"> <span class="comment"># time.sleep(1)</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> left.close()</span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> left,right=Pipe()</span><br><span class="line"></span><br><span class="line"> c1=Process(target=consumer,args=((left,right),<span class="string">'c1'</span>))</span><br><span class="line"> c1.start()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> seq=(i <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>))</span><br><span class="line"> producer(seq,(left,right))</span><br><span class="line"></span><br><span class="line"> right.close()</span><br><span class="line"> left.close()</span><br><span class="line"></span><br><span class="line"> c1.join()</span><br><span class="line"> print(<span class="string">'主进程'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 基于管道实现进程间通信（与队列的方式是类似的，队列就是管道加锁实现的）</span></span><br></pre></td></tr></table></figure>
<p>注意：生产者和消费者都没有使用管道的某个端点，就应该将其关闭，如在生产者中关闭管道的右端，在消费者中关闭管道的左端。如果忘记执行这些步骤，程序可能再消费者中的recv()操作上挂起。管道是由操作系统进行引用计数的,必须在所有进程中关闭管道后才能生产EOFError异常。因此在生产者中关闭管道不会有任何效果，付费消费者中也关闭了相同的管道端点。</p>
<p>管道可以用于双向通信，利用通常在客户端/服务器中使用的请求／响应模型或远程过程调用，就可以使用管道编写与进程交互的程序</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Process,Pipe</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> time,os</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">adder</span>(<span class="params">p,name</span>):</span></span><br><span class="line"> server,client=p</span><br><span class="line"> client.close()</span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> x,y=server.recv()</span><br><span class="line"> <span class="keyword">except</span> EOFError:</span><br><span class="line"> server.close()</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> res=x+y</span><br><span class="line"> server.send(res)</span><br><span class="line"> print(<span class="string">'server done'</span>)</span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> server,client=Pipe()</span><br><span class="line"></span><br><span class="line"> c1=Process(target=adder,args=((server,client),<span class="string">'c1'</span>))</span><br><span class="line"> c1.start()</span><br><span class="line"></span><br><span class="line"> server.close()</span><br><span class="line"></span><br><span class="line"> client.send((<span class="number">10</span>,<span class="number">20</span>))</span><br><span class="line"> print(client.recv())</span><br><span class="line"> client.close()</span><br><span class="line"></span><br><span class="line"> c1.join()</span><br><span class="line"> print(<span class="string">'主进程'</span>)</span><br><span class="line"><span class="comment">#注意：send()和recv()方法使用pickle模块对对象进行序列化。</span></span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category>2. Python</category>
<category>基本知识</category>
</categories>
<tags>
<tag>Python</tag>
<tag>多进程</tag>
</tags>
</entry>
</search>