Python开发的HTTP库requests详解


Posted in Python onAugust 29, 2017

Requests 是使用 Apache2 Licensed 许可证的 基于Python开发的HTTP 库,其在Python内置模块的基础上进行了高度的封装,从而使得Pythoner进行网络请求时,变得美好了许多,使用Requests可以轻而易举的完成浏览器可有的任何操作。

1. GET请求

# 1、无参数实例
  
 import requests
  
 ret = requests.get('https://github.com/timeline.json')
  
 print(ret.url)
 print(ret.text)

 # 2、有参数实例
  
 import requests
  
 payload = {'key1': 'value1', 'key2': 'value2'}
 ret = requests.get("http://httpbin.org/get", params=payload)
  
 print(ret.url)
 print(ret.text)

2. POST请求

# 1、基本POST实例
  
 import requests
  
 payload = {'key1': 'value1', 'key2': 'value2'}
 ret = requests.post("http://httpbin.org/post", data=payload)
  
 print(ret.text)
  
  
 # 2、发送请求头和数据实例
  
 import requests
 import json
  
 url = 'https://api.github.com/some/endpoint'
 payload = {'some': 'data'}
 headers = {'content-type': 'application/json'}
  
 ret = requests.post(url, data=json.dumps(payload), headers=headers)
  
 print(ret.text)
 print(ret.cookies)

3. 其它请求

requests.get(url, params=None, **kwargs)
 requests.post(url, data=None, json=None, **kwargs)
 requests.put(url, data=None, **kwargs)
 requests.head(url, **kwargs)
 requests.delete(url, **kwargs)
 requests.patch(url, data=None, **kwargs)
 requests.options(url, **kwargs)
  
 # 以上方法均是在此方法的基础上构建
 requests.request(method, url, **kwargs)

4. 请求参数

def request(method, url, **kwargs):
  """Constructs and sends a :class:`Request <Request>`.

  :param method: method for the new :class:`Request` object.
  :param url: URL for the new :class:`Request` object.
  :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
  :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
  :param json: (optional) json data to send in the body of the :class:`Request`.
  :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
  :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
  :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload.
    ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')``
    or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string
    defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers
    to add for the file.
  :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
  :param timeout: (optional) How long to wait for the server to send data
    before giving up, as a float, or a :ref:`(connect timeout, read
    timeout) <timeouts>` tuple.
  :type timeout: float or tuple
  :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
  :type allow_redirects: bool
  :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
  :param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``.
  :param stream: (optional) if ``False``, the response content will be immediately downloaded.
  :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
  :return: :class:`Response <Response>` object
  :rtype: requests.Response

  Usage::

   >>> import requests
   >>> req = requests.request('GET', 'http://httpbin.org/get')
   <Response [200]>
  """

5. 参数示例

def param_method_url():
  # requests.request(method='get', url='http://127.0.0.1:8000/test/')
  # requests.request(method='post', url='http://127.0.0.1:8000/test/')
  pass


def param_param():
  # - 可以是字典
  # - 可以是字符串
  # - 可以是字节(ascii编码以内)

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params={'k1': 'v1', 'k2': '水电费'})

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params="k1=v1&k2=水电费&k3=v3&k3=vv3")

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params=bytes("k1=v1&k2=k2&k3=v3&k3=vv3", encoding='utf8'))

  # 错误
  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params=bytes("k1=v1&k2=水电费&k3=v3&k3=vv3", encoding='utf8'))
  pass


def param_data():
  # 可以是字典
  # 可以是字符串
  # 可以是字节
  # 可以是文件对象

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data={'k1': 'v1', 'k2': '水电费'})

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data="k1=v1; k2=v2; k3=v3; k3=v4"
  # )

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data="k1=v1;k2=v2;k3=v3;k3=v4",
  # headers={'Content-Type': 'application/x-www-form-urlencoded'}
  # )

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data=open('data_file.py', mode='r', encoding='utf-8'), # 文件内容是:k1=v1;k2=v2;k3=v3;k3=v4
  # headers={'Content-Type': 'application/x-www-form-urlencoded'}
  # )
  pass


def param_json():
  # 将json中对应的数据进行序列化成一个字符串,json.dumps(...)
  # 然后发送到服务器端的body中,并且Content-Type是 {'Content-Type': 'application/json'}
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           json={'k1': 'v1', 'k2': '水电费'})


def param_headers():
  # 发送请求头到服务器端
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           json={'k1': 'v1', 'k2': '水电费'},
           headers={'Content-Type': 'application/x-www-form-urlencoded'}
           )


def param_cookies():
  # 发送Cookie到服务器端
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           data={'k1': 'v1', 'k2': 'v2'},
           cookies={'cook1': 'value1'},
           )
  # 也可以使用CookieJar(字典形式就是在此基础上封装)
  from http.cookiejar import CookieJar
  from http.cookiejar import Cookie

  obj = CookieJar()
  obj.set_cookie(Cookie(version=0, name='c1', value='v1', port=None, domain='', path='/', secure=False, expires=None,
             discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False,
             port_specified=False, domain_specified=False, domain_initial_dot=False, path_specified=False)
          )
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           data={'k1': 'v1', 'k2': 'v2'},
           cookies=obj)


def param_files():
  # 发送文件
  # file_dict = {
  # 'f1': open('readme', 'rb')
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 发送文件,定制文件名
  # file_dict = {
  # 'f1': ('test.txt', open('readme', 'rb'))
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 发送文件,定制文件名
  # file_dict = {
  # 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf")
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 发送文件,定制文件名
  # file_dict = {
  #   'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf", 'application/text', {'k1': '0'})
  # }
  # requests.request(method='POST',
  #         url='http://127.0.0.1:8000/test/',
  #         files=file_dict)

  pass


def param_auth():
  from requests.auth import HTTPBasicAuth, HTTPDigestAuth

  ret = requests.get('https://api.github.com/user', auth=HTTPBasicAuth('wupeiqi', 'sdfasdfasdf'))
  print(ret.text)

  # ret = requests.get('http://192.168.1.1',
  # auth=HTTPBasicAuth('admin', 'admin'))
  # ret.encoding = 'gbk'
  # print(ret.text)

  # ret = requests.get('http://httpbin.org/digest-auth/auth/user/pass', auth=HTTPDigestAuth('user', 'pass'))
  # print(ret)
  #


def param_timeout():
  # ret = requests.get('http://google.com/', timeout=1)
  # print(ret)

  # ret = requests.get('http://google.com/', timeout=(5, 1))
  # print(ret)
  pass


def param_allow_redirects():
  ret = requests.get('http://127.0.0.1:8000/test/', allow_redirects=False)
  print(ret.text)


def param_proxies():
  # proxies = {
  # "http": "61.172.249.96:80",
  # "https": "http://61.185.219.126:3128",
  # }

  # proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'}

  # ret = requests.get("http://www.proxy360.cn/Proxy", proxies=proxies)
  # print(ret.headers)


  # from requests.auth import HTTPProxyAuth
  #
  # proxyDict = {
  # 'http': '77.75.105.165',
  # 'https': '77.75.105.165'
  # }
  # auth = HTTPProxyAuth('username', 'mypassword')
  #
  # r = requests.get("http://www.google.com", proxies=proxyDict, auth=auth)
  # print(r.text)

  pass


def param_stream():
  ret = requests.get('http://127.0.0.1:8000/test/', stream=True)
  print(ret.content)
  ret.close()

  # from contextlib import closing
  # with closing(requests.get('http://httpbin.org/get', stream=True)) as r:
  # # 在此处理响应。
  # for i in r.iter_content():
  # print(i)


def requests_session():
  import requests

  session = requests.Session()

  ### 1、首先登陆任何页面,获取cookie

  i1 = session.get(url="http://dig.chouti.com/help/service")

  ### 2、用户登陆,携带上一次的cookie,后台对cookie中的 gpsd 进行授权
  i2 = session.post(
    url="http://dig.chouti.com/login",
    data={
      'phone': "8615131255089",
      'password': "xxxxxx",
      'oneMonth': ""
    }
  )

  i3 = session.post(
    url="http://dig.chouti.com/link/vote?linksId=8589623",
  )
  print(i3.text)

6. requests模拟登陆GitHub

import requests
 from bs4 import BeautifulSoup
 
 
 def login_github():
   """
   通过requests模块模拟浏览器登陆GitHub
   :return: 
   """
   # 获取csrf_token
   r1 = requests.get('https://github.com/login')  # 获得get请求的对象
   s1 = BeautifulSoup(r1.text, 'html.parser')   # 使用bs4解析HTML对象
   token = s1.find('input', attrs={'name': 'authenticity_token'}).get('value')   # 获取登陆授权码,即csrf_token
   get_cookies = r1.cookies.get_dict()   # 获取get请求的cookies,post请求时必须携带
   
   # 发送post登陆请求
   '''
   post登陆参数
   commit  Sign+in
   utf8  ✓
   authenticity_token  E961jQMIyC9NPwL54YPj70gv2hbXWJ…fTUd+e4lT5RAizKbfzQo4eRHsfg==
   login  JackUpDown(用户名)
   password  **********(密码)
   '''
   r2 = requests.post(
     'https://github.com/session',
     data={
       'commit': 'Sign+in',
       'utf8': '✓',
       'authenticity_token': token,
       'login': 'JackUpDown',
       'password': '**********'
     },
     cookies=get_cookies   # 携带get请求的cookies
            )
   login_cookies = r2.cookies.get_dict()  # 获得登陆成功的cookies,携带此cookies就可以访问任意GitHub页面
 
   # 携带post cookies跳转任意页面
   r3 = requests.get('https://github.com/settings/emails', cookies=login_cookies)
   print(r3.text)
Python 相关文章推荐
使用python实现扫描端口示例
Mar 29 Python
Python卸载模块的方法汇总
Jun 07 Python
Python中死锁的形成示例及死锁情况的防止
Jun 14 Python
python多进程和多线程究竟谁更快(详解)
May 29 Python
python3使用matplotlib绘制散点图
Mar 19 Python
Python字符串匹配之6种方法的使用详解
Apr 08 Python
selenium2.0中常用的python函数汇总
Aug 05 Python
python通过robert、sobel、Laplace算子实现图像边缘提取详解
Aug 21 Python
python修改FTP服务器上的文件名
Sep 11 Python
Pygame的程序开始示例代码
May 07 Python
Python基础之常用库常用方法整理
Apr 30 Python
python scipy 稀疏矩阵的使用说明
May 26 Python
Python实现按学生年龄排序的实际问题详解
Aug 29 #Python
详解Python进程间通信之命名管道
Aug 28 #Python
基于Python闭包及其作用域详解
Aug 28 #Python
利用Python查看目录中的文件示例详解
Aug 28 #Python
Python如何通过subprocess调用adb命令详解
Aug 27 #Python
Python中序列的修改、散列与切片详解
Aug 27 #Python
Python正确重载运算符的方法示例详解
Aug 27 #Python
You might like
在PHP中利用XML技术构造远程服务(上)
2006/10/09 PHP
php is_file()和is_dir()用于遍历目录时用法注意事项
2010/03/02 PHP
php中memcache 基本操作实例
2015/05/17 PHP
Zend Framework基本页面布局分析
2016/03/19 PHP
PHP基于imagick扩展实现合成图片的两种方法【附imagick扩展下载】
2017/11/14 PHP
js window.onload 加载多个函数的方法
2009/11/02 Javascript
获取客户端网卡MAC地址和IP地址实现JS代码
2013/03/17 Javascript
JavaScript操作HTML元素和样式的方法详解
2015/10/21 Javascript
jquery仿QQ登录账号选择下拉框效果
2016/03/22 Javascript
Vue实现固定定位图标滑动隐藏效果
2019/05/30 Javascript
判断“命令按钮”是否被鼠标单击详解
2019/07/31 Javascript
浅谈vue异步数据影响页面渲染
2019/10/29 Javascript
vue 百度地图(vue-baidu-map)绘制方向箭头折线实例代码详解
2020/04/28 Javascript
vant自定义二级菜单操作
2020/11/02 Javascript
vue+vant 上传图片需要注意的地方
2021/01/03 Vue.js
Python简单计算文件夹大小的方法
2015/07/14 Python
5种Python单例模式的实现方式
2016/01/14 Python
python多进程并发demo实例解析
2019/12/13 Python
利用OpenCV和Python实现查找图片差异
2019/12/19 Python
Python抓包程序mitmproxy安装和使用过程图解
2020/03/02 Python
Python openpyxl 插入折线图实例
2020/04/17 Python
python中导入 train_test_split提示错误的解决
2020/06/19 Python
python安装及变量名介绍详解
2020/12/12 Python
Python 将代码转换为可执行文件脱离python环境运行(步骤详解)
2021/01/25 Python
Shell如何接收变量输入
2016/08/06 面试题
Linux开机引导的步骤是什么
2014/02/26 面试题
护士个人自我鉴定
2014/03/24 职场文书
小学生期末评语
2014/04/21 职场文书
《谁的本领大》教后反思
2014/04/25 职场文书
暖通工程师岗位职责
2014/06/12 职场文书
大学生违纪检讨书300字
2014/10/25 职场文书
高中社区服务活动报告
2015/02/05 职场文书
大学生十八大感想
2015/08/11 职场文书
班主任班级管理心得体会
2016/01/07 职场文书
毕业欢送晚会主持词
2019/06/25 职场文书
详解Spring Bean的配置方式与实例化
2022/06/10 Java/Android