详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库


Posted in Python onJanuary 24, 2021

获取要爬取的URL

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

爬虫前期工作

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

用Pycharm打开项目开始写爬虫文件

字段文件items

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class NbaprojectItem(scrapy.Item):
  # define the fields for your item here like:
  # name = scrapy.Field()
  # pass
  # 创建字段的固定格式-->scrapy.Field()
  # 英文名
  engName = scrapy.Field()
  # 中文名
  chName = scrapy.Field()
  # 身高
  height = scrapy.Field()
  # 体重
  weight = scrapy.Field()
  # 国家英文名
  contryEn = scrapy.Field()
  # 国家中文名
  contryCh = scrapy.Field()
  # NBA球龄
  experience = scrapy.Field()
  # 球衣号码
  jerseyNo = scrapy.Field()
  # 入选年
  draftYear = scrapy.Field()
  # 队伍英文名
  engTeam = scrapy.Field()
  # 队伍中文名
  chTeam = scrapy.Field()
  # 位置
  position = scrapy.Field()
  # 东南部
  displayConference = scrapy.Field()
  # 分区
  division = scrapy.Field()

爬虫文件

import scrapy
import json
from nbaProject.items import NbaprojectItem

class NbaspiderSpider(scrapy.Spider):
  name = 'nbaSpider'
  allowed_domains = ['nba.com']
  # 第一次爬取的网址,可以写多个网址
  # start_urls = ['http://nba.com/']
  start_urls = ['https://china.nba.com/static/data/league/playerlist.json']
  # 处理网址的response
  def parse(self, response):
    # 因为访问的网站返回的是json格式,首先用第三方包处理json数据
    data = json.loads(response.text)['payload']['players']
    # 以下列表用来存放不同的字段
    # 英文名
    engName = []
    # 中文名
    chName = []
    # 身高
    height = []
    # 体重
    weight = []
    # 国家英文名
    contryEn = []
    # 国家中文名
    contryCh = []
    # NBA球龄
    experience = []
    # 球衣号码
    jerseyNo = []
    # 入选年
    draftYear = []
    # 队伍英文名
    engTeam = []
    # 队伍中文名
    chTeam = []
    # 位置
    position = []
    # 东南部
    displayConference = []
    # 分区
    division = []
    # 计数
    count = 1
    for i in data:
      # 英文名
      engName.append(str(i['playerProfile']['firstNameEn'] + i['playerProfile']['lastNameEn']))
      # 中文名
      chName.append(str(i['playerProfile']['firstName'] + i['playerProfile']['lastName']))
      # 国家英文名
      contryEn.append(str(i['playerProfile']['countryEn']))
      # 国家中文
      contryCh.append(str(i['playerProfile']['country']))
      # 身高
      height.append(str(i['playerProfile']['height']))
      # 体重
      weight.append(str(i['playerProfile']['weight']))
      # NBA球龄
      experience.append(str(i['playerProfile']['experience']))
      # 球衣号码
      jerseyNo.append(str(i['playerProfile']['jerseyNo']))
      # 入选年
      draftYear.append(str(i['playerProfile']['draftYear']))
      # 队伍英文名
      engTeam.append(str(i['teamProfile']['code']))
      # 队伍中文名
      chTeam.append(str(i['teamProfile']['displayAbbr']))
      # 位置
      position.append(str(i['playerProfile']['position']))
      # 东南部
      displayConference.append(str(i['teamProfile']['displayConference']))
      # 分区
      division.append(str(i['teamProfile']['division']))

      # 创建item字段对象,用来存储信息 这里的item就是对应上面导的NbaprojectItem
      item = NbaprojectItem()
      item['engName'] = str(i['playerProfile']['firstNameEn'] + i['playerProfile']['lastNameEn'])
      item['chName'] = str(i['playerProfile']['firstName'] + i['playerProfile']['lastName'])
      item['contryEn'] = str(i['playerProfile']['countryEn'])
      item['contryCh'] = str(i['playerProfile']['country'])
      item['height'] = str(i['playerProfile']['height'])
      item['weight'] = str(i['playerProfile']['weight'])
      item['experience'] = str(i['playerProfile']['experience'])
      item['jerseyNo'] = str(i['playerProfile']['jerseyNo'])
      item['draftYear'] = str(i['playerProfile']['draftYear'])
      item['engTeam'] = str(i['teamProfile']['code'])
      item['chTeam'] = str(i['teamProfile']['displayAbbr'])
      item['position'] = str(i['playerProfile']['position'])
      item['displayConference'] = str(i['teamProfile']['displayConference'])
      item['division'] = str(i['teamProfile']['division'])
      # 打印爬取信息
      print("传输了",count,"条字段")
      count += 1
      # 将字段交回给引擎 -> 管道文件
      yield item

配置文件->开启管道文件

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

# Scrapy settings for nbaProject project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#   https://docs.scrapy.org/en/latest/topics/settings.html
#   https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#   https://docs.scrapy.org/en/latest/topics/spider-middleware.html
# ----------不做修改部分---------
BOT_NAME = 'nbaProject'

SPIDER_MODULES = ['nbaProject.spiders']
NEWSPIDER_MODULE = 'nbaProject.spiders'
# ----------不做修改部分---------

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'nbaProject (+http://www.yourdomain.com)'

# Obey robots.txt rules
# ----------修改部分(可以自行查这是啥东西)---------
# ROBOTSTXT_OBEY = True
# ----------修改部分---------

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#  'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#  'nbaProject.middlewares.NbaprojectSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#  'nbaProject.middlewares.NbaprojectDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#  'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# 开启管道文件
# ----------修改部分---------
ITEM_PIPELINES = {
  'nbaProject.pipelines.NbaprojectPipeline': 300,
}
# ----------修改部分---------
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

管道文件 -> 将字段写进mysql

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
from itemadapter import ItemAdapter

import pymysql
class NbaprojectPipeline:
	# 初始化函数
  def __init__(self):
    # 连接数据库 注意修改数据库信息
    self.connect = pymysql.connect(host='域名', user='用户名', passwd='密码',
                    db='数据库', port=端口号) 
    # 获取游标
    self.cursor = self.connect.cursor()
    # 创建一个表用于存放item字段的数据
    createTableSql = """
              create table if not exists `nbaPlayer`(
              playerId INT UNSIGNED AUTO_INCREMENT,
              engName varchar(80),
              chName varchar(20),
              height varchar(20),
              weight varchar(20),
              contryEn varchar(50),
              contryCh varchar(20),
              experience int,
              jerseyNo int,
              draftYear int,
              engTeam varchar(50),
              chTeam varchar(50),
              position varchar(50),
              displayConference varchar(50),
              division varchar(50),
              primary key(playerId)
              )charset=utf8;
              """
    # 执行sql语句
    self.cursor.execute(createTableSql)
    self.connect.commit()
    print("完成了创建表的工作")
	#每次yield回来的字段会在这里做处理
  def process_item(self, item, spider):
  	# 打印item增加观赏性
  	print(item)
    # sql语句
    insert_sql = """
    insert into nbaPlayer(
    playerId, engName, 
    chName,height,
    weight,contryEn,
    contryCh,experience,
    jerseyNo,draftYear
    ,engTeam,chTeam,
    position,displayConference,
    division
    ) VALUES (null,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
    """
    # 执行插入数据到数据库操作
    # 参数(sql语句,用item字段里的内容替换sql语句的占位符)
    self.cursor.execute(insert_sql, (item['engName'], item['chName'], item['height'], item['weight']
                     , item['contryEn'], item['contryCh'], item['experience'], item['jerseyNo'],
                     item['draftYear'], item['engTeam'], item['chTeam'], item['position'],
                     item['displayConference'], item['division']))
    # 提交,不进行提交无法保存到数据库
    self.connect.commit()
    print("数据提交成功!")

启动爬虫

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

屏幕上滚动的数据

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

去数据库查看数据

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

简简单单就把球员数据爬回来啦~

到此这篇关于详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库的文章就介绍到这了,更多相关Scrapy爬虫员数据存放到Mysql内容请搜索三水点靠木以前的文章或继续浏览下面的相关文章希望大家以后多多支持三水点靠木!

Python 相关文章推荐
python使用scrapy解析js示例
Jan 23 Python
Python单元测试框架unittest使用方法讲解
Apr 13 Python
Django视图之ORM数据库查询操作API的实例
Oct 27 Python
Python3实现爬取指定百度贴吧页面并保存页面数据生成本地文档的方法
Apr 22 Python
一些Centos Python 生产环境的部署命令(推荐)
May 07 Python
python+pyqt5实现24点小游戏
Jan 24 Python
Python画图实现同一结点多个柱状图的示例
Jul 07 Python
python多线程与多进程及其区别详解
Aug 08 Python
python3.6编写的单元测试示例
Aug 17 Python
Python OpenCV去除字母后面的杂线操作
Jul 05 Python
Python random模块的使用示例
Oct 10 Python
python实现过滤敏感词
May 08 Python
Ubuntu20下的Django安装的方法步骤
Jan 24 #Python
selenium+超级鹰实现模拟登录12306
Jan 24 #Python
使用numpngw和matplotlib生成png动画的示例代码
Jan 24 #Python
详解如何修改jupyter notebook的默认目录和默认浏览器
Jan 24 #Python
详解修改Anaconda中的Jupyter Notebook默认工作路径的三种方式
Jan 24 #Python
浅析python字符串前加r、f、u、l 的区别
Jan 24 #Python
python 图像增强算法实现详解
Jan 24 #Python
You might like
通过对服务器端特性的配置加强php的安全
2006/10/09 PHP
php对图像的各种处理函数代码小结
2013/07/08 PHP
PHP的文件操作与算法实现的面试题示例
2015/08/10 PHP
如何打开php的gd2库
2017/02/09 PHP
php新建文件的方法实例
2019/09/26 PHP
解决在laravel中auth建立时候遇到的问题
2019/10/15 PHP
Nigma vs Liquid BO3 第二场2.13
2021/03/10 DOTA
js停止输出代码
2008/07/20 Javascript
简单的Jquery遮罩层代码实例
2013/11/14 Javascript
js如何判断输入字符串长度
2015/12/16 Javascript
Js 获取、判断浏览器版本信息的简单方法
2016/08/08 Javascript
JS实现一个简单的日历
2017/02/22 Javascript
详解vue.js根据不同环境(正式、测试)打包到不同目录
2018/07/13 Javascript
Javascript格式化并高亮xml字符串的方法及注意事项
2018/08/13 Javascript
JS实现的Object数组去重功能示例【数组成员为Object对象】
2019/02/01 Javascript
基于JavaScript 实现拖放功能
2019/09/12 Javascript
VUE项目axios请求头更改Content-Type操作
2020/07/24 Javascript
[38:23]2014 DOTA2国际邀请赛中国区预选赛 5 23 CIS VS LGD第一场
2014/05/24 DOTA
Flask SQLAlchemy一对一,一对多的使用方法实践
2013/02/10 Python
给Python初学者的一些编程技巧
2015/04/03 Python
Python Selenium Cookie 绕过验证码实现登录示例代码
2018/04/10 Python
Python Image模块基本图像处理操作小结
2019/04/13 Python
Python3.5面向对象程序设计之类的继承和多态详解
2019/04/24 Python
python实现图片九宫格分割
2021/03/07 Python
Python Socket TCP双端聊天功能实现过程详解
2020/06/15 Python
10个示例带你掌握python中的元组
2020/11/23 Python
TensorFlow的环境配置与安装方法
2021/02/20 Python
Matplotlib animation模块实现动态图
2021/02/25 Python
联想香港官方网站及网店:Lenovo香港
2018/04/13 全球购物
英国电信商店:BT Shop
2019/12/17 全球购物
简历的自荐信
2013/12/19 职场文书
服装设计专业自荐书范文
2013/12/30 职场文书
党章学习思想汇报
2014/01/14 职场文书
少年闰土教学反思
2014/02/22 职场文书
尊老爱亲美德少年事迹材料
2014/08/14 职场文书
Python if else条件语句形式详解
2022/03/24 Python