首页
所有分类
TAGS
文字工具 EMOJI BIBLE
编程 Python

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

Posted in Python onJanuary 24, 2021

获取要爬取的URL

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

爬虫前期工作

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

用Pycharm打开项目开始写爬虫文件

字段文件items

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class NbaprojectItem(scrapy.Item):
  # define the fields for your item here like:
  # name = scrapy.Field()
  # pass
  # 创建字段的固定格式-->scrapy.Field()
  # 英文名
  engName = scrapy.Field()
  # 中文名
  chName = scrapy.Field()
  # 身高
  height = scrapy.Field()
  # 体重
  weight = scrapy.Field()
  # 国家英文名
  contryEn = scrapy.Field()
  # 国家中文名
  contryCh = scrapy.Field()
  # NBA球龄
  experience = scrapy.Field()
  # 球衣号码
  jerseyNo = scrapy.Field()
  # 入选年
  draftYear = scrapy.Field()
  # 队伍英文名
  engTeam = scrapy.Field()
  # 队伍中文名
  chTeam = scrapy.Field()
  # 位置
  position = scrapy.Field()
  # 东南部
  displayConference = scrapy.Field()
  # 分区
  division = scrapy.Field()

爬虫文件

import scrapy
import json
from nbaProject.items import NbaprojectItem

class NbaspiderSpider(scrapy.Spider):
  name = 'nbaSpider'
  allowed_domains = ['nba.com']
  # 第一次爬取的网址,可以写多个网址
  # start_urls = ['http://nba.com/']
  start_urls = ['https://china.nba.com/static/data/league/playerlist.json']
  # 处理网址的response
  def parse(self, response):
    # 因为访问的网站返回的是json格式,首先用第三方包处理json数据
    data = json.loads(response.text)['payload']['players']
    # 以下列表用来存放不同的字段
    # 英文名
    engName = []
    # 中文名
    chName = []
    # 身高
    height = []
    # 体重
    weight = []
    # 国家英文名
    contryEn = []
    # 国家中文名
    contryCh = []
    # NBA球龄
    experience = []
    # 球衣号码
    jerseyNo = []
    # 入选年
    draftYear = []
    # 队伍英文名
    engTeam = []
    # 队伍中文名
    chTeam = []
    # 位置
    position = []
    # 东南部
    displayConference = []
    # 分区
    division = []
    # 计数
    count = 1
    for i in data:
      # 英文名
      engName.append(str(i['playerProfile']['firstNameEn'] + i['playerProfile']['lastNameEn']))
      # 中文名
      chName.append(str(i['playerProfile']['firstName'] + i['playerProfile']['lastName']))
      # 国家英文名
      contryEn.append(str(i['playerProfile']['countryEn']))
      # 国家中文
      contryCh.append(str(i['playerProfile']['country']))
      # 身高
      height.append(str(i['playerProfile']['height']))
      # 体重
      weight.append(str(i['playerProfile']['weight']))
      # NBA球龄
      experience.append(str(i['playerProfile']['experience']))
      # 球衣号码
      jerseyNo.append(str(i['playerProfile']['jerseyNo']))
      # 入选年
      draftYear.append(str(i['playerProfile']['draftYear']))
      # 队伍英文名
      engTeam.append(str(i['teamProfile']['code']))
      # 队伍中文名
      chTeam.append(str(i['teamProfile']['displayAbbr']))
      # 位置
      position.append(str(i['playerProfile']['position']))
      # 东南部
      displayConference.append(str(i['teamProfile']['displayConference']))
      # 分区
      division.append(str(i['teamProfile']['division']))

      # 创建item字段对象,用来存储信息 这里的item就是对应上面导的NbaprojectItem
      item = NbaprojectItem()
      item['engName'] = str(i['playerProfile']['firstNameEn'] + i['playerProfile']['lastNameEn'])
      item['chName'] = str(i['playerProfile']['firstName'] + i['playerProfile']['lastName'])
      item['contryEn'] = str(i['playerProfile']['countryEn'])
      item['contryCh'] = str(i['playerProfile']['country'])
      item['height'] = str(i['playerProfile']['height'])
      item['weight'] = str(i['playerProfile']['weight'])
      item['experience'] = str(i['playerProfile']['experience'])
      item['jerseyNo'] = str(i['playerProfile']['jerseyNo'])
      item['draftYear'] = str(i['playerProfile']['draftYear'])
      item['engTeam'] = str(i['teamProfile']['code'])
      item['chTeam'] = str(i['teamProfile']['displayAbbr'])
      item['position'] = str(i['playerProfile']['position'])
      item['displayConference'] = str(i['teamProfile']['displayConference'])
      item['division'] = str(i['teamProfile']['division'])
      # 打印爬取信息
      print("传输了",count,"条字段")
      count += 1
      # 将字段交回给引擎 -> 管道文件
      yield item

配置文件->开启管道文件

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

# Scrapy settings for nbaProject project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#   https://docs.scrapy.org/en/latest/topics/settings.html
#   https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#   https://docs.scrapy.org/en/latest/topics/spider-middleware.html
# ----------不做修改部分---------
BOT_NAME = 'nbaProject'

SPIDER_MODULES = ['nbaProject.spiders']
NEWSPIDER_MODULE = 'nbaProject.spiders'
# ----------不做修改部分---------

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'nbaProject (+http://www.yourdomain.com)'

# Obey robots.txt rules
# ----------修改部分(可以自行查这是啥东西)---------
# ROBOTSTXT_OBEY = True
# ----------修改部分---------

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#  'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#  'nbaProject.middlewares.NbaprojectSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#  'nbaProject.middlewares.NbaprojectDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#  'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# 开启管道文件
# ----------修改部分---------
ITEM_PIPELINES = {
  'nbaProject.pipelines.NbaprojectPipeline': 300,
}
# ----------修改部分---------
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

管道文件 -> 将字段写进mysql

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
from itemadapter import ItemAdapter

import pymysql
class NbaprojectPipeline:
	# 初始化函数
  def __init__(self):
    # 连接数据库 注意修改数据库信息
    self.connect = pymysql.connect(host='域名', user='用户名', passwd='密码',
                    db='数据库', port=端口号) 
    # 获取游标
    self.cursor = self.connect.cursor()
    # 创建一个表用于存放item字段的数据
    createTableSql = """
              create table if not exists `nbaPlayer`(
              playerId INT UNSIGNED AUTO_INCREMENT,
              engName varchar(80),
              chName varchar(20),
              height varchar(20),
              weight varchar(20),
              contryEn varchar(50),
              contryCh varchar(20),
              experience int,
              jerseyNo int,
              draftYear int,
              engTeam varchar(50),
              chTeam varchar(50),
              position varchar(50),
              displayConference varchar(50),
              division varchar(50),
              primary key(playerId)
              )charset=utf8;
              """
    # 执行sql语句
    self.cursor.execute(createTableSql)
    self.connect.commit()
    print("完成了创建表的工作")
	#每次yield回来的字段会在这里做处理
  def process_item(self, item, spider):
  	# 打印item增加观赏性
  	print(item)
    # sql语句
    insert_sql = """
    insert into nbaPlayer(
    playerId, engName, 
    chName,height,
    weight,contryEn,
    contryCh,experience,
    jerseyNo,draftYear
    ,engTeam,chTeam,
    position,displayConference,
    division
    ) VALUES (null,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
    """
    # 执行插入数据到数据库操作
    # 参数(sql语句,用item字段里的内容替换sql语句的占位符)
    self.cursor.execute(insert_sql, (item['engName'], item['chName'], item['height'], item['weight']
                     , item['contryEn'], item['contryCh'], item['experience'], item['jerseyNo'],
                     item['draftYear'], item['engTeam'], item['chTeam'], item['position'],
                     item['displayConference'], item['division']))
    # 提交，不进行提交无法保存到数据库
    self.connect.commit()
    print("数据提交成功!")

启动爬虫

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

屏幕上滚动的数据

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

去数据库查看数据

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

简简单单就把球员数据爬回来啦~

到此这篇关于详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库的文章就介绍到这了,更多相关Scrapy爬虫员数据存放到Mysql内容请搜索三水点靠木以前的文章或继续浏览下面的相关文章希望大家以后多多支持三水点靠木！

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

- Author -

我不是秃头哆??咪

声明：登载此文出于传递更多信息之目的，并不意味着赞同其观点或证实其描述。

Python 相关文章推荐
python爬取网站数据保存使用的方法
Nov 20 Python
在Python 3中实现类型检查器的简单方法
Jul 03 Python
浅谈python中的占位符
Nov 09 Python
Python中循环引用（import）失败的解决方法
Apr 22 Python
python 实现批量xls文件转csv文件的方法
Oct 23 Python
python 图片二值化处理（处理后为纯黑白的图片）
Nov 01 Python
Python ArgumentParse的subparser用法说明
Apr 20 Python
神经网络训练采用gpu设置的方式
Mar 03 Python
Python爬虫：从m3u8文件里提取小视频的正确操作
May 14 Python
python调用ffmpeg命令行工具便捷操作视频示例实现过程
Nov 01 Python
Pandas数据结构之Series的使用
Mar 31 Python
分享python函数常见关键字
Apr 26 Python
Ubuntu20下的Django安装的方法步骤
Jan 24 #Python
selenium+超级鹰实现模拟登录12306
Jan 24 #Python
使用numpngw和matplotlib生成png动画的示例代码
Jan 24 #Python
详解如何修改jupyter notebook的默认目录和默认浏览器
Jan 24 #Python
详解修改Anaconda中的Jupyter Notebook默认工作路径的三种方式
Jan 24 #Python
浅析python字符串前加r、f、u、l 的区别
Jan 24 #Python
python 图像增强算法实现详解
Jan 24 #Python
慢日志(1) 面向对象(3) Mint UI(1) exists(2) 相对路径 (1) REST(1) 相对路径(1) mt-swipe(1) cmd(1) 远程桌面(2)
You might like
php获取文件名称和扩展名的方法
2017/02/07 PHP
Mootools 1.2教程 正则表达式
2009/09/15 Javascript
JAVASCRIPT车架号识别/验证函数代码 汽车车架号验证程序
2012/01/08 Javascript
Javascript实现滚动图片新闻的实例代码
2013/11/27 Javascript
jqplot通过ajax动态画折线图的方法及思路
2013/12/08 Javascript
用js传递value默认值的示例代码
2014/09/11 Javascript
简单谈谈javascript Date类型
2015/09/06 Javascript
JavaScript实现的浮动层框架用法实例分析
2015/10/10 Javascript
Angularjs中的ui-bootstrap的使用教程
2017/02/19 Javascript
基于Bootstrap框架菜鸟入门教程(推荐)
2017/09/17 Javascript
node.js中路由，中间件，ge请求和post请求的参数详解
2017/12/26 Javascript
jquery动态添加以及遍历option并获取特定样式名称的option方法
2018/01/29 jQuery
浅谈Vue 性能优化之深挖数组
2018/12/11 Javascript
vue使用video插件vue-video-player详解
2020/10/23 Javascript
Python定时执行之Timer用法示例
2015/05/27 Python
Python创建对称矩阵的方法示例【基于numpy模块】
2017/10/12 Python
Python实现列表删除重复元素的三种常用方法分析
2017/11/24 Python
python使用zip将list转为json的方法
2018/12/31 Python
python爬虫快速响应服务器的做法
2020/11/24 Python
python 基于PYMYSQL使用MYSQL数据库
2020/12/24 Python
利用CSS3实现毛玻璃效果示例源码
2016/09/25 HTML / CSS
HTML5中Localstorage的使用教程
2015/07/09 HTML / CSS
详解HTML5中的Communication API基本使用方法
2016/01/29 HTML / CSS
您熟悉ORM（Object-Relation Mapping）吗？请谈谈您所理解的ORM
2016/02/08 面试题
北京天润融通.net面试题笔试题
2012/02/20 面试题
汽车维修专业毕业生的求职信分享
2013/12/04 职场文书
标准自荐信范文
2014/01/29 职场文书
幼儿园八一建军节活动方案
2014/08/27 职场文书
医学检验专业自荐信
2014/09/18 职场文书
2014年学校财务工作总结
2014/12/06 职场文书
展览会邀请函
2015/02/02 职场文书
爱岗敬业先进典型事迹材料（2016推荐版）
2016/02/26 职场文书
辞职报告（范文三篇）
2019/08/27 职场文书
关于nginx 实现jira反向代理的问题
2021/09/25 Servers
MySQL8.0升级的踩坑历险记
2021/11/01 MySQL
使用 Apache Dubbo 实现远程通信(微服务架构)
2022/02/12 Servers