编程 Python

使用Python爬虫库BeautifulSoup遍历文档树并对标签进行操作详解

Posted in Python onJanuary 25, 2020

下面就是使用Python爬虫库BeautifulSoup对文档树进行遍历并对标签进行操作的实例，都是最基础的内容

html_doc = """
<html><head><title>The Dormouse's story</title></head>

<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc,'lxml')

一、子节点

一个Tag可能包含多个字符串或者其他Tag，这些都是这个Tag的子节点.BeautifulSoup提供了许多操作和遍历子结点的属性。

1.通过Tag的名字来获得Tag

print(soup.head)
print(soup.title)

<head><title>The Dormouse's story</title></head>
<title>The Dormouse's story</title>

通过名字的方法只能获得第一个Tag，如果要获得所有的某种Tag可以使用find_all方法

soup.find_all('a')

[<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
 <a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
 <a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

2.contents属性：将Tag的子节点通过列表的方式返回

head_tag = soup.head
head_tag.contents

[<title>The Dormouse's story</title>]

title_tag = head_tag.contents[0]
title_tag

<title>The Dormouse's story</title>

title_tag.contents

["The Dormouse's story"]

3.children：通过该属性对子节点进行循环

for child in title_tag.children:
  print(child)

The Dormouse's story

4.descendants：不论是contents还是children都是返回直接子节点，而descendants对所有tag的子孙节点进行递归循环

for child in head_tag.children:
  print(child)

<title>The Dormouse's story</title>

for child in head_tag.descendants:
  print(child)

<title>The Dormouse's story</title>
The Dormouse's story

5.string 如果tag只有一个NavigableString类型的子节点，那么tag可以使用.string得到该子节点

title_tag.string

"The Dormouse's story"

如果一个tag只有一个子节点，那么使用.string可以获得其唯一子结点的NavigableString.

head_tag.string

"The Dormouse's story"

如果tag有多个子节点，tag无法确定.string对应的是那个子结点的内容，故返回None

print(soup.html.string)

None

6.strings和stripped_strings

如果tag包含多个字符串，可以使用.strings循环获取

for string in soup.strings:
  print(string)

The Dormouse's story


The Dormouse's story


Once upon a time there were three little sisters; and their names were

Elsie
,

Lacie
 and

Tillie
;
and they lived at the bottom of a well.


...

.string输出的内容包含了许多空格和空行，使用strpped_strings去除这些空白内容

for string in soup.stripped_strings:
  print(string)

The Dormouse's story
The Dormouse's story
Once upon a time there were three little sisters; and their names were
Elsie
,
Lacie
and
Tillie
;
and they lived at the bottom of a well.
...

二、父节点

1.parent：获得某个元素的父节点

title_tag = soup.title
title_tag.parent

<head><title>The Dormouse's story</title></head>

字符串也有父节点

title_tag.string.parent

<title>The Dormouse's story</title>

2.parents：递归的获得所有父辈节点

link = soup.a
for parent in link.parents:
  if parent is None:
    print(parent)
  else:
    print(parent.name)

p
body
html
[document]

三、兄弟结点

sibling_soup = BeautifulSoup("<a><b>text1</b><c>text2</c></b></a>",'lxml')
print(sibling_soup.prettify())

<html>
 <body>
 <a>
  <b>
  text1
  </b>
  <c>
  text2
  </c>
 </a>
 </body>
</html>

1.next_sibling和previous_sibling

sibling_soup.b.next_sibling

<c>text2</c>

sibling_soup.c.previous_sibling

<b>text1</b>

在实际文档中.next_sibling和previous_sibling通常是字符串或者空白符

soup.find_all('a')

[<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
 <a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
 <a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

soup.a.next_sibling # 第一个<a></a>的next_sibling是,\n

',\n'

soup.a.next_sibling.next_sibling

<a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>

2.next_siblings和previous_siblings

for sibling in soup.a.next_siblings:
  print(repr(sibling))

',\n'
<a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>
' and\n'
<a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>
';\nand they lived at the bottom of a well.'

for sibling in soup.find(id="link3").previous_siblings:
  print(repr(sibling))

' and\n'
<a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>
',\n'
<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>
'Once upon a time there were three little sisters; and their names were\n'

四、回退与前进

1.next_element和previous_element

指向下一个或者前一个被解析的对象(字符串或tag)，即深度优先遍历的后序节点和前序节点

last_a_tag = soup.find("a", id="link3")
print(last_a_tag.next_sibling)
print(last_a_tag.next_element)

;
and they lived at the bottom of a well.
Tillie

last_a_tag.previous_element

' and\n'

2.next_elements和previous_elements

通过.next_elements和previous_elements可以向前或向后访问文档的解析内容，就好像文档正在被解析一样

for element in last_a_tag.next_elements:
  print(repr(element))

'Tillie'
';\nand they lived at the bottom of a well.'
'\n'
<p class="story">...</p>
'...'
'\n'

更多关于使用Python爬虫库BeautifulSoup遍历文档树并对标签进行操作的方法与文章大家可以点击下面的相关文章

使用Python爬虫库BeautifulSoup遍历文档树并对标签进行操作详解

- Author -

BQW_

声明：登载此文出于传递更多信息之目的，并不意味着赞同其观点或证实其描述。

Python 相关文章推荐

python分割文件的常用方法

Nov 01 Python

python文件与目录操作实例详解

Feb 22 Python

Python爬虫：通过关键字爬取百度图片

Feb 17 Python

python cx_Oracle的基础使用方法(连接和增删改查)

Nov 19 Python

Scrapy框架使用的基本知识

Oct 21 Python

Python考拉兹猜想输出序列代码实践

Jul 05 Python

Python CVXOPT模块安装及使用解析

Aug 01 Python

Python 存取npy格式数据实例

Jul 01 Python

python实现磁盘日志清理的示例

Nov 05 Python

Python识别处理照片中的条形码

Nov 16 Python

解决pytorch 保存模型遇到的问题

Mar 03 Python

有趣的二维码：使用MyQR和qrcode来制作二维码

May 10 Python

Python爬虫库BeautifulSoup获取对象(标签)名,属性,内容,注释

Jan 25 #Python

Python爬虫库BeautifulSoup的介绍与简单使用实例

Jan 25 #Python

使用Python爬虫库requests发送表单数据和JSON数据

Jan 25 #Python

Python爬虫库requests获取响应内容、响应状态码、响应头

Jan 25 #Python

使用Python爬虫库requests发送请求、传递URL参数、定制headers

Jan 25 #Python

flask框架自定义url转换器操作详解

Jan 25 #Python

常用python爬虫库介绍与简要说明

Jan 25 #Python

You might like

PHP 年龄计算函数(精确到天)

2012/06/07 PHP

php计算整个目录大小的方法

2015/06/01 PHP

smarty的section嵌套循环用法示例

2016/05/28 PHP

PHP语言对接抖音快手小红书视频/图片去水印API接口源码

2020/08/11 PHP

JTrackBar水平拖动效果

2007/07/15 Javascript

jQuery.Validate 使用笔记(jQuery Validation范例 )

2010/06/25 Javascript

Jquery Uploadify多文件上传带进度条且传递自己的参数

2013/08/28 Javascript

jquery.autocomplete修改实现键盘上下键自动填充示例

2013/11/19 Javascript

js定时调用方法成功后并停止调用示例

2014/04/08 Javascript

PHP开发者必须掌握的6个关键字

2014/04/14 Javascript

jquery弹窗插件colorbox绑定动态生成元素的方法

2014/06/20 Javascript

jQuery .tmpl() 用法示例介绍

2014/08/21 Javascript

关于编写性能高效的javascript事件的技术

2014/11/28 Javascript

JavaScript基本语法讲解

2015/06/03 Javascript

jQuery实现按钮点击遮罩加载及处理完后恢复的效果

2016/06/07 Javascript

JS实现的打字机效果完整实例

2016/06/20 Javascript

Bootstrap实现圆角、圆形头像和响应式图片

2016/12/14 Javascript

浅析从vue源码看观察者模式

2018/01/29 Javascript

vue-cli3+ts+webpack实现多入口多出口功能

2019/05/30 Javascript

JavaScript中CreateTextFile函数

2020/08/30 Javascript

VUE-ElementUI 自定义Loading图操作

2020/11/11 Javascript

vue iview 隐藏Table组件里的某一列操作

2020/11/13 Javascript

微信小程序实现登录注册功能

2020/12/29 Javascript

[54:08]LGD女子刀塔学院 DOTA2炼金术士教学

2014/01/09 DOTA

Python2.7读取PDF文件的方法示例

2017/07/13 Python

python 命令行传入参数实现解析

2019/08/30 Python

iphoneX 适配客户端H5页面的方法教程

2017/12/08 HTML / CSS

跑鞋、网球鞋、网球拍、服装及装备：Holabird Sports

2016/09/19 全球购物

敏捷开发的主要原则都有哪些

2015/04/26 面试题

门诊挂号室室长岗位职责

2013/11/27 职场文书

金属材料工程毕业生个人的自我评价

2013/11/28 职场文书

解除施工合同协议书

2014/10/17 职场文书

给老师的感谢信

2015/01/20 职场文书

社区工作者个人总结

2015/02/28 职场文书

寻衅滋事罪辩护词

2015/05/21 职场文书

Matplotlib可视化之添加让统计图变得简单易懂的注释

2021/06/11 Python