目標(biāo)網(wǎng)站

采用的數(shù)據(jù)解析方式：xpath、bs4、re正則

獲取網(wǎng)站中所有的五言絕句詩詞鏈接

from bs4 import BeautifulSoup import re# 獲取五言絕句代碼鏈接，以列表的形式返回 def _getLink(html):html_data = BeautifulSoup(html, 'lxml')five_html = str(html_data.find_all('div', class_="typecont")[0])ret = re.findall('<span><a href="(.*?)" target="_blank">', five_html)return ret

獲取對應(yīng)古詩頁面中，古詩的題目和內(nèi)容

from lxml import etree# 合并列表中所有字符串 def GetMsg(iterable):ret = ""for i in iterable:ret += ireturn ret# 獲取古詩題目，作者 def _getPoetryTitle(data):html = etree.HTML(data)title = GetMsg(html.xpath('//*[@id="sonsyuanwen"]/div[1]/h1/text()'))msg = GetMsg(html.xpath('//*[@id="sonsyuanwen"]/div[1]/p/a/text()'))return title + " " + msg# 獲取古詩正文 def _getPoetryText(data):html = etree.HTML(data)# //*[@id = "sonsyuanwen"]/div[1]/div[2]msg = GetMsg(html.xpath('//*[@id ="sonsyuanwen"]/div[1]/div[2]/text()'))return msg

主函數(shù)爬取過程

import requestsfrom getLink import _getLink from getPoetry import _getPoetryText, _getPoetryTitleurl_root = "https://so.gushiwen.cn"main_page = "/gushi/tangshi.aspx"headers = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 ''Safari/537.36 ' }if __name__ == '__main__':response = requests.get(url_root + main_page, headers=headers)file = open("五言絕句.txt", 'w', encoding='utf-8')if response.status_code != 200:print('獲取主頁面失敗，程序退出')exit(-1)html = response.text# _getLink獲取所有要請求的鏈接link = _getLink(html)# 請求每一個鏈接for _url in link:_response = requests.get(url_root + _url, headers=headers)if response.status_code != 200:print('獲取子頁面失敗，程序退出')exit(-1)data = _response.textif file is None:exit(-1)print(f'{_getPoetryTitle(data)}爬取開始')file.write(f" {_getPoetryTitle(data)}\n")# 寫入取詩歌內(nèi)容file.write(_getPoetryText(data))file.write("====================================================\n")print(f'{_getPoetryTitle(data)}爬取結(jié)束')# 寫入完成后關(guān)閉文件file.close()

總結(jié)

以上是生活随笔為你收集整理的Python-爬虫（爬虫练习 爬取古诗文网五言绝句）的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。