爬取起點中文網小說

Scrapy框架結構

• 引擎（ENGINE）
• 調度器（SCHEDULER）
• 下載器（DOWNLOADER）
• 爬蟲（SPIDERS）
• 項目管道（ITEM PIPELINES）
• 下載器中間件（Downloader Middlewares）
• 爬蟲中間件（Spider Middlewares）

需求分析

目標網站 https://www.qidian.com/rank/hotsales?style=1&page=1

提取內容為：小說名稱、作者、類型和形式

項目

創建項目，在命令行定位到放項目的目錄

scrapy startproject qidian_hot

打開pycharm

在spiders目錄下新建爬蟲源文件qidian_hot_spider.py，代碼如下

# !/user/bin/env python # -*- coding:utf-8 -*- # author:Zfy date:2021/7/4 20:30from scrapy import Request from scrapy.spiders import Spider from qidian_hot.items import QidianHotItemclass HotSalesSpider(Spider):name = 'hot'qidian_headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"}current_page = 1def start_requests(self):url = "https://www.qidian.com/rank/hotsales?style=1&page=1"yield Request(url, headers=self.qidian_headers, callback=self.parse)def parse(self, response, **kwargs): # 數據解析list_selector = response.xpath("//div[@class='book-mid-info']")for one_selector in list_selector:# 獲取小說信息name = one_selector.xpath("h4/a/text()").extract()[0]# 獲取作者author = one_selector.xpath("p[1]/a[1]/text()").extract()[0]# 獲取類型type = one_selector.xpath("p[1]/a[2]/text()").extract()[0]# 獲取形式form = one_selector.xpath("p[1]/span/text()").extract()[0]item = QidianHotItem()item["name"] = nameitem["author"] = authoritem["type"] = typeitem["form"] = formyield item# 獲取下一頁urlself.current_page += 1if self.current_page <= 5:new_url = "https://www.qidian.com/rank/hotsales?style=1&page=%d" % self.current_pageyield Request(new_url)

item.py

class QidianHotItem(scrapy.Item):# define the fields for your item here like:name = scrapy.Field()author = scrapy.Field()type = scrapy.Field()form = scrapy.Field()

pipelines.py

先在setting.py中把管道的注釋去掉，大概在67行

from itemadapter import ItemAdapter from scrapy.exceptions import DropItemclass QidianHotPipeline:def __init__(self):self.author_set = set()def process_item(self, item, spider):if item["name"] in self.author_set:raise DropItem("查找到重復姓名的項目：%s" % item)return item

運行爬蟲，命令行模式下輸入，自動生成csv文件

scrapy crawl hot -o hot.csv

保存到MySQL數據庫

在本機MySQL添加數據庫qidian，添加表hot（表名稱和爬蟲源文件qidian_hot_spider.py中的類HotSalesSpider的name屬性要一致），然后添加字段

安裝mysqlclient

pip install mysqlclient

先在配置文件settings.py添加數據庫的變量

MYSQL_DB_NAME = 'qidian' # 對應數據庫中的表名稱 MYSQL_HOST = 'localhost' MYSQL_USER = 'root' MYSQL_PASSWORD = '123456'

然后在pipelines.py中添加代碼

import MySQLdb from itemadapter import ItemAdapter from scrapy.exceptions import DropItemclass QidianHotPipeline:def __init__(self):self.author_set = set()def process_item(self, item, spider):if item["name"] in self.author_set:raise DropItem("查找到重復姓名的項目：%s" % item)return itemclass MySQlPipeline(object):def open_spider(self, spider): # 在spider開始之前，調用一次db_name = spider.settings.get('MYSQL_DB_NAME', 'qidian')host = spider.settings.get('MYSQL_HOST', 'localhost')user = spider.settings.get('MYSQL_USER', 'root')pwd = spider.settings.get('MYSQL_PASSWORD', '123456')# 鏈接數據庫self.db_conn = MySQLdb.connect(db=db_name,host=host,user=user,password=pwd,charset="utf8")self.db_cursor = self.db_conn.cursor() # 得到游標def process_item(self, item, spider): # 處理每一個itemvalues = (item["name"], item["author"], item["type"], item["form"])# 確定sqlsql = "insert into hot(name,author,type,form) values(%s, %s, %s, %s)"self.db_cursor.execute(sql, values)return itemdef close_spider(self, spider): # 在spider結束時，調用一次self.db_conn.commit() # 提交數據self.db_cursor.close()self.db_conn.close()

在配置文件settings.py添加管道，67行

ITEM_PIPELINES = {'qidian_hot.pipelines.QidianHotPipeline': 300,'qidian_hot.pipelines.MySQlPipeline': 400, }

新建start.py，添加代碼

from scrapy import cmdlinecmdline.execute("scrapy crawl hot".split())

運行start.py，實現了數據的永久保存，打開數據表查看

保存到MongoDB數據庫

安裝 pymongo

settings.py中配置

ITEM_PIPELINES = {'qidian_hot.pipelines.QidianHotPipeline': 300,# 'qidian_hot.pipelines.MySQlPipeline': 400,'qidian_hot.pipelines.MongoDBPipeline': 400, }# MongoDB MONGODB_HOST = "localhost" MONGODB_PORT = 27017 MONGODB_NAME = "qidian" MONGODB_COLLECTION = "hot"

pipelines.py添加代碼

import pymongoclass MongoDBPipeline(object):def open_spider(self, spider): # 在spider開始之前，調用一次host = spider.settings.get("MONGODB_HOST", "localhost")port = spider.settings.get("MONGODB_PORT", 27017)db_name = spider.settings.get("MONGODB_NAME", "qidian")collection_name = spider.settings.get("MONGODB_COLLECTION", "hot")self.db_client = pymongo.MongoClient(host=host, port=port) # 客戶端對象# 指定數據庫self.db = self.db_client[db_name]# 指定集合self.db_collection = self.db[collection_name]def process_item(self, item, spider): # 處理每一個itemitem_dict = dict(item)self.db_collection.insert_one(item_dict)def close_spider(self, spider): # 在spider結束時，調用一次self.db_client.close()

打開start.py，運行，打開 MongoDB compass 查看

《從零開始學Scrapy網絡爬蟲》，這本書挺不錯的，有原理，有源碼，有視頻，還有ppt

總結

以上是生活随笔為你收集整理的Scrapy(1) 爬取起点中文网小说，并保存到数据库的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。