這是一個簡單爬蟲
打開糗事百科首頁https://www.qiushibaike.com/
拉到最下面點擊下一頁,觀察url變化

由此可以構造生成url的函數

def getUrls(self,pages):url1 = 'https://www.qiushibaike.com/text/page/'for i in range(1,pages):url = url1 + str(i)self.urls.append(url)

我們選擇一個內容,右擊鼠標點擊檢查

可以發現所有的糗事內容都包含在'div',attrs = {'class':'col1'}
然后再找這個標簽下面所有的'div',attrs={'class':'content'}
之后遍歷取出每個標簽里面的<span></span>里的內容即可
全部代碼如下
(這里為了測試只爬取了5頁內容)

from bs4 import BeautifulSoup import requestsclass JokeItem(object):author = Nonecontent = Noneclass GetJoke(object):def __init__(self):self.urlBase = 'https://www.qiushibaike.com/text/'self.urls = []self.item = []self.getUrls(5)self.spider(self.urls)def getHTMLText(self,url):r = requests.get(url)r.raise_for_status()r.encoding = r.apparent_encodingreturn r.text# def getPages(self):# html = self.getHTMLText(self.urlBase)# soup = BeautifulSoup(html,'html.parser')def getUrls(self,pages):url1 = 'https://www.qiushibaike.com/text/page/'for i in range(1,pages):url = url1 + str(i)self.urls.append(url)def spider(self,urls):for url in urls:htmlContent = self.getHTMLText(url)soup = BeautifulSoup(htmlContent,'lxml')anchorTag = soup.find('div',attrs = {'class':'col1'})#tags = anchorTag.find('span')tags = anchorTag.find_all('div',attrs={'class':'content'})for tag in tags:item = JokeItem()item.author = tag.find('span').getText()# item.content = tag.find('h2').getText()self.item.append(item)# print(soup.encode('GBK', 'ignore'))print(tag.find('span').getText())if __name__ == '__main__':g = GetJoke()

總結

以上是生活随笔為你收集整理的爬取糗事百科的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。