我們?cè)谠L問網(wǎng)站的時(shí)候，發(fā)現(xiàn)有些網(wǎng)頁(yè)ID 是按順序排列的數(shù)字，這個(gè)時(shí)候我們就可以使用ID遍歷的方式來爬取內(nèi)容。但是局限性在于有些ID數(shù)字在10位數(shù)左右，那么這樣爬取效率就會(huì)很低很低！



import itertools
from common import download


def iteration():
max_errors = 5 # maximum number of consecutive download errors allowed
num_errors = 0 # current number of consecutive download errors
for page in itertools.count(1):
url = 'http://example.webscraping.com/view/-{}'.format(page)
html = download(url)
if html is None:
# received an error trying to download this webpage
num_errors += 1
if num_errors == max_errors:
# reached maximum amount of errors in a row so exit
break
# so assume have reached the last country ID and can stop downloading
else:
# success - can scrape the result
# ...
num_errors = 0

轉(zhuǎn)載于:https://www.cnblogs.com/mrruning/p/7638459.html

總結(jié)

以上是生活随笔為你收集整理的用python写网络爬虫 -从零开始 3 编写ID遍历爬虫的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。