當(dāng)前位置：首頁 > 前端技术 > HTML >内容正文

HTML

爬虫521错误(又是一次和可爱的前端vs的故事)

發(fā)布時(shí)間：2023/12/2 HTML 20 豆豆

生活随笔收集整理的這篇文章主要介紹了爬虫521错误(又是一次和可爱的前端vs的故事) 小編覺得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.

起因:

　　今天突然想重構(gòu)一下代理池,并且想擴(kuò)充一下代理,所以就想著爬點(diǎn)代理IP,然后就有了下面的故事

一上來先進(jìn)行了一頓操作:

def get_xxdaili(url):headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36',"Host": 'www.66ip.cn',"Referer": 'http://www.66ip.cn/index.html',"Upgrade-Insecure-Requests": '1',}res = requests.get(url=url, headers=headers)

然后看都沒看狀態(tài)碼直接xpath取:過了一會(huì)黑人問號(hào)??????,喵喵喵,為啥是空,點(diǎn)開源代碼,啥都有,哦,可能是xpath寫的有問題,又進(jìn)行了微調(diào),還是取不到,突然感覺這個(gè)網(wǎng)站好騷,怎么就取不到呢.有重新分析了一次源代碼與Network, 然后看了眼返回狀態(tài)碼,521,進(jìn)過分析以后得出了問題的原因:

　　發(fā)生 521 錯(cuò)誤是因?yàn)樵捶?wù)器拒絕來自 Cloudflare 的連接。更具體地說，Cloudflare 嘗試通過端口 80 或 443 連接到您的源服務(wù)器，但卻收到連接被拒絕的錯(cuò)誤。

我發(fā)現(xiàn)cookie的參數(shù)很有問題,所以估計(jì)是cookie的問題(之前沒遇到521,所以一開始也不清楚哪里的問題),網(wǎng)上整理了一下資料,原來是進(jìn)行了cookie加密(js),所以接下來思路是很清晰了,就是分析js,然后拿到加密后的數(shù)據(jù).所以我直接拿到相應(yīng)信息

經(jīng)過js優(yōu)化:

經(jīng)過參考資料,和自己的研究,發(fā)現(xiàn)關(guān)鍵地方

于是我把 eval 替換成 console.log

經(jīng)過整理得到(上圖與下面js代碼聲明的不一樣,但是基本上一樣,):

從上面可以看出網(wǎng)站在得到cookie之后又進(jìn)行了一次加密.所以我們?cè)诎焉厦娴拇a?document.cookie 中的數(shù)據(jù)得到就是我們想要的cookie了

__jsl_clearance=1558954345.795|0|V%2Bp1UYNNA%2Fc4wboCF4SQoA%2Fy9j0%3D;Expires=Mon, 27-May-19 11:52:25 GMT;Path=/;? 這就是我們要得到的數(shù)據(jù),在加上第一次我們需要的cookie ,然后將它們進(jìn)行拼接就是我們要的cookie了,

想要在python下運(yùn)行js,有很多包,這里我們使用 js2py 與 execjs (這兩個(gè)都可以)

pip install Js2Py pip install PyExecJS

?兩個(gè)代碼基本類似,而且由于時(shí)間關(guān)系,很多地方?jīng)]有優(yōu)化,只是實(shí)現(xiàn)的功能,希望大家見諒(后期優(yōu)化)

js2py 實(shí)現(xiàn)?

#!/usr/bin/env python # -*- coding: utf-8 -*- # @Time : 2019/5/27 15:19 # @Author : yhl # @Software: PyCharmimport re import time import js2py import random import requests from lxml import etreeheaders = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36',"Host": 'www.66ip.cn',# "Referer": 'http://www.66ip.cn/index.html',"Upgrade-Insecure-Requests": '1', }def get_521_content(url):req = requests.get(url=url, headers=headers)cookies = req.cookiescookies = '; '.join(['='.join(item) for item in cookies.items()])txt_521 = req.texttxt_521 = ''.join(re.findall('<script>(.*?)</script>', txt_521))return (txt_521, cookies, req)def fixed_fun(function,url):print(function)js = function.replace("<script>", "").replace("</script>", "").replace("{eval(", "{var my_data_1 = (")# print(js)# 使用js2py的js交互功能獲得剛才賦值的data1對(duì)象context = js2py.EvalJs()context.execute(js)js_temp = context.my_data_1print(js_temp)index1 = js_temp.find("document.")index2 = js_temp.find("};if((")js_temp = js_temp[index1:index2].replace("document.cookie", "my_data_2")new_js_temp = re.sub(r'document.create.*?firstChild.href', '"{}"'.format(url), js_temp)# print(new_js_temp)# print(type(new_js_temp)) context.execute(new_js_temp)data = context.my_data_2# print(data)__jsl_clearance = str(data).split(';')[0]return __jsl_clearancedef get_66daili(url):txt_521, cookies, req = get_521_content(url)print(req.status_code)if req.status_code == 521:__jsl_clearance = fixed_fun(txt_521,url)headers['Cookie'] = __jsl_clearance + ';' + cookiesres1 = requests.get(url=url, headers=headers)else:res1 = reqres1.encoding = 'gb2312'html = etree.HTML(res1.text)tr_list = html.xpath('//table//tr')for num, tr in enumerate(tr_list, 1):proxy_ip_dict = {}if num != 1:proxy_ip_dict['proxy_ip'] = ''.join(tr.xpath('.//td[1]/text()'))proxy_ip_dict['proxy_port'] = ''.join(tr.xpath('.//td[2]/text()'))proxy_ip_dict['proxy_local'] = ''.join(tr.xpath('.//td[3]/text()'))proxy_ip_dict['proxy_anonymous'] = ''.join(tr.xpath('.//td[4]/text()'))print(proxy_ip_dict) #proxy_type 網(wǎng)頁沒有,自己添加+代理檢測(cè)def main():for i in range(1, 2000):get_66daili('http://www.66ip.cn/%s.html' % (i))if __name__ == '__main__':for i in range(2, 2000):get_66daili('http://www.66ip.cn/%s.html' % (i))time.sleep(random.uniform(1, 2))

execjs 實(shí)現(xiàn)(容易出bug,但是還是可以出來的,親測(cè))

#!/usr/bin/env python # -*- coding: utf-8 -*- # @Time : 2019/5/27 17:18 # @Author : yhl # @Software: PyCharmimport re import execjs import js2py import requestsheaders = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36' }def get_521_content():req = requests.get('http://www.66ip.cn/1.html', headers=headers)cookies = req.cookiescookies = '; '.join(['='.join(item) for item in cookies.items()])txt_521 = req.texttxt_521 = ''.join(re.findall('<script>(.*?)</script>', txt_521))return (txt_521, cookies)def fixed_fun(function):print(function)func_return = function.replace('eval', 'return')resHtml = "function getClearance(){" + func_return + "};"ctx = execjs.compile(resHtml)temp1 = ctx.call('getClearance')print(temp1)s = 'var a' + temp1.split('document.cookie')[1].split("Path=/;'")[0] + "Path=/;';return a;"s = re.sub(r'document.create.*?firstChild.href', '"{}"'.format('http://www.66ip.cn/1.html'), s) print('s--->',s)resHtml = "function getnewClearance(){" + s + "};"ctx = execjs.compile(resHtml)jsl_clearance = ctx.call('getnewClearance')__jsl_clearance = str(jsl_clearance).split(';')[0]print(jsl_clearance)return __jsl_clearanceif __name__ == '__main__':func = get_521_content()content = func[0]cookie_id = func[1]cookie_id1 = fixed_fun(content)headers['Cookie'] = cookie_id + ';' + cookie_id1res1 = requests.get(url='http://www.66ip.cn/1.html', headers=headers)res1.encoding = 'gb2312'print(res1.text)

?基于execjs實(shí)現(xiàn)結(jié)果

轉(zhuǎn)載于:https://www.cnblogs.com/yhll/p/10932349.html

總結(jié)

以上是生活随笔為你收集整理的爬虫521错误(又是一次和可爱的前端vs的故事)的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。