壓縮文件可以直接放到下載器里面下載的
you-get 連接 下載任意文件 重點

用python下載文件的若干種方法匯總

用python下載文件的若干種方法匯總

import requests
url = 'https://www.python.org/static/img/python-logo@2x.png'
myfile = requests.get(url)
open('PythonImage.png', 'wb').write(myfile.content)

import wget
url = "https://www.python.org/static/img/python-logo@2x.png"
wget.download(url, 'pythonLogo.png')

import requests
url="******"
try:r=requests.get(url)r.raise_for_status()  #如果不是200，產生異常requests.HTTPErrorr.encoding=r.apparent_encodingprint(r.text)
except:print("爬取失敗...")

import requests
url = 'https://readthedocs.org/projects/python-guide/downloads/pdf/latest/'
myfile = requests.get(url, allow_redirects=True)
open('hello.pdf', 'wb').write(myfile.content)

import requests
url = 'https://buildmedia.readthedocs.org/media/pdf/python-guide/latest/python-guide.pdf'
r = requests.get(url, stream = True)
with open("PythonBook.pdf", "wb") as Pypdf:for chunk in r.iter_content(chunk_size = 1024): # 1024 bytesif chunk:Pypdf.write(chunk)

import os
import requests
from time import time
from multiprocessing.pool import ThreadPool

def url_response(url):
path, url = url
r = requests.get(url, stream=True)
with open(path, ‘wb’) as f:
for ch in r:
f.write(ch)

urls = [(“Event1”, “https://www.python.org/events/python-events/805/”),
(“Event2”, “https://www.python.org/events/python-events/801/”),
(“Event3”, “https://www.python.org/events/python-events/790/”),
(“Event4”, “https://www.python.org/events/python-events/798/”),
(“Event5”, “https://www.python.org/events/python-events/807/”),
(“Event6”, “https://www.python.org/events/python-events/807/”),
(“Event7”, “https://www.python.org/events/python-events/757/”),
(“Event8”, “https://www.python.org/events/python-user-group/816/”)]

start = time()
for x in urls:
url_response(x)

print(f"Time to download: {time() - start}")

Time to download: 7.306085824966431

并行版本，只需改動一行代碼ThreadPool(9).imap_unordered(url_response, urls)，時間會大幅度減少：

import os

import requests
from time import time
from multiprocessing.pool import ThreadPool

def url_response(url):
path, url = url
r = requests.get(url, stream=True)
with open(path, ‘wb’) as f:
for ch in r:
f.write(ch)

urls = [(“Event1”, “https://www.python.org/events/python-events/805/”),
(“Event2”, “https://www.python.org/events/python-events/801/”),
(“Event3”, “https://www.python.org/events/python-events/790/”),
(“Event4”, “https://www.python.org/events/python-events/798/”),
(“Event5”, “https://www.python.org/events/python-events/807/”),
(“Event6”, “https://www.python.org/events/python-events/807/”),
(“Event7”, “https://www.python.org/events/python-events/757/”),
(“Event8”, “https://www.python.org/events/python-user-group/816/”)]

start = time()
ThreadPool(9).imap_unordered(url_response, urls)
print(f"Time to download: {time() - start}")

Time to download: 0.0064961910247802734

5. 使用urllib獲取html頁面

import urllib.request

urllib.request.urlretrieve(‘url’, ‘path’)

urllib.request.urlretrieve(‘https://www.python.org/’, ‘PythonOrganization.html’)

6. python下載視頻的神器

you-get[2]，目前you-get所支持的網站包含國內外幾十個網站(youtube、twitter、騰訊、愛奇藝、優酷、bilibili等)。

pip install you-get

測試一下：

you-get https://www.bilibili.com/video/av52694584/?spm_id_from=333.334.b_686f6d655f706f70756c6172697a65.3

youtube-dl[3]也是一個類似的工具。

7. 舉個例子

批量下載： NOAA-CIRES 20th Century 2m氣溫再分析資料[4]。一個個點手會點殘，這時候可以借助Python來批量化下載數據。

首先打開頁面，按F12查看網頁源碼：

可以看出，對應下載文件的鏈接都在div標簽下的a標簽中，需要將這些鏈接一一獲取然后就可以進行批量化下載了。

# -- coding: utf-8 --

import urllib

from bs4 import BeautifulSoup

rawurl=‘https://www.esrl.noaa.gov/psd/cgi-bin/db_search/DBListFiles.pl?did=118&tid=40290&vid=2227’
content = urllib.request.urlopen(rawurl).read().decode(‘ascii’) # 獲取頁面的HTML

soup = BeautifulSoup(content, ‘lxml’)
url_cand_html=soup.find_all(id=‘content’) # 定位到存放url的標號為content的div標簽
list_urls=url_cand_html[0].find_all(“a”) # 定位到a標簽，其中存放著文件的url
urls=[]

for i in list_urls[1:]:
urls.append(i.get(‘href’)) # 取出鏈接

for i,url in enumerate(urls):
print(“This is file”+str(i+1)+" downloading! You still have “+str(142-i-1)+” files waiting for downloading!!")
file_name = “./ncfile/”+url.split(’/’)[-1] # 文件保存位置+文件名
urllib.request.urlretrieve(url, file_name)

本文分享自微信公眾號 - MeteoAI（meteoai）

原文出處及轉載信息見文內詳細說明，如有侵權，請聯系 yunjia_community@tencent.com刪除。

原始發表時間：2019-07-22

本文參與騰訊云自媒體分享計劃，歡迎正在閱讀的你也加入，一起分享。

• 分享文章到朋友圈

• 分享文章到 QQ
• 分享文章到微博
• 復制文章鏈接到剪貼板

掃碼關注云+社區

領取騰訊云代金券

• Keras系列（五） ConvLSTM 空間特征深度學習
• Python 命令行參數解析庫argparse
• 還在用matplotlib畫圖？你out啦
• Keras 系列（六） CNN 分類及fit_generator函數
• xarray尾聲：TIFF與GRIB處理

• 上一篇：python 面向對象編程 - 小游戲
• 下一篇：爬取斗圖網圖片,使用xpath格式來匹配內容,對請求偽裝成瀏覽器, Referer 防跨域請求

總結

以上是生活随笔為你收集整理的用python下载文件的若干种方法汇总的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。