前面豆子學習了基本的urllib的模塊，通過這個模塊可以寫一些簡單的爬蟲文件。如果要處理大中型的爬蟲項目，urllib就顯得比較low了，這個時候可以使用scrapy框架來實現(xiàn)，很多基本的處理在scrapy里面已經(jīng)做好了。

首先來安裝一下。推薦的流程如下：

首先升級一下pip

C:\WINDOWS\system32>python -m pip install --upgrade pip Requirement already up-to-date: pip in c:\python36\lib\site-packages

安裝wheel C:\WINDOWS\system32>pip install wheel Requirement already satisfied: wheel in c:\python36\lib\site-packages

3.安裝lxml

C:\WINDOWS\system32>pip install lxml Collecting lxmlDownloading lxml-4.1.1-cp36-cp36m-win32.whl (3.2MB)100% |████████████████████████████████| 3.2MB 307kB/s Installing collected packages: lxml Successfully installed lxml-4.1.1

安裝twisted（如果在線安裝報錯的話，可以考慮離線安裝）
從https://www.lfd.uci.edu 上搜索twisted, 可以看見他有很多個版本

執(zhí)行一下python命令看看當前的版本，可以看見我的版本是3.6.2,32位的

c:\Users\yuan.li\Downloads>python Python 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.

這樣的話可以下載對應的版本twisted-17.9.0-cp36-win32.whl就行了。
下載之后，手動安裝

c:\Users\yuan.li\Downloads>pip install Twisted-17.9.0-cp36-cp36m-win32.whl Processing c:\users\yuan.li\downloads\twisted-17.9.0-cp36-cp36m-win32.whl Requirement already satisfied: incremental>=16.10.1 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: Automat>=0.3.0 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: zope.interface>=4.0.2 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: hyperlink>=17.1.1 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: constantly>=15.1 in c:\python36\lib\site-packages (from Twisted==17.9.0) Requirement already satisfied: six in c:\python36\lib\site-packages (from Automat>=0.3.0->Twisted==17.9.0) Requirement already satisfied: attrs in c:\python36\lib\site-packages (from Automat>=0.3.0->Twisted==17.9.0) Requirement already satisfied: setuptools in c:\python36\lib\site-packages (from zope.interface>=4.0.2->Twisted==17.9.0) Installing collected packages: Twisted Successfully installed Twisted-17.9.0

最后安裝scrapy

c:\Users\yuan.li\Downloads>pip install scrapy Collecting scrapyDownloading Scrapy-1.5.0-py2.py3-none-any.whl (251kB)100% |████████████████████████████████| 256kB 2.3MB/s Collecting pyOpenSSL (from scrapy)Downloading pyOpenSSL-17.5.0-py2.py3-none-any.whl (53kB)100% |████████████████████████████████| 61kB 4.5MB/s Collecting cssselect>=0.9 (from scrapy)Downloading cssselect-1.0.3-py2.py3-none-any.whl Collecting parsel>=1.1 (from scrapy)

安裝完成之后，執(zhí)行一下scrapy, 看看是否工作

c:\Users\yuan.li\Downloads>scrapy Scrapy 1.5.0 - no active projectUsage:scrapy <command> [options] [args]Available commands:bench Run quick benchmark testfetch Fetch a URL using the Scrapy downloadergenspider Generate new spider using pre-defined templatesrunspider Run a self-contained spider (without creating a project)settings Get settings valuesshell Interactive scraping consolestartproject Create new projectversion Print Scrapy versionview Open URL in browser, as seen by Scrapy[ more ] More commands available when run from project directory

總結(jié)

以上是生活随笔為你收集整理的初识 scrapy 框架 - 安装的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。