荊軻刺秦王

因業(yè)務需求，需要做一個網(wǎng)頁的信息采集功能。這個網(wǎng)頁就是安居客的新房的列表頁。

第一步：一開始，我用最基本的采集，采集一點很基本的內(nèi)容，就是網(wǎng)頁 html 的的<title>標簽的內(nèi)容，采集出來的是亂碼

問過同事后才明白：原來有些網(wǎng)站為了優(yōu)化，會使用 gzip 壓縮，這樣就導致我們采集的信息一直是亂碼。

如何檢測網(wǎng)頁是否使用了 gzip 壓縮？

1.谷歌瀏覽器? F12打開頁面

2.右鍵點擊?Waterfall? > Response Headers > Content-Encoding

3. 如何開啟了 gzip 則會顯示 gzip 沒有則為空

解決辦法：使用下面的函數(shù)：

function https_request($url='',$gz=false){$curl=curl_init();if($gz){curl_setopt($curl,CURLOPT_ENCODING,'gzip');}//加入gzip解析curl_setopt($curl,CURLOPT_URL,$url);curl_setopt($curl,CURLOPT_SSL_VERIFYPEER,FALSE);curl_setopt($curl,CURLOPT_SSL_VERIFYHOST,FALSE);curl_setopt($curl,CURLOPT_RETURNTRANSFER,1);$data=curl_exec($curl);if(curl_errno($curl)){return 'ERROR'.curl_error($curl);}curl_close($curl);return $data;}

第一個參數(shù)傳入需要抓取頁面的 url 第二個參數(shù)設為 true

然后，我將抓取頁面信息的函數(shù)封裝成一個方法，如果同學需要抓取其他頁面，需要修改其中的正則表達式！

//this is collect functionpublic function getCollectContent($url){$content = $this->https_request($url,true);//echo $content;exit();$title_preg='#<h3><span class="items-name">(.*)</span></h3>#i';preg_match_all($title_preg,$content,$titles);//print_r($res[1]);exit;$newarray = array();foreach($titles[1] as $k=>$v){$newarray[$k] = array('ktitle' => $v,);}//print_r($newarray);//this is address and area collect$address_preg='#<span class="list-map" target="_blank">(.*)</span>#i';preg_match_all($address_preg,$content,$address);//print_r($address[1]);exit();foreach($address[1] as $k=>$v){foreach($newarray as $key=>$value){if($k == $key){$newarray[$key]['address']=substr(str_replace('&nbsp;','',$v),strpos(str_replace('&nbsp;','',$v),']')+1);$newarray[$key]['area']=substr(str_replace('&nbsp;','',$v),1,(strpos(str_replace('&nbsp;','',$v), ']') -1));}}}//this is sole status$sale_preg='#<i class="status-icon .*s.*">(.*)</i>#i';preg_match_all($sale_preg,$content,$sales);//print_r($sales[1]);exit;foreach($sales[1] as $k=>$v){foreach($newarray as $key=>$value){if($k == $key){$newarray[$key]['isflag']=$v;}}}//this is web_id$web_info_preg = '/<div class="item-mod " data-link="(.*?)" data-soj="(.*?)" rel="nofollow">/';preg_match_all($web_info_preg,$content,$web_infos);//print_r($web_infos[1]);exit;foreach($web_infos[1] as $k=>$v){foreach($newarray as $key=>$value){if($k == $key){$array = explode('/', $v);$newarray[$key]['web_id'] = substr($array[2],0,strpos($array[2], '.'));$newarray[$key]['web_domain'] = substr($array[4],0,strpos($array[4], '.'));$newarray[$key]['web_fenqi'] = '';$newarray[$key]['fromweb'] = 2;$newarray[$key]['fromurl'] = $v;$newarray[$key]['city_id'] = $this->city_id;$newarray[$key]['user_id'] = UID;$newarray[$key]['addtime'] = date("Y-m-d H:i",time());$newarray[$key]['uptime'] = 0;$newarray[$key]['deltime'] = 0;$newarray[$key]['state'] = 1;$newarray[$key]['cai_id'] = null;}}}return $newarray;}

其實我想重點講解一下這個采集的方法，或者說是函數(shù)！

里面在采集 address 的時候，使用的字符串截取和去除 &nbsp; 是寫到一塊了，所以千萬別被迷惑了。

主要內(nèi)容就是將：[&nbsp;湯陰&nbsp;湯陰&nbsp;]&nbsp;乾坤大道與德賢路交匯處? 處理為：湯陰湯陰? 和? 乾坤大道與德賢路交匯處? 將網(wǎng)頁中的內(nèi)容截分為兩個數(shù)據(jù)庫的字段。

?

上面在采集具體某一個樓盤鏈接的時候，我順帶將需要插入數(shù)據(jù)庫的所有字段都做了補充，方便后面直接使用 TP5 的 insertAll

方法一次性全部插入。不過就這個項目而言這個思路被我廢棄了。不管怎樣，你只需要知道 我寫的這個方法返回的是一個二維數(shù)組就可以了！

然后，考慮到列表頁肯定不止一頁數(shù)據(jù)，而且城市不一樣 樓盤數(shù)量肯定也不一樣，我專門做了一個 getPage 方法，這個方法也很簡單，就是先使用采集獲取到一共有多少條數(shù)據(jù)，

這個 getPage 的思路也很簡單，就是獲取這個所有的樓盤數(shù)量，然后除以每頁顯示的數(shù)量，然后再向上取整 :?

//this is for get page functionpublic function getPage(){$domain = $this->getCityDomain();//echo $domain;exit();$url = "https://".$domain.".fang.anjuke.com/loupan/?from=navigation";//echo $url;exit();$content = $this->https_request($url,true);//echo $content;exit();$em_preg='#<span class="result">.*<em>(.*)</em>#isU';preg_match_all($em_preg,$content,$result);//print_r($result);exit;$res = $result[1][0];$allPage = ceil($res/60);return $allPage;}

這個 getPage 函數(shù)主要也是為了獲取第二頁，第三頁等等的鏈接地址的。

for($i=1;$i<=$this->getPage(); $i++) {sleep($wait_sec);$url = "https://".$domain.".fang.anjuke.com/loupan/all/p$i/";$res = $this->getCollectContent($url);//print_r($res);exit();$this->checkData($res);}

可以看到，我每次采集都會先使程序 sleep 幾秒，我寫這個是因為項目需要，一般情況是可以省略的。

然后使用 foreach 循環(huán)獲取第一頁，第二頁，第三頁等等的內(nèi)容。

最后我還寫了一個 checkData? 解釋一下這個函數(shù)：這個是為了解決：客戶第一次采集完數(shù)據(jù)之后，如果再次采集，重復的數(shù)據(jù)作為更新，不重復的數(shù)據(jù)將作為新增數(shù)據(jù)，新增到數(shù)據(jù)庫。

代碼：

//this is for check data is existspublic function checkData($obj_array){//$data = Db::table('tb_house_cai')->select();//print_r($obj_array);foreach($obj_array as $k=>$v){$result = Db::table('tb_house_cai')->where('ktitle',$v['ktitle'])->select();if(empty($result)){Db::table('tb_house_cai')->insert($v);}else{$res = Db::table('tb_house_cai')->where('ktitle',$v['ktitle'])->find();$v['cai_id'] = $res['cai_id'];Db::table('tb_house_cai')->update($v);}}}

注意！

這里 個人認為非常不理想，我一開始的思路是這樣的：

public function checkDta($obj_array){$data = Db::table('tb_house_cai')->select();//print_r($obj_array);foreach($data as $k=>$v){foreach($obj_array as $key=>$value){if($v['ktitle'] == $value['ktitle']){$obj_array[$key]['cai_id'] = $data[$k]['cai_id'];Db::table('tb_house_cai')->update($v);}else{Db::table('tb_house_cai')->insert($v);}}}}

我認為不好的地方也很明顯，就是連接數(shù)據(jù)庫的次數(shù)太多，我一開始的思路按照上面的代碼操作，但是結(jié)果并不理想，或者說結(jié)果與我預期的效果不符合。我最后沒辦法，只能放棄這種思路，如果有同學可以幫我指出問題，可以在評論區(qū)里討論。

最后，上整個控制器的代碼：

這里面有個別地方需要借鑒的同學修改！

<?php namespace app\web\admin;use app\admin\controller\Admin; use app\common\builder\ZBuilder;//引入Zbuilder; use app\sys\model\City as CityModel; use think\Db;//引入Bb類/** * 樓盤信息采集管理 */ class Collect extends Admin {private $city_id;public function index(){if ($this->request->isPost()) {// 接收表單數(shù)據(jù)$data = $this->request->post();if($data['city_id'] == null)$this->error('城市不能為空');$wait_sec = $data['wait_sec'];//echo $wait_sec;exit();if($data['city_id'] == null){$city_id = 1;}else{$city_id = $data['city_id'];}$this->city_id = $city_id;$domain = $this->getCityDomain();//https://zz.fang.anjuke.com/loupan/all/p1///echo $this->getPage();exit();for($i=1;$i<=$this->getPage(); $i++) {sleep($wait_sec);$url = "https://".$domain.".fang.anjuke.com/loupan/all/p$i/";$res = $this->getCollectContent($url);//print_r($res);exit();$this->checkData($res);}$this->success('采集完成');}$city_list = CityModel::getCityList();$wait_list = array('3'=>'三秒','5'=>'五秒','10'=>'十秒','15'=>'十五秒','20'=>'二十秒','25'=>'二十五秒','30'=>'三十秒','45'=>'四十五秒','60'=>'一分鐘','120'=>'兩分鐘',);return ZBuilder::make('form')->setPageTitle('采集鏈接')->addFormItem('select', 'city_id', '請選擇城市', '', $city_list)->addFormItem('select', 'wait_sec', '請選擇采集等待時間', '', $wait_list)//->addFormItem('text', 'url', '請輸入需要采集的鏈接')->fetch();}function https_request($url='',$gz=false){$curl=curl_init();if($gz){curl_setopt($curl,CURLOPT_ENCODING,'gzip');}//加入gzip解析curl_setopt($curl,CURLOPT_URL,$url);curl_setopt($curl,CURLOPT_SSL_VERIFYPEER,FALSE);curl_setopt($curl,CURLOPT_SSL_VERIFYHOST,FALSE);curl_setopt($curl,CURLOPT_RETURNTRANSFER,1);$data=curl_exec($curl);if(curl_errno($curl)){return 'ERROR'.curl_error($curl);}curl_close($curl);return $data;}function DeleteHtml($str){$str = trim($str); //清除字符串兩邊的空格$str = preg_replace("/\t/","",$str); //使用正則表達式替換內(nèi)容，如：空格，換行，并將替換為空。$str = preg_replace("/\r\n/","",$str);$str = preg_replace("/\r/","",$str);$str = preg_replace("/\n/","",$str);$str = preg_replace("/ /","",$str);$str = preg_replace("/ /","",$str); //匹配html中的空格return trim($str); //返回字符串}function SubStr($tinfo=NULL,$ks=NULL,$js=NULL,$s=1,$e=0) {if(empty($tinfo)){return '';}if(strpos($tinfo, $ks) == FALSE) {return '';}if(!empty($ks)){$arry_tinfo=explode($ks,$tinfo);if(!empty($arry_tinfo)){$tinfo=$arry_tinfo[$s];}}if(!empty($js)){$arry_tinfo=explode($js,$tinfo);if(!empty($arry_tinfo)){$tinfo=$arry_tinfo[$e];}}return $tinfo;}public function getCityDomain(){$array = array('1'=>'zz','2'=>'kf','3'=>'ly','4'=>'pds','5'=>'ay','6'=>'jz','7'=>'hb','8'=>'xx','9'=>'py','10'=>'xc','11'=>'lh','12'=>'smx','13'=>'ny','14'=>'sq','15'=>'zk','16'=>'xy','17'=>'zmd','18'=>'jy','19'=>'zm');return $array[$this->city_id];}//this is collect functionpublic function getCollectContent($url){$content = $this->https_request($url,true);//echo $content;exit();$title_preg='#<h3><span class="items-name">(.*)</span></h3>#i';preg_match_all($title_preg,$content,$titles);//print_r($res[1]);exit;$newarray = array();foreach($titles[1] as $k=>$v){$newarray[$k] = array('ktitle' => $v,);}//print_r($newarray);//this is address and area collect$address_preg='#<span class="list-map" target="_blank">(.*)</span>#i';preg_match_all($address_preg,$content,$address);//print_r($address[1]);exit();foreach($address[1] as $k=>$v){foreach($newarray as $key=>$value){if($k == $key){$newarray[$key]['address']=substr(str_replace('&nbsp;','',$v),strpos(str_replace('&nbsp;','',$v),']')+1);$newarray[$key]['area']=substr(str_replace('&nbsp;','',$v),1,(strpos(str_replace('&nbsp;','',$v), ']') -1));}}}//this is sole status$sale_preg='#<i class="status-icon .*s.*">(.*)</i>#i';preg_match_all($sale_preg,$content,$sales);//print_r($sales[1]);exit;foreach($sales[1] as $k=>$v){foreach($newarray as $key=>$value){if($k == $key){$newarray[$key]['isflag']=$v;}}}//this is web_id$web_info_preg = '/<div class="item-mod " data-link="(.*?)" data-soj="(.*?)" rel="nofollow">/';preg_match_all($web_info_preg,$content,$web_infos);//print_r($web_infos[1]);exit;foreach($web_infos[1] as $k=>$v){foreach($newarray as $key=>$value){if($k == $key){$array = explode('/', $v);$newarray[$key]['web_id'] = substr($array[2],0,strpos($array[2], '.'));$newarray[$key]['web_domain'] = substr($array[4],0,strpos($array[4], '.'));$newarray[$key]['web_fenqi'] = '';$newarray[$key]['fromweb'] = 2;$newarray[$key]['fromurl'] = $v;$newarray[$key]['city_id'] = $this->city_id;$newarray[$key]['user_id'] = UID;$newarray[$key]['addtime'] = date("Y-m-d H:i",time());$newarray[$key]['uptime'] = 0;$newarray[$key]['deltime'] = 0;$newarray[$key]['state'] = 1;$newarray[$key]['cai_id'] = null;}}}return $newarray;}//this is for get page functionpublic function getPage(){$domain = $this->getCityDomain();//echo $domain;exit();$url = "https://".$domain.".fang.anjuke.com/loupan/?from=navigation";//echo $url;exit();$content = $this->https_request($url,true);//echo $content;exit();$em_preg='#<span class="result">.*<em>(.*)</em>#isU';preg_match_all($em_preg,$content,$result);//print_r($result);exit;$res = $result[1][0];$allPage = ceil($res/60);return $allPage;}//this is for check data is existspublic function checkData($obj_array){//$data = Db::table('tb_house_cai')->select();//print_r($obj_array);foreach($obj_array as $k=>$v){$result = Db::table('tb_house_cai')->where('ktitle',$v['ktitle'])->select();if(empty($result)){Db::table('tb_house_cai')->insert($v);}else{$res = Db::table('tb_house_cai')->where('ktitle',$v['ktitle'])->find();$v['cai_id'] = $res['cai_id'];Db::table('tb_house_cai')->update($v);}}}public function checkDta($obj_array){$data = Db::table('tb_house_cai')->select();//print_r($obj_array);foreach($data as $k=>$v){foreach($obj_array as $key=>$value){if($v['ktitle'] == $value['ktitle']){$obj_array[$key]['cai_id'] = $data[$k]['cai_id'];Db::table('tb_house_cai')->update($v);}else{Db::table('tb_house_cai')->insert($v);}}}}}

關(guān)于采集：

有興趣的同學可以學習使用一下： phpQuery?

不過這個 phpQuery v3 版本的用著沒有 v4 版本的好用，但是 v4 版本的需要 php 7.0以上，所以我放棄了，但是如果條件允許，phpQuery 將是個非常好用的采集插件！

?

?

?

?

總結(jié)

以上是生活随笔為你收集整理的ThinkPHP5 采集网页的指定内容的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。