日韩性视频-久久久蜜桃-www中文字幕-在线中文字幕av-亚洲欧美一区二区三区四区-撸久久-香蕉视频一区-久久无码精品丰满人妻-国产高潮av-激情福利社-日韩av网址大全-国产精品久久999-日本五十路在线-性欧美在线-久久99精品波多结衣一区-男女午夜免费视频-黑人极品ⅴideos精品欧美棵-人人妻人人澡人人爽精品欧美一区-日韩一区在线看-欧美a级在线免费观看

歡迎訪問 生活随笔!

生活随笔

當前位置: 首頁 > 编程资源 > 编程问答 >内容正文

编程问答

solr文档索引最佳实践

發布時間:2024/1/23 编程问答 28 豆豆
生活随笔 收集整理的這篇文章主要介紹了 solr文档索引最佳实践 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

solr文檔索引最佳實踐

@(OTHERS)[solr]

  • solr文檔索引最佳實踐
    • 一直接提交
    • 二AutoCommit
    • 三 commitWithin
    • 四建議及結論
      • 1單線程情況
      • 2多線程情況

solr的文檔生成后,需要將其提交到solr集群,提交的方法有以下三種:

(一)直接提交

每生成一個文檔就直接提交至solr:

CloudSolrClient client = new CloudSolrClient(SOLR_ZK); SolrInputDocument doc2 = new SolrInputDocument(); doc2.addField("id", "ljhtest3"); doc2.addField("key_ss", map); client.add(doc2); client.commit();

即每add一次就commit一次,這種實現方案實現簡單,但性能不高。

* 注意,不只是commit()效率不高,client.add()的效率也是非常低的,因此需要將所有文檔先add進一個collection,然后client.add(collection) *

List<SolrInputDocument> docList = new LinkedList<SolrInputDocument>();for (int i = 0; i < DOC_NUM; i++) {SolrInputDocument doc2 = new SolrInputDocument();doc2.addField("id", "way2" + i);Set set = new HashSet();for (String s : "abc,edf,kkk,lll".split(",")) {set.add(s);}Map map = new HashMap();map.put("set", set);doc2.addField("key_ss", map);docList.add(doc2);}client.add(docList);client.commit();

(二)AutoCommit

可以在solrConfig.xml中的updateHandler設置自動提交機制:


<!-- Enables a transaction log, used for real-time get, durability, andand solr cloud replica recovery. The log can grow as big asuncommitted changes to the index, so use of a hard autoCommitis recommended (see below)."dir" - the target directory for transaction logs, defaults to thesolr data directory."numVersionBuckets" - sets the number of buckets used to keeptrack of max version values when checking for re-orderedupdates; increase this value to reduce the cost ofsynchronizing access to version buckets during high-volumeindexing, this requires 8 bytes (long) * numVersionBucketsof heap space per Solr core. --> <updateLog><str name="dir">${solr.ulog.dir:}</str><int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int> </updateLog><!-- AutoCommitPerform a hard commit automatically under certain conditions.Instead of enabling autoCommit, consider using "commitWithin"when adding documents. http://wiki.apache.org/solr/UpdateXmlMessagesmaxDocs - Maximum number of documents to add since the lastcommit before automatically triggering a new commit.maxTime - Maximum amount of time in ms that is allowed to passsince a document was added before automaticallytriggering a new commit. openSearcher - if false, the commit causes recent index changesto be flushed to stable storage, but does not cause a newsearcher to be opened to make those changes visible.If the updateLog is enabled, then it's highly recommended tohave some sort of hard autoCommit to limit the log size.--><autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <openSearcher>false</openSearcher> </autoCommit><!-- softAutoCommit is like autoCommit except it causes a'soft' commit which only ensures that changes are visiblebut does not ensure that data is synced to disk. This isfaster and more near-realtime friendly than a hard commit.--><autoSoftCommit> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>

可以指定多長時間間隔或者多少文檔就會提交一次。
有hard & soft2種,后者不會將數據同步到disk。

(三) commitWithin

client.add(DEFAULT_COLLECTION, doc,1000);

client.add()的文檔會在1000ms內被提交到solr中。

(四)建議及結論

1、單線程情況

1、將需要提交的文檔add到一個collection中。
2、client add這個collection,而不是add文檔 。
3、client指定commitWithin參數。
參考代碼如下:

List<SolrInputDocument> docList = new LinkedList<SolrInputDocument>();for (int i = 0; i < DOC_NUM; i++) {SolrInputDocument doc2 = new SolrInputDocument();doc2.addField("id", "way2" + i);Set set = new HashSet();for (String s : "abc,edf,kkk,lll".split(",")) {set.add(s);}Map map = new HashMap();map.put("set", set);doc2.addField("key_ss", map);docList.add(doc2);}client.add(docList);client.commit();

2、多線程情況

(1)hbase寫時要有多線程
(2)coprocessor會在多個分區中并行執行。

創作挑戰賽新人創作獎勵來咯,堅持創作打卡瓜分現金大獎

總結

以上是生活随笔為你收集整理的solr文档索引最佳实践的全部內容,希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯,歡迎將生活随笔推薦給好友。