一 LightGBM分布式支持說明

使用源碼編譯過的二進制lightgbm來運行分布式

分布式worker之間通信可以使用Socket與MPI方式（MPI通信更快，建議使用）

?

二 LightGBM分布式環境安裝

分布式訓練環境是Ubuntu

一 Socket環境支持

On Linux LightGBM can be built using CMake and gcc or Clang.

Install CMake.

Run the following commands:

git clone --recursive https://github.com/microsoft/LightGBM cd LightGBM mkdir build cd build cmake .. make -j4

Note: glibc >= 2.14 is required.

Also, you may want to read gcc Tips.

?

二 MPI環境支持

The default build version of LightGBM is based on socket. LightGBM also supports MPI. MPI is a high performance communication approach with RDMA support.

If you need to run a distributed learning application with high performance communication, you can build the LightGBM with MPI support.

On Linux an MPI version of LightGBM can be built using Open MPI, CMake and gcc or Clang.

1. OpenMPI安裝方法

???? 1. OpenSSH安裝

apt-get update apt-get install openssh-server

??? 如果是多臺機器使用MPI通信，請配置節點間ssh免密登錄 并設置 StrictHostKeyChecking=no

# ~/.ssh/config 中添加如下信息 Host *StrictHostKeyChecking no

2.安裝openmpi （建議安裝4.1.0版本）

wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.0.tar.gz tar zxvf openmpi-4.1.0.tar.gz cd openmpi-4.1.0/ ./configure --prefix=/usr/local/openmpi # 如果環境直接安裝在/usr/local， 這樣可以不用設置之后的環境變量 #./configure --prefix=/usr/localmake -j8 make install

?

3. 配置環境變量（~/.bashrc）

export PATH=$PATH:/usr/local/openmpi/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/openmpi/lib/ source ~/.bashrc sudo ldconfig

???? 3. 驗證安裝成功

cd examples make ./hello_c #mpirun -np 8 hello_c Hello, world, I am 0 of 1, (Open MPI v3.1.0, package: Open MPI root@ssli_centos7 Distribution, ident: 3.1.0, repo rev: v3.1.0, May 07, 2018, 112)

示例程序正確運行，說明安裝成功

?

2. Install CMake.

Run the following commands:

git clone --recursive https://github.com/microsoft/LightGBM cd LightGBM mkdir build cd build cmake -DUSE_MPI=ON .. make -j4

Note: glibc >= 2.14 is required.

?

三 分布式環境測試

相關數據與參數配置請參考如下：

https://github.com/microsoft/LightGBM/blob/master/examples/parallel_learning/README.md

Node1:

Node2:

?

Copy data file, executable file, config file and mlist.txt to all machines.

Note: MPI needs to be run in the same path on all machines.

Run following command on one machine (not need to run on all machines), need to change your_config_file to real config file.

mpiexec --machinefile mlist.txt ./lightgbm config=your_config_file

在實驗中（openMPI 4.1.0），發現用上面的方法，不會在每臺機器上分配1個進程，可以使用如下方法啟動分布式

mpiexec -np 2 -H 172.200.24.6,172.200.25.10 lightgbm config=train.conf

?

?

?

總結

以上是生活随笔為你收集整理的[机器学习]LightGBM分布式使用完全手册的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。