利用多個空閑GPU，提高模型訓練推理速度

import torch from gpuinfo import GPUInfo# 查詢空閑GPU gpu_ids = GPUInfo.check_empty() # 設置模型初始化設備 device = torch.device(f"cuda:{str(gpu_ids[0])}" if gpu_ids else "cpu") # 數據并行 model = xxxxxx model = torch.nn.DataParallel(model, device_ids=gpu_ids) # 模型加載到初始化設備中 model = model.to(device) # 模型計算數據并行 for line in dataloader:# 數據不需要加載到相同的設備中moded(line) # ......

DistributedDataParallel 相對于 DataParallel的優勢主要在模型訓練階段，模型推理時由于不涉及參數更新不需要進程間通信，所以推理速度應該相差不大

# The difference between DistributedDataParallel and DataParallel is: # DistributedDataParallel uses multiprocessing where a process is created for each GPU, # while DataParallel uses multithreading. By using multiprocessing, each GPU has its dedicated process, # this avoids the performance overhead caused by GIL of Python interpreter.

模型訓練時分布式多進程數據并行可結合這兩個類改寫
torch.nn.parallel.DistributedDataParallel
torch.utils.data.distributed.DistributedSampler
Distributed Data Parallel
（有時間再寫一版）

總結

以上是生活随笔為你收集整理的Pytorch 多GPU数据并行(DataParallel)的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。