keras核心已轉儲

This article is the follow up from Part 1. Here, I will compare the tf.data and Keras.ImageDataGeneratoractual training times using the mobilenetmodel.

本文是第1部分的后續文章。在這里，我將使用mobilenet模型比較tf.data和Keras.ImageDataGenerator實際訓練時間。

YouTube VideoYouTube視頻

In Part 1, I showed that loading images using tf.data is approximately 5 times faster in comparison toKeras.ImageDataGenerator. The dataset considered was Kaggle- dogs_and_cats (217 MB) having 10000 images distributed among 2 different classes.

在第1部分中，我展示了使用tf.data加載圖像的速度比Keras.ImageDataGenerator快5倍。 所考慮的數據集是Kaggledogs_and_cats (217 MB)，具有10000張分布在2個不同類別中的 圖像 。

In this Part 2, I have considered a bigger dataset which is commonly used for image classification problems. The dataset chosen is COCO2017 (18 GB) having 117266 images distributed among 80 different classes. Various versions of COCO datasets are freely available to try and test at this link. The reason for choosing the bigger dataset of 18 GB is to have better comparison results. For a practical image classification problem, datasets can be even bigger ranging from 100 GB (gigabytes)to a few TB (terabytes). In our case, 18 GB of data is enough to understand the comparison as using the dataset in TB would significantly increase the training times and computational resources.

在第2部分中，我考慮了一個較大的數據集，該數據集通常用于圖像分類問題。 選擇的數據集為COCO2017 (18 GB) ，其中117266張圖像分布在80個不同的類別中。 可通過此鏈接免費獲得各種版本的COCO數據集進行嘗試和測試。 選擇更大的18 GB數據集的原因是為了獲得更好的比較結果。 對于實際的圖像分類問題，數據集甚至可能更大，范圍從100 GB(千兆字節)到幾TB(TB)。 在我們的案例中，18 GB的數據足以理解比較，因為在TB中使用數據集會大大增加訓練時間和計算資源。

培訓時間結果(提前) (Training times Results (in advance))

The above results are compared on a workstation having 16-GB RAM, 2.80 GHz with Core i7 using GPU version of TensorFlow 2.x. The dataset considered is COCO2017 (18 GB) having 117266 images distributed among 80 different classes.

以上結果在使用GPU版本的TensorFlow 2.x，具有Core i7的具有16 GB RAM，2.80 GHz的工作站上進行了比較。 所考慮的數據集為COCO2017 (18 GB) ，其中117266張圖像分布在80個不同的類別中。

• When using Keras.ImageDataGenerator, it took approximately 58 minutes during training for each epoch using COCO2017 dataset.

  使用Keras.ImageDataGenerator ，使用COCO2017數據集訓練每個歷時大約需要58分鐘 。

• When using tf.data with variablecache=True , the program crashes. The reason behind this crash is that the dataset considered (size 18 GB) is bigger than the RAM of the workstation. With cache=True the program starts storing the images in the RAM for quick access and when this surpasses the RAM (16 GB in our case), program crashes. I have tested the same option for a smaller dataset Kaggle- dogs_and_cats and it worked fine. This shows that we shouldn’t use cache=True option when the size of the considered dataset is bigger than the RAM.

  當將tf.data與變量cache=True ，程序崩潰。 發生此崩潰的原因是，所考慮的數據集( 大小為 18 GB)大于工作站的RAM 。 當cache=True ，程序開始將圖像存儲在RAM中以進行快速訪問，當超過RAM( 在我們的情況下為16 GB)時，程序崩潰。 我已經對較小的數據集Kaggledogs_and_cats測試了相同的選項，并且效果很好。 這表明當所考慮的數據集的大小大于RAM時，我們不應使用cache=True選項。

• When using tf.data with variablecache=False , the program takes approximately 23 minutes, which is 2.5 times faster in comparison to Keras.ImageDataGenerator.

  當使用帶有變量cache=False tf.data時，該程序大約需要23分鐘，比Keras.ImageDataGenerator 快2.5倍 。

• When using cache='some_path.tfcache' , during the first epoch tf.data will make a dump of the dataset/images in your computer directory. This is why it is slower during the first epoch and takes around 42 minutes. In successive epochs, it doesn’t have to store the images again on the computer but use the already created dump during the first epoch, which ultimately speeds up the training times. During successive epochs, it took only 14 minutes for each epoch. Creating the dump is only 1-time process. For the hyperparameter tuning, it takes around 14 minutes in comparison to 58 minutes with Keras , which is approximately 4.14 times faster.

  當使用cache='some_path.tfcache' ，在第一個時期tf.data將轉儲計算機目錄中的數據集/圖像。 這就是為什么它在第一個時期比較慢并且需要大約42分鐘的原因 。 在連續的時期中，它不必將圖像再次存儲在計算機上，而是在第一個時期中使用已經創建的轉儲，從而最終加快了訓練時間。 在連續的時期中， 每個時期僅花費14分鐘 。 創建轉儲只是1次過程。 對于超參數調整，大約需要14分鐘 ，而使用Keras則需要58分鐘 ，這大約快了4.14倍。

Note: The dump created in the memory when using cache='some_path.tfcache'is approximately of size 90 GB which is actually far greater than the original size (18 GB) of the dataset. I couldn’t exactly understand the reason for this as there is no clear documentation from TensorFlow about this. I hope this is the glich which will be sorted out in the future.

注意：使用cache='some_path.tfcache'時在內存中創建的轉儲大小約為90 GB，實際上遠大于數據集的原始大小(18 GB)。 我無法完全理解其原因，因為TensorFlow沒有明確的文檔。 我希望這是將來可以解決的問題。

For smaller datasets like Kaggle- dogs_and_cats which is only 217 MB, there will not be a noticeable difference in speed or training times with tf.data and Keras as the RAM is big enough to store all the images at once.

對于較小的數據集像Kaggle- dogs_and_cats這是只有217 MB，不會有與速度或訓練次數明顯的區別tf.data和Keras的RAM足夠大到所有圖像保存一次。

訓練模型代碼 (Training model code)

Here, the initial code for creating and training the model is shown. Pretrained model mobilenet is used to perform the transfer learning with base layers of pretrained model are frozen.

在這里，顯示了用于創建和訓練模型的初始代碼。 預訓練模型mobilenet用于凍結已預訓練模型基礎層的轉移學習。

所有的代碼在一起 (All the Code together)

Here I have combined the code from both Part 1 and Part 2. The comments and docstrings for functions will help in understanding the code.

在這里，我結合了第1部分和第2部分中的代碼。函數的注釋和文檔字符串將有助于理解代碼。

結論 (Conclusion)

This article shows thattf.data is 2.5(when using cache=False) to 4.14 (when using cache=`some_path.tfcache`) times faster in comparison to Keras.ImageDataGenerator for any practical image classification problem. I think it is worth to try tf.data.

本文表明， tf.data為2.5 (when using cache=False) 至4.14 (when using cache= ` some_path.tfcache`)相比倍的速度 Keras.ImageDataGenerator為任何實用的圖像分類問題。 我認為值得嘗試tf.data 。

翻譯自: https://medium.com/@sunnychugh/dump-keras-imagedatagenerator-start-using-tensorflow-tf-data-part-2-fba7cda81203

keras核心已轉儲

總結

以上是生活随笔為你收集整理的keras核心已转储_转储Keras-ImageDataGenerator。 开始使用TensorFlow-tf.data（第2部分）的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。