無監督學習 k-means

有關深層學習的FAU講義 (FAU LECTURE NOTES ON DEEP LEARNING)

These are the lecture notes for FAU’s YouTube Lecture “Deep Learning”. This is a full transcript of the lecture video & matching slides. We hope, you enjoy this as much as the videos. Of course, this transcript was created with deep learning techniques largely automatically and only minor manual modifications were performed. Try it yourself! If you spot mistakes, please let us know!

這些是FAU YouTube講座“ 深度學習 ”的 講義 。 這是演講視頻和匹配幻燈片的完整記錄。 我們希望您喜歡這些視頻。 當然，此成績單是使用深度學習技術自動創建的，并且僅進行了較小的手動修改。 自己嘗試！ 如果發現錯誤，請告訴我們！

導航 (Navigation)

Previous Lecture / Watch this Video / Top Level / Next Lecture

上一個講座 / 觀看此視頻 / 頂級 / 下一個講座

A common AI myth is that GANs are mainly used to worship cats. Image created using gifify. Source: YouTube常見的AI神話是GAN主要用于崇拜貓。 使用gifify創建的圖像 。 資料來源： YouTube

Welcome back to deep learning! So today, we finally want to look into the generative adversarial networks which are a key technology in unsupervised deep learning. So, let’s see what I have for you here.

歡迎回到深度學習！ 因此，今天，我們終于要研究生成對抗網絡，這是無監督深度學習中的一項關鍵技術。 所以，讓我們在這里看看我有什么。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

Well, the unsupervised deep learning part generative adversarial networks come from the key idea that GANs to play the following game: You have a generator and a discriminator. Now the generator, one could argue is somebody who generates a fake image. Then, the discrimination has to figure out whether the generator actually produced something that’s real or something which is fake. So, the discriminator can decide fake or real and in order to train the discriminator, he has access to many real data observations. So, the outcome of the discriminator then is whether the input was real or fake. Well, of course, this is difficult to ask persons and artists to draw things. So, we replace the tool with deep neural networks and we have D that is the discriminator and we have G that is the generator. The generator receives some latent input some noise variable z and from the noise variable and the parameters, it produces some image. The discriminator then tries to figure out whether this was a real or fake image. So, the output of the discriminator is going to be 1 for real and 0 for fake.

好吧，無監督的深度學習部分生成對抗網絡來自GAN玩以下游戲的關鍵思想：您有一個生成器和一個鑒別器。 現在，生成器可能會爭辯說是某人生成了虛假圖像。 然后，必須辨別出生成器實際上是生成的是真實的還是假的。 因此，鑒別者可以決定是假的還是真實的，并且為了訓練鑒別者，他可以訪問許多真實的數據觀察結果。 因此，判別器的結果就是輸入是真實的還是假的。 好吧，當然，這很難問個人和藝術家畫畫。 因此，我們用深度神經網絡代替了該工具，并且D是鑒別符，G是生成器。 發生器接收一些潛在的輸入一些噪聲變量z，并從噪聲變量和參數中產生一些圖像。 然后，鑒別者試圖弄清楚這是真實圖像還是偽圖像。 因此，鑒別器的輸出對于真實將為1，對于偽造為0。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

Once we have found this kind of neural network representation, we are also able to describe a loss. The loss of our discriminator is to minimize the following function that is dependent on the parameters of the discriminator and the parameters of the generator. It is essentially minimizing the expected value of x from the data. This is simply the logarithm of the output of our discriminator for real samples minus the expected value of some generated noise and that is the logarithm of 1 minus the discriminator of the generator of some noise. So, it’s trained to distinguish real data samples from fake ones. Now, if you want to train the generator you minimize the loss of the generator that is the negative loss of the discriminator. So, the generator minimizes the probability of the discriminator being correct. You train to generate domain images to fool D. Optionally, you can run k steps of one player for every step of the other player and the equilibrium is a saddle point of the discriminator loss.

一旦找到了這種神經網絡表示形式，我們就能夠描述損失。 鑒別器的損失是使取決于鑒別器參數和發生器參數的以下功能最小化。 從本質上講，它是從數據中最小化x的期望值。 這僅是我們的鑒權器對真實樣本的輸出的對數減去一些產生的噪聲的期望值，也就是1的對數減去某些噪聲的生成器的對數。 因此，它經過訓練可以區分真實數據樣本和假數據樣本。 現在，如果您要訓練發電機，則可以將發電機的損耗(即鑒別器的負損耗)降至最低。 因此，生成器將鑒別器正確的可能性降到最低。 您訓練以生成要欺騙D的域圖像。(可選)您可以對一個參與者的每一步運行k步，而對另一參與者的每一步進行平衡，則均衡是鑒別器損失的鞍點。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

If you look into this in more detail, you can find that the loss of the generator is directly tied to the negative loss of the discriminator. So, you can summarize this game with a value function specifying the discriminator’s payoff that is given as V. This is the negative loss of the discriminator and this then results in the following minimax game: So the optimal parameter set of the generator can be determined by maximizing V with respect to the discriminator nested into a minimization of the parameters of G with respect to the same value function.

如果您對此進行更詳細的研究，您會發現發生器的損耗與鑒別器的負損耗直接相關。 因此，您可以使用值函數來概括該游戲，該函數指定以V給出的鑒別器的收益。這是鑒別器的負損失，因此導致以下最小極大游戲：因此，可以確定發生器的最佳參數集通過相對于鑒別器最大化V，嵌套到相對于相同值函數的G參數最小化。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

So, let’s have a look at the optimal discriminator. There a key assumption is that is both densities are nonzero everywhere. Otherwise, some input values would never be trained and the discriminator would have undetermined behavior in those areas. Then you solve with respect to the gradient of the discriminator loss with respect to the discriminator to be zero. You can find the optimal discriminator for any data distribution and any model distribution in the following way: the optimal discriminator is the distribution of the data divided by the distribution of the data plus the distribution of the model over your entire input domain of x. Unfortunately, this optimal discriminate is theoretical and unachievable. So, it’s key for GANs to have an estimation mechanism. You can use supervised learning to estimate this ratio. Then this leads to the problem of underfitting and overfitting.

因此，讓我們看一下最佳鑒別器。 一個關鍵的假設是，兩個密度在各處都不都是零。 否則，將永遠不會訓練某些輸入值，并且判別器在那些區域中的行為不確定。 然后，根據鑒別器損耗相對于鑒別器的梯度為零進行求解。 您可以通過以下方式找到任何數據分布和模型分布的最佳判別器：最佳判別器是數據的分布除以數據分布再加上模型在整個x輸入域上的分布。 不幸的是，這種最佳區分是理論上無法實現的。 因此，GAN具有評估機制的關鍵。 您可以使用監督學習來估計此比率。 然后，這會導致擬合不足和擬合過度的問題。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

Now, what else can we do? We can do non-saturating games where we modify the generator’s loss. Then, in this example, we are no longer using the same function for both. Instead, we have a new loss for the generator where we simply compute the expected value of the logarithm of the discriminator of the generator given some input noise. In minimax, G minimizes the log probability of D being correct. In this solution, G minimizes the log probability of D being mistaken. It’s heuristically motivated because it fights the vanishing gradient of G when D is too smart. This is particularly a problem in the beginning. However, the equilibrium is no longer describable using a single loss.

現在，我們還能做什么？ 我們可以做非飽和游戲，在此我們可以修改生成器的損失。 然后，在此示例中，我們不再為兩者使用相同的功能。 取而代之的是，我們對發生器產生了新的損失，在這種情況下，只要輸入噪聲，我們就可以簡單地計算出發生器鑒別器對數的期望值。 在minimax中，G使D正確的對數概率最小。 在此解決方案中，G使D被錯誤記錄的對數概率最小。 啟發式動機是因為當D太聰明時，它可以克服G消失的梯度。 一開始這尤其是一個問題。 但是，不再使用單個損失來描述平衡。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

So, there are a lot of things like extensions that are quite popular like the feature matching loss or the perceptual loss. Here, G is trying to match the expected value of features f(x) of some intermediate layer of D. You’ve seen this already that f can be for example some other network and some layer 3 or layer 5 representation. Then, you want the expected values of these representations to be the same given for real inputs as well as for generated noise images. So, here you want to prevent the overtraining of the generator on the current discriminator. By the way, this is also a popular loss in many other domains.

因此，有很多諸如擴展之類的東西很流行，例如特征匹配損失或感知損失。 在這里，G試圖匹配D的某個中間層的特征f( x )的期望值。您已經看到，f可以例如是其他網絡和第3層或第5層表示形式。 然后，您希望這些表示的期望值與實際輸入以及生成的噪聲圖像的期望值相同。 因此，在這里您要防止發電機在當前鑒別器上過度訓練。 順便說一句，這在許多其他領域也是一種普遍的損失。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

What else can be done? Well, there’s the so-called Wasserstein loss. It’s derived from the Wasserstein distance which is also known as the earth movers distance. Here, you learn a discriminator that maximizes the discrepancy between the real and fake samples, and at the same time, you restrict the gradient to stay behind a certain limit. So, you essentially limit the gradient towards a specific Lipschitz constant which is the maximum slope of the gradient. Here, in the image on the right-hand side, you can see that out of the red discrimination curve which saturated very quickly, you can then create a discriminator that has this non-saturated loss. This way, you will always be able to find good gradients, even in areas where you’re already saturated with your discriminator. Again, this helps to counter vanishing gradients in the discriminator. Many more loss functions exist like the KL divergence. Then, the GANs actually do maximum likelihood, but the approximation strategy matters much more than the loss.

還有什么可以做的？ 好吧，這就是所謂的Wasserstein損失。 它源自Wasserstein距離，也稱為推土機距離。 在這里，您將學習一個鑒別器，該鑒別器可以使真實樣本與假樣本之間的差異最大化，同時，您可以將梯度限制在一定限度內。 因此，您實際上將梯度限制在特定的Lipschitz常數上，該常數是梯度的最大斜率。 在這里，在右側的圖像中，您可以看到在很快Swift飽和的紅色辨別曲線中，您可以創建一個具有這種非飽和損耗的鑒別器。 這樣一來，即使在已經充滿識別器的區域中，您也始終可以找到良好的漸變。 同樣，這有助于抵消鑒別器中消失的梯度。 存在更多的損失函數，例如KL散度。 然后，GAN確實具有最大的可能性，但是近似策略比損失要重要得多。

Faces can be generated extremely well with GANS. Image created using gifify. Source: YouTube使用GANS可以很好地生成人臉。 使用gifify創建的圖像 。 資料來源： YouTube

So, how do we evaluate GANs? Well, we can, of course, look at the image and say: “Yeah, they look realistic! Or not?” But this is kind of intractable for large data sets. So, you have to use a score for images. One idea is the inception score.

那么，我們如何評估GAN？ 好吧，我們當然可以看一下圖像說：“是的，它們看起來很逼真！ 或不？” 但這對于大型數據集來說是很難處理的。 因此，您必須對圖像使用分數。 一種想法是起始分數。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

The inception score is based on two goals. One goal is that the generated images should be recognizable. So, you use, for example, an Inception v3 pre-trained Network on ImageNet and you want the score distribution to be dominated by one class. The image-wise class distribution should have low entropy. At the same time, you want the generated images to be diverse. So the overall class distribution should be more or less uniform. The entropy should be high. So, you can then express this inception score as e to the power of the expected value of the KL divergence between p(y|x) and p(y).

初始分數基于兩個目標。 一個目標是生成的圖像應該是可識別的。 因此，例如，您使用ImageNet上的Inception v3預訓練網絡，并且希望分數分布由一個類別控制。 按圖像分類分布應具有較低的熵。 同時，您希望生成的圖像多樣化。 因此，整個班級分布應大致相同。 熵應該很高。 因此，您可以將該初始得分表示為p(y | x )和p(y)之間KL散布的期望值的冪。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

Another measurement is the Fréchet inception distance which is using an intermediate layer. So, the last pooling layer of Inception v3 pretrained on ImageNet, for example. Then, you model the data distribution by multivariate Gaussians. The FID score between the real images x and the generated images g can be expressed as the difference between the mean values of x and g in an l2 norm plus the trace of the covariance matrices of x and g minus two times the square root of covariance matrix x times covariance matrix g. This is more robust than the inception score. We don’t need the class concept. In this case, we can simply work on multivariate Gaussians in order to model the distributions.

另一個度量是使用中間層的弗雷謝特起始距離。 因此，例如，在ImageNet上對Inception v3的最后一個池化層進行了預訓練。 然后，通過多元高斯模型對數據分布進行建模。 實像x和生成的圖像g之間的FID分數可以表示為l2范數中x和g的平均值之差加上x和g的協方差矩陣的跡線減去協方差平方根的兩倍矩陣x乘方差矩陣g 。 這比初始分數更健壯。 我們不需要類的概念。 在這種情況下，我們可以簡單地處理多元高斯模型以對分布進行建模。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

A big advantage of GANs is that they are able to generate samples in parallel. There are very few restrictions. For example, compared to the Boltzmann machines that have plenty of restrictions: You don’t need a Markov chain in this model. There are also no variational bounds needed. GANs are known to be asymptotically consistent since the model families are universal function approximators. So, this was a very first introduction to GANs.

GAN的一大優勢在于它們能夠并行生成樣本。 限制很少。 例如，與具有很多限制的Boltzmann機器相比：在此模型中不需要馬爾可夫鏈。 也沒有變化的界限。 由于模型族是通用函數逼近器，因此已知GAN漸近一致。 因此，這是GAN的第一個介紹。

CC BY 4.0 from the 深度學習講座中 Deep Learning Lecture.CC BY 4.0下的圖像。

In the next video, we want to talk a bit about more advanced GAN concepts like the conditional GANs where we can also start and modeling constraints and conditions into the generation process. People also looked into a very cool technique that is called the cycle GAN which allows unpaired to domain translation. So, you can translate images from day to night. You can even translate horses to zebras and zebras to horses. So, a very, very cool technique coming up. I hope you enjoyed this video and I’m looking forward to seeing you in the next one. Thank you very much!

在下一個視頻中，我們想談一些更高級的GAN概念，例如條件GAN，我們還可以在其中開始并將約束和條件建模到生成過程中。 人們還研究了一種稱為循環GAN的非常酷的技術，該技術允許不配對的域翻譯。 因此，您可以將圖像白天轉換為晚上。 您甚至可以將馬翻譯成斑馬，再將斑馬翻譯成馬。 因此，出現了一種非常非常酷的技術。 希望您喜歡這個視頻，并期待在下一個視頻中見到您。 非常感謝你！

Simpson GAN creates your favourite cartoon characters. Image created using gifify. Source: YouTube辛普森·甘(Simpson GAN)創建您最喜歡的卡通人物。 使用gifify創建的圖像 。 資料來源： YouTube

If you liked this post, you can find more essays here, more educational material on Machine Learning here, or have a look at our Deep LearningLecture. I would also appreciate a follow on YouTube, Twitter, Facebook, or LinkedIn in case you want to be informed about more essays, videos, and research in the future. This article is released under the Creative Commons 4.0 Attribution License and can be reprinted and modified if referenced. If you are interested in generating transcripts from video lectures try AutoBlog.

如果你喜歡這篇文章，你可以找到這里更多的文章 ，更多的教育材料，機器學習在這里 ，或看看我們的深入 學習 講座 。 如果您希望將來了解更多文章，視頻和研究信息，也歡迎關注YouTube ， Twitter ， Facebook或LinkedIn 。 本文是根據知識共享4.0署名許可發布的 ，如果引用，可以重新打印和修改。 如果您對從視頻講座中生成成績單感興趣，請嘗試使用AutoBlog 。

鏈接 (Links)

Link — Variational Autoencoders: Link — NIPS 2016 GAN Tutorial of GoodfellowLink — How to train a GAN? Tips and tricks to make GANs work (careful, noteverything is true anymore!) Link - Ever wondered about how to name your GAN?

鏈接 —可變自動編碼器： 鏈接 — Goodfellow的NIPS 2016 GAN教程鏈接 —如何訓練GAN？ 使GAN正常工作的提示和技巧(小心，什么都沒了！) 鏈接 -是否想知道如何命名GAN？

翻譯自: https://towardsdatascience.com/unsupervised-learning-part-3-7b15038bb884

無監督學習 k-means

總結

以上是生活随笔為你收集整理的无监督学习 k-means_无监督学习-第3部分的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。