靜音抑制

Tech giants like Google and Microsoft have taken notice of exciting new AI research such as GPT-3, which can write articles, website markup, and even software code. But will their bottom lines stifle any real progress?

像Google和Microsoft這樣的科技巨頭已經(jīng)注意到了激動人心的AI新研究，例如GPT-3，它可以編寫文章，網(wǎng)站標(biāo)記甚至軟件代碼。 但是，他們的底線會扼殺任何實際進展嗎？

By Ben Dickson

通過本迪克森

A recent article in The Guardian stirred up a lot of excitement-and a little fear-on social media. The reason: The initial draft was reportedly written by GPT-3, OpenAI’s new text generator.

《衛(wèi)報》上最近的一篇文章激起了社會媒體的興奮和恐懼。 原因：據(jù)報道，最初的草案是由OpenAI的新文本生成器GPT-3編寫的。

Since its beta release, GPT-3, an artificial intelligence system that takes a cue and generates text, has captivated the tech community and the media. Developers and computer scientists have been using it to write articles, website markup, and even software code. Some entrepreneurs are contemplating creating new products on GPT-3.

自測試版發(fā)布以來，具有提示功能并生成文本的人工智能系統(tǒng)GPT-3吸引了技術(shù)界和媒體。 開發(fā)人員和計算機科學(xué)家一直在使用它編寫文章，網(wǎng)站標(biāo)記甚至軟件代碼。 一些企業(yè)家正在考慮在GPT-3上開發(fā)新產(chǎn)品。

While flawed in fundamental ways, GPT-3 still shows how far advances in natural language processing have come. This is by far the largest and most coherent text-generation algorithm ever created.

盡管GPT-3在根本上存在缺陷，但它仍然顯示出自然語言處理技術(shù)已經(jīng)取得了多大進步。 這是迄今為止創(chuàng)建的最大，最一致的文本生成算法。

But it also highlights some of the problems the AI research community faces, including its growing dependence on the wealth of large tech companies. This is a problem that could endanger the scientific mission for which OpenAI and other AI research labs were founded.

但這也凸顯了AI研究界面臨的一些問題，包括其對大型科技公司財富的日益依賴。 這個問題可能會危及OpenAI和其他AI研究實驗室建立的科學(xué)使命。

GPT-3的費用 (The Cost of GPT-3)

GPT-3 is a massive deep-learning model. Deep learning is a type of AI system that develops its behavior through experience. Every deep learning model is composed of many layers of parameters that start at random values and gradually tune themselves as the model is trained on examples.

GPT-3是一個大規(guī)模的深度學(xué)習(xí)模型。 深度學(xué)習(xí)是一種通過經(jīng)驗來發(fā)展其行為的AI系統(tǒng)。 每個深度學(xué)習(xí)模型均由多層參數(shù)組成，這些參數(shù)層從隨機值開始，并隨著在實例上進行訓(xùn)練而逐漸進行自我調(diào)整。

Before deep learning, programmers and domain experts had to manually write the commands that defined the logic and rules to parse and make sense of text. With deep learning, you provide a model with a large corpus of text-say, Wikipedia articles-and it adjusts its parameters to capture the relations between the different words. You can then use the model for a variety of language tasks such as answering questions, automatic email-reply suggestions, and advanced search.

在進行深度學(xué)習(xí)之前，程序員和領(lǐng)域?qū)＜冶仨毷謩泳帉懚x邏輯和規(guī)則的命令，以解析和理解文本。 通過深度學(xué)習(xí)，您可以為模型提供大量的語料庫，例如Wikipedia文章，并且可以調(diào)整其參數(shù)以捕獲不同單詞之間的關(guān)系。 然后，您可以將該模型用于各種語言任務(wù)，例如回答問題，自動電子郵件回復(fù)建議和高級搜索。

Research and development in the past few years has shown that in general, the performance of deep-learning models improves as they are given larger numbers of parameters and trained on bigger data sets.

過去幾年的研究和開發(fā)表明，一般而言，深度學(xué)習(xí)模型的性能會得到提高，因為它們會被賦予更多的參數(shù)并在更大的數(shù)據(jù)集上進行訓(xùn)練。

In this respect, GPT-3 has broken all records: It is composed of 175 billion parameters, which makes it more than a hundred times larger than its predecessor, GPT-2. And the data set used to train the AI is at least 10 times larger than GPT-2’s 40-gigabyte training corpus. Although there’s much debate about whether larger neural networks will solve the fundamental problem of understanding the context of language, GPT-3 has outperformed all of its predecessors in language-related tasks.

在這方面，GPT-3打破了所有記錄：它由1,750億個參數(shù)組成，這使其比其前身GPT-2大100倍以上。 用于訓(xùn)練AI的數(shù)據(jù)集至少比GPT-2的40 GB訓(xùn)練語料庫大10倍。 盡管關(guān)于大型神經(jīng)網(wǎng)絡(luò)是否能夠解決理解語言上下文的根本問題仍有很多爭議，但GPT-3在與語言有關(guān)的任務(wù)中勝過了所有其前任。

But the benefits of larger neural networks come with trade-offs. The more parameters and layers you add to a neural network, the more expensive its training becomes. According to an estimate by Chuan Li, the Chief Science Officer of Lambda, a provider of hardware and cloud resources for deep learning, it could take up to 355 years and $4.6 million to train GPT-3 on a server with a V100 graphics card.

但是，更大的神經(jīng)網(wǎng)絡(luò)的好處在于權(quán)衡取舍。 添加到神經(jīng)網(wǎng)絡(luò)的參數(shù)和層越多，其訓(xùn)練費用就越高。 根據(jù)用于深度學(xué)習(xí)的硬件和云資源提供商Lambda的首席科學(xué)官Chuan Li的估計，在配備V100顯卡的服務(wù)器上訓(xùn)練GPT-3可能需要長達355年的時間和460萬美元。

“Our calculation with a V100 GPU is extremely simplified. In practice, you can’t train GPT-3 on a single GPU, but with a distributed system with many GPUs like the one OpenAI used,” Li says. “One will never get perfect scaling in a large distributed system due to the overhead of device-to-device communication. So in practice, it will take more than $4.6 million to finish the training cycle.”

“我們使用V100 GPU的計算得到了極大簡化。 在實踐中，您不能在單個GPU上訓(xùn)練GPT-3，但要在具有許多GPU的分布式系統(tǒng)(例如使用的一個OpenAI)上進行訓(xùn)練，” Li說。 “由于設(shè)備到設(shè)備通信的開銷，在大型分布式系統(tǒng)中永遠無法獲得完美的擴展。 因此，在實踐中，完成培訓(xùn)周期將需要超過460萬美元。”

This estimate is still simplified. Training a neural network is hardly a one-shot process. It involves a lot of trial and error, and engineers must often change the settings and retrain the network to obtain optimal performance.

該估計仍被簡化。 訓(xùn)練神經(jīng)網(wǎng)絡(luò)幾乎不是一shot而就的過程。 它涉及很多試驗和錯誤，工程師必須經(jīng)常更改設(shè)置并重新培訓(xùn)網(wǎng)絡(luò)以獲得最佳性能。

“There are certainly behind-the-scenes costs as well: parameter tuning, the prototyping that it takes to get a finished model, the cost of researchers, so it certainly was expensive to create GPT-3,” says Nick Walton, the co-founder of Latitude and the creator of AI dungeon, a text-based game created on GPT-2.

“當(dāng)然還有幕后的成本：參數(shù)調(diào)整，獲得完整模型所需的原型，研究人員的成本，因此創(chuàng)建GPT-3肯定是昂貴的，”合作伙伴Nick Walton說道。 -Latitude的創(chuàng)始人和AI地下城的創(chuàng)造者，這是一款基于GPT-2創(chuàng)建的基于文本的游戲。

Walton said that the real cost of the research behind GPT-3 could be anywhere between 1.5 to 5 times the cost of training the final model, but he added, “It’s really hard to say without knowing what their process looks like internally.”

沃爾頓說，GPT-3背后的研究的實際成本可能是訓(xùn)練最終模型的成本的1.5到5倍之間，但他補充說：“在不知道內(nèi)部流程是什么樣的情況下，很難說。”

追求盈利模式 (Going to a For-Profit Model)

OpenAI was founded in late 2015 as a nonprofit research lab with the mission to develop human-level AI for the benefit of all humanity. Among its founders were Tesla CEO Elon Musk and Sam Altman, former Y Combinator president, who collectively donated $1 billion to the lab’s research. Altman later became the CEO of OpenAI.

OpenAI成立于2015年末，是一個非營利性研究實驗室，其使命是開發(fā)造福全人類的人類級AI 。 其創(chuàng)始人包括特斯拉首席執(zhí)行官埃隆·馬斯克(Elon Musk)和前Y Combinator總裁薩姆·奧特曼(Sam Altman)，他們共同為該實驗室的研究捐款10億美元。 奧特曼后來成為OpenAI的首席執(zhí)行官。

But given the huge costs of training deep-learning models and hiring AI talent, $1 billion would cover only a few years’ worth of OpenAI’s research. It was clear from the beginning that the lab would run into cash problems long before it reached its goal.

但是考慮到培訓(xùn)深度學(xué)習(xí)模型和雇用AI人才的巨額成本，10億美元將僅涵蓋OpenAI研究的幾年價值。 從一開始就很明顯，實驗室在達到目標(biāo)之前就陷入了現(xiàn)金問題。

“We’ll need to invest billions of dollars in upcoming years into large-scale cloud compute, attracting and retaining talented people, and building AI supercomputers,” the lab declared in 2019, when it renamed itself OpenAI LP and restructured to a “capped-profit” company. The change allowed venture capital firms and large tech companies to invest in OpenAI for returns “capped” at a hundred times their initial investment.

該實驗室在2019年宣布時將其命名為OpenAI LP，并改組為“封閉式”實驗室，該實驗室于2019年宣布：“未來幾年，我們需要投資數(shù)十億美元用于大規(guī)模云計算，吸引和留住人才，并建造AI超級計算機。” -利潤”公司。 這一變化使風(fēng)險投資公司和大型科技公司可以在OpenAI上進行投資，以將其“最高”回報限制為初始投資的100倍。

Shortly after the announcement, Microsoft invested $1 billion in OpenAI. The infusion of cash allowed the company to continue to work on GPT-3 and other expensive deep-learning projects. But investor money always comes with strings attached.

宣布后不久，微軟向OpenAI投資了10億美元。 注入的現(xiàn)金使公司得以繼續(xù)從事GPT-3和其他昂貴的深度學(xué)習(xí)項目。 但是投資者的錢總是附帶條件。

向模糊過渡 (Shifting Toward Obscurity)

In June, when it announced GPT-3, the company did not release its AI model to the public, as is the norm in scientific research. Instead, it released an application programming interface (API) that allows developers to give GPT-3 input and obtain the results. In the future, the company will commercialize GPT-3 by renting out access to the API.

6月，當(dāng)它宣布GPT-3時，該公司并未像科學(xué)研究的規(guī)范那樣向公眾發(fā)布其AI模型。 相反，它發(fā)布了一個應(yīng)用程序編程接口(API)，允許開發(fā)人員提供GPT-3輸入并獲取結(jié)果。 將來，該公司將通過出租API的訪問權(quán)來實現(xiàn)GPT-3的商業(yè)化。

“Commercializing the technology helps us pay for our ongoing AI research, safety, and policy efforts,” OpenAI wrote in a blog post announcing the GPT-3 API.

OpenAI在宣布GPT-3 API的博客文章中寫道：“將該技術(shù)商業(yè)化有助于我們?yōu)檎谶M行的AI研究，安全和政策工作付費。”

But to make GPT-3 profitable, OpenAI will have to make sure other companies can’t replicate it, which is why it is not making the source code and trained model public. Organizations and individuals can request access to the GPT-3 API-but not every request is approved.

但是，要使GPT-3盈利，OpenAI必須確保其他公司不能復(fù)制它，這就是為什么它不公開源代碼和經(jīng)過訓(xùn)練的模型。 組織和個人可以請求訪問GPT-3 API，但并非所有請求都得到批準(zhǔn)。

Among those who weren’t given access to GPT-3 API are Gary Marcus, cognitive scientist and AI researcher, and Ernest Davis, computer science professor at New York University, who were interested in testing the capabilities and limits of GPT-3.

未被授予使用GPT-3 API權(quán)限的人包括認知科學(xué)家和AI研究人員Gary Marcus和紐約大學(xué)計算機科學(xué)教授Ernest Davis，他們對測試GPT-3的功能和限制感興趣。

“OpenAI has thus far not allowed us research access to GPT-3, despite both the company’s name and the nonprofit status of its oversight organization. Instead, OpenAI put us off indefinitely despite repeated requests-even as it made access widely available to the media,” Marcus and Davis wrote in an article published in MIT Technology Review. “OpenAI’s striking lack of openness seems to us to be a serious breach of scientific ethics, and a distortion of the goals of the associated nonprofit.”

“盡管公司的名稱和其監(jiān)督組織的非營利組織身份，OpenAI迄今仍不允許我們對GPT-3進行研究訪問。 相反，盡管反復(fù)提出要求，OpenAI仍然無限期地推遲了我們的工作，即使它使媒體可以廣泛使用它。” Marcus和Davis在《麻省理工科技評論》上發(fā)表的一篇文章中寫道。 “ OpenAI明顯缺乏開放性，在我們看來，這嚴重違反了科學(xué)道德規(guī)范，并扭曲了相關(guān)非營利組織的目標(biāo)。”

The two scientists managed to run the experiments through a colleague who had access to the API, but their research was limited to a small number of tests. Marcus had been a vocal critic of the hype surrounding GPT-3’s predecessor.

兩位科學(xué)家設(shè)法通過可以訪問API的同事進行了實驗，但是他們的研究僅限于少量測試。 馬庫斯曾大聲批評圍繞GPT-3的前身大肆宣傳。

可以保存AI研究嗎？ (Can AI Research Be Saved?)

GPT-3 shows the growing challenges of scientific AI research. The focus on creating larger and larger neural networks is increasing the costs of research. And, for the moment, the only organizations that can dispense that kind of money are large tech companies such as Google, Microsoft, and SoftBank.

GPT-3顯示了AI科學(xué)研究日益嚴峻的挑戰(zhàn)。 對創(chuàng)建越來越大的神經(jīng)網(wǎng)絡(luò)的關(guān)注增加了研究成本。 而且，目前，唯一可以分配這類資金的組織是大型科技公司，例如Google，Microsoft和SoftBank。

But those companies are interested in short-term returns on investment, not long-term goals that benefit humanity in its entirety.

但是那些公司對短期投資回報感興趣，而不是對整個人類有利的長期目標(biāo)。

OpenAI now has a commitment to Microsoft and other potential investors, and it must show proof that it is a profitable company to ensure future funding. At the same time, it wants to pursue its scientific mission of creating beneficial AGI (artificial general intelligence, essentially human-level AI), which does not have short-term returns and is at least decades away.

OpenAI現(xiàn)在對微軟和其他潛在投資者有承諾，并且必須證明它是確保未來資金投入的盈利公司。 同時，它希望履行其科學(xué)使命，即創(chuàng)建有益的AGI(人工通用情報，本質(zhì)上是人類水平的AI)，該投資沒有短期回報，而且至少需要幾十年的時間。

Those two goals conflict in other ways. Scientific research is predicated on transparency and information sharing among different communities of scientists. In contrast, creating profitable products requires hiding research and hoarding company secrets to keep the edge over competitors.

這兩個目標(biāo)在其他方面存在沖突。 科學(xué)研究基于不同科學(xué)家社區(qū)之間的透明度和信息共享。 相反，創(chuàng)造有利潤的產(chǎn)品需要隱藏研究和ho積公司機密，以保持領(lǐng)先于競爭對手的優(yōu)勢。

Finding the right balance between the nonprofit mission and the for-profit commitment will be extremely difficult. And OpenAI’s situation is not an isolated example. DeepMind, the UK-based research lab that is considered one of OpenAI’s peers, faced similar problems after it was acquired by Google in 2014.

在非營利組織的使命與營利性承諾之間找到適當(dāng)?shù)钠胶鈱⒎浅＠щy。 而且OpenAI的情況并非孤立的例子。 DeepMind是位于英國的研究實驗室，被認為是OpenAI的同行之一，它在2014年被Google收購后也面臨類似的問題。

Many scientists believe that AGI-if ever achieved-will be one of the most impactful inventions of humanity. If this is true, then achieving AGI will require the concerted efforts and contributions of the international community, not merely the deep pockets of companies whose main focus is their bottom line.

許多科學(xué)家認為，如果能夠?qū)崿F(xiàn)AGI，它將是人類最有影響力的發(fā)明之一。 如果這是真的，那么實現(xiàn)AGI將需要國際社會的共同努力和貢獻，而不僅僅是主要關(guān)注其底線的公司的財大氣粗。

A good model might be the Large Hadron Collider project, which obtained a $9 billion budget from funding agencies in CERN’s member and non-member states. While member states will eventually benefit from the results of CERN’s work, they don’t expect the organization to turn in profits in the short term.

一個很好的模型可能是大型強子對撞機項目，該項目從CERN成員國和非成員國的供資機構(gòu)獲得了90億美元的預(yù)算。 盡管成員國最終將從CERN的工作成果中受益，但他們并不希望該組織在短期內(nèi)實現(xiàn)盈利。

A similar initiative might help OpenAI and other research labs to continue chasing the dream of human-level AI without having to worry about returning investor money.

類似的舉措可能會幫助OpenAI和其他研究實驗室繼續(xù)追逐人類級AI的夢想，而不必擔(dān)心會退還投資者的錢。

Originally published at https://www.pcmag.com.

最初發(fā)布在https://www.pcmag.com 。

翻譯自: https://medium.com/pcmag-access/is-research-for-profit-holding-back-ai-innovation-c335a8dc44f7

靜音抑制

總結(jié)

以上是生活随笔為你收集整理的静音抑制_正在研究利润以抑制创新的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。