apache mesos

by Marco Massenzio

由Marco Massenzio

試用Apache Mesos HTTP API獲得樂(lè)趣和收益 (Experimenting with the Apache Mesos HTTP API for Fun and Profit)

Apache Mesos is a tool used in production at large-scale services like Twitter and Airbnb. Here’s its textbook description:

Apache Mesos是Twitter和Airbnb等大規(guī)模服務(wù)中用于生產(chǎn)的工具。 這是它的教科書描述：

The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with API’s for resource management and scheduling across entire datacenter and cloud environments. — from the Apache Mesos project site.

Mesos內(nèi)核在每臺(tái)機(jī)器上運(yùn)行，并為應(yīng)用程序(例如Hadoop，Spark，Kafka，Elasticsearch)提供API，用于在整個(gè)數(shù)據(jù)中心和云環(huán)境中進(jìn)行資源管理和調(diào)度。 —從Apache Mesos項(xiàng)目站點(diǎn)。

This is the first of a series of three articles that show how to setup a Vagrant-based Apache Mesos test/development environment on your laptop, how to run a Python notebook against the HTTP API, and how to launch Docker containers on the running Agent VM.

這是三篇系列文章中的第一篇，該系列文章介紹如何在筆記本電腦上設(shè)置基于Vagrant的Apache Mesos測(cè)試/開(kāi)發(fā)環(huán)境，如何針對(duì)HTTP API運(yùn)行Python筆記本以及如何在正在運(yùn)行的Agent上啟動(dòng)Docker容器虛擬機(jī)。

This series is an extended (and updated) version of the talk I gave at MesosCon Europe 2015 updated for Apache Mesos 1.0.0, which has just been released (August 2016) — you can also find the slides there.

本系列是我在MesosCon Europe 2015上 演講的擴(kuò)展(和更新)版本，該演講針對(duì)Apache Mesos 1.0.0(已于2016年8月發(fā)布)進(jìn)行了更新-您也可以在此處找到幻燈片 。

This post is pretty jam-packed, and will require you to be familiarity with some concepts around containers, VMs, and Mesos. But I’ll take the time to show all the intermediate steps (hence, the 3-parts). It should be easy to follow, even if you’ve never used Vagrant, Mesos, or even Jupyter notebooks before.

這篇文章擠滿了人，需要您熟悉有關(guān)容器，VM和Mesos的一些概念。 但是，我將花一些時(shí)間介紹所有中間步驟(因此分為3部分)。 即使您以前從未使用過(guò)Vagrant，Mesos甚至Jupyter筆記本電腦，也應(yīng)該易于遵循。

I recommend you first have a basic familiarity with Python and handling HTTP requests and responses, as we will not be going those details there.

我建議您首先對(duì)Python和處理HTTP請(qǐng)求和響應(yīng)有一個(gè)基本的了解，因?yàn)槲覀冊(cè)谶@里不再贅述。

All the code is available on the zk-mesos git repository:

所有代碼都可以在zk-mesos git存儲(chǔ)庫(kù)中找到 ：

git clone git@github.com:massenz/zk-mesos.git

And you can also view the README.

您還可以查看README 。

入門 (Getting Started)

In order to follow along, you will need to clone the repository (as shown above) and install Virtualbox and Vagrant. Follow the instructions on their respective sites and you’ll be up and running in no time.

為了進(jìn)行后續(xù)操作，您將需要克隆存儲(chǔ)庫(kù)(如上所示)并安裝Virtualbox和Vagrant 。 按照其各自站點(diǎn)上的說(shuō)明進(jìn)行操作，您將立即啟動(dòng)并運(yùn)行。

I also recommend quickly scanning the Vagrant documentation. A knowledge of Vagrant beyond `vagrant up` is not really required to get the most out of this series, but it may help if you get stuck (or would like to experiment and improve on our Vagrantfile).

我還建議快速掃描Vagrant文??檔。 要充分利用本系列的知識(shí)，并不是真正需要“ vagrant up ”以外的Vagrant知識(shí)，但是如果您陷入困境(或者想對(duì)我們的Vagrantfile進(jìn)行試驗(yàn)和改進(jìn))，則可能會(huì)有所幫助。

If you’re not familiar with Apache Mesos I recommend taking a look at the project’s site. I recommend reading Mesos in Action (note that I was one of the manuscript’s reviewers).

如果您不熟悉Apache Mesos ，建議您查看該項(xiàng)目的站點(diǎn)。 我建議閱讀《 Mesos in Action》 (請(qǐng)注意，我是手稿的審閱者之一)。

We will not be building Mesos from source here, but will instead use Mesosphere packages. You don’t need to download them. The Vagrantfile will automatically download and install on the VMs.

我們不會(huì)在這里從源代碼構(gòu)建Mesos，而是使用Mesosphere軟件包 。 您不需要下載它們。 Vagrantfile將自動(dòng)下載并安裝在VM上。

To run the Python notebook, we’ll take advantage of the Jupyter packages, and use a virtualenv to run all of our code. Virtualenv isn’t strictly necessary, but will prevent you messing up your system Python.

要運(yùn)行Python筆記本，我們將利用Jupyter軟件包，并使用virtualenv運(yùn)行所有代碼。 Virtualenv不是嚴(yán)格必需的，但是可以防止您弄亂系統(tǒng)Python。

If you‘ve never used virtualenv before:

如果您以前從未使用過(guò)virtualenv ：

$ sudo pip install virtualenv

And then create and run a virtualenv:

然后創(chuàng)建并運(yùn)行一個(gè)virtualenv ：

$ cd zk-mesos $ virtualenv mesos-demo$ source mesos-demo/bin/activate $ pip install -r requirements.txt

Finally, verify that you can run and load the Jupyter notebook:

最后，確認(rèn)您可以運(yùn)行并加載Jupyter筆記本：

$ jupyter notebook

This should automatically open your browser and point it to http://localhost:8888. From here you can select the notebooks/Demo-API.ipynb file. Don’t run it just yet, but if it shows up, it will confirm that your Python setup is just fine.

這應(yīng)該會(huì)自動(dòng)打開(kāi)瀏覽器，并將其指向http：// localhost：8888 。 在這里，您可以選擇notebooks / Demo-API.ipynb文件。 暫時(shí)不要運(yùn)行它，但是如果顯示出來(lái)，它將確認(rèn)您的Python設(shè)置很好。

構(gòu)建和安裝Apache Mesos (Building and installing Apache Mesos)

This is where the beauty of Vagrant shines in all its glory. Installing Apache Mesos Master and Agent are not trivial tasks, but in our case, it’s just a matter of:

這就是流浪者的美麗在所有榮耀中閃耀的地方。 安裝Apache Mesos Master和Agent并不是一件容易的事，但就我們而言，這只是一個(gè)問(wèn)題：

$ cd vagrant $ vagrant up

Make sure that you’re in the same directory as the Vagrantfile when issuing any of the Vagrant commands, or it will complain about it.

發(fā)出任何Vagrant命令時(shí)，請(qǐng)確保與Vagrantfile位于同一目錄中，否則它將抱怨。

It is worth noting that we are building two Vagrant boxes, so any command will operate on both unless specified. To avoid this, you can specify the name of the VM after the command. For example, to SSH onto the Agent:

值得注意的是，我們正在構(gòu)建兩個(gè) Vagrant框，因此，除非指定，否則任何命令都將對(duì)這 兩個(gè)框進(jìn)行操作。 為避免這種情況，您可以在命令后指定VM的名稱。 例如，要SSH到代理上：

$ vagrant ssh agent

This should log you in on that box. From there, you can explore, experiment, and diagnose any issues.

這將使您登錄該框。 從那里，您可以探索，試驗(yàn)和診斷任何問(wèn)題。

The vagrant up command will take some time to execute, but it should eventually lead your Virtualbox to have two VMs, named respectively mesos-master and mesos-agent. Incidentally, you should never need to use VirtualBox to manage them. All the tasks can be undertaken via Vagrant commands. But you can manage them manually if necessary or desired.

vagrant up命令將花費(fèi)一些時(shí)間來(lái)執(zhí)行，但最終將導(dǎo)致您的Virtualbox具有兩個(gè)VM，分別命名為mesos-master和mesos-agent 。 順便說(shuō)一句，您永遠(yuǎn)不需要使用VirtualBox來(lái)管理它們。 所有任務(wù)都可以通過(guò)Vagrant命令執(zhí)行。 但是，如有必要或期望，您可以手動(dòng)管理它們。

Once your VMs are built, ensure you can access Mesos HTTP UI at:

構(gòu)建完虛擬機(jī)后，請(qǐng)確保可以通過(guò)以下方式訪問(wèn)Mesos HTTP UI：

http://192.168.33.10:5050

You should also see one agent running, accessible either via the Master UI.

您還應(yīng)該看到一個(gè)正在運(yùn)行的代理，可以通過(guò)主UI進(jìn)行訪問(wèn)。

Or directly at:

或直接在：

http://192.168.33.11:5051/state

Note that the Agent runs not only on a different IP address than the Master, but also on a different port (5051 instead of 5050).

請(qǐng)注意，代理不僅在與主服務(wù)器不同的IP地址上運(yùn)行，而且在不同的端口(5051而不是5050)上運(yùn)行。

Look into vagrant/run-agent.sh to see a few of the command line flags that we use to run the Agent (and in run-master.sh for the Master).

查看vagrant / run-agent.sh，以查看我們用于運(yùn)行代理的一些命令行標(biāo)志(對(duì)于Master，在run-master.sh中)。

動(dòng)物園管理員 (Zookeeper)

It’s worth noting that we are also running an instance of Zookeeper (for Leader election and Master/Agent coordination) on the mesos-master VM, inside a Docker container: partly because we can, but also to show how easy it is to do so using containers.

值得注意的是，我們還在Docker容器內(nèi)的mesos-master VM上運(yùn)行了Zookeeper實(shí)例(用于Leader選舉和Master / Agent協(xié)調(diào))：部分原因是我們可以這樣做，但同時(shí)也表明這樣做很容易使用容器。

This one line (in run-master.sh), will give you a perfectly good ZK instance (albeit, a catastrophically unreliable one in a production environment, where you’d want to run at least 3–5 nodes, at least, on physically separate machines/racks):

這一行(在run-master.sh中 )將為您提供一個(gè)非常好的ZK實(shí)例(盡管在生產(chǎn)環(huán)境中，這是一個(gè)災(zāi)難性的不可靠實(shí)例)，在該生產(chǎn)環(huán)境中，您至少要在3-5個(gè)節(jié)點(diǎn)上運(yùn)行物理上分開(kāi)的機(jī)器/機(jī)架)：

docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 \ -p 3888:3888 jplock/zookeeper:3.4.8

And because we expose the ports (in particular, 2181) to the host VM, we can connect to it via the Zookeeper CLI utility (zkCli.sh) and explore it. From your development machine (you will need to first download Zookeeper) you can use:

并且由于我們將端口(特別是2181)公開(kāi)給主機(jī)VM，因此我們可以通過(guò)Zookeeper CLI實(shí)用程序( zkCli.sh )連接到它并進(jìn)行瀏覽。 在開(kāi)發(fā)機(jī)器(您需要首先下載Zookeeper)中，可以使用：

$ zkCli.sh -server 192.168.33.10:2181...[zk: 192.168.33.10:2181(CONNECTED) 4] get /mesos/vagrant/json.info_0000000000# Formatted for better readability:{"address": { "hostname": "mesos-master", "ip":"192.168.33.10", "port":5050 }, "hostname":"mesos-master", "id":"7eb34f10-b07c-4921-aece-bbaece09dfd1", "ip":169978048, "pid":"master@192.168.33.10:5050", "port":5050, "version":"1.0.0"}cZxid = 0xbctime = Sat Aug 27 14:00:44 PDT 2016...

This is how Agents get information about how to connect to the Master node.

這是代理如何獲取有關(guān)如何連接到主節(jié)點(diǎn)的信息。

The _000000 suffix gets incremented every time a new Leader gets elected, so depending on how long the ZK instance has been running and whether the Master was restarted, it may become something like _0000005. That is an “ephemeral node” in Zookeeper’s parlance.

每次選舉新的Leader時(shí)，后綴_000000都會(huì)增加，因此取決于ZK實(shí)例運(yùn)行了多長(zhǎng)時(shí)間以及是否重新啟動(dòng)了主服務(wù)器，它的后綴可能類似于_0000005。 在Zookeeper的說(shuō)法中，這是一個(gè)“ 短暫節(jié)點(diǎn)” 。

In that record above, it is worth noting that “pid” is libprocess unique identifier and “ip” as a compressed 4-byte representation of an IPv4 octect quadruple. These legacy fields and may eventually be removed.

在上面的記錄中，值得注意的是，“ pid”是libprocess唯一標(biāo)識(shí)符，“ ip”是IPv4 octect四元組的壓縮4字節(jié)表示形式。 這些舊字段可能最終會(huì)被刪除。

結(jié)語(yǔ) (Wrap Up)

You are now the proud owner of a Master/Agent 2-node Apache Mesos deployment. Welcome in the same league as Twitter and Airbnb production wizards.

您現(xiàn)在是Master / Agent 2節(jié)點(diǎn)Apache Mesos部署的驕傲擁有者。 歡迎與Twitter和Airbnb制作向?qū)恕?

In Part 2, we’ll run our Python notebook against the Master API and will accept the Agent’s offers to launch a Docker container.

在第2部分中，我們將針對(duì)Master API運(yùn)行Python筆記本，并將接受代理提供的啟動(dòng)Docker容器的報(bào)價(jià)。

If you’ve got time, let’s jump on in and learn how to connect to Mesos Master and accept Resource Offers.

如果您有時(shí)間，請(qǐng)繼續(xù)學(xué)習(xí)如何連接到Mesos Master并接受Resource Offer 。

Originally published at codetrips.com on August 27, 2016.

最初于2016年8月27日發(fā)布在codetrips.com 。

翻譯自: https://www.freecodecamp.org/news/experimenting-with-the-apache-mesos-http-api-for-fun-and-profit-part-1-of-3-cf5736e84f85/

apache mesos

總結(jié)

以上是生活随笔為你收集整理的apache mesos_试用Apache Mesos HTTP API获得乐趣和收益的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。