Yarn資源分配示例

1 簡(jiǎn)介

此手冊(cè)應(yīng)用于yarn資源按照隊(duì)列分配的情景。

2 準(zhǔn)備

2.1 環(huán)境說明

hadoop版本: hadoop-2.5.0-cdh5.3.3

3 配置

3.1 配置說明

1)? Yarn-site.xml里配置yanr能用到的資源總數(shù)以及公平調(diào)度策略所在路徑

#在虛擬機(jī)關(guān)機(jī)情況下，選中后點(diǎn)擊設(shè)置，在控制器SATA處點(diǎn)擊右邊的加號(hào)以創(chuàng)建虛擬硬盤。

?<property>

<name>yarn.nodemanager.resource.memory-mb</name>

<value>32760</value>

</property>

?

<property>

<name>yarn.scheduler.minimum-allocation-mb</name>

<value>2048</value>

</property>

?

<property>

<name>yarn.scheduler.maximum-allocation-mb</name>

<value>16384</value>

</property>

?

<property>

<name>yarn.nodemanager.resource.cpu-vcores</name>

<value>21</value>

</property>

?

?

?

<property>

???????? <name>yarn.scheduler.fair.allow-undeclared-pools</name>

???????? <value>false</value>

???????? <description>如果這是true,新的隊(duì)列可以在提交申請(qǐng)時(shí)被創(chuàng)建,falsessh使用默認(rèn)隊(duì)列</description>

</property>

<property>

???????? <name>yarn.scheduler.fair.user-as-default-queue</name>

???????? <value>false</value>

???????? <description>使用跟用戶名相同的隊(duì)列名,如果設(shè)置為false或者未設(shè)置,則所有未知隊(duì)列的應(yīng)用程序?qū)⒈惶峤坏絛efault隊(duì)列中，默認(rèn)值為true</description>

</property>

<property>

???????? <name>yarn.resourcemanager.scheduler.class</name>

???????? <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>

???????? <description>公平配置隊(duì)列配置在fair-scheduler</description>

</property>

<property>

???????? <name>yarn.scheduler.fair.allocation.file</name>

???????? <value>/home/hadoop/hadoop-2.7.1/etc/hadoop/fair-allocation.xml</value>

???????? <description>公平配置文件路徑</description>

</property>

<property>

???????? <name>yarn.scheduler.fair.preemption</name>

???????? <value>true</value>

???????? <description>是否支持搶占,默認(rèn)值為false</description>

</property>

<property>

???????? <name>yarn.scheduler.fair.sizebasedweight</name>

???????? <value>false</value>

???????? <description>是否啟用按應(yīng)用程序資源需求分配資源,默認(rèn)值為false即采用公平輪詢的方法分配資源</description>

</property>

<property>

???????? <name>yarn.scheduler.increment-allocation-mb</name>

???????? <value>1024</value>

???????? <description>僅fair有效,內(nèi)存規(guī)整化單位,墨認(rèn)值1024.(示例一個(gè)container請(qǐng)求1.5G,則調(diào)度器規(guī)整化為2G)</description>

</property>

2)? 配置隊(duì)列的資源限額及應(yīng)用數(shù)量

#隊(duì)列可指定到用戶或則用戶組,如果是用戶組則需要在aclSubmitApps參數(shù)前加空格。

<allocations>

???????? <queuename="root">

???????? <aclSubmitApps>hadoop,dpi</aclSubmitApps>

???????? <aclAdministerApps>hadoop</aclAdministerApps>

?????????????????? <queuename="default">

??????????????????????????? <minResources>2000mb,1vcores</minResources>

??????????????????????????? <maxResources>8000mb,8vcores</maxResources>

??????????????????????????? <maxRunningApps>40</maxRunningApps>

??????????????????????????? <schedulingMode>fair</schedulingMode>

??????????????????????????? <aclSubmitApps>*</aclSubmitApps>

??????????????????????????? <aclAdministerApps>hadoop</aclAdministerApps>

?????????????????? </queue>

?????????????????? <queuename="test">

??????????????????????????? <minResources>2000mb,10vcores</minResources>

??????????????????????????? <maxResources>8000mb,8vcores</maxResources>

??????????????????????????? <maxRunningApps>4</maxRunningApps>

??????????????????????????? <schedulingMode>fair</schedulingMode>

??????????????????????????? <aclSubmitApps>test,hadoop</aclSubmitApps>

??????????????????????????? <aclAdministerApps>hadoop</aclAdministerApps>

?????????????????? </queue>

?????????????????? <queuename="hadoop">

??????????????????????????? <minResources>2000mb,1vcores</minResources>

??????????????????????????? <maxResources>8000mb,8vcores</maxResources>

??????????????????????????? <maxRunningApps>4</maxRunningApps>

??????????????????????????? <schedulingMode>fair</schedulingMode>

??????????????????????????? <aclSubmitApps>hadoop</aclSubmitApps>

??????????????????????????? <aclAdministerApps>hadoop</aclAdministerApps>

?????????????????? </queue>

???????? </queue>

???????? <username="hadoop"> <!-- 對(duì)于特定用戶的配置 -->

???????? ?<maxRunningApps>100</maxRunningApps>?

???????? </user>?

???????? <userMaxAppsDefault>50</userMaxAppsDefault><!--默認(rèn)的用戶最多可以同時(shí)運(yùn)行的任務(wù) -->

???????? <queuePlacementPolicy>

?????????????????? <rulename="user" create="false" />

?????????????????? <rulename="specified" />

?????????????????? <rulename="default"/>

???????? </queuePlacementPolicy>

</allocations>

3.2配置問題

暫無.

#修改隊(duì)列信息后會(huì)動(dòng)態(tài)刷新，以yarn 8088端口里的資源為準(zhǔn)。

4 使用

set mapreduce.job.queuename為某個(gè)隊(duì)列到8088端口查看任務(wù)是否提交到該隊(duì)列.

5 總結(jié)

暫無.

總結(jié)

以上是生活随笔為你收集整理的Yarn资源分配示例的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。