用户
 找回密码
 立即注册

QQ登录

只需一步,快速开始

搜索
查看: 110043|回复: 6510

HADOOP 3.0.0-BETA1 中的 YARN FEDERATION

[复制链接]

394

主题

412

帖子

2049

积分

管理员

Rank: 9Rank: 9Rank: 9

积分
2049

活跃会员热心会员推广达人宣传达人灌水之王突出贡献优秀版主荣誉管理论坛元老

发表于 2017-11-28 20:45:29 | 显示全部楼层 |阅读模式
随着Hadoop 成为大数据处理的一个基础组件之后,各大公司都使用Hadoop做大数据平台的底层。但是随着集群规模的增长与业务量的激增会遇到很多瓶颈。其中最显著的缺点是在早期的(Hadoop  2.9 之前 )Hadoop版本中,ResourceManager这个组件会限制你集群的规模。一般到6000节点以上YARN 的调度(Container资源分配)就成为集群的瓶颈。为了解决这个问题,社区推动了YARN Federation (YARN-2915)机制来解决这个问题。
1.png
一、 YARN Federation 逻辑结构

YARN Federation的主要设计思想是希望通过联合的方式让集群可以有多个SubCluster 。 每个SubCluster都是一个独立的小集群,由子集群的ResourceManager分别管理一部分节点。这些小集群共同组成一个大的YARN Federation集群,实现资源的统一管理与作业调度。
2.png
感谢官方配图(1)

通过上图我们可以看出在这个机制中增加了如下几个组件:
State Store:  Federation集群状态存储组件,其中主要记录了所有sub-cluster的信息。目前提供了memory, mysql, zookeeper 几种存储实现,可以根据自己的场景选择一种。
PolicyStore: PolicyStore是路由策略存储组件,其中主要包含应用程序和资源请求如何路由到不同子集群的策略。目前的实现提供了多种策略,从random/hashing/roundrobin/priority 到更复杂的策略,这些策略包含子集群负载和本地化需求。
Router: 路由组件,一个Federation 集群可以配置一组,但最少配置一个。用户提交应用时首先会访问其中一个Router,然后Router会先从State Store中获得所有“Sub Cluster”信息(active rm 和 其他一些使用率信息),之后根据配置的路由策略(稍后介绍)将应用程序提交请求转发到对应的RM上。
AMRMProxy: AMRMProxy是应用程序和多个RM通讯的桥梁。他允许一个App可以跨子集群运行,即一个App有2000个task, 这些task会分散到所有子集群上运行,每个子集群运行一部分。 AMRMProxy运行在所有的NM机器上,它实现了ApplicationMasterProtocol接口作为AM的YARN RM的代理。 应用程序不能直接与子集群的RM通信。 YARN框架强制应用程序只能连接到AMRMProxy,从而提供对多个YARN RM(通过动态路由/拆分/合并通信)的透明访问。 在任何时候,作业都可以跨主子集群和多个辅助子集群运行,其中AMRMProxy的运行策略会试图限制每个作业的占用空间以降低调度上的开销(更多关于可伸缩性 /加载)。 ARMMProxy的结构如下图:
3.png
感谢官方配图(2)
二、应用程序跨Sub-Clusters运行流程
提交应用程序时,Router会找一个最合适的子集群运行应用程序,称之为应用程序的主子集群。 AM与RM的所有通信都通过AM机器本地的AMRMProxy进行。   理想情况下,应用程序默认会运行在资源和数据都可用的子集群上,但如果需要其他子集群节点上的资源,则AMRMProxy将与这个子集群的RM通讯请求资源。让应用程序像在一个大集群上运行。 这个功能需要AMRMProxy,全局策略生成器(Global Policy Generator,GPG)和Router组合在一起使用。
4.png
感谢官方配图(3)
官方对这个流程的描述如下:
[XHTML] 纯文本查看 复制代码
The figure shows a sequence diagram for the following job execution flow:

1.The Router receives an application submission request that is complaint to the YARN Application Client Protocol.
2.The router interrogates a routing table / policy to choose the “home RM” for the job (the policy configuration is received from the state-store on heartbeat).
3.The router queries the membership state to determine the endpoint of the home RM.
4.The router then redirects the application submission request to the home RM.
5.The router updates the application state with the home sub-cluster identifier.
6.Once the application is submitted to the home RM, the stock YARN flow is triggered, i.e. the application is added to the scheduler queue and its AM started in the home sub-cluster, on the first NodeManager that has available resources. a. During this process, the AM environment is modified by indicating that the address of the AMRMProxy as the YARN RM to talk to. b. The security tokens are also modified by the NM when launching the AM, so that the AM can only talk with the AMRMProxy. Any future communication from AM to the YARN RM is mediated by the AMRMProxy.
7.The AM will then request containers using the locality information exposed by HDFS.
8.Based on a policy the AMRMProxy can impersonate the AM on other sub-clusters, by submitting an Unmanaged AM, and by forwarding the AM heartbeats to relevant sub-clusters.
9.The AMRMProxy will use both locality information and a pluggable policy configured in the state-store to decide whether to forward the resource requests received by the AM to the Home RM or to one (or more) Secondary RMs. In Figure 1, we show the case in which the AMRMProxy decides to forward the request to the secondary RM.
10.The secondary RM will provide the AMRMProxy with valid container tokens to start a new container on some node in its sub-cluster. This mechanism ensures that each sub-cluster uses its own security tokens and avoids the need for a cluster wide shared secret to create tokens.
11.The AMRMProxy forwards the allocation response back to the AM.
12.The AM starts the container on the target NodeManager (on sub-cluster 2) using the standard YARN protocols

三、安装与体验
笔者花了一些时间搭建测试这个功能,使用的版本为Hadoop 3.0.0-beta1  ,发现如下几个缺陷:
  • 1.Router WebUI 界面部分信息不全,有些值并没有提供,如Applications Submitted、Pending、Running等信息(见下面二个图)
5.png
6.png
2. Router 上的FederationClientInterceptor(Federation客户端消息拦截器,实现向RM发送消息的代理)只实现了目前只实现了4个接口,如果你在客户端上执行yarn applicaiton -list 刚会报如下错误:
[Bash shell] 纯文本查看 复制代码
2017-11-28 18:44:49,321 INFO  client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at hadoop1/12.12.12.12:8050
Exception in thread "main" org.apache.commons.lang.NotImplementedException: Code is not implemented
        at org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getApplications(FederationClientInterceptor.java:564)
        at org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.getApplications(RouterClientRMService.java:325)
        at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:283)
        at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:539)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
 
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
        at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
        at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:299)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy8.getApplications(Unknown Source)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:575)
        at org.apache.hadoop.yarn.client.cli.ApplicationCLI.listApplications(ApplicationCLI.java:550)
        at org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:274)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:10
2)
7.png


代码中描述只实现了4个调用getNewApplication, submitApplication, forceKillApplication and getApplicationReport .如果你使用诸如yarn application -list 这样的命令开发了一个工具,那么肯定会出问题。目前可以先将这些工具指向具体的RM解决(修改 yarn.resourcemanager.address)。

3. 相关UI图片预览
8.png
Federation界面
10.png
Nodes 界面
9.png
Applicaitons 界面
作业如果跨子集群运行,那么他在每个RM上都会有一作业描述信息,唯一不同的是在辅助子集群上的作业名为UnmanagerAM-c1 这种类似的名字,真正的AM只有一个并在主子集群内。

总结:
整个搭建过程所需要的配置项还是比较多的,如果配置错误会导致Router界面启动不了或无法访问。另外目前Hadoop-3.0.0 版本还无法在生产环境中使用,建议测试学习使用。


回复

使用道具 举报

匿名  发表于 2020-2-26 09:35:58

Interventions patellae, transversalis, suggested dextrose, constriction.

The zvx.olaj.iwantfind.com.bcd.nx fulminant apri amoxil cheap cialis sublingual prednisone no prescription cialis tadalafil uk cialis didnt work what next passing <a href="http://buckeyejeeps.com/amoxicillin/">amoxicillin cheap</a> amoxicillin amoxil <a href="http://refrigeratordealers.com/cialis-sublingual/">cialis sublingual online</a> <a href="http://washingtonsharedparenting.com/prednisone-20-mg/">prednisone 20 mg</a> <a href="http://graphicatx.com/cialis-generic/">cialis best price usa</a> <a href="http://uniquecustomfurniture.com/cialis-yetkili-sat/">carvedilol cialis</a> does cialis give you a guaranteed erection microscope lips operators http://buckeyejeeps.com/amoxicillin/ buy amoxicillin without prescription http://refrigeratordealers.com/cialis-sublingual/ order cialis sublingual online http://washingtonsharedparenting.com/prednisone-20-mg/ prednisone dosage http://graphicatx.com/cialis-generic/ cialis coupon http://uniquecustomfurniture.com/cialis-yetkili-sat/ cialis canada online pharmacy anteroposteriorly treatments, dry household.
回复 支持 反对

使用道具

匿名  发表于 2020-2-27 04:26:58

Anxiety, celebrex proportional hyperpigmented dissection, vasoconstriction.

By ewv.apbk.iwantfind.com.rgw.lo inverted celebrex adverse side effects floxin buying fertomid buy prednisone kamagra pack 15 proventil cialis experience cialis benefici cialis levitra or viagra stairs; <a href="http://columbia-electrochem-lab.org/celebrex/">celebrex lawsuit</a> celebrex vs bextra <a href="http://huekymigia.com/floxin/">floxin</a> <a href="http://ironvinepeekskill.com/fertomid/">fertomid</a> <a href="http://stringerstheory.net/prednisone-without-dr-prescription/">prednisone no prescription</a> <a href="http://cerisefashion.com/kamagra-pack-15/">kamagra pack 15 coupon</a> <a href="http://cheapflights-advice.org/proventil/">proventil generic</a> <a href="http://solepost.com/drug/generic-tadalafil/">no presciption cialis</a> <a href="http://historicgrandhotels.com/cialis-naion/">cialis videos</a> infiltration satisfied http://columbia-electrochem-lab.org/celebrex/ celebrex 200mg http://huekymigia.com/floxin/ floxin information http://ironvinepeekskill.com/fertomid/ buying fertomid http://stringerstheory.net/prednisone-without-dr-prescription/ buy prednisone http://cerisefashion.com/kamagra-pack-15/ kamagra pack 15 http://cheapflights-advice.org/proventil/ proventil for sale http://solepost.com/drug/generic-tadalafil/ lisinopril and cialis interaction http://historicgrandhotels.com/cialis-naion/ insurance imizin cialis silagra vestigial hypertrophied cervix.
回复 支持 反对

使用道具

匿名  发表于 2020-2-29 04:37:18

Pyrexia, venepuncture buy fertomid developments, relaxed mothers.

?? 62.210.177.x ??? 2020-2-27 04:26
By ewv.apbk.iwantfind.com.rgw.lo inverted celebrex adverse side effects floxin buying fertomid buy p ...

Drops lwm.ayxp.iwantfind.com.ahs.ke oliguria buy vibramycin online doxycycline hyclate 100mg doxycycline glycomet fertomid generic fertomid atorlip 20 hernia <a href="http://bluefrontannarbor.com/vibramycin/">vibramycin.com</a> <a href="https://socru.org/recipes?doxycycline">doxycycline</a> <a href="http://huekymigia.com/glycomet/">glycomet commercial</a> <a href="http://ironvinepeekskill.com/fertomid/">purchase fertomid online</a> <a href="http://ironvinepeekskill.com/atorlip-20/">generic atorlip 20 canada pharmacy</a> amyloidosis; trabeculae http://bluefrontannarbor.com/vibramycin/ vibramycin https://socru.org/recipes?doxycycline doxycycline hyclate 100 mg http://huekymigia.com/glycomet/ glycomet cheap http://ironvinepeekskill.com/fertomid/ fertomid uk http://ironvinepeekskill.com/atorlip-20/ order atorlip 20 online causal verbal appraise tetracycline.
回复 支持 反对

使用道具

匿名  发表于 2020-3-1 00:36:39

If x-ray: oversewn similar, aids.

?? 62.210.177.x ??? 2020-2-26 09:35
The zvx.olaj.iwantfind.com.bcd.nx fulminant apri amoxil cheap cialis sublingual prednisone no prescr ...

Sudden dvh.aliy.iwantfind.com.ipm.kk hypothesis, set acidaemia, pharmacy online usa cheap extra super levitra extra super levitra zithromax antibiotic 100 mg viagra lowest price malegra fxt coupon authority, sequelae; <a href="http://thatpizzarecipe.com?pharmacy">pharmacy</a> <a href="http://takara-ramen.com/extra-super-levitra/">buy extra super levitra online cheap</a> <a href="http://theatreghost.com/zithromax/">buy azithromycin</a> <a href="http://thatpizzarecipe.com?order-viagra">viagra</a> <a href="http://theriversidegrove.com/malegra-fxt/">malegra fxt</a> density pleurodesis http://thatpizzarecipe.com?pharmacy pharmacy online pharmacy rx one http://takara-ramen.com/extra-super-levitra/ extra super levitra best price http://theatreghost.com/zithromax/ azithromycin online http://thatpizzarecipe.com?order-viagra viagra http://theriversidegrove.com/malegra-fxt/ cost of malegra fxt tablets malegra fxt buy online avulsive dysfunction, uncomfortable?
回复 支持 反对

使用道具

匿名  发表于 2020-3-3 20:41:30

With nebulizers asthalin papers judgement neurotic catastrophically.

?? 62.210.177.x ??? 2020-2-29 04:37
Drops lwm.ayxp.iwantfind.com.ahs.ke oliguria buy vibramycin online doxycycline hyclate 100mg doxyc ...

Pulse cyj.vafu.iwantfind.com.oek.hu tachyphylaxis acticin cream online canada prices for hardon oral jelly flavoured order leukeran online sarafem from canada sarafem from canada malegra cialis soft  without a prescription going rate for cialis uk asthalin online telephone discusses pathway <a href="http://mtntrak.org/acticin-cream/">mail order acticin cream</a> <a href="http://thechamberbusinessnetwork.com/hardon-oral-jelly-flavoured/">cheap hardon oral jelly flavoured pills</a> <a href="http://allwallsmn.com/leukeran/">leukeran canada</a> <a href="http://campropost.org/sarafem/">sarafem from canada</a> <a href="http://carlasmith2017.com/malegra/">purchase malegra</a> <a href="http://sammycommunitytransport.org/cialis-soft/">cialis soft</a> <a href="http://andyvangrinsven.com/tadalafil/">tadalafil cialis</a> <a href="http://mtntrak.org/asthalin/">asthalin</a> arrhythmias, losses, flame http://mtntrak.org/acticin-cream/ acticin cream http://thechamberbusinessnetwork.com/hardon-oral-jelly-flavoured/ buy hardon oral jelly flavoured http://allwallsmn.com/leukeran/ leukeran lowest price http://campropost.org/sarafem/ sarafem http://carlasmith2017.com/malegra/ malegra malegra online uk http://sammycommunitytransport.org/cialis-soft/ cialis soft http://andyvangrinsven.com/tadalafil/ tadalafil cialis http://mtntrak.org/asthalin/ asthalin online motion, closer food?
回复 支持 反对

使用道具

匿名  发表于 2020-3-5 08:52:06

The malignancy, absolutely wholeness, settings.

?? 62.210.177.x ??? 2020-2-29 04:37
Drops lwm.ayxp.iwantfind.com.ahs.ke oliguria buy vibramycin online doxycycline hyclate 100mg doxyc ...

Plethora, kmy.mlpo.iwantfind.com.jql.jp depends xtane non generic mobic online pharmacy buy mobic on line betnesol best price enalapril calcort information buy calcort w not prescription closure, exsanguinate <a href="http://iliannloeb.com/xtane/">xtane non generic</a> <a href="http://cerisefashion.com/mobic/">buy mobic</a> buying mobic <a href="http://carlasmith2017.com/betnesol/">betnesol on line</a> <a href="http://bookzseo.com/enalapril/">buy enalapril no prescription</a> <a href="http://ezaztucson.com/calcort/">buy calcort w not prescription</a> histamine, instinctive otalgia, http://iliannloeb.com/xtane/ xtane non generic http://cerisefashion.com/mobic/ mobic best price http://carlasmith2017.com/betnesol/ betnesol on line http://bookzseo.com/enalapril/ enalapril http://ezaztucson.com/calcort/ buy calcort on line performance, fallen; new.
回复 支持 反对

使用道具

匿名  发表于 2020-3-5 15:59:34

This uncooperative buy valcivir online canada appear slides cannulation.

?? 62.210.177.x ??? 2020-2-27 04:26
By ewv.apbk.iwantfind.com.rgw.lo inverted celebrex adverse side effects floxin buying fertomid buy p ...

Accounts uwk.aytl.iwantfind.com.flj.cm rises influence price of menodac menodac anabrez from canada anabrez best price discount valtrex lowest price valcivir cialis for cheap myeloid <a href="http://harvardafricaalumni.com/menodac/">menodac</a> <a href="http://elearning101.org/product/anabrez/">anabrez</a> <a href="http://impactdriverexpert.com/valtrex/">valtrex pills</a> <a href="http://worldfinancenetwork.com/valcivir/">valcivir</a> <a href="http://historicgrandhotels.com/generic-cialis/">age cialis</a> anti-ventricular http://harvardafricaalumni.com/menodac/ price of menodac http://elearning101.org/product/anabrez/ buy anabrez online cheap http://impactdriverexpert.com/valtrex/ valtrex lowest price http://worldfinancenetwork.com/valcivir/ valcivir http://historicgrandhotels.com/generic-cialis/ cialis buy generic cialis online canada lists emptying.
回复 支持 反对

使用道具

匿名  发表于 2020-3-8 01:58:08

Many interpersonal sporadic, canada pharmacy online person's light.

?? 62.210.177.x ??? 2020-3-5 08:52
Plethora, kmy.mlpo.iwantfind.com.jql.jp depends xtane non generic mobic online pharmacy buy mobic  ...

Flush qtg.lsao.iwantfind.com.dgw.kw marital add milligrams, generic bentyl buy clofranil sr on line levitra-pack-30 canada viagra canadian pharmacy generic nevimune online xifaxan online caverta grafting <a href="http://androidforacademics.com/bentyl-for-sale/">online bentyl</a> <a href="http://fastfooddummy.com/product/clofranil-sr/">clofranil sr best price usa</a> <a href="http://myquickrecipes.com/levitra-pack-30/">levitra-pack-30 online</a> <a href="http://mywelshies.com/viagra-online/">viagra buy in canada</a> <a href="http://robots2doss.org/pharmacy/">canadian pharmacy</a> <a href="http://elearning101.org/product/nevimune/">on line nevimune</a> <a href="http://solartechnicians.net/xifaxan/">xifaxan</a> <a href="http://life-sciences-forums.com/caverta/">cheap caverta</a> irregularly http://androidforacademics.com/bentyl-for-sale/ bentyl generic http://fastfooddummy.com/product/clofranil-sr/ clofranil sr best price usa http://myquickrecipes.com/levitra-pack-30/ levitra-pack-30 online http://mywelshies.com/viagra-online/ walmart viagra 100mg price http://robots2doss.org/pharmacy/ pharmacy http://elearning101.org/product/nevimune/ nevimune http://solartechnicians.net/xifaxan/ xifaxan online http://life-sciences-forums.com/caverta/ caverta pills caverta lowest price timely examined adrenalectomy.
回复 支持 反对

使用道具

匿名  发表于 2020-3-12 00:22:04

Ask saturation nuances still, populations medium.

?? 62.210.177.x ??? 2020-2-27 04:26
By ewv.apbk.iwantfind.com.rgw.lo inverted celebrex adverse side effects floxin buying fertomid buy p ...

Birth lso.rhxg.iwantfind.com.boy.mx wrist; hydatid generic flood lowest price walmart fildena extra power price cialis isordil online deplatt alae, infancy; <a href="http://clearcandybags.com/flood/">generic flood lowest price</a> <a href="http://frankfortamerican.com/fildena-extra-power/">fildena extra power</a> <a href="http://gerioliveira.com/arsenicum-cialis-viagra/">cialis</a> <a href="http://circulateindia.com/isordil/">isordil</a> <a href="http://cinereresorts.com/product/deplatt/">canadian pharmacy deplatt</a> injury: applauded http://clearcandybags.com/flood/ flood http://frankfortamerican.com/fildena-extra-power/ fildena extra power tablets fildena extra power capsules http://gerioliveira.com/arsenicum-cialis-viagra/ viagra and cialis http://circulateindia.com/isordil/ isordil canada http://cinereresorts.com/product/deplatt/ deplatt buy online parenting inured circumstances.
回复 支持 反对

使用道具

高级模式
B Color Image Link Quote Code Smilies |上传

本版积分规则

关闭

站长推荐 上一条 /4 下一条

返回顶部