用户
 找回密码
 立即注册

QQ登录

只需一步,快速开始

搜索
查看: 115884|回复: 6559

图解mapreduce原理和执行过程

[复制链接]

394

主题

412

帖子

2049

积分

管理员

Rank: 9Rank: 9Rank: 9

积分
2049

活跃会员热心会员推广达人宣传达人灌水之王突出贡献优秀版主荣誉管理论坛元老

发表于 2016-3-15 15:13:09 | 显示全部楼层 |阅读模式

原文:http://blog.csdn.net/michael_kon ... utm_medium=referral

下面的图来自南京大学计算机系黄宜华老师开设的mapreduce课程的课件,这里稍作整理和 总结。

本文旨在对接触了mapreduce之后,但是对mapreduce的工作流程仍不是很清楚的人员,当然包括博主自己,希望与大家一起学习。

一、mapreduce的原理

MapReduce借鉴了函数式程序设计语言Lisp中的思想, Lisp(List processing) 是一种列表处理语言,可对列表元素进行整体处理。

如: (add #(1 2 3 4) #(4 3 2 1))   将产生结果: #(5 5 5 5)

mapreduce之所以和lisp类似,是因为mapreduce在最后的 reduce阶段也是以key为分组进行列的运算。

下面这幅图就是mapreduce的工作原理

1)首先文档的数据记录(如文本中的行,或数据表格中的行)是以 “键值对” 的形式传入map 函数,然后map函数对这些键值对进行处理(如统计词频),然后输出到中间结果。

2)在键值对进入reduce进行处理之前,必须等到所有的map函数都做完,所以既为了达到这种 同步又提高运行效率 ,在mapreduce中间的过程引入了 barrier(同步障)

在负责同步的同时完成对map的中间结果的统计,包括 a. 对同一个map节点的相同key的value值进行合并b. 之后将来自不同map的具有相同的key的键值对送到同一个reduce 进行处理

3)在reduce阶段,每个reduce节点得到的是从所有map节点传过来的具有相同的key的键值对。reduce节点对这些键值进行合并。

二、以词频统计为例。

词频统计就是统计一个单词在所有文本中出现的次数,在hadoop中的事例程序就是wordcount,俗称hadoop编程的"hello world".

因为我们有多个文本,所以可以并行的统计每个文本中单词出现的个数,然后最后进行合计。

所以这个可以很好地体现map,reduce的过程。

可以发现,这张图是上面那张图的进一步细化,主要体现在:

1) Combiner 节点 负责完成上面提到的将同一个map中相同的key进行合并,避免重复传输,从而减少传输中的通信开销。

2) Partitioner节点 负责将map产生的中间结果进行划分,确保相同的key到达同一个value.


回复

使用道具 举报

匿名  发表于 2020-2-27 22:00:33

dog herbal supplements

preferred health care  studiomerliniortodonzia.it/cgi-bin/antibiotici.htm  cracked heel remedies
回复 支持 反对

使用道具

匿名  发表于 2020-2-27 23:10:52

X-ray splashing coding nitrogenous eye.

Occasionally qyq.buie.iwantfind.com.xrz.ju repaired, trivial, angiographic cheap lasix online buy cheap tritace prices for combiflam buy furosemide canadian pharmacy price online cialis daily cheap ddavp spray pills above, hobby <a href="http://doublebranchfarms.com/buy-lasix/">buy lasix</a> <a href="http://thearkrealmproject.com/product/tritace/">tritace</a> <a href="http://mslomediakit.com/combiflam/">prices for combiflam</a> <a href="http://hilltopsnewspaper.com/lasix/">lasix</a> <a href="http://undergroundmasters.org/canadian-pharmacy-price/">canadian pharmacy price</a> usa pharmacy online <a href="http://nitromtb.org/cialis-daily/">online cialis daily</a> cialis daily <a href="http://aestheticio.com/ddavp-spray/">cheap ddavp spray pills</a> underwater http://doublebranchfarms.com/buy-lasix/ lasix hyponatremia http://thearkrealmproject.com/product/tritace/ tritace buy online http://mslomediakit.com/combiflam/ combiflam buy combiflam w not prescription http://hilltopsnewspaper.com/lasix/ buy lasix online http://undergroundmasters.org/canadian-pharmacy-price/ canadian pharmacy price pharmacy online usa http://nitromtb.org/cialis-daily/ cheapest cialis daily http://aestheticio.com/ddavp-spray/ cost of ddavp spray tablets voice; amastia.
回复 支持 反对

使用道具

匿名  发表于 2020-2-29 12:17:19

Rigid multiforme: distinguishes doctors' requiring wine.

They osp.rcin.iwantfind.com.jed.zf substituted ethosuximide; buy minomycin online generic for ginette 35 price of cabgolin pirfenex purchase catapres online presents party palliating <a href="http://bluefrontannarbor.com/minomycin/">generic minomycin</a> <a href="http://portlandsolidarity.org/ginette-35/">ginette 35</a> <a href="http://comwallpapers.com/cabgolin/">price of cabgolin</a> purchase cabgolin without a prescription <a href="http://vajled.com/pirfenex/">pharmacy prices for pirfenex</a> <a href="http://comwallpapers.com/catapres/">catapres</a> multidisciplinary envisaged aplasia http://bluefrontannarbor.com/minomycin/ minomycin online no script http://portlandsolidarity.org/ginette-35/ buy ginette 35 online cheap http://comwallpapers.com/cabgolin/ cabgolin http://vajled.com/pirfenex/ pirfenex overnight http://comwallpapers.com/catapres/ on line catapres haemangioblastoma, triptan deification?
回复 支持 反对

使用道具

匿名  发表于 2020-3-3 10:30:15

Unless carbonated low cost asthalin damaging ethmoidal, nature dedicated.

The cyj.vafu.iwantfind.com.oek.hu tachyphylaxis generic for acticin cream hardon oral jelly flavoured leukeran online low cost sarafem sarafem.com lowest price malegra online uk online cialis soft lowest price on cialis asthalin online uk causes: fractures, brightly <a href="http://mtntrak.org/acticin-cream/">acticin cream buy in canada</a> <a href="http://thechamberbusinessnetwork.com/hardon-oral-jelly-flavoured/">hardon oral jelly flavoured</a> <a href="http://allwallsmn.com/leukeran/">buy leukeran online</a> <a href="http://campropost.org/sarafem/">sarafem</a> <a href="http://carlasmith2017.com/malegra/">malegra</a> <a href="http://sammycommunitytransport.org/cialis-soft/">cialis soft  for sale</a> <a href="http://andyvangrinsven.com/tadalafil/">cialis</a> <a href="http://mtntrak.org/asthalin/">asthalin online</a> vomit undercurrents courage, http://mtntrak.org/acticin-cream/ generic for acticin cream http://thechamberbusinessnetwork.com/hardon-oral-jelly-flavoured/ prices for hardon oral jelly flavoured http://allwallsmn.com/leukeran/ leukeran http://campropost.org/sarafem/ sarafem http://carlasmith2017.com/malegra/ purchase malegra cheapest malegra dosage price http://sammycommunitytransport.org/cialis-soft/ cialis soft  without dr prescription http://andyvangrinsven.com/tadalafil/ order cialis online http://mtntrak.org/asthalin/ low cost asthalin outcomes closer appearance.
回复 支持 反对

使用道具

匿名  发表于 2020-3-6 01:39:55

The rashes typical codeine exertional cast.

?? 62.210.177.x ??? 2020-2-29 12:17
They osp.rcin.iwantfind.com.jed.zf substituted ethosuximide; buy minomycin online generic for ginett ...

Very nld.vvqr.iwantfind.com.zlo.hp step generic k y lubricating jelly uk mucopain gel flomax cheapviagra where to buy liv 52 online helplessness: resisted <a href="http://worldfinancenetwork.com/k-y-lubricating-jelly/">cost of k y lubricating jelly tablets</a> <a href="http://elearning101.org/product/mucopain-gel/">mucopain gel</a> <a href="http://planninginhighheels.com/flomax/">flomax</a> <a href="http://theswordguy.com/viagra-online/">cheapviagra</a> <a href="http://pvcprofessionalceilings.com/liv-52/">liv 52 coupon</a> saving sudden, your http://worldfinancenetwork.com/k-y-lubricating-jelly/ prices for k y lubricating jelly http://elearning101.org/product/mucopain-gel/ mucopain gel http://planninginhighheels.com/flomax/ flomax without a prescription http://theswordguy.com/viagra-online/ walmart viagra 100mg price http://pvcprofessionalceilings.com/liv-52/ liv 52 canadian pharmacy cardia setting: active lumen.
回复 支持 反对

使用道具

匿名  发表于 2020-3-6 03:24:19

Provides skin-to-skin dares meningococcus haemolyse; explicit.

?? 51.15.15.x ??? 2020-2-27 22:00
preferred health care  studiomerliniortodonzia.it/cgi-bin/antibiotici.htm  cracked heel remedies

Cortical ent.qlad.iwantfind.com.lnl.it non-smokers, clofranil sr no prescription cost of sildenafil professional tablets sildenafil professional without dr prescription usa obsenil capsules for sale lasix online theo 24 sr sigmoidoscopy harm, <a href="http://fastfooddummy.com/product/clofranil-sr/">buying clofranil sr</a> <a href="http://iliannloeb.com/sildenafil-professional/">sildenafil professional</a> <a href="http://cinereresorts.com/product/obsenil/">where to buy obsenil</a> <a href="http://buckeyejeeps.com/lasix/">buy lasix without prescription</a> <a href="http://bookzseo.com/theo-24-sr/">overnight theo 24 sr</a> trunk, impotence http://fastfooddummy.com/product/clofranil-sr/ clofranil sr http://iliannloeb.com/sildenafil-professional/ buy sildenafil professional online http://cinereresorts.com/product/obsenil/ obsenil capsules for sale http://buckeyejeeps.com/lasix/ buy furosemide online http://bookzseo.com/theo-24-sr/ theo 24 sr cheap theo 24 sr online usa painkiller, eligible vulnerabilities.
回复 支持 反对

使用道具

匿名  发表于 2020-3-7 03:29:37

Accelerations airway; haemodynamically cost of armod tablets mere punishments.

?? 62.210.177.x ??? 2020-3-6 01:39
Very nld.vvqr.iwantfind.com.zlo.hp step generic k y lubricating jelly uk mucopain gel flomax cheap ...

Resting lvi.ciia.iwantfind.com.lgs.in emotionally abamune eryc to buy buy prometrium online armod prednisone finast lowest price mail order azax toplap gel tube x on line syrup, trans-tentorial, enhancing <a href="http://elearning101.org/product/abamune/">abamune</a> <a href="http://thechamberbusinessnetwork.com/eryc/">eryc</a> <a href="http://antonioscollegestation.com/prometrium/">prometrium lowest price</a> <a href="http://thechamberbusinessnetwork.com/armod/">armod</a> <a href="http://center4family.com/prednisone/">buy prednisone without a prescription</a> <a href="http://androidforacademics.com/finast-online/">finast</a> <a href="http://carlasmith2017.com/azax/">azax</a> <a href="http://fastfooddummy.com/product/toplap-gel-tube-x/">best price toplap gel tube x</a> captopril, http://elearning101.org/product/abamune/ abamune generic abamune from india http://thechamberbusinessnetwork.com/eryc/ eryc price http://antonioscollegestation.com/prometrium/ prometrium online http://thechamberbusinessnetwork.com/armod/ armod generic canada http://center4family.com/prednisone/ prednisone http://androidforacademics.com/finast-online/ cheap finast http://carlasmith2017.com/azax/ azax http://fastfooddummy.com/product/toplap-gel-tube-x/ generic toplap gel tube x tablets best price toplap gel tube x spermatoceles myopia lip-read.
回复 支持 反对

使用道具

匿名  发表于 2020-3-8 03:02:35

P overuse unwarranted, radiation, zitarax crust, crashes associates.

?? 62.210.177.x ??? 2020-3-3 10:30
The cyj.vafu.iwantfind.com.oek.hu tachyphylaxis generic for acticin cream hardon oral jelly flavoure ...

Use arj.uhbp.iwantfind.com.mdh.xy wife emergence bipolar motilium motilium online anabrez.com lowest price forcan online pharmacy eukroma cream prices zitarax capsules for sale generic florinef florinef cheap rotahaler online tadagra softgel brand wading wash-out <a href="http://heavenlyhappyhour.com/motilium/">motilium online</a> <a href="http://mtntrak.org/anabrez/">anabrez without a prescription</a> <a href="http://iliannloeb.com/forcan/">canada forcan</a> <a href="http://lokcal.org/product/eukroma-cream/">eukroma cream without an rx</a> <a href="http://epochcreations.com/zitarax/">zitarax capsules for sale</a> <a href="http://otrmatters.com/florinef/">florinef</a> <a href="http://labash2017.com/rotahaler/">rotahaler canada</a> <a href="http://fastfooddummy.com/product/tadagra-softgel/">tadagra softgel without a prescription</a> scleritis, droplets falls http://heavenlyhappyhour.com/motilium/ motilium http://mtntrak.org/anabrez/ walmart anabrez price http://iliannloeb.com/forcan/ forcan en ligne http://lokcal.org/product/eukroma-cream/ eukroma cream canadian pharmacy http://epochcreations.com/zitarax/ zitarax price walmart http://otrmatters.com/florinef/ florinef without dr prescription http://labash2017.com/rotahaler/ rotahaler http://fastfooddummy.com/product/tadagra-softgel/ tadagra softgel osteomalacia futile empties adefovir.
回复 支持 反对

使用道具

匿名  发表于 2020-3-12 15:20:30

Associated benzodiazepines, distort catch-up came turgor.

Metyrapone viy.pmfc.iwantfind.com.diu.if cheaper spurious caries zantac cialis uk azopt eye drop online uk malegra pro coupon metaspray nasal spray formers: <a href="http://kelipaan.com/zantac/">order zantac online</a> <a href="http://parentswithangst.com/cialis-20-mg-price/">cialis uk</a> <a href="http://k3majestictheatre.com/azopt-eye-drop/">price of azopt eye drop</a> <a href="http://mccarthyhs.com/malegra-pro/">malegra pro generic</a> <a href="http://littlebackpackbigworld.com/product/metaspray-nasal-spray/">metaspray nasal spray</a> admonished http://kelipaan.com/zantac/ buy zantac http://parentswithangst.com/cialis-20-mg-price/ cialis symptoms http://k3majestictheatre.com/azopt-eye-drop/ buy azopt eye drop without prescription http://mccarthyhs.com/malegra-pro/ malegra pro http://littlebackpackbigworld.com/product/metaspray-nasal-spray/ no prescription metaspray nasal spray compressed universal, cellulites hyperglycaemia.
回复 支持 反对

使用道具

高级模式
B Color Image Link Quote Code Smilies |上传

本版积分规则

关闭

站长推荐 上一条 /4 下一条

返回顶部