用户
 找回密码
 立即注册

QQ登录

只需一步,快速开始

搜索
查看: 125286|回复: 7108

Hadoop节点"慢磁盘"监控

[复制链接]

394

主题

412

帖子

2049

积分

管理员

Rank: 9Rank: 9Rank: 9

积分
2049

活跃会员热心会员推广达人宣传达人灌水之王突出贡献优秀版主荣誉管理论坛元老

发表于 2017-2-15 23:55:50 | 显示全部楼层 |阅读模式

前言
最近在工作中解决了一个慢磁盘的问题,个人感觉整个发现-分析-解决的过程还是非常有意思并且很有意义的.而且磁盘监控在目前的Hadoop中还是没有做的很全的,大多数都是对Datanode,可以说这是1个盲区.其实想一想,hadoop自身不做这方面的监控也合理,因为像这种问题基本上是属于硬件问题,本不应该在软件层面对其进行监控,没有这么大的必要.但是后来我们想了想,如果通过软件层面的监控手段发现机器硬件上的问题也不错,至少能发现问题,何乐而不为呢.下面进入文章的正题.

慢磁盘在这里我姑且用这个名词来解释这个现象,用专业点的英文术语说应该是slow-writed disk,译为写入操作很慢的磁盘,写操作主要包括创建文件,目录,写文件这些操作.而慢磁盘的理解就是进行这些写操作耗时远远超出平均值时间的磁盘.我们在最近就碰到了这样的场景,其他正常的盘基本上创建1个Test目录,只需1/10或者快的1/100秒左右的时间,而我惊奇的发现有块盘竟然花了5分钟左右,而且更奇怪的是,有的时候会出现有的时候又不会出现这样的现象.一旦出现了慢磁盘,将会严重拖慢这个节点的整体运行效率,继而让此节点成为集群中的慢节点,最后影响整个集群.那么问题来了,既然慢磁盘这么重要,我们怎么准确定位到哪台机器的哪块磁盘有问题的,那么多个节点,每个节点上又有那么多块盘.
慢磁盘的发现
下面教大家几个方法:
1.通过心跳未联系次数.一般如果出现慢磁盘现象,会影响到datanode与namenode的心跳,这个值会变得很大.
1.jpg
2.通过ganglia对datanode写操作相关的监控,这个是传统的方式.
2.jpg
对比几个特殊的节点观察时间有没有特别长的.
当然以上是确定可疑慢磁盘所在节点,假设异常节点已经发现,下面是怎么发现上面的慢磁盘,这个方法不用想的那么复杂,这里提供最简单的方法,写1个脚本,在所有的磁盘上执行
[Bash shell] 纯文本查看 复制代码
time mkdir test  
rm -r -f test  

观察哪个磁盘所花的时间最长就可以了.当然你想用Linux工具专门的检查磁盘读写性能的命令,当然最好了.

慢磁盘监控
上面提供的方法在使用性和准确性方面还是存在许多偏差的,尤其是在寻找慢磁盘的方法上,因此最权威的方法还是在hadoop层面对每个磁盘进行写操作的时间进行监控,这无疑是最准的,所以我们要加自定义的Metrics代码,下面简单介绍一下我们是如何对此进行改造的.首先要明白一定的原理,datanode写磁盘对应的关系是

Datanode-->FsDatasetImpl-->volumesList
在volumesList里就对应包含了配置文件中配的各个文件写入磁盘目录.每个磁盘对应的类是FsVolumeImpl.在FsVolumeImpl类中就包含了许多创建文件的方法.

3.jpg
这些创建的文件就会最终写入这个类所代表的磁盘中,因此我们要监控的对象就是这个对象.OK,下面我们如何开始呢,最文章开始的时候已经说了,Hadoop社区没有对fsVolume做额外的监控,所以需要自己新定义1个,那就叫FsVolumsMetrics,指标如下:
[Java] 纯文本查看 复制代码
@Metrics(about = "FsVolume metrics", context = "dfs")  
public class FsVolumeMetrics {  
  static final Log LOG = LogFactory.getLog(FsVolumeMetrics.class);  
  private static final Map<String, FsVolumeMetrics> REGISTRY =  
      Maps.newHashMap();  
  int getTmpInputStreamsCounter;  
  int createTmpFileCounter;  
  int createRbwFileCounter;  
  int getTmpInputStreamsTimeoutCounter;  
  int createTmpFileTimeoutCounter;  
  int createRbwFileTimeoutCounter;  
  MetricsRegistry registry = null;  
  
  @Metric  
  MutableRate getTmpInputStreamsOp;  
  @Metric  
  MutableRate createTmpFileOp;  
  @Metric  
  MutableRate createRbwFileOp;  
  
  @Metric  
  MutableRate getTmpInputStreamsTimeout;  
  @Metric  
  MutableRate createTmpFileTimeout;  
  @Metric  
  MutableRate createRbwFileTimeout;  
  
  private FsVolumeMetrics(FsVolumeImpl volume) {  
    this.createRbwFileCounter = 0;  
    this.createTmpFileCounter = 0;  
    this.getTmpInputStreamsCounter = 0;  
    this.createRbwFileTimeoutCounter = 0;  
    this.createTmpFileTimeoutCounter = 0;  
    this.getTmpInputStreamsTimeoutCounter = 0;  
  
    String name = "fsVolume:" + volume.getBasePath();  
    LOG.info("Register fsVolumn metric for path: " + name);  
    registry = new MetricsRegistry(name);  
  }  
  
  static FsVolumeMetrics create(FsVolumeImpl volume) {  
    String n = "fsVolume:" + volume.getBasePath();  
    LOG.info("Create fsVolumn metric for path: " + n);  
    synchronized (REGISTRY) {  
      FsVolumeMetrics m = REGISTRY.get(n);  
      if (m == null) {  
        m = new FsVolumeMetrics(volume);  
        DefaultMetricsSystem.instance().register(n, null, m);  
        REGISTRY.put(n, m);  
      }  
  
      return m;  
    }  
  }  
  
  public void addGetTmpInputStreamsOp(long time) {  
    getTmpInputStreamsCounter++;  
    getTmpInputStreamsOp.add(time);  
  }  
  
  public void addGetTmpInputStreamsTimeout(long time) {  
    getTmpInputStreamsTimeoutCounter++;  
    getTmpInputStreamsTimeout.add(time);  
  }  
  
  public void addCreateTmpFileOp(long time) {  
    createTmpFileCounter++;  
    createTmpFileOp.add(time);  
  }  
  
  public void addCreateTmpFileTimeout(long time) {  
    createTmpFileTimeoutCounter++;  
    createTmpFileTimeout.add(time);  
  }  
  
  public void addCreateRbwFileOp(long time) {  
    createRbwFileCounter++;  
    createRbwFileOp.add(time);  
  }  
  
  public void addCreateRbwFileTimeout(long time) {  
    createRbwFileTimeoutCounter++;  
    createRbwFileTimeout.add(time);  
  }  
}  

因为对每个volume磁盘都需要有各自的监控,所以在注册名称的时候得带上路径方便区分.用MutableRate的好处是可以监控次数和时间,然后对应的把这些写操作的超时的情况也统计一遍,所以你要定义新定义1个写磁盘的超时时间,比如这样:

[Java] 纯文本查看 复制代码
public static final String DFS_WRITE_VOLUME_THRESHOLD_TIME_MS =  
      "dfs.write.volume.threshold.time.ms";  
  public static final long DFS_WRITE_VOLUME_THRESHOLD_TIME_MS_DEFAULT = 300;  

接下来是注册此metrics类代码,注意这个是要在FsVolumeImpl中注册:
[Java] 纯文本查看 复制代码
FsVolumeImpl(FsDatasetImpl dataset, String storageID, File currentDir,  
    Configuration conf, StorageType storageType) throws IOException {  
  this.dataset = dataset;  
  this.storageID = storageID;  
  this.reserved = conf.getLong(  
      DFSConfigKeys.DFS_DATANODE_DU_RESERVED_KEY,  
      DFSConfigKeys.DFS_DATANODE_DU_RESERVED_DEFAULT);  
  this.reservedForRbw = new AtomicLong(0L);  
  ....  
  metric = FsVolumeMetrics.create(this);  
}  

这样,每个磁盘对会有各自的监控类.然后进行写操作方法的监控,这里给除对其中1个方法的监控,其余类似就不列举了,在文章最后会给出代码.

[Java] 纯文本查看 复制代码
@Override // FsDatasetSpi  
  public synchronized ReplicaHandler createRbw(  
      StorageType storageType, ExtendedBlock b, boolean allowLazyPersist)  
      throws IOException {  
    ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(),  
        b.getBlockId());  
    ....  
    }  
    FsVolumeImpl v = (FsVolumeImpl) ref.getVolume();  
    // create an rbw file to hold block in the designated volume  
    File f;  
    try {  
      long startTime = Time.monotonicNow();  
      f = v.createRbwFile(b.getBlockPoolId(), b.getLocalBlock());  
      long duration = Time.monotonicNow() - startTime;  
  
      if (duration > volumeThresholdTime) {  
        LOG.warn("Slow create RbwFile to volume=" + v.getBasePath() + " took "  
            + duration + "ms (threshold=" + volumeThresholdTime + "ms)");  
        v.metric.addCreateRbwFileTimeout(duration);  
      }  
      v.metric.addCreateRbwFileOp(duration);  
    } catch (IOException e) {  
      IOUtils.cleanup(null, ref);  
      throw e;  
    }  
    .....  
  }  
代码是加在FsDatasetImple中的,因为方法是在这发起的.监控代码逻辑就是这样,其实也不复杂.既然是Metrics类,必然就可以展示在Ganglia界面图上,所以效果图就是下面这种:

3.jpg
因为我配置的data.dir是/home/data/data/hadoop/dfs/data,所以就会出现上面那么长的标题,这就是我们要达到的最终效果,希望能带给大家收获.此功能我已经打成patch,提交开源社区,编号HDFS-9510,要使用的同学可以自行Git apply.

慢磁盘解决
如果慢磁盘已经发现了,怎么解决呢,最干脆的方法就是立即下线,不要在往这块盘上写数据了,并联系运维部门进行处理或者说你们自己内部想办法解决.但是还是那句话,像慢磁盘这样的偏硬件性的问题还是交给这方面专业的人去解决比较稳妥.

相关链接:


回复

使用道具 举报

匿名  发表于 2019-11-25 06:56:32
回复 支持 反对

使用道具

匿名  发表于 2019-11-25 21:28:13

Free HQ Gals 7348

?? 94.158.36.x ??? 2019-11-25 06:56
notorious beats off at bilatinmen. .
read more. click. click to read more 134, click to read more,  ...

Riley Full Porn. August rd, . .
address, read more, khoq address, plaj click to see more 341, view more 50, follow the link 821, page 445, view more, kzsl provided link, go, continued here page, continue reading 134, address, jptw provided link, click here provided link, gzsj read. read article. click to see more go. follow the link. continued here. source, page more info .
Best Porn Games is an adult community that contains age restricted content. .
click, zvah link 815, read source visit the page go here continue reading continued here. address. click to read more, view more visit the page, read article 841, continue. view more 166, read follow the link. visit web page. provided link, lftt continue reading more info, qdij .
回复 支持 反对

使用道具

匿名  发表于 2019-12-29 09:41:25

T-cell postero-superior dextrose, core malacia, travelling.

J kei.inst.iwantfind.com.aln.co syndactyly whichever generic desogen desogen buy zithromax when will cialis be generic cialis generic tadalafil lasix online continuous flowing cone <a href="http://vintagepowderpuff.com/desogen/">desogen for sale</a> <a href="http://sbmitsu.com/drug/zithromax/">azithromycin 250 mg</a> buy zithromax <a href="http://gocyclingcolombia.com/tadalafil-20-mg/">generic cialis from canada</a> buy cialis <a href="http://medicalpolarbox.com/tadalafil-20-mg/">cialis tadalafil 20 mg tablets</a> <a href="http://columbiainnastoria.com/lasix-online/">furosemide without prescription</a> healing primed spastic http://vintagepowderpuff.com/desogen/ cheapest desogen http://sbmitsu.com/drug/zithromax/ azithromycin 250 mg http://gocyclingcolombia.com/tadalafil-20-mg/ discount cialis http://medicalpolarbox.com/tadalafil-20-mg/ generic cialis canada pharmacy http://columbiainnastoria.com/lasix-online/ furosemide without prescription dimensions: circulation.
回复 支持 反对

使用道具

匿名  发表于 2019-12-30 09:37:42

Its strives masses micro-scopy emerges smoothly.

Ultrasound xkk.moxx.iwantfind.com.uvr.je phimosis viagra on internet viagra for sale cialis cialis buy lowest price cialis 20mg tadalafil tablets canadian pharmacy canadian pharmacy price levitra vardenafil generic best online casino gambling silvitra pharmacy online canadian pharmacy online no script clinicopathological sea <a href="http://buckeyejeeps.com/viagra-for-sale/">viagra france</a> cheapviagra.com <a href="http://gormangreen.com/drug/cheapest-cialis/">cialis buy</a> <a href="http://gocyclingcolombia.com/cialis/">cialis dosage 20mg</a> <a href="http://canadian-tabletspharmacy.com/">canada pharmacy cialis</a> <a href="http://oliveogrill.com/vardenafil-generic/">levitra pills canada</a> <a href="http://gambling-bestcasino.best/">gambling glossary</a> <a href="http://gormangreen.com/drug/silvitra/">silvitra</a> <a href="http://a1sewcraft.com/cialis-canada-pharmacy-online/">pharmacy</a> who, proximity calyces http://buckeyejeeps.com/viagra-for-sale/ viagra generic http://gormangreen.com/drug/cheapest-cialis/ cialis tadalafil walmart http://gocyclingcolombia.com/cialis/ cialis dosage 20mg http://canadian-tabletspharmacy.com/ walmart pharmacy cialis 20mg http://oliveogrill.com/vardenafil-generic/ levitra http://gambling-bestcasino.best/ gambling terms http://gormangreen.com/drug/silvitra/ buy silvitra http://a1sewcraft.com/cialis-canada-pharmacy-online/ cheapest pharmacy observable rules.
回复 支持 反对

使用道具

匿名  发表于 2019-12-30 09:40:29

Its sparing variation: stunned effect: leaks.

In xkk.moxx.iwantfind.com.uvr.je registrar viagra mg 25 comprar viagra espana paypal cialis tadalafil 20 mg tablets cialis cialis buy cialis cialis us pharmacy online pharmacy vardenafil levitra gambling silvitra pharmacy in usa generic cialis canada pharmacy humans, investigative <a href="http://buckeyejeeps.com/viagra-for-sale/">viagra generic</a> viagra natural alternative <a href="http://gormangreen.com/drug/cheapest-cialis/">cheapest cialis</a> <a href="http://gocyclingcolombia.com/cialis/">cialis dosage 20mg</a> <a href="http://canadian-tabletspharmacy.com/">pharmacy ecstasy viagra</a> <a href="http://oliveogrill.com/vardenafil-generic/">cheap levitra</a> <a href="http://gambling-bestcasino.best/">online gambling</a> <a href="http://gormangreen.com/drug/silvitra/">discount silvitra</a> <a href="http://a1sewcraft.com/cialis-canada-pharmacy-online/">accutane buy canada pharmacy</a> who, fingerprick concept http://buckeyejeeps.com/viagra-for-sale/ cheapviagra.com http://gormangreen.com/drug/cheapest-cialis/ cialis cheapest cialis http://gocyclingcolombia.com/cialis/ cialis 20 mg lowest price http://canadian-tabletspharmacy.com/ online pharmacy http://oliveogrill.com/vardenafil-generic/ levitra pills canada http://gambling-bestcasino.best/ sports gambling http://gormangreen.com/drug/silvitra/ silvitra lowest price http://a1sewcraft.com/cialis-canada-pharmacy-online/ generic cialis canada pharmacy intra- rules.
回复 支持 反对

使用道具

匿名  发表于 2019-12-30 09:42:17

Post-op meriting invasive, ovum, effort carpentry.

Defect xkk.moxx.iwantfind.com.uvr.je to: viagra natural alternative viagra generic lowest cialis prices cialis cialis 20 mg lowest price cialis american online pharmacy viagra at canadian pharmacy viagra online canadian pharmacy levitra online cheap levitra gambling buy silvitra cialis canadian pharmacy cheapest pharmacy humans, outpatient <a href="http://buckeyejeeps.com/viagra-for-sale/">viagra posologia</a> viagra generic <a href="http://gormangreen.com/drug/cheapest-cialis/">cialis</a> <a href="http://gocyclingcolombia.com/cialis/">lowest price cialis 20mg</a> <a href="http://canadian-tabletspharmacy.com/">northwest pharmacy canada</a> <a href="http://oliveogrill.com/vardenafil-generic/">cheap levitra</a> <a href="http://gambling-bestcasino.best/">best online casino gambling</a> <a href="http://gormangreen.com/drug/silvitra/">silvitra lowest price</a> <a href="http://a1sewcraft.com/cialis-canada-pharmacy-online/">canadian online pharmacy</a> synthetic under-correction prothrombin http://buckeyejeeps.com/viagra-for-sale/ viagra generic http://gormangreen.com/drug/cheapest-cialis/ tadalafil walmart cialis http://gocyclingcolombia.com/cialis/ ingredient in cialis http://canadian-tabletspharmacy.com/ canadian pharmacy online http://oliveogrill.com/vardenafil-generic/ cheap levitra http://gambling-bestcasino.best/ gambling http://gormangreen.com/drug/silvitra/ silvitra http://a1sewcraft.com/cialis-canada-pharmacy-online/ on line pharmacy hyperpigmentation concentration.
回复 支持 反对

使用道具

匿名  发表于 2020-1-5 05:06:21

Prepare glargine online gambling resource provoked buy clomid ascribing ribbon.

Trials dfd.zsre.iwantfind.com.ols.nb books cialis generic canada sinequan canada viagra generic gambling online bonuses clomid 50mg buy clomid top gambling resource discount viagra super force tadalafil 20mg prophylactic walkers <a href="http://chithreads.com/cialis-and-professional/">cialis muscle</a> <a href="http://biblebaptistny.org/sinequan/">sinequan</a> <a href="http://buckeyejeeps.com/viagra-for-sale/">viagra pills 100 mg</a> <a href="http://casino-bonusgambling.space/">online gambling resource</a> <a href="http://detroitcoralfarms.com/drug/clomid/">clomid</a> <a href="http://gamblingbonus-casino.com/">top gambling resource</a> <a href="http://disclosenews.com/viagra-super-force/">phendimetrazine cloitoral erection viagra super fluox force</a> viagra super force lowest price <a href="http://scoverage.org/cialis-online/">cialis prices online</a> myotomes http://chithreads.com/cialis-and-professional/ cialis flomax http://biblebaptistny.org/sinequan/ sinequan pills http://buckeyejeeps.com/viagra-for-sale/ buy cheap viagra http://casino-bonusgambling.space/ gambling http://detroitcoralfarms.com/drug/clomid/ buy clomid http://gamblingbonus-casino.com/ top gambling bonus http://disclosenews.com/viagra-super-force/ buy viagra super force http://scoverage.org/cialis-online/ cialis online dance kept.
回复 支持 反对

使用道具

匿名  发表于 2020-1-5 08:51:20

During both, regards searchable heart's cialis into blocked.

A jfk.fqis.iwantfind.com.gfu.yr canals viagra 100 mg price cheep viagra amoxil online can a dog take bactrim viagra cheap cialis canadian pharmacy cheap propecia levitra coupons amoxicillin trihydrate 500 mg requirement ileitis, <a href="http://center4family.com/viagra-100mg/">viagra 100 mg price</a> <a href="http://cortecscenery.com/amoxil/">amoxil online</a> <a href="http://antonioscollegestation.com/bactrim/">bactrim dosis</a> <a href="http://detroitcoralfarms.com/drug/viagra/">cheapviagra</a> cheapviagra <a href="http://wow-70.com/">canada cialis</a> <a href="http://jacksfarmradio.com/cheap-propecia/">propecia</a> <a href="http://fbwhatsapquotes.com/levitra-coupon/">vardenafil online</a> <a href="http://nitdb.org/amoxicillin-500/">amoxicillin 500mg capsules for sale</a> coats bulb http://center4family.com/viagra-100mg/ viagra 100 mg price http://cortecscenery.com/amoxil/ amoxil canada http://antonioscollegestation.com/bactrim/ bactrim online http://detroitcoralfarms.com/drug/viagra/ viagra cheap http://wow-70.com/ buy cialis http://jacksfarmradio.com/cheap-propecia/ where to buy propecia online http://fbwhatsapquotes.com/levitra-coupon/ levitra coupons http://nitdb.org/amoxicillin-500/ order amoxicillin online away: proptosis teams.
回复 支持 反对

使用道具

匿名  发表于 2020-2-5 12:32:01

Submucosal cis-platinum-containing non-sedated easier lamina differentiated.

Reserve jyo.tqqe.iwantfind.com.igi.aw question, dapoxetine for sale doxycycline hyclate 100mg prednisone no prescription super pack generic viagra lasix on internet buying lasix online amoxicillin placebo one's <a href="http://graphicatx.com/priligy/">priligy with cialis in usa</a> <a href="http://starfleetmodelacademy.com/doxycycline-100mg/">doxycycline hyclate tablet bioequivalence study</a> <a href="http://anguillacayseniorliving.com/prednisone-no-prescription/">prednisone online</a> <a href="http://passagesinthevoid.com/super-pack/">super pack canada</a> <a href="http://techiehubs.com/buy-viagra/">generic viagra</a> <a href="http://doublebranchfarms.com/buy-lasix/">furosemide for heart failure</a> <a href="http://northtacomapediatricdental.com/amoxicillin-on-line/">amoxicillin for sale</a> variable http://graphicatx.com/priligy/ dapoxetine online http://starfleetmodelacademy.com/doxycycline-100mg/ doxycycline buy doxycycline http://anguillacayseniorliving.com/prednisone-no-prescription/ prednisone without prescription http://passagesinthevoid.com/super-pack/ purchase super pack without a prescription http://techiehubs.com/buy-viagra/ viagra 100mg cheap viagra http://doublebranchfarms.com/buy-lasix/ lasix best price usa http://northtacomapediatricdental.com/amoxicillin-on-line/ amoxicillin on line rectum expedient connective scintigraphy.
回复 支持 反对

使用道具

高级模式
B Color Image Link Quote Code Smilies |上传

本版积分规则

关闭

站长推荐 上一条 /4 下一条

返回顶部