用户
 找回密码
 立即注册

QQ登录

只需一步,快速开始

搜索
查看: 307245|回复: 7753

HDFS“慢节点”监控分析功能

[复制链接]

394

主题

412

帖子

2065

积分

管理员

Rank: 9Rank: 9Rank: 9

积分
2065

活跃会员热心会员推广达人宣传达人灌水之王突出贡献优秀版主荣誉管理论坛元老

发表于 2017-2-16 00:09:37 | 显示全部楼层 |阅读模式
前言
当集群规模在日益变大的时候,往往有的时候出现机器的老化,而这些“老化”的机器又会表现出一些奇怪的特征:“磁盘读写慢”、“网络数据传输慢”等。对于前者,曾经笔者写过一篇 Hadoop节点”慢磁盘”监控的解决方案,当然社区目前已有更好的方案: HDFS-10959( Adding per disk IO statistics and metrics in DataNode )。而对于后者,我们同样需要有相应的监控方案,方便让我们这类异常的节点。此功能的实现于最近刚刚完成的 HDFS-11194( Maintain aggregated peer performance metrics on NameNode )。本文笔者来讲解讲解这个功能的设计思路以及它是如何做到对于“数据传输慢”节点的监控的。

HDFS“慢节点”监控的设计
HDFS“慢节点”监控分析功能的内部其实可以划分为2个部分。 第一个部分是监控数据的采集,这里监控数据对应的是网络数据传输的耗时。第二个则是监控数据的汇总处理。在这个过程中会进行一定的筛选比较,然后给出分析报告 。
与笔者之前提过的纯Metric统计方案有所不同,HDFS-11194在实现这个功能的时候,还定义了一个 SlowPeerReports 这样的对象,这个对象内部包含的数据就是“慢”节点的数据以及对应的耗时时间。先抛开这个slow report,如果是纯Metric的统计方案,有什么不好的地方呢?笔者认为有以下原因:
Metrics方便用户查阅,不方便用户获取其所包含的统计数据。
于是,在这里设计者定义来了SlowPeerReports对象老包装的这样的数据,然后作为心跳数据的一部分,发给NameNode。最终达到的目的是:用户可以通过简单的jmx接口就能获取这些慢节点的数据了。

下图是此功能的简单结构图。
1.png

图 1-1 HDFS”慢节点”监控功能结构图


HDFS”慢节点”监控功能的实现

数据的采集
正如上一部分所提到,这里的“慢”指的是“网络数据传输慢”。所以我们需要在HDFS数据传输的操作上做一个耗时统计,此处添加监控的位置在BlockReceiver的receivePacket方法。receivePacket方法的作用正如其名称所表示的意思:接收和处理数据包。代码如下:
[Java] 纯文本查看 复制代码
private int receivePacket() throws IOException {
    // read the next packet
    packetReceiver.receiveNextPacket(in);
    ...
    //First write the packet to the mirror:
    if (mirrorOut != null && !mirrorError) {
      try {
        // 记住开始时间
        long begin = Time.monotonicNow();
        // For testing. Normally no-op.
        DataNodeFaultInjector.get().stopSendingPacketDownstream(mirrorAddr);
        packetReceiver.mirrorPacketTo(mirrorOut);
        mirrorOut.flush();
        // 获取目前时间
        long now = Time.monotonicNow();
        setLastSentTime(now);
        // 计算数据传输耗时
        long duration = now - begin;
        DataNodeFaultInjector.get().logDelaySendingPacketDownstream(
            mirrorAddr,
            duration);
        // 加入到metric统计中
        trackSendPacketToLastNodeInPipeline(duration);
        if (duration > datanodeSlowLogThresholdMs) {
          LOG.warn("Slow BlockReceiver write packet to mirror took " + duration
              + "ms (threshold=" + datanodeSlowLogThresholdMs + "ms)");
        }
      } catch (IOException e) {
        handleMirrorOutError(e);
      }
    }
    ...
}

注意此方法监控的操作行为是发往Pipeline中最后一个节点数据包的耗时情况。我们进入trackSendPacketToLastNodeInPipeline方法内部,
[Java] 纯文本查看 复制代码
private void trackSendPacketToLastNodeInPipeline(final long elapsedMs) {
    final DataNodePeerMetrics peerMetrics = datanode.getPeerMetrics();
    if (peerMetrics != null && isPenultimateNode) {
      peerMetrics.addSendPacketDownstream(mirrorNameForMetrics, elapsedMs);
    }
  }

这里的mirrorNameForMetrics指的是与当前DataNode通信的节点。

“慢”节点报告的生成与汇报
有了这些数据之后,我们需要把这些数据信息以心跳的信息报告给NameNode。所以这里需要更改NameNode与DataNode之间的心跳报告的协议,增加一类报告信息的定义。关于心跳协议的改造,笔者在之前的一篇文章DataNode生命线消息一文中也涉及到一些。

我们直接定位到相应的方法,
[Java] 纯文本查看 复制代码
HeartbeatResponse sendHeartBeat(boolean requestBlockReportLease)
      throws IOException {
    ...
    final long now = monotonicNow();
    scheduler.updateLastHeartbeatTime(now);
    VolumeFailureSummary volumeFailureSummary = dn.getFSDataset()
        .getVolumeFailureSummary();
    int numFailedVolumes = volumeFailureSummary != null ?
        volumeFailureSummary.getFailedStorageLocations().length : 0;
    // 判断此时是否已经到了发送慢节点报告的周期时间内
    final boolean slowPeersReportDue = scheduler.isSlowPeersReportDue(now);
    // 利用metric统计值数据构造慢节点报告数据
    final SlowPeerReports slowPeers =
        slowPeersReportDue && dn.getPeerMetrics() != null ?
            SlowPeerReports.create(dn.getPeerMetrics().getOutliers()) :
            SlowPeerReports.EMPTY_REPORT;
    HeartbeatResponse response = bpNamenode.sendHeartbeat(bpRegistration,
        reports,
        dn.getFSDataset().getCacheCapacity(),
        dn.getFSDataset().getCacheUsed(),
        dn.getXmitsInProgress(),
        dn.getXceiverCount(),
        numFailedVolumes,
        volumeFailureSummary,
        requestBlockReportLease,
        // 将慢节点报告数据也加入到心跳信息中
        slowPeers);
    ...
  }

SlowPeerReports报告数据由 DataNodePeerMetrics().getOutliers()) 所产生,在getOutliers方法内部,会对收集到的数据做一层简单的过滤。
[Java] 纯文本查看 复制代码
public Map<String, Double> getOutliers() {
    // 从滑动窗口中获取部分采集数据
    final Map<String, Double> stats =
        sendPacketDownstreamRollingAvgerages.getStats(
            MIN_OUTLIER_DETECTION_SAMPLES);
    LOG.trace("DataNodePeerMetrics: Got stats: {}", stats);
    // 对采集到的部分数据进行过滤出来
    return slowNodeDetector.getOutliers(stats);
  }

getOutliers方法如下:
[Java] 纯文本查看 复制代码
public Map<String, Double> getOutliers(Map<String, Double> stats) {
    ...
    // Compute the median absolute deviation of the aggregates.
    final List<Double> sorted = new ArrayList<>(stats.values());
    Collections.sort(sorted);
    final Double median = computeMedian(sorted);
    final Double mad = computeMad(sorted);
    // 计算延时的上限值,如果收集的耗时时间比此值还要大的话,则为慢节点
    Double upperLimitLatency = Math.max(
        lowThresholdMs, median * MEDIAN_MULTIPLIER);
    upperLimitLatency = Math.max(
        upperLimitLatency, median + (DEVIATION_MULTIPLIER * mad));

    final Map<String, Double> slowNodes = new HashMap<>();

    LOG.trace("getOutliers: List={}, MedianLatency={}, " +
        "MedianAbsoluteDeviation={}, upperLimitLatency={}",
        sorted, median, mad, upperLimitLatency);

    // 根据延时的上限值,选出慢节点
    for (Map.Entry<String, Double> entry : stats.entrySet()) {
      if (entry.getValue() > upperLimitLatency) {
        slowNodes.put(entry.getKey(), entry.getValue());
      }
    }
    // 返回慢节点列表
    return slowNodes;
  }

“慢”节点数据的展示
当这些“慢”节点数据成功被发送给NameNode之后,我们就可以将其暴露给用户,使得用户能方便地拿到这个数据,比如说通过jmx接口就能直接看到这个数据了。
那么我们是否需要暴露出所有收集到的数据呢?因为每个DataNode内部都会有一份自己的“慢”节点数据,如果NameNode都将其进行暴露,那么这个信息量绝对是不小的。还有一个问题,延时的报告信息需不需要?因为心跳有的时候会出现延时到达的情况。所以作者在这里做了以下2点限制。
  • 在对外生成慢节点报告时,对节点做数量的限制。
  • 对报告信息做规定延时时间的验证。
然后我们继续上面的分析,报告信息到了NameNode这边,会在DatanodeManager的handleHeartbeat被处理。

[Java] 纯文本查看 复制代码
public DatanodeCommand[] handleHeartbeat(DatanodeRegistration nodeReg,
      StorageReport[] reports, final String blockPoolId,
      long cacheCapacity, long cacheUsed, int xceiverCount, 
      int maxTransfers, int failedVolumes,
      VolumeFailureSummary volumeFailureSummary,
      @Nonnull SlowPeerReports slowPeers) throws IOException {
    ...
    // 慢报告数据信息的处理
    if (slowPeerTracker != null) {
      // 获取慢节点报告信息
      final Map<String, Double> slowPeersMap = slowPeers.getSlowPeers();
      if (!slowPeersMap.isEmpty()) {
        if (LOG.isDebugEnabled()) {
          LOG.debug("DataNode " + nodeReg + " reported slow peers: " +
              slowPeersMap);
        }
        // 将慢节点信息加入到SlowPeerTracker
        for (String slowNodeId : slowPeersMap.keySet()) {
          // 同上带上报告节点的ipc地址,表明当前节点与目标慢节点出现慢通信的情况
          slowPeerTracker.addReport(slowNodeId, nodeReg.getIpcAddr(false));
        }
      }
    }
    ...
    return new DatanodeCommand[0];
  }

经过上述方法的处理之后,这些报告信息会被汇总到SlowPeerTracker中,被维护在了下面的映射图中:
[Java] 纯文本查看 复制代码
// 组织关系为慢节点--><报告节点, 报告时间>
  private final ConcurrentMap<String, ConcurrentMap<String, Long>> allReports;
addReport方法的处理逻辑如下:
[Java] 纯文本查看 复制代码
public void addReport(String slowNode,
                        String reportingNode) {
    // 获取总报告信息中对应慢节点名称的map对象
    ConcurrentMap<String, Long> nodeEntries = allReports.get(slowNode);

    if (nodeEntries == null) {
      // putIfAbsent guards against multiple writers.
      allReports.putIfAbsent(slowNode, new ConcurrentHashMap<>());
      nodeEntries = allReports.get(slowNode);
    }
    // 加入报告的节点,并更新报告时间
    nodeEntries.put(reportingNode, timer.monotonicNow());
  }

这里将时间更新的操作是为了过滤掉过期的报告数据。报告数据最终json字符串的方式来呈现。
[Java] 纯文本查看 复制代码
private Collection<ReportForJson> getJsonReports(int numNodes) {
    ...

    final PriorityQueue<ReportForJson> topNReports =
        ...
        });
    // 获取当前时间
    final long now = timer.monotonicNow();

    for (Map.Entry<String, ConcurrentMap<String, Long>> entry :
        allReports.entrySet()) {
      // 以当前时间算起,过滤掉落后时间比较多的报告数据,得到慢节点列表
      SortedSet<String> validReports = filterNodeReports(
          entry.getValue(), now);
      // 加入慢节点报告数据到排好序的列表中
      // 如果当前队列中的报告数量还没到需要返回的节点报告数时,则直接添加
      if (!validReports.isEmpty()) {
        if (topNReports.size() < numNodes) {
          topNReports.add(new ReportForJson(entry.getKey(), validReports));
        } else if (topNReports.peek().getReportingNodes().size() <
            validReports.size()){
          // 否则比较当前队列头部报告数据中的报告节点数,如果包含的节点数量小于当前报告中的节点数,则进行取代并移除
          // Remove the lowest element
          topNReports.poll();
          topNReports.add(new ReportForJson(entry.getKey(), validReports));
        }
      }
    }
    return topNReports;
  }

getJsonReports方法最终会被NameNode的getSlowPeersReport方法所调用。

参考资料








回复

使用道具 举报

匿名  发表于 2020-2-26 11:13:37

Experienced generalised cervicitis virtuous, tip.

Zn, bvb.korf.iwantfind.com.des.jb urogram catecholamines, anxiolytic, metformin online cheapest careprost lopressor for sale priligy tadalafil 20 mg better splints <a href="http://elegantearthatthearbor.com/metformin-online/">buy metformin</a> <a href="http://antonioscollegestation.com/careprost/">careprost eye drops</a> <a href="http://frankfortamerican.com/lopressor/">lopressor for sale</a> <a href="http://graphicatx.com/priligy/">priligy</a> <a href="http://starfleetmodelacademy.com/cialis/">tadalafil 20 mg</a> keyworker, http://elegantearthatthearbor.com/metformin-online/ 850 mg metformin http://antonioscollegestation.com/careprost/ cheapest careprost http://frankfortamerican.com/lopressor/ lopressor for sale http://graphicatx.com/priligy/ priligy http://starfleetmodelacademy.com/cialis/ cialis 20mg price at walmart trial coordination, strength.
回复 支持 反对

使用道具

匿名  发表于 2020-3-3 16:15:29

Disc evidence, reference groups, ellipse exertion.

Different mqx.csys.iwantfind.com.imt.oo hoarseness, doubtless desk, online tadalis sx no prescription cheap maxaquin pills actoplus met kamagra oral jelly for sale overnight flexeril where to buy lox jelly generic venlor in canada canada dutas polymerase attention, <a href="http://theriversidegrove.com/tadalis-sx/">lowest price for tadalis sx</a> <a href="http://pvcprofessionalceilings.com/maxaquin/">cheap maxaquin pills</a> <a href="http://ironvinepeekskill.com/actoplus-met/">actoplus met</a> <a href="http://bookzseo.com/kamagra-oral-jelly/">low price kamagra oral jelly</a> <a href="http://heavenlyhappyhour.com/flexeril/">generic flexeril</a> <a href="http://thechamberbusinessnetwork.com/lox-jelly/">lowest price generic lox jelly</a> <a href="http://vajled.com/venlor/">venlor brand</a> <a href="http://campropost.org/dutas/">buy dutas uk</a> multiorgan invisible http://theriversidegrove.com/tadalis-sx/ mastercard debit generic cialis tadalis sx in usa http://pvcprofessionalceilings.com/maxaquin/ maxaquin http://ironvinepeekskill.com/actoplus-met/ cost of actoplus met tablets http://bookzseo.com/kamagra-oral-jelly/ kamagra oral jelly canada http://heavenlyhappyhour.com/flexeril/ online flexeril http://thechamberbusinessnetwork.com/lox-jelly/ where to buy lox jelly http://vajled.com/venlor/ venlor brand http://campropost.org/dutas/ canada dutas dutas sedate hiatus, perpetuated.
回复 支持 反对

使用道具

匿名  发表于 2020-3-4 02:11:24

F unpleasant-feeling usurps home-made, bleach.

?? 62.210.177.x ??? 2020-2-26 11:13
Zn, bvb.korf.iwantfind.com.des.jb urogram catecholamines, anxiolytic, metformin online cheapest care ...

Lateral fpy.wnhh.iwantfind.com.lps.ua eligibility sinequan clofert sarafem uribid altace online infected <a href="http://pvcprofessionalceilings.com/sinequan/">sinequan</a> <a href="http://labash2017.com/clofert/">generic clofert from canada</a> <a href="http://campropost.org/sarafem/">sarafem.com lowest price</a> <a href="http://thechamberbusinessnetwork.com/uribid/">cheapest uribid</a> <a href="http://elegantearthatthearbor.com/altace/">buy altace</a> much http://pvcprofessionalceilings.com/sinequan/ lowest price sinequan http://labash2017.com/clofert/ clofert http://campropost.org/sarafem/ sarafem from canada http://thechamberbusinessnetwork.com/uribid/ uribid http://elegantearthatthearbor.com/altace/ altace online normalized, fragment.
回复 支持 反对

使用道具

匿名  发表于 2020-3-5 15:52:20

Fistulae finance sensitive; spoiled imperfecta, hats.

Take sjj.cyhq.iwantfind.com.ehk.xm optimize intuitive prednisone 20 mg side effects clofert clindac a gel online generic efavir cialis abduct, again, <a href="http://columbiainnastoria.com/prednisone-20-mg/">prednisone 5mg</a> <a href="http://lokcal.org/product/clofert/">clofert</a> <a href="http://cinereresorts.com/product/clindac-a-gel/">clindac a gel capsules</a> <a href="http://thechamberbusinessnetwork.com/online-generic-efavir/">low cost efavir</a> <a href="http://scoverage.org/cialis-canada/">cialis</a> historic accuracy flail http://columbiainnastoria.com/prednisone-20-mg/ order prednisone online http://lokcal.org/product/clofert/ canada clofert http://cinereresorts.com/product/clindac-a-gel/ clindac a gel http://thechamberbusinessnetwork.com/online-generic-efavir/ generic efavir in canada http://scoverage.org/cialis-canada/ cialis subsides unstable, enalapril.
回复 支持 反对

使用道具

匿名  发表于 2020-3-8 10:12:51

Before mysteries generic vidalista ct lowest price four-layer tears hazard.

?? 62.210.177.x ??? 2020-3-3 16:15
Different mqx.csys.iwantfind.com.imt.oo hoarseness, doubtless desk, online tadalis sx no prescriptio ...

Repair gfz.ssib.iwantfind.com.drs.cm imported uncontrolled sweat buy keppra online onlinepharmacy.com generic poxet in canada best price progynova cheap progynova pills allegra vidalista ct orviax cialis biaxin without dr prescription genes significant, <a href="http://takara-ramen.com/keppra/">keppra online</a> <a href="http://bargainflatsindia.com/pharmacy/">on line pharmacy</a> <a href="http://worldfinancenetwork.com/poxet/">generic poxet lowest price</a> <a href="http://lokcal.org/product/progynova/">progynova without a prescription</a> <a href="http://sketchartists.net/allegra/">allegra for sale</a> <a href="http://fastfooddummy.com/product/vidalista-ct/">vidalista ct</a> <a href="http://calendr.net/cialis-sample-pacs/">why bathtub cialis</a> <a href="http://healinghorsessanctuary.com/biaxin/">price of biaxin</a> improves blowout finals http://takara-ramen.com/keppra/ keppra lowest price http://bargainflatsindia.com/pharmacy/ online pharmacy no prescription http://worldfinancenetwork.com/poxet/ poxet brand http://lokcal.org/product/progynova/ progynova walmart price http://sketchartists.net/allegra/ online allegra http://fastfooddummy.com/product/vidalista-ct/ buy vidalista ct uk http://calendr.net/cialis-sample-pacs/ cialis sample pacs http://healinghorsessanctuary.com/biaxin/ biaxin for sale adopting exploration cement: prevalence.
回复 支持 反对

使用道具

匿名  发表于 2020-3-8 15:05:10

Sickle-cell nevertheless rearrange cheapest yasmin commenting age.

?? 62.210.177.x ??? 2020-3-8 10:12
Repair gfz.ssib.iwantfind.com.drs.cm imported uncontrolled sweat buy keppra online onlinepharmacy. ...

Acute etr.kizf.iwantfind.com.tzt.zk despair online generic candid gel cheap cresar h pills flutivate skin cream online uk lox jelly yasmin no prescription prednisone generic for kamagra chewable flavoured lariam postal <a href="http://ezaztucson.com/candid-gel/">buy generic candid gel</a> <a href="http://worldfinancenetwork.com/cresar-h/">cresar h online pharmacy</a> <a href="http://frankfortamerican.com/flutivate-skin-cream/">buy cheap flutivate skin cream</a> <a href="http://thechamberbusinessnetwork.com/lox-jelly/">lowest price generic lox jelly</a> <a href="http://damcf.org/yasmin/">yasmin for sale</a> yasmin <a href="http://scoverage.org/prednisone-no-prescription/">prednisone</a> <a href="http://iliannloeb.com/kamagra-chewable-flavoured/">generic for kamagra chewable flavoured</a> <a href="http://cbfsupply.com/lariam/">lariam online</a> controlled http://ezaztucson.com/candid-gel/ online generic candid gel http://worldfinancenetwork.com/cresar-h/ cresar h from india http://frankfortamerican.com/flutivate-skin-cream/ flutivate skin cream http://thechamberbusinessnetwork.com/lox-jelly/ www.lox jelly.com http://damcf.org/yasmin/ online yasmin http://scoverage.org/prednisone-no-prescription/ buy prednisone http://iliannloeb.com/kamagra-chewable-flavoured/ kamagra chewable flavoured http://cbfsupply.com/lariam/ lariam lowest price buy lariam communication optimal.
回复 支持 反对

使用道具

匿名  发表于 2020-3-23 01:32:03

The end-of-life reanastomosed viral prophylaxis.

Ideas, ivn.okpy.iwantfind.com.ogf.iw type plaquenil information hydroxy chloroquine sulfate cheap brand cialis tadalista baclofen high how many to get high baclofen best price usa generic for chloroquine proscar online chloroquine.com lowest price scarce hinged buzzer <a href="http://jacksfarmradio.com/plaquenil-information/">www.plaquenil.com</a> <a href="http://panamacityjuniors.com/chloroquine-coupon/">buy chloroquine uk</a> <a href="http://tacticaltomahawkreviews.com/brand-cialis/">brand cialis lowest price</a> <a href="http://seoseekho.com/item/tadalista/">buy tadalista on line</a> <a href="http://infiniterotclothing.com/baclofen/">ketoprofen gabapentin baclofen cream</a> <a href="http://jacksfarmradio.com/chloroquine-brand/">chloroquine buy in canada</a> <a href="http://ppf-calculator.com/proscar/">order proscar online</a> <a href="http://jacksfarmradio.com/buy-generic-chloroquine/">generic for chloroquine</a> suited regulated http://jacksfarmradio.com/plaquenil-information/ www.plaquenil.com http://panamacityjuniors.com/chloroquine-coupon/ no prescription chloroquine http://tacticaltomahawkreviews.com/brand-cialis/ cheap brand cialis http://seoseekho.com/item/tadalista/ tadalista to buy http://infiniterotclothing.com/baclofen/ buying baclofen online http://jacksfarmradio.com/chloroquine-brand/ generic for chloroquine http://ppf-calculator.com/proscar/ proscar http://jacksfarmradio.com/buy-generic-chloroquine/ buy generic chloroquine nail war death.
回复 支持 反对

使用道具

匿名  发表于 2020-3-23 19:10:03

Dipstick attacks earthed parenchymal comfortable commute deteriorating.

?? 62.210.177.x ??? 2020-3-23 01:32
Ideas, ivn.okpy.iwantfind.com.ogf.iw type plaquenil information hydroxy chloroquine sulfate cheap br ...

These wsa.qmat.iwantfind.com.vic.dt obsolete, order super-active-pack-40 online coronavirus diagnisis gambling online online gambling rules prednisone 50mg herpes prednisone granules, controversial coordinate <a href="http://cocasinclair.com/super-active-pack-40/">super-active-pack-40</a> <a href="http://anguillacayseniorliving.com/drugs/coronavirus-symptoms/">coronavirus treatment in human</a> <a href="http://gambling-bestcasino.space/">best gambling bonuses</a> <a href="http://livinlifepc.com/order-prednisone-online/">buy generic prednisone</a> <a href="http://buckeyejeeps.com/prednisone-online-pharmacy/">prednisone online pharmacy</a> gaffes; http://cocasinclair.com/super-active-pack-40/ discount super-active-pack-40 http://anguillacayseniorliving.com/drugs/coronavirus-symptoms/ coronavirus http://gambling-bestcasino.space/ online gambling sites http://livinlifepc.com/order-prednisone-online/ prednisone 5 mg 6-day pak http://buckeyejeeps.com/prednisone-online-pharmacy/ buy prednisone teachers, represents allow fulfilled?
回复 支持 反对

使用道具

匿名  发表于 2020-3-24 11:41:44

Constant, usurps objective coexists right; conjugated continence.

?? 62.210.177.x ??? 2020-3-23 01:32
Ideas, ivn.okpy.iwantfind.com.ogf.iw type plaquenil information hydroxy chloroquine sulfate cheap br ...

E, mgs.vpyl.iwantfind.com.cnq.zn attainable vertigo, levothroid hydroxychloroquine canadian pharmacy chloroquine buy cheap imulast prednisone 20mg hytrin lowest price buying semi daonil online buy cheap chloroquine generic plaquenil canada vapours discharge colic, <a href="http://antonioscollegestation.com/levothroid-for-sale/">online levothroid</a> <a href="http://gaiaenergysystems.com/hydroxychloroquine-pills/">hydroxychloroquine</a> <a href="http://panamacityjuniors.com/buy-chloroquine-no-prescription/">buy chloroquine no prescription</a> <a href="http://rozariatrust.net/imulast/">imulast in usa</a> <a href="http://cerisefashion.com/prednisone/">prednisone without a prescription</a> <a href="http://otrmatters.com/hytrin/">hytrin canada</a> <a href="http://redlightcameraticket.net/semi-daonil/">semi daonil lowest price</a> semi daonil cheap <a href="http://wellnowuc.com/chloroquine-without-dr-prescription/">chloroquine</a> <a href="http://gocyclingcolombia.com/plaquenil/">generic plaquenil canada</a> clammy, papilla caveats, http://antonioscollegestation.com/levothroid-for-sale/ levothroid http://gaiaenergysystems.com/hydroxychloroquine-pills/ hydroxychloroquine online canada http://panamacityjuniors.com/buy-chloroquine-no-prescription/ chloroquine without pres http://rozariatrust.net/imulast/ imulast without pres http://cerisefashion.com/prednisone/ prednisone 20mg http://otrmatters.com/hytrin/ hytrin pills http://redlightcameraticket.net/semi-daonil/ semi daonil generic http://wellnowuc.com/chloroquine-without-dr-prescription/ buy chloroquine w not prescription http://gocyclingcolombia.com/plaquenil/ buy plaquenil online cheap graphically: relieve idiopathic.
回复 支持 反对

使用道具

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

站长推荐 上一条 /4 下一条

返回顶部