Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9879

Erasure Coding : schedule striped blocks to be cached on DataNodes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: caching, erasure-coding
    • Labels:
      None

      Description

      This jira to discuss and implement the caching of striped block objects on the appropriate datanode.

      Presently it is checking block group size and scheduling the blockGroupId to the datanode, which needs to be refined by checking the StripedBlockUtil.getInternalBlockLength() and schedule proper blockId to the datanode.

      CacheReplicationMonitor.java
      
            if (pendingCapacity < blockInfo.getNumBytes()) {
              LOG.trace("Block {}: DataNode {} is not a valid possibility " +
                  "because the block has size {}, but the DataNode only has {} " +
                  "bytes of cache remaining ({} pending bytes, {} already cached.)",
                  blockInfo.getBlockId(), datanode.getDatanodeUuid(),
                  blockInfo.getNumBytes(), pendingCapacity, pendingBytes,
                  datanode.getCacheRemaining());
              outOfCapacity++;
              continue;
            }
      
          for (DatanodeDescriptor datanode : chosen) {
            LOG.trace("Block {}: added to PENDING_CACHED on DataNode {}",
                blockInfo.getBlockId(), datanode.getDatanodeUuid());
            pendingCached.add(datanode);
            boolean added = datanode.getPendingCached().add(cachedBlock);
            assert added;
          }
      

        Attachments

          Activity

            People

            • Assignee:
              rakeshr Rakesh Radhakrishnan
              Reporter:
              rakeshr Rakesh Radhakrishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: