Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10448

CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: caching
    • Labels:
      None
    • Target Version/s:

      Description

      The logic in CacheManager#checkLimit is not correct. In this method, it does with these three logic:

      First, it will compute needed bytes for the specific path.

      CacheDirectiveStats stats = computeNeeded(path, replication);
      

      But the param replication is not used here. And the bytesNeeded is just one replication's vaue.

      return new CacheDirectiveStats.Builder()
              .setBytesNeeded(requestedBytes)
              .setFilesCached(requestedFiles)
              .build();
      

      Second, then it should be multiply by the replication to compare the limit size because the method computeNeeded was not used replication.

      pool.getBytesNeeded() + (stats.getBytesNeeded() * replication) > pool.getLimit()
      

      Third, if we find the size was more than the limit value and then print warning info. It divided by replication here, while the stats.getBytesNeeded() was just one replication value.

            throw new InvalidRequestException("Caching path " + path + " of size "
                + stats.getBytesNeeded() / replication + " bytes at replication "
                + replication + " would exceed pool " + pool.getPoolName()
                + "'s remaining capacity of "
                + (pool.getLimit() - pool.getBytesNeeded()) + " bytes.");
      

        Attachments

          Activity

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              linyiqun Yiqun Lin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: