[HDFS-10448] CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: caching
• Target Version/s: 2.8.0
• Assignee: Yiqun Lin
• Reporter: Yiqun Lin
• Created: 23/May/16 02:54
• Updated: 30/Aug/16 01:15
• Resolved: 21/Jun/16 01:32

Description

The logic in CacheManager#checkLimit is not correct. In this method, it does with these three logic:

First, it will compute needed bytes for the specific path.

CacheDirectiveStats stats = computeNeeded(path, replication);

But the param replication is not used here. And the bytesNeeded is just one replication's vaue.

return new CacheDirectiveStats.Builder()
        .setBytesNeeded(requestedBytes)
        .setFilesCached(requestedFiles)
        .build();

Second, then it should be multiply by the replication to compare the limit size because the method computeNeeded was not used replication.

pool.getBytesNeeded() + (stats.getBytesNeeded() * replication) > pool.getLimit()

Third, if we find the size was more than the limit value and then print warning info. It divided by replication here, while the stats.getBytesNeeded() was just one replication value.

      throw new InvalidRequestException("Caching path " + path + " of size "
          + stats.getBytesNeeded() / replication + " bytes at replication "
          + replication + " would exceed pool " + pool.getPoolName()
          + "'s remaining capacity of "
          + (pool.getLimit() - pool.getBytesNeeded()) + " bytes.");