[HDFS-14547] DirectoryWithQuotaFeature.quota costs additional memory even the storage type quota is not set.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 2.9.3, 3.1.3
• Assignee: Jinglun
• Reporter: Jinglun
• Created: 05/Jun/19 04:05
• Updated: 22/Sep/21 06:02
• Resolved: 16/Jul/19 16:16

Description

Our XiaoMi HDFS is considering upgrading from 2.6 to 3.1. We notice the storage type quota 'tsCounts' is instantiated to EnumCounters<StorageType>(StorageType.class), so it will cost a long[5] even if we don't have any storage type quota on this inode(only space quota or name quota).

In our cluster we have many dirs with quota and the NameNode's memory is in tension, so the additional cost will be a problem.
See DirectoryWithQuotaFeature.Builder().

 

class DirectoryWithQuotaFeature$Builder {
  public Builder() {
   this.quota = new QuotaCounts.Builder().nameSpace(DEFAULT_NAMESPACE_QUOTA).
   storageSpace(DEFAULT_STORAGE_SPACE_QUOTA).
   typeSpaces(DEFAULT_STORAGE_SPACE_QUOTA).build();// set default value -1.
   this.usage = new QuotaCounts.Builder().nameSpace(1).build();
  }
  public Builder typeSpaces(long val) {// set default value.
   this.tsCounts.reset(val);
   return this;
  }
}
class QuotaCounts$Builder {
  public Builder() {
    this.nsSsCounts = new EnumCounters<Quota>(Quota.class);
    this.tsCounts = new EnumCounters<StorageType>(StorageType.class);
  }
}
class EnumCounters {
  public EnumCounters(final Class<E> enumClass) {
    final E[] enumConstants = enumClass.getEnumConstants();
    Preconditions.checkNotNull(enumConstants);
    this.enumClass = enumClass;
    this.counters = new long[enumConstants.length];// new a long array here.
  }
}

Related to HDFS-14542.

 

Issue Links

• causes: HDFS-16233 Do not use exception handler to implement copy-on-write for EnumCounters
• is a child of: HDFS-14566 Quota improvement in HDFS