[AMBARI-20670] Node manager start extremely slow when YARN NM local dirs are very large

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.5.1
• Fix Version/s: trunk, 2.5.1
• Component/s: ambari-server
• Assignee: Dmytro Grinenko
• Reporter: Dmytro Grinenko
• Created: 04/Apr/17 14:24
• Updated: 16/Feb/18 13:14
• Resolved: 06/Apr/17 12:23

Description

On the cluster with the YARN NM, where local dirs are 100 GB+ with lot of small files - NM starts slow with timeouts

Reason could be in this specific call in yarn.py

def create_local_dir(dir_name):
  import params
  Directory(dir_name,
            create_parents = True,
            cd_access="a",
            mode=0755,
            owner=params.yarn_user,
            group=params.user_group,
            ignore_failures=True,
            recursive_mode_flags = {'f': 'a+rw', 'd': 'a+rwx'},
  )

was taking ~15 minutes per mount.

Issue Links

• links to: Review Board