Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15571

Multiple FileContexts created with the same configuration object should be allowed to have different umask

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 3.2.0, 3.1.1, 3.0.4
    • None
    • None

    Description

      Ran into a super hard-to-debug issue due to this. [Edit: Turns out the same issue as YARN-5749 that Tao Yang ran into]

      Issue

      Configuration conf = new Configuration();
      fc1 = FileContext.getFileContext(uri1, conf);
      fc2 = FileContext.getFileContext(uri2, conf);
      fc.setUMask(umask_for_fc1); // Screws up umask for fc2 also!

      This was not the case before HADOOP-13440.

      Symptoms:

      Scenario I ran into

      When trying to localize a HDFS directory (hdfs:///my/dir/1.txt), NodeManager tries to replicate the directory structure on the local file-system ($yarn-local-dirs/filecache/my/dir/1.txt).

      Now depending on whether NM has ever done a log-aggregation (completely unrelated code that sets umask to be 137 for its own files on HDFS), the directories /my and /my/dir on local-fs may have different permissions. In the specific case where NM did log-aggregation, /my/dir was created with 137 umask and so localization of 1.txt completely failed due to absent directory executable permissions!

      Previous scenarios:

      We ran into this before in test-cases and instead of fixing the root-cause, we just fixed the test-cases: YARN-5679 / YARN-5749

      Attachments

        1. HADOOP-15571.txt
          3 kB
          Vinod Kumar Vavilapalli
        2. HADOOP-15571.1.txt
          6 kB
          Vinod Kumar Vavilapalli

        Issue Links

          Activity

            People

              vinodkv Vinod Kumar Vavilapalli
              vinodkv Vinod Kumar Vavilapalli
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: