Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15571

Multiple FileContexts created with the same configuration object should be allowed to have different umask

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0, 3.1.1, 3.0.4
    • Component/s: None
    • Labels:
      None

      Description

      Ran into a super hard-to-debug issue due to this. [Edit: Turns out the same issue as YARN-5749 that Tao Yang ran into]

      Issue

      Configuration conf = new Configuration();
      fc1 = FileContext.getFileContext(uri1, conf);
      fc2 = FileContext.getFileContext(uri2, conf);
      fc.setUMask(umask_for_fc1); // Screws up umask for fc2 also!

      This was not the case before HADOOP-13440.

      Symptoms:

      Scenario I ran into

      When trying to localize a HDFS directory (hdfs:///my/dir/1.txt), NodeManager tries to replicate the directory structure on the local file-system ($yarn-local-dirs/filecache/my/dir/1.txt).

      Now depending on whether NM has ever done a log-aggregation (completely unrelated code that sets umask to be 137 for its own files on HDFS), the directories /my and /my/dir on local-fs may have different permissions. In the specific case where NM did log-aggregation, /my/dir was created with 137 umask and so localization of 1.txt completely failed due to absent directory executable permissions!

      Previous scenarios:

      We ran into this before in test-cases and instead of fixing the root-cause, we just fixed the test-cases: YARN-5679 / YARN-5749

        Attachments

        1. HADOOP-15571.1.txt
          6 kB
          Vinod Kumar Vavilapalli
        2. HADOOP-15571.txt
          3 kB
          Vinod Kumar Vavilapalli

          Issue Links

            Activity

              People

              • Assignee:
                vinodkv Vinod Kumar Vavilapalli
                Reporter:
                vinodkv Vinod Kumar Vavilapalli
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: