Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5359

FileSystemTimelineReader/Writer uses unix-specific default storage path

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      FileSystemTimelineReaderImpl and FileSystemTimelineWriterImpl use a unix-specific default. It won't work on Windows.

      Also, TestFileSystemTimelineReaderImpl uses this default directly, which is also brittle against concurrent tests.

      1. YARN-5359.01.patch
        12 kB
        Sangjin Lee
      2. YARN-5359.02.patch
        12 kB
        Sangjin Lee
      3. YARN-5359.03.patch
        22 kB
        Sangjin Lee

        Issue Links

          Activity

          Hide
          varun_saxena Varun Saxena added a comment -

          Committed the latest patch to trunk.
          Thanks Sangjin Lee for your contribution and Joep Rottinghuis for the reviews.

          Show
          varun_saxena Varun Saxena added a comment - Committed the latest patch to trunk. Thanks Sangjin Lee for your contribution and Joep Rottinghuis for the reviews.
          Hide
          varun_saxena Varun Saxena added a comment -

          Patch looks fine to me. Will commit it later today.
          I will however go through the timelineservice code once again and see if we aren't using unix style paths somewhere else.

          Show
          varun_saxena Varun Saxena added a comment - Patch looks fine to me. Will commit it later today. I will however go through the timelineservice code once again and see if we aren't using unix style paths somewhere else.
          Hide
          sjlee0 Sangjin Lee added a comment -

          The failed tests are unrelated known issues.

          Show
          sjlee0 Sangjin Lee added a comment - The failed tests are unrelated known issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 7m 11s trunk passed
          +1 compile 1m 37s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 48s trunk passed
          +1 mvneclipse 0m 34s trunk passed
          +1 findbugs 0m 57s trunk passed
          +1 javadoc 0m 27s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 37s the patch passed
          +1 compile 1m 33s the patch passed
          +1 javac 1m 33s the patch passed
          +1 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7)
          +1 mvnsite 0m 44s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 9s the patch passed
          +1 javadoc 0m 23s the patch passed
          +1 unit 0m 46s hadoop-yarn-server-timelineservice in the patch passed.
          -1 unit 4m 28s hadoop-yarn-server-tests in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          24m 22s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817588/YARN-5359.03.patch
          JIRA Issue YARN-5359
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ecbabe9affe0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 06c56ff
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12300/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12300/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12300/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12300/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 11s trunk passed +1 compile 1m 37s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 48s trunk passed +1 mvneclipse 0m 34s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 27s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 37s the patch passed +1 compile 1m 33s the patch passed +1 javac 1m 33s the patch passed +1 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7) +1 mvnsite 0m 44s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 9s the patch passed +1 javadoc 0m 23s the patch passed +1 unit 0m 46s hadoop-yarn-server-timelineservice in the patch passed. -1 unit 4m 28s hadoop-yarn-server-tests in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 24m 22s Reason Tests Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817588/YARN-5359.03.patch JIRA Issue YARN-5359 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ecbabe9affe0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 06c56ff Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12300/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12300/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12300/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/12300/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.4.

          Addressed Joep's comments. Also, I found more places where the default storage location is used for the filesystem writer/reader. They run the risk of stepping on each other's foot when multiple unit tests are running possibly from multiple builds. They have been fixed.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.4. Addressed Joep's comments. Also, I found more places where the default storage location is used for the filesystem writer/reader. They run the risk of stepping on each other's foot when multiple unit tests are running possibly from multiple builds. They have been fixed.
          Hide
          jrottinghuis Joep Rottinghuis added a comment -

          Patch looks good.
          There may be one additional unit test using Linux style paths:
          TestFileSystemTimelineWriterImpl#testWriteEntityToFile
          Around line 87

                String fileName = fsi.getOutputRoot() +
                    "/entities/cluster_id/user_id/flow_name/flow_version/12345678/" +
                    "app_id/" + type + "/" + id +
                    FileSystemTimelineWriterImpl.TIMELINE_SERVICE_STORAGE_EXTENSION;
                Path path = Paths.get(fileName);
                File f = new File(fileName);
                assertTrue(f.exists() && !f.isDirectory());
          

          Similar around line 102

                String fileName2 = fsi.getOutputRoot() +
                    "/entities/cluster_id/user_id/flow_name/flow_version/12345678/app_id/"
                    + type2 + "/" + id2 +
                    FileSystemTimelineWriterImpl.TIMELINE_SERVICE_STORAGE_EXTENSION;
          
          Show
          jrottinghuis Joep Rottinghuis added a comment - Patch looks good. There may be one additional unit test using Linux style paths: TestFileSystemTimelineWriterImpl#testWriteEntityToFile Around line 87 String fileName = fsi.getOutputRoot() + "/entities/cluster_id/user_id/flow_name/flow_version/12345678/" + "app_id/" + type + "/" + id + FileSystemTimelineWriterImpl.TIMELINE_SERVICE_STORAGE_EXTENSION; Path path = Paths.get(fileName); File f = new File(fileName); assertTrue(f.exists() && !f.isDirectory()); Similar around line 102 String fileName2 = fsi.getOutputRoot() + "/entities/cluster_id/user_id/flow_name/flow_version/12345678/app_id/" + type2 + "/" + id2 + FileSystemTimelineWriterImpl.TIMELINE_SERVICE_STORAGE_EXTENSION;
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 33s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 57s trunk passed
          +1 compile 0m 19s trunk passed
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 24s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 31s trunk passed
          +1 javadoc 0m 15s trunk passed
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4)
          +1 mvnsite 0m 21s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 34s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 0m 45s hadoop-yarn-server-timelineservice in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          13m 12s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817503/YARN-5359.02.patch
          JIRA Issue YARN-5359
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e82348f398a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7705812
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12295/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12295/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 57s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 31s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 45s hadoop-yarn-server-timelineservice in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 13m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817503/YARN-5359.02.patch JIRA Issue YARN-5359 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e82348f398a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7705812 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12295/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice Console output https://builds.apache.org/job/PreCommit-YARN-Build/12295/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.2 to address the checkstyle complaint.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.2 to address the checkstyle complaint.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 33s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 17s trunk passed
          +1 compile 0m 20s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 26s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 16s trunk passed
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 19s the patch passed
          +1 javac 0m 19s the patch passed
          -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 1 new + 3 unchanged - 1 fixed = 4 total (was 4)
          +1 mvnsite 0m 22s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 39s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 0m 45s hadoop-yarn-server-timelineservice in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          13m 44s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817472/YARN-5359.01.patch
          JIRA Issue YARN-5359
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fbe076c6d142 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7705812
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12291/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12291/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12291/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 17s trunk passed +1 compile 0m 20s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 19s the patch passed +1 compile 0m 19s the patch passed +1 javac 0m 19s the patch passed -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 1 new + 3 unchanged - 1 fixed = 4 total (was 4) +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 45s hadoop-yarn-server-timelineservice in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 13m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817472/YARN-5359.01.patch JIRA Issue YARN-5359 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fbe076c6d142 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7705812 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12291/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12291/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice Console output https://builds.apache.org/job/PreCommit-YARN-Build/12291/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.1. Moved to use hadoop.tmp.dir as the parent for the storage directory.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.1. Moved to use hadoop.tmp.dir as the parent for the storage directory.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I'll get to this after YARN-5354 and MAPREDUCE-6731 are committed.

          Show
          sjlee0 Sangjin Lee added a comment - I'll get to this after YARN-5354 and MAPREDUCE-6731 are committed.

            People

            • Assignee:
              sjlee0 Sangjin Lee
              Reporter:
              sjlee0 Sangjin Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development