Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2819

NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.6.0
    • 2.6.0
    • timelineserver

    Description

      Caused by: java.lang.NullPointerException
              at java.lang.String.<init>(String.java:554)
              at org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.put(LeveldbTimelineStore.java:873)
              at org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.put(LeveldbTimelineStore.java:1014)
              at org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:330)
              at org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.postEntities(TimelineWebServices.java:260)
      

      triggered by

      entity.getRelatedEntities();
      ...
                  } else {
                    byte[] domainIdBytes = db.get(createDomainIdKey(
                        relatedEntityId, relatedEntityType, relatedEntityStartTime));
                    // This is the existing entity
                    String domainId = new String(domainIdBytes);
                    if (!domainId.equals(entity.getDomainId())) {
      

      The new String(domainIdBytes); throws an NPE.

      Attachments

        1. YARN-2819.1.patch
          21 kB
          Zhijie Shen

        Activity

          zjshen Zhijie Shen added a comment -

          The NPE happens because the data integrity assumes that no entity has a null domainId. However, if leveldb already contains the timeline data that are generated by prior 2.6 timeline server, the integrity is broken. Previously, the entity doesn't have the domain information. Will work on a fix to be compatible to the existing store.

          zjshen Zhijie Shen added a comment - The NPE happens because the data integrity assumes that no entity has a null domainId. However, if leveldb already contains the timeline data that are generated by prior 2.6 timeline server, the integrity is broken. Previously, the entity doesn't have the domain information. Will work on a fix to be compatible to the existing store.
          zjshen Zhijie Shen added a comment -

          Create a patch to make the leveldb store be compatible to existing data. Basically, we're going to treat the entity without domain Id as the one having a DEFAULT domain.

          zjshen Zhijie Shen added a comment - Create a patch to make the leveldb store be compatible to existing data. Basically, we're going to treat the entity without domain Id as the one having a DEFAULT domain.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12679989/YARN-2819.1.patch
          against trunk revision 75b820c.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5761//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5761//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12679989/YARN-2819.1.patch against trunk revision 75b820c. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5761//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5761//console This message is automatically generated.
          zjshen Zhijie Shen added a comment -

          I've done the following experiments locally:

          1. Run timeline server 2.5 to generate the old timeline data without domainId field in leveldb.

          2. Run timeline server 2.6 (current trunk actually), and try to update the old entity and relate to it. I can reproduce the same NPE as is mentioned in the description.

          3. Run timeline server 2.6 with the attached patch, and try to update the old entity and relate to it. The problem is gone.

          zjshen Zhijie Shen added a comment - I've done the following experiments locally: 1. Run timeline server 2.5 to generate the old timeline data without domainId field in leveldb. 2. Run timeline server 2.6 (current trunk actually), and try to update the old entity and relate to it. I can reproduce the same NPE as is mentioned in the description. 3. Run timeline server 2.6 with the attached patch, and try to update the old entity and relate to it. The problem is gone.
          gtcarrera9 Li Lu added a comment -

          Hi zjshen! I reviewed this patch, and it looks good to me. Specifically, I agree that we can treat nulls from old data equal to DEFAULT_DOMAIN in new data, by masking them during read out time. The special design for related entity in level db store is fine because only entities stored in level db will face this problem (any persistent data storage needs to consider domains afterwards).

          I noticed that you've added several tests. I think they look good for now. In future if we're adding support to hbase or something else, we may want to share some of our existing tests like testRelatingToNonExistingEntity.

          In conclusion, patch LGTM. You may want to find some committers to go through it once more.

          gtcarrera9 Li Lu added a comment - Hi zjshen ! I reviewed this patch, and it looks good to me. Specifically, I agree that we can treat nulls from old data equal to DEFAULT_DOMAIN in new data, by masking them during read out time. The special design for related entity in level db store is fine because only entities stored in level db will face this problem (any persistent data storage needs to consider domains afterwards). I noticed that you've added several tests. I think they look good for now. In future if we're adding support to hbase or something else, we may want to share some of our existing tests like testRelatingToNonExistingEntity . In conclusion, patch LGTM. You may want to find some committers to go through it once more.
          xgong Xuan Gong added a comment -

          +1 Looks good. Checking this in

          xgong Xuan Gong added a comment - +1 Looks good. Checking this in
          xgong Xuan Gong added a comment -

          Committed into trunk/branch-2/branch-2.6. Thanks, zhijie.

          xgong Xuan Gong added a comment - Committed into trunk/branch-2/branch-2.6. Thanks, zhijie.
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6489 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6489/)
          YARN-2819. NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen (xgong: rev 4a114dd67aae83e5bb2d65470166de954acf36a2)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6489 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6489/ ) YARN-2819 . NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen (xgong: rev 4a114dd67aae83e5bb2d65470166de954acf36a2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #737 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/737/)
          YARN-2819. NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen (xgong: rev 4a114dd67aae83e5bb2d65470166de954acf36a2)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #737 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/737/ ) YARN-2819 . NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen (xgong: rev 4a114dd67aae83e5bb2d65470166de954acf36a2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1927 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1927/)
          YARN-2819. NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen (xgong: rev 4a114dd67aae83e5bb2d65470166de954acf36a2)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/CHANGES.txt
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1927 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1927/ ) YARN-2819 . NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen (xgong: rev 4a114dd67aae83e5bb2d65470166de954acf36a2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/CHANGES.txt
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1951 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1951/)
          YARN-2819. NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen (xgong: rev 4a114dd67aae83e5bb2d65470166de954acf36a2)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1951 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1951/ ) YARN-2819 . NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen (xgong: rev 4a114dd67aae83e5bb2d65470166de954acf36a2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java

          People

            zjshen Zhijie Shen
            gopalv Gopal Vijayaraghavan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: