Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3207

secondary filter matches entites which do not have the key being filtered for.

    Details

      Description

      in the leveldb implementation of the TimelineStore the secondary filter matches entities where the key being searched for is not present.

      ex query from tez ui
      http://uvm:8188/ws/v1/timeline/TEZ_DAG_ID/?limit=1&secondaryFilter=foo:bar

      will match and return the entity even though there is no entity with otherinfo.foo defined.

      the issue seems to be in

      LeveldbTimelineStore:675
      if (vs != null && !vs.contains(filter.getValue())) {
        filterPassed = false;
        break;
      }
      

      this should be IMHO
      vs == null || !vs.contains(filter.getValue())

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Pulled this into 2.6.1. Ran compilation and TestLeveldbTimelineStore, TestMemoryTimelineStore, TestTimelineDataManager before the push. Patch applied cleanly.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1. Ran compilation and TestLeveldbTimelineStore, TestMemoryTimelineStore, TestTimelineDataManager before the push. Patch applied cleanly.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2059 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2059/)
          YARN-3207. Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2059 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2059/ ) YARN-3207 . Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #109 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/109/)
          YARN-3207. Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #109 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/109/ ) YARN-3207 . Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2040 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2040/)
          YARN-3207. Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2040 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2040/ ) YARN-3207 . Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/99/)
          YARN-3207. Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/99/ ) YARN-3207 . Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #108 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/108/)
          YARN-3207. Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #108 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/108/ ) YARN-3207 . Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #842 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/842/)
          YARN-3207. Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #842 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/842/ ) YARN-3207 . Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java hadoop-yarn-project/CHANGES.txt
          Hide
          zjshen Zhijie Shen added a comment -

          was just thinking in terms of any scenario to store different values in primary filter(just like a tag) and other info(more detailed)

          In this case, according to the code path, other info decides the result if the key appears in both sides, but the value differs. Thanks, Naga for raising this special case explicitly.

          Show
          zjshen Zhijie Shen added a comment - was just thinking in terms of any scenario to store different values in primary filter(just like a tag) and other info(more detailed) In this case, according to the code path, other info decides the result if the key appears in both sides, but the value differs. Thanks, Naga for raising this special case explicitly.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          If the user put the same info in both primary filter and other info, and make the value null, it will return false, but it's more like a fault use case, isn't it?

          Yes i agree this scenario is not practical but was just thinking in terms of any scenario to store different values in primary filter(just like a tag) and other info(more detailed), hence asked.

          Show
          Naganarasimha Naganarasimha G R added a comment - If the user put the same info in both primary filter and other info, and make the value null, it will return false, but it's more like a fault use case, isn't it? Yes i agree this scenario is not practical but was just thinking in terms of any scenario to store different values in primary filter(just like a tag) and other info(more detailed), hence asked.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7139 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7139/)
          YARN-3207. Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7139 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7139/ ) YARN-3207 . Secondary filter matches entites which do not have the key (xgong: rev 57db50cbe3ce42618ad6d6869ae337d15b261f4e) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java hadoop-yarn-project/CHANGES.txt
          Hide
          xgong Xuan Gong added a comment -

          Committed into trunk/branch-2. Thanks, zhijie !

          Show
          xgong Xuan Gong added a comment - Committed into trunk/branch-2. Thanks, zhijie !
          Hide
          xgong Xuan Gong added a comment -

          +1 LGTM. Will commit.

          Show
          xgong Xuan Gong added a comment - +1 LGTM. Will commit.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12699388/YARN-3207.1.patch
          against trunk revision 685af8a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6654//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6654//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699388/YARN-3207.1.patch against trunk revision 685af8a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6654//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6654//console This message is automatically generated.
          Hide
          zjshen Zhijie Shen added a comment -

          Create the patch to fix the issue.

          Show
          zjshen Zhijie Shen added a comment - Create the patch to fix the issue.
          Hide
          zjshen Zhijie Shen added a comment -

          Varun, thanks for reassign it back to me. I'm currently working on this issue.

          Also can there be a possibility that a given secondary filter key is present in both other info and primary filters,

          Practically, if the user put some info in primary filter, it should be put into other info of the entity again. "primary filter" is somehow a special info, but will be used as the index.

          If the user put the same info in both primary filter and other info, and make the value null, it will return false, but it's more like a fault use case, isn't it?

          Show
          zjshen Zhijie Shen added a comment - Varun, thanks for reassign it back to me. I'm currently working on this issue. Also can there be a possibility that a given secondary filter key is present in both other info and primary filters, Practically, if the user put some info in primary filter, it should be put into other info of the entity again. "primary filter" is somehow a special info, but will be used as the index. If the user put the same info in both primary filter and other info, and make the value null, it will return false, but it's more like a fault use case, isn't it?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Zhijie Shen,
          Also can there be a possibility that a given secondary filter key is present in both other info and primary filters, but value is present only in primary filters ?
          In that case below code returns false.

                      Object v = entity.getOtherInfo().get(filter.getName());
                      if (v == null) {
                        Set<Object> vs = entity.getPrimaryFilters()
                            .get(filter.getName());
                        if (vs != null && !vs.contains(filter.getValue())) {
                          filterPassed = false;
                          break;
                        }
                      } else if (!v.equals(filter.getValue())) {
                        filterPassed = false;
                        break;
                      }
          

          Willing to work on this if you have not already started .

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Zhijie Shen , Also can there be a possibility that a given secondary filter key is present in both other info and primary filters, but value is present only in primary filters ? In that case below code returns false. Object v = entity.getOtherInfo().get(filter.getName()); if (v == null ) { Set< Object > vs = entity.getPrimaryFilters() .get(filter.getName()); if (vs != null && !vs.contains(filter.getValue())) { filterPassed = false ; break ; } } else if (!v.equals(filter.getValue())) { filterPassed = false ; break ; } Willing to work on this if you have not already started .
          Hide
          zjshen Zhijie Shen added a comment -

          Prakash Ramachandran, thanks for reporting the issue. I think it makes sense. The record without secondary filter info shouldn't be qualified for the query filter too.

          Show
          zjshen Zhijie Shen added a comment - Prakash Ramachandran , thanks for reporting the issue. I think it makes sense. The record without secondary filter info shouldn't be qualified for the query filter too.

            People

            • Assignee:
              zjshen Zhijie Shen
              Reporter:
              pramachandran Prakash Ramachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development