Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: timelineserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For large applications, the majority of the time in LeveldbTimelineStore is spent deleting old entities record at a time. An exclusive write lock is held during the entire deletion phase which in practice can be hours. If we are to relax some of the consistency constraints, other performance enhancing techniques can be employed to maximize the throughput and minimize locking time.

      Split the 5 sections of the leveldb database (domain, owner, start time, entity, index) into 5 separate databases. This allows each database to maximize the read cache effectiveness based on the unique usage patterns of each database. With 5 separate databases each lookup is much faster. This can also help with I/O to have the entity and index databases on separate disks.

      Rolling DBs for entity and index DBs. 99.9% of the data are in these two sections 4:1 ration (index to entity) at least for tez. We replace DB record removal with file system removal if we create a rolling set of databases that age out and can be efficiently removed. To do this we must place a constraint to always place an entity's events into it's correct rolling db instance based on start time. This allows us to stitching the data back together while reading and artificial paging.

      Relax the synchronous writes constraints. If we are willing to accept losing some records that we not flushed in the operating system during a crash, we can use async writes that can be much faster.

      Prefer Sequential writes. sequential writes can be several times faster than random writes. Spend some small effort arranging the writes in such a way that will trend towards sequential write performance over random write performance.

      1. YARN-3448.9.patch
        116 kB
        Jonathan Eagles
      2. YARN-3448.8.patch
        115 kB
        Jonathan Eagles
      3. YARN-3448.7.patch
        96 kB
        Jonathan Eagles
      4. YARN-3448.5.patch
        96 kB
        Jonathan Eagles
      5. YARN-3448.4.patch
        98 kB
        Jonathan Eagles
      6. YARN-3448.3.patch
        99 kB
        Jonathan Eagles
      7. YARN-3448.2.patch
        94 kB
        Jonathan Eagles
      8. YARN-3448.17.patch
        135 kB
        Jonathan Eagles
      9. YARN-3448.16.patch
        134 kB
        Jonathan Eagles
      10. YARN-3448.15.patch
        134 kB
        Jonathan Eagles
      11. YARN-3448.14.patch
        133 kB
        Jonathan Eagles
      12. YARN-3448.13.patch
        127 kB
        Jonathan Eagles
      13. YARN-3448.12.patch
        119 kB
        Jonathan Eagles
      14. YARN-3448.10.patch
        112 kB
        Jonathan Eagles
      15. YARN-3448.1.patch
        92 kB
        Jonathan Eagles

        Issue Links

          Activity

          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Jonathan. Interesting approach, and this should drastically improve performance for retention processing. Some comments on the patch so far:

          I think the code would be easier to follow if we didn't abuse Map.Entry as a pair class to associate a WriteBatch to the corresponding DB. Creating a custom utility class to associate these would make the code a lot more readable than always needing to deduce that getKey() is a database and getValue() is a WriteBatch.

          The underlying database throws a runtime exception, and the existing leveldb store translates these to IOExceptions. I think we want to do the same here. For example, put has a try..finally block with no catch clauses yet the method says it does not throw exceptions like IOException. Arguably it should throw IOException when the database has an error.

          The original leveldb code had locking around entities but I don't see it here. Since updating entities often involves a read/modify/write operation on the database, are we sure it's OK to remove that synchronization?

          computeCheckMillis says it needs to be called synchronously, but it looks like it can be called without a lock via a number of routes, e.g.:
          put -> putIndex -> computeCurrentCheckMillis -> computeCheckMillis
          put -> putEntities -> computeCurrentCheckMillis -> computeCheckMillis

          These should probably be debug statements, otherwise I think they could be quite spammy in the server log. Also the latter one will always be followed by the former because of the loop and may not be that useful in practice, even at the debug level.

          +        LOG.info("Trying the  db" + db);
          ...
          +        LOG.info("Trying the previous db" + db);
          

          This will NPE on entityUpdate if db is null, and the code explicity checks for that possibility:

          +      Map.Entry<DB, WriteBatch> entityUpdate = entityUpdates.get(roundedStartTime);
          +      if (entityUpdate == null) {
          +        DB db = entitydb.getDBForStartTime(startAndInsertTime.startTime);
          +        if (db != null) {
          +          WriteBatch writeBatch = db.createWriteBatch();
          +          entityUpdate = new AbstractMap.SimpleImmutableEntry<DB, WriteBatch>(db, writeBatch);
          +          entityUpdates.put(roundedStartTime, entityUpdate);
          +        };
          +      }
          +      WriteBatch writeBatch = entityUpdate.getValue();
          

          In the following code we lookup relatedEntityUpdate but then after checking if it's null never use it again. I think we're supposed to be setting up relatedEntityUpdate in the block if it's null rather than re-assigning entityUpdate. Then after the null check we should be using relatedEntityUpdate rather than entityUpdate to get the proper write batch.

          +            Map.Entry<DB, WriteBatch> relatedEntityUpdate = entityUpdates.get(relatedRoundedStartTime);
          +            if (relatedEntityUpdate == null) {
          +              DB db = entitydb.getDBForStartTime(relatedStartTimeLong);
          +              if (db != null) {
          +                WriteBatch relatedWriteBatch = db.createWriteBatch();
          +                entityUpdate = new AbstractMap.SimpleImmutableEntry<DB, WriteBatch>(
          +                    db, relatedWriteBatch);
          +                entityUpdates.put(relatedRoundedStartTime, entityUpdate);
          +              }
          +              ;
          +            }
          +            WriteBatch relatedWriteBatch = entityUpdate.getValue();
          

          This code is commented out. Should have been deleted or is there something left to do here with respect to related entitites?

          +    /*
          +    for (EntityIdentifier relatedEntity : relatedEntitiesWithoutStartTimes) {
          +      try {
          +        StartAndInsertTime relatedEntityStartAndInsertTime =
          +            getAndSetStartTime(relatedEntity.getId(), relatedEntity.getType(),
          +            readReverseOrderedLong(revStartTime, 0), null);
          +        if (relatedEntityStartAndInsertTime == null) {
          +          throw new IOException("Error setting start time for related entity");
          +        }
          +        byte[] relatedEntityStartTime = writeReverseOrderedLong(
          +            relatedEntityStartAndInsertTime.startTime);
          +          // This is the new entity, the domain should be the same
          +        byte[] key = createDomainIdKey(relatedEntity.getId(),
          +            relatedEntity.getType(), relatedEntityStartTime);
          +        writeBatch.put(key, entity.getDomainId().getBytes());
          +        ++putCount;
          +        writeBatch.put(createRelatedEntityKey(relatedEntity.getId(),
          +            relatedEntity.getType(), relatedEntityStartTime,
          +            entity.getEntityId(), entity.getEntityType()), EMPTY_BYTES);
          +        ++putCount;
          +        writeBatch.put(createEntityMarkerKey(relatedEntity.getId(),
          +            relatedEntity.getType(), relatedEntityStartTime),
          +            writeReverseOrderedLong(relatedEntityStartAndInsertTime
          +                .insertTime));
          +        ++putCount;
          +      } catch (IOException e) {
          +        LOG.error("Error putting related entity " + relatedEntity.getId() +
          +            " of type " + relatedEntity.getType() + " for entity " +
          +            entity.getEntityId() + " of type " + entity.getEntityType(), e);
          +        TimelinePutError error = new TimelinePutError();
          +        error.setEntityId(entity.getEntityId());
          +        error.setEntityType(entity.getEntityType());
          +        error.setErrorCode(TimelinePutError.IO_EXCEPTION);
          +        response.addError(error);
          +      }
          +    }
          +    */
          

          Similar NPE potential on indexUpdate if db is null:

          +      Map.Entry<DB, WriteBatch> indexUpdate = indexUpdates.get(roundedStartTime);
          +      if (indexUpdate == null) {
          +        DB db = indexdb.getDBForStartTime(startAndInsertTime.startTime);
          +        if (db != null) {
          +          WriteBatch writeBatch = db.createWriteBatch();
          +          indexUpdate = new AbstractMap.SimpleImmutableEntry<DB, WriteBatch>(db, writeBatch);
          +          indexUpdates.put(roundedStartTime, indexUpdate);
          +        };
          +      }
          +      WriteBatch writeBatch = indexUpdate.getValue();
          

          getAndSetStartTime and checkStartTimeInDb comments refer to obtaining a lock on the entity but no entity-level locking appears to be used.

          Nit: put and putWithNoDomainId should be refactored to call a parameterized form of common code since they are so similar. Would also help fix the inconsistency where one debug logs and the other info logs.

          Nit: there are some extraneous semicolons in the patch, either after braces or on lines by themselves.

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Jonathan. Interesting approach, and this should drastically improve performance for retention processing. Some comments on the patch so far: I think the code would be easier to follow if we didn't abuse Map.Entry as a pair class to associate a WriteBatch to the corresponding DB. Creating a custom utility class to associate these would make the code a lot more readable than always needing to deduce that getKey() is a database and getValue() is a WriteBatch. The underlying database throws a runtime exception, and the existing leveldb store translates these to IOExceptions. I think we want to do the same here. For example, put has a try..finally block with no catch clauses yet the method says it does not throw exceptions like IOException. Arguably it should throw IOException when the database has an error. The original leveldb code had locking around entities but I don't see it here. Since updating entities often involves a read/modify/write operation on the database, are we sure it's OK to remove that synchronization? computeCheckMillis says it needs to be called synchronously, but it looks like it can be called without a lock via a number of routes, e.g.: put -> putIndex -> computeCurrentCheckMillis -> computeCheckMillis put -> putEntities -> computeCurrentCheckMillis -> computeCheckMillis These should probably be debug statements, otherwise I think they could be quite spammy in the server log. Also the latter one will always be followed by the former because of the loop and may not be that useful in practice, even at the debug level. + LOG.info( "Trying the db" + db); ... + LOG.info( "Trying the previous db" + db); This will NPE on entityUpdate if db is null, and the code explicity checks for that possibility: + Map.Entry<DB, WriteBatch> entityUpdate = entityUpdates.get(roundedStartTime); + if (entityUpdate == null ) { + DB db = entitydb.getDBForStartTime(startAndInsertTime.startTime); + if (db != null ) { + WriteBatch writeBatch = db.createWriteBatch(); + entityUpdate = new AbstractMap.SimpleImmutableEntry<DB, WriteBatch>(db, writeBatch); + entityUpdates.put(roundedStartTime, entityUpdate); + }; + } + WriteBatch writeBatch = entityUpdate.getValue(); In the following code we lookup relatedEntityUpdate but then after checking if it's null never use it again. I think we're supposed to be setting up relatedEntityUpdate in the block if it's null rather than re-assigning entityUpdate. Then after the null check we should be using relatedEntityUpdate rather than entityUpdate to get the proper write batch. + Map.Entry<DB, WriteBatch> relatedEntityUpdate = entityUpdates.get(relatedRoundedStartTime); + if (relatedEntityUpdate == null ) { + DB db = entitydb.getDBForStartTime(relatedStartTimeLong); + if (db != null ) { + WriteBatch relatedWriteBatch = db.createWriteBatch(); + entityUpdate = new AbstractMap.SimpleImmutableEntry<DB, WriteBatch>( + db, relatedWriteBatch); + entityUpdates.put(relatedRoundedStartTime, entityUpdate); + } + ; + } + WriteBatch relatedWriteBatch = entityUpdate.getValue(); This code is commented out. Should have been deleted or is there something left to do here with respect to related entitites? + /* + for (EntityIdentifier relatedEntity : relatedEntitiesWithoutStartTimes) { + try { + StartAndInsertTime relatedEntityStartAndInsertTime = + getAndSetStartTime(relatedEntity.getId(), relatedEntity.getType(), + readReverseOrderedLong(revStartTime, 0), null ); + if (relatedEntityStartAndInsertTime == null ) { + throw new IOException( "Error setting start time for related entity" ); + } + byte [] relatedEntityStartTime = writeReverseOrderedLong( + relatedEntityStartAndInsertTime.startTime); + // This is the new entity, the domain should be the same + byte [] key = createDomainIdKey(relatedEntity.getId(), + relatedEntity.getType(), relatedEntityStartTime); + writeBatch.put(key, entity.getDomainId().getBytes()); + ++putCount; + writeBatch.put(createRelatedEntityKey(relatedEntity.getId(), + relatedEntity.getType(), relatedEntityStartTime, + entity.getEntityId(), entity.getEntityType()), EMPTY_BYTES); + ++putCount; + writeBatch.put(createEntityMarkerKey(relatedEntity.getId(), + relatedEntity.getType(), relatedEntityStartTime), + writeReverseOrderedLong(relatedEntityStartAndInsertTime + .insertTime)); + ++putCount; + } catch (IOException e) { + LOG.error( "Error putting related entity " + relatedEntity.getId() + + " of type " + relatedEntity.getType() + " for entity " + + entity.getEntityId() + " of type " + entity.getEntityType(), e); + TimelinePutError error = new TimelinePutError(); + error.setEntityId(entity.getEntityId()); + error.setEntityType(entity.getEntityType()); + error.setErrorCode(TimelinePutError.IO_EXCEPTION); + response.addError(error); + } + } + */ Similar NPE potential on indexUpdate if db is null: + Map.Entry<DB, WriteBatch> indexUpdate = indexUpdates.get(roundedStartTime); + if (indexUpdate == null ) { + DB db = indexdb.getDBForStartTime(startAndInsertTime.startTime); + if (db != null ) { + WriteBatch writeBatch = db.createWriteBatch(); + indexUpdate = new AbstractMap.SimpleImmutableEntry<DB, WriteBatch>(db, writeBatch); + indexUpdates.put(roundedStartTime, indexUpdate); + }; + } + WriteBatch writeBatch = indexUpdate.getValue(); getAndSetStartTime and checkStartTimeInDb comments refer to obtaining a lock on the entity but no entity-level locking appears to be used. Nit: put and putWithNoDomainId should be refactored to call a parameterized form of common code since they are so similar. Would also help fix the inconsistency where one debug logs and the other info logs. Nit: there are some extraneous semicolons in the patch, either after braces or on lines by themselves.
          Hide
          zjshen Zhijie Shen added a comment -

          Jonathan, thanks for your contribution. It sounds an interesting proposal. I'd like to take a look at the patch too.

          Show
          zjshen Zhijie Shen added a comment - Jonathan, thanks for your contribution. It sounds an interesting proposal. I'd like to take a look at the patch too.
          Hide
          jeagles Jonathan Eagles added a comment -

          Jason Lowe, addressed you comments. One thing to note is that the entity based read write lock I didn't add back in. Theoretically this is possible for applications that have multiple writers to update to both get and set the start times for an entity or related entity. For applications like Tez (one writer) this is not possible AFAIK. It probably isn't a huge over head, I just haven't had the time to benchmark before and after with entity locking.

          Show
          jeagles Jonathan Eagles added a comment - Jason Lowe , addressed you comments. One thing to note is that the entity based read write lock I didn't add back in. Theoretically this is possible for applications that have multiple writers to update to both get and set the start times for an entity or related entity. For applications like Tez (one writer) this is not possible AFAIK. It probably isn't a huge over head, I just haven't had the time to benchmark before and after with entity locking.
          Hide
          zjshen Zhijie Shen added a comment - - edited

          Jonathan, I've several high questions about the design and the implementation:

          Split the 5 sections of the leveldb database (domain, owner, start time, entity, index) into 5 separate databases.

          According to the official document, LevelDb a single process (possibly multi-threaded). Therefore, instead of 5 separate (logic) tables, 5 separate databases is used to increase concurrency, isn't it?

          However, this approach may raise the inconsistency issue. For example, if I upload an entity with primary filter defined, I may run into a scenario that some I/O exception happens when timeline server tries to write into entity db, while the index record is persisted without any problem. In scenario, the entity is searchable by primary filter, but cannot be got by its identifier.

          Rolling DBs for entity and index DBs. 99.9% of the data are in these two sections 4:1 ration (index to entity) at least for tez.

          If I understand it correct, ownerdb can be treated as the secondary index of domaindb. If we want to lookup for the domains of one owner, we have two steps: 1) get all domain IDs from ownerdb and then 2) pull each individual domains from domaindb.

          I think we could adopt the similar approach for entitydb and indexdb. Instead of a full copy of entity content in indexdb, we could just record the entity identifier there, and do two-step lookup to answer the query. By doing this, we should be able to significantly shrink indexdb size, and improve write performance. In contrast, the previous leveldb index implementation seems to optimize towards the query.

          3. I'm wondering if we need a separate configuration of rolling period or we should use ttl as the rolling period. The reason is if we set ttl smaller than the rolling period, in the most recent database, there will still exist old data. Therefore, we still need the deletion thread to remove these entities/index entries, or the query has to exclude them from result set.

          On the other side, it may be also not good to set ttl greater than rolling period. This is because if period now is smaller than ttl, we still need to wait until ttl to delete the database. Therefore, setting small rolling period along won't shrink the total database size if ttl is kept large.

          Combining the two points above, it seems to be better to let rolling period = ttl. And I think it may simplify the implementation with it, because we know current database will have all the live data, and previous databases are sure to have the old data to be discarded. Thoughts?

          4. I assume that roll() method is going to be processed quickly, right? Otherwise, during the transit state of rolling a database, write performance will degrade somehow.

          Show
          zjshen Zhijie Shen added a comment - - edited Jonathan, I've several high questions about the design and the implementation: Split the 5 sections of the leveldb database (domain, owner, start time, entity, index) into 5 separate databases. According to the official document , LevelDb a single process (possibly multi-threaded). Therefore, instead of 5 separate (logic) tables, 5 separate databases is used to increase concurrency, isn't it? However, this approach may raise the inconsistency issue. For example, if I upload an entity with primary filter defined, I may run into a scenario that some I/O exception happens when timeline server tries to write into entity db, while the index record is persisted without any problem. In scenario, the entity is searchable by primary filter, but cannot be got by its identifier. Rolling DBs for entity and index DBs. 99.9% of the data are in these two sections 4:1 ration (index to entity) at least for tez. If I understand it correct, ownerdb can be treated as the secondary index of domaindb. If we want to lookup for the domains of one owner, we have two steps: 1) get all domain IDs from ownerdb and then 2) pull each individual domains from domaindb. I think we could adopt the similar approach for entitydb and indexdb. Instead of a full copy of entity content in indexdb, we could just record the entity identifier there, and do two-step lookup to answer the query. By doing this, we should be able to significantly shrink indexdb size, and improve write performance. In contrast, the previous leveldb index implementation seems to optimize towards the query. 3. I'm wondering if we need a separate configuration of rolling period or we should use ttl as the rolling period. The reason is if we set ttl smaller than the rolling period, in the most recent database, there will still exist old data. Therefore, we still need the deletion thread to remove these entities/index entries, or the query has to exclude them from result set. On the other side, it may be also not good to set ttl greater than rolling period. This is because if period now is smaller than ttl, we still need to wait until ttl to delete the database. Therefore, setting small rolling period along won't shrink the total database size if ttl is kept large. Combining the two points above, it seems to be better to let rolling period = ttl. And I think it may simplify the implementation with it, because we know current database will have all the live data, and previous databases are sure to have the old data to be discarded. Thoughts? 4. I assume that roll() method is going to be processed quickly, right? Otherwise, during the transit state of rolling a database, write performance will degrade somehow.
          Hide
          jeagles Jonathan Eagles added a comment -

          Zhijie Shen, Interesting idea about index just beings pointers into the entity db. I'll have to investigate what the write and read performance implications are.

          As for rolling period vs ttl. I think rolling period should always be a smaller than ttl. One thing to consider is that unlike traditional rolling files, there are more than one active at a time. In fact, all rolling dbs from now unto ttl may be active. That is due to stitching of data back together on the reads. All events for the same entity id will go into the same database.

          My current setup includes rolling every hour and a ttl of one day.

          As far as what roll does, it only schedules the db to be deleted and removes the old entity and index from being found. This does mean that there will some start times associated that are old that are still active. That will get eventually consistent once the ttl eviction period finishes.

          Show
          jeagles Jonathan Eagles added a comment - Zhijie Shen , Interesting idea about index just beings pointers into the entity db. I'll have to investigate what the write and read performance implications are. As for rolling period vs ttl. I think rolling period should always be a smaller than ttl. One thing to consider is that unlike traditional rolling files, there are more than one active at a time. In fact, all rolling dbs from now unto ttl may be active. That is due to stitching of data back together on the reads. All events for the same entity id will go into the same database. My current setup includes rolling every hour and a ttl of one day. As far as what roll does, it only schedules the db to be deleted and removes the old entity and index from being found. This does mean that there will some start times associated that are old that are still active. That will get eventually consistent once the ttl eviction period finishes.
          Hide
          zjshen Zhijie Shen added a comment -

          In fact, all rolling dbs from now unto ttl may be active.

          Yeah, actually this is the point I'd like make. For example, if ttl = 10h and rolling period = 1h, we will have 10 active rolling dbs. Though 2 - 10 dbs are not current, but they can't be deleted because they contain the data that is still alive. Only rolling dbs from 11 and son will be deleted. While ttl = 10h, we change rolling period = 10h, and I will only have 1 active 10 rolling db, and its size should be equivalent to prior 10 1h rolling dbs. Therefore, my point is that if rolling period smaller than ttl, we still need to keep all the data alive, it's not necessary to separate them into multiple dbs rather than keeping them together in the current db.

          One benefit I can think of about multiple-rolling-db approach (as well as different dbs for different data type) is to increase concurrency. However, I didn't see we have multiple threads to write different dbs concurrently.

          Show
          zjshen Zhijie Shen added a comment - In fact, all rolling dbs from now unto ttl may be active. Yeah, actually this is the point I'd like make. For example, if ttl = 10h and rolling period = 1h, we will have 10 active rolling dbs. Though 2 - 10 dbs are not current, but they can't be deleted because they contain the data that is still alive. Only rolling dbs from 11 and son will be deleted. While ttl = 10h, we change rolling period = 10h, and I will only have 1 active 10 rolling db, and its size should be equivalent to prior 10 1h rolling dbs. Therefore, my point is that if rolling period smaller than ttl, we still need to keep all the data alive, it's not necessary to separate them into multiple dbs rather than keeping them together in the current db. One benefit I can think of about multiple-rolling-db approach (as well as different dbs for different data type) is to increase concurrency. However, I didn't see we have multiple threads to write different dbs concurrently.
          Hide
          jeagles Jonathan Eagles added a comment -

          Couple of more bug fixes in patch 4. Next I try out the index change you suggest above.

          Show
          jeagles Jonathan Eagles added a comment - Couple of more bug fixes in patch 4. Next I try out the index change you suggest above.
          Hide
          jeagles Jonathan Eagles added a comment -

          Zhijie Shen and Jason Lowe, here is the patch that is ready for review. It has the great improvement zhijie suggested about using index as just pointers. This this only used to read a few hundred at max entity when the UI is loaded, the UI still seems to load at the same rate.

          Show
          jeagles Jonathan Eagles added a comment - Zhijie Shen and Jason Lowe , here is the patch that is ready for review. It has the great improvement zhijie suggested about using index as just pointers. This this only used to read a few hundred at max entity when the UI is loaded, the UI still seems to load at the same rate.
          Hide
          zjshen Zhijie Shen added a comment -

          Jonathan Eagles, thanks for updating the patch. It looks good to me overall. I've some additional comments about it:

          1. Should RollingLevelDB extend AbstractService?

          2. In computeCheckMillis(), shall we prevent execution flow across different cases in the switch block?

          3. The javadoc of RollingLevelDBTimelineStore needs to be updated accordingly.

          4. I'm still wondering if writing entitydb and indexdb is atomic. What if entitydb is written, but indexdb isn't for an entity? Shall we delete the data in entitydb too in this case?

          5. I think another improvement could be having one thread per different type of db and per db inside rollingdb, and writing those dbs simultaneously to increase concurrency, but I'm not sure if it's difficult to implement. Thoughts?

          6. Can we make sure TimelineStoreTestUtils' test cases are passed with RollingLevelDBTimelineStore too?

          7. Is TimelineDataManager code change related or necessary?

          8. Is it better to cleanup the directory after completing the test cases in TestRollingLevelDB?

          Show
          zjshen Zhijie Shen added a comment - Jonathan Eagles , thanks for updating the patch. It looks good to me overall. I've some additional comments about it: 1. Should RollingLevelDB extend AbstractService? 2. In computeCheckMillis(), shall we prevent execution flow across different cases in the switch block? 3. The javadoc of RollingLevelDBTimelineStore needs to be updated accordingly. 4. I'm still wondering if writing entitydb and indexdb is atomic. What if entitydb is written, but indexdb isn't for an entity? Shall we delete the data in entitydb too in this case? 5. I think another improvement could be having one thread per different type of db and per db inside rollingdb, and writing those dbs simultaneously to increase concurrency, but I'm not sure if it's difficult to implement. Thoughts? 6. Can we make sure TimelineStoreTestUtils' test cases are passed with RollingLevelDBTimelineStore too? 7. Is TimelineDataManager code change related or necessary? 8. Is it better to cleanup the directory after completing the test cases in TestRollingLevelDB?
          Hide
          jeagles Jonathan Eagles added a comment -

          I fixed the case statement bug earlier today, but hadn't uploaded a new patch.

          Show
          jeagles Jonathan Eagles added a comment - I fixed the case statement bug earlier today, but hadn't uploaded a new patch.
          Hide
          jeagles Jonathan Eagles added a comment -

          Zhijie Shen, Jason Lowe, updated the patch to include tests.

          Show
          jeagles Jonathan Eagles added a comment - Zhijie Shen , Jason Lowe , updated the patch to include tests.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12725620/YARN-3448.8.patch
          against trunk revision fddd552.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 10 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7347//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7347//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7347//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725620/YARN-3448.8.patch against trunk revision fddd552. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 10 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7347//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7347//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7347//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12725873/YARN-3448.9.patch
          against trunk revision 2e8ea78.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7357//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7357//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7357//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725873/YARN-3448.9.patch against trunk revision 2e8ea78. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7357//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7357//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7357//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12725932/YARN-3448.10.patch
          against trunk revision 1fa8075.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7361//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7361//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7361//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725932/YARN-3448.10.patch against trunk revision 1fa8075. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7361//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7361//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7361//console This message is automatically generated.
          Hide
          jeagles Jonathan Eagles added a comment -

          Improved the patch further by running the code with a java profile. This patch is 25% faster and generates roughly 20% smaller database than the previous version.

          • Removed unnecessary PREFIX since each type is in its own database and is not needed to distinguish.
          • Removed unused invisible related entities to reduce to reduce further operations.
          • Changed database serialization method to more quickly generate a smaller serialized size of the primary filter values and other info. Library introduced is verified Apache License 2.0 from fast-serialization.
          • Profile show much time spent converting Strings to byte arrays. Converted the strings once and reused for all the database keys.
          • Reduced the read cache and write buffer size to take into consideration the 7 day default retention.
          • Removed insert time from start time database. This feature is used to detect changes since last query, but is not functional since it forces a scan of all data entries. Could be added back at a later time.
          Show
          jeagles Jonathan Eagles added a comment - Improved the patch further by running the code with a java profile. This patch is 25% faster and generates roughly 20% smaller database than the previous version. Removed unnecessary PREFIX since each type is in its own database and is not needed to distinguish. Removed unused invisible related entities to reduce to reduce further operations. Changed database serialization method to more quickly generate a smaller serialized size of the primary filter values and other info. Library introduced is verified Apache License 2.0 from fast-serialization. Profile show much time spent converting Strings to byte arrays. Converted the strings once and reused for all the database keys. Reduced the read cache and write buffer size to take into consideration the 7 day default retention. Removed insert time from start time database. This feature is used to detect changes since last query, but is not functional since it forces a scan of all data entries. Could be added back at a later time.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 37s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 5m 23s The applied patch generated 6 additional checkstyle issues.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 8s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api.
          +1 yarn tests 3m 17s Tests passed in hadoop-yarn-server-applicationhistoryservice.
              45m 35s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727487/YARN-3448.12.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a100be6
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7465/artifact/patchprocess/checkstyle-result-diff.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7465/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7465/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7465/testReport/
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7465//console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 37s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 23s The applied patch generated 6 additional checkstyle issues. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 8s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api. +1 yarn tests 3m 17s Tests passed in hadoop-yarn-server-applicationhistoryservice.     45m 35s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727487/YARN-3448.12.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a100be6 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7465/artifact/patchprocess/checkstyle-result-diff.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7465/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7465/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7465/testReport/ Console output https://builds.apache.org/job/PreCommit-YARN-Build/7465//console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 21s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 8m 8s There were no new javac warning messages.
          +1 javadoc 9m 45s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 6m 1s The applied patch generated 3 additional checkstyle issues.
          +1 install 1m 39s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 12s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
          +1 yarn tests 3m 18s Tests passed in hadoop-yarn-server-applicationhistoryservice.
              50m 49s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728314/YARN-3448.13.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 618ba70
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7508/artifact/patchprocess/checkstyle-result-diff.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7508/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7508/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7508/testReport/
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7508/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 21s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 8m 8s There were no new javac warning messages. +1 javadoc 9m 45s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 6m 1s The applied patch generated 3 additional checkstyle issues. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 12s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. +1 yarn tests 3m 18s Tests passed in hadoop-yarn-server-applicationhistoryservice.     50m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728314/YARN-3448.13.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 618ba70 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7508/artifact/patchprocess/checkstyle-result-diff.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7508/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7508/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7508/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7508/console This message was automatically generated.
          Hide
          mitdesai Mit Desai added a comment -

          Thanks for the patch Jonathan Eagles.

          Few minor comments on the patch v13

          Any reason why this is hard coded and not using DEFAULT_TIMELINE_SERVICE_TTL_ENABLE

          this.ttlEnabled = conf.getBoolean(
            YarnConfiguration.TIMELINE_SERVICE_TTL_ENABLE, true);
          

          The comment need to be fixed here. It should be 6 hours instead of 12

          case QUARTER_DAILY: {
            // round down to 12 hour interval
            int hour = (cal.get(Calendar.HOUR) / 6) * 6;
          

          Extra semicolon here

          // seek to the first start time entry
          iterator.seekToFirst();
          ;
          

          Is just a log message enough when we catch the exception? 'db' will be null if an exception is thrown

          DB db = null;
          try {
            db = factory.open(new File(rollingDBPath.toUri().getPath()), options);
          } catch (IOException ioe) {
            LOG.warn("Failed to open rolling leveldb instance :"
                + new File(rollingDBPath.toUri().getPath()), ioe);
          }
          rollingdbs.put(dbStartTime, db);
          String dbName = fdf.format(dbStartTime);
          

          The TreeMap rollingdbs is mostly used for iteration. Would be faster to use an ArrayList instead

            /** Collection of all active rolling leveldb instances. */
           private final TreeMap<Long, DB> rollingdbs;
          

          package-private would be more appropriate rather than marking private here

          @VisibleForTesting
          @Private
          public synchronized long getStartTimeFor(DB db) {
            long startTime = -1;
            for (Map.Entry<Long, DB> entry : rollingdbs.entrySet()) {
              if (entry.getValue() == db) {
                startTime = entry.getKey();
              }
            }
            return startTime;
          }
          
          Show
          mitdesai Mit Desai added a comment - Thanks for the patch Jonathan Eagles . Few minor comments on the patch v13 Any reason why this is hard coded and not using DEFAULT_TIMELINE_SERVICE_TTL_ENABLE this .ttlEnabled = conf.getBoolean( YarnConfiguration.TIMELINE_SERVICE_TTL_ENABLE, true ); The comment need to be fixed here. It should be 6 hours instead of 12 case QUARTER_DAILY: { // round down to 12 hour interval int hour = (cal.get(Calendar.HOUR) / 6) * 6; Extra semicolon here // seek to the first start time entry iterator.seekToFirst(); ; Is just a log message enough when we catch the exception? 'db' will be null if an exception is thrown DB db = null ; try { db = factory.open( new File(rollingDBPath.toUri().getPath()), options); } catch (IOException ioe) { LOG.warn( "Failed to open rolling leveldb instance :" + new File(rollingDBPath.toUri().getPath()), ioe); } rollingdbs.put(dbStartTime, db); String dbName = fdf.format(dbStartTime); The TreeMap rollingdbs is mostly used for iteration. Would be faster to use an ArrayList instead /** Collection of all active rolling leveldb instances. */ private final TreeMap< Long , DB> rollingdbs; package-private would be more appropriate rather than marking private here @VisibleForTesting @Private public synchronized long getStartTimeFor(DB db) { long startTime = -1; for (Map.Entry< Long , DB> entry : rollingdbs.entrySet()) { if (entry.getValue() == db) { startTime = entry.getKey(); } } return startTime; }
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 47s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 40s There were no new checkstyle issues.
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 51s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 12s Pre-build of native portion
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
          -1 yarn tests 3m 1s Tests failed in hadoop-yarn-server-applicationhistoryservice.
          +1 hdfs tests 0m 15s Tests passed in hadoop-hdfs-client.
              45m 59s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730269/YARN-3448.14.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bf70c5a
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7690/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7690/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-YARN-Build/7690/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7690/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7690/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 47s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 40s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 51s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 12s Pre-build of native portion +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. -1 yarn tests 3m 1s Tests failed in hadoop-yarn-server-applicationhistoryservice. +1 hdfs tests 0m 15s Tests passed in hadoop-hdfs-client.     45m 59s   Reason Tests Failed unit tests hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730269/YARN-3448.14.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bf70c5a hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7690/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7690/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-YARN-Build/7690/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7690/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7690/console This message was automatically generated.
          Hide
          jeagles Jonathan Eagles added a comment -

          Ported YARN-3530 to RollingLevelDBTimelineStore as part of patch 15

          Show
          jeagles Jonathan Eagles added a comment - Ported YARN-3530 to RollingLevelDBTimelineStore as part of patch 15
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 37s There were no new checkstyle issues.
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 51s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 15s Pre-build of native portion
          +1 yarn tests 0m 31s Tests passed in hadoop-yarn-api.
          +1 yarn tests 3m 8s Tests passed in hadoop-yarn-server-applicationhistoryservice.
          +1 hdfs tests 0m 15s Tests passed in hadoop-hdfs-client.
              46m 1s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730372/YARN-3448.15.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 318081c
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7702/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7702/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-YARN-Build/7702/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7702/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7702/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 37s There were no new checkstyle issues. +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 51s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 15s Pre-build of native portion +1 yarn tests 0m 31s Tests passed in hadoop-yarn-api. +1 yarn tests 3m 8s Tests passed in hadoop-yarn-server-applicationhistoryservice. +1 hdfs tests 0m 15s Tests passed in hadoop-hdfs-client.     46m 1s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730372/YARN-3448.15.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 318081c hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7702/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7702/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-YARN-Build/7702/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7702/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7702/console This message was automatically generated.
          Hide
          jeagles Jonathan Eagles added a comment -

          Addressing Mit Desai's comments. There is no default ttl enabled and it will be better to have in an array, but it will be a challenge to rewrite and the main bottleneck is still db writes so it won't affect overall performance.

          Show
          jeagles Jonathan Eagles added a comment - Addressing Mit Desai's comments. There is no default ttl enabled and it will be better to have in an array, but it will be a challenge to rewrite and the main bottleneck is still db writes so it won't affect overall performance.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 40s There were no new checkstyle issues.
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 50s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 14s Pre-build of native portion
          +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api.
          +1 yarn tests 3m 26s Tests passed in hadoop-yarn-server-applicationhistoryservice.
          +1 hdfs tests 0m 15s Tests passed in hadoop-hdfs-client.
              46m 17s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730542/YARN-3448.16.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 9b01f81
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7710/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7710/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-YARN-Build/7710/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7710/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7710/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 40s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 50s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api. +1 yarn tests 3m 26s Tests passed in hadoop-yarn-server-applicationhistoryservice. +1 hdfs tests 0m 15s Tests passed in hadoop-hdfs-client.     46m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730542/YARN-3448.16.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9b01f81 hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7710/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7710/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-YARN-Build/7710/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7710/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7710/console This message was automatically generated.
          Hide
          jeagles Jonathan Eagles added a comment -

          Zhijie Shen, Jason Lowe, Can you have another look now that I have gotten my hadoopqa +1?

          Show
          jeagles Jonathan Eagles added a comment - Zhijie Shen , Jason Lowe , Can you have another look now that I have gotten my hadoopqa +1?
          Hide
          zjshen Zhijie Shen added a comment -

          Sure I'll try to get back to you by tomorrow.

          Show
          zjshen Zhijie Shen added a comment - Sure I'll try to get back to you by tomorrow.
          Hide
          zjshen Zhijie Shen added a comment -

          Jonathan, the patch looks good to me overall. I tried it locally and everything seems to work fine so far. Just some minor comments:

          1. Why do we need to change hadoop-hdfs-client/pom.xml?

          2. Why do we need fstConf?

          184	      <groupId>de.ruedigermoeller</groupId>
          185	      <artifactId>fst</artifactId>
          

          3. We import * in RollingLevelDBTimelineStore.

          4. Move it into serviceInit phase?

          309	    if (conf.getBoolean(TIMELINE_SERVICE_TTL_ENABLE, true)) {
          310	      deletionThread = new EntityDeletionThread(conf);
          311	      deletionThread.start();
          312	    }
          

          5. I'm wondering it will have some compatibility problem, as previous user don't know they need to handle this error code. On the other side, we will state ATSv1 stable in YARN-3539. Thoughts?

          136	    public static final int EXPIRED_ENTITY = 7;
          
          Show
          zjshen Zhijie Shen added a comment - Jonathan, the patch looks good to me overall. I tried it locally and everything seems to work fine so far. Just some minor comments: 1. Why do we need to change hadoop-hdfs-client/pom.xml ? 2. Why do we need fstConf? 184 <groupId>de.ruedigermoeller</groupId> 185 <artifactId>fst</artifactId> 3. We import * in RollingLevelDBTimelineStore. 4. Move it into serviceInit phase? 309 if (conf.getBoolean(TIMELINE_SERVICE_TTL_ENABLE, true )) { 310 deletionThread = new EntityDeletionThread(conf); 311 deletionThread.start(); 312 } 5. I'm wondering it will have some compatibility problem, as previous user don't know they need to handle this error code. On the other side, we will state ATSv1 stable in YARN-3539 . Thoughts? 136 public static final int EXPIRED_ENTITY = 7;
          Hide
          jeagles Jonathan Eagles added a comment -

          Thanks for taking another look, Zhijie Shen. I have a few question below, can you help me to address these?

          1. Why do we need to change hadoop-hdfs-client/pom.xml?

          removed. that was part of another change I am working on HADOOP-11883.

          2. Why do we need fstConf?
          184 <groupId>de.ruedigermoeller</groupId>
          185 <artifactId>fst</artifactId>

          This is a new dependency introduced for serializing/deserializing otherInfos and primaryValues into the db.

          3. We import * in RollingLevelDBTimelineStore.

          Expanded these imports

          4. Move it into serviceInit phase?
          309	    if (conf.getBoolean(TIMELINE_SERVICE_TTL_ENABLE, true)) {
          310	      deletionThread = new EntityDeletionThread(conf);
          311	      deletionThread.start();
          312	    }
          

          Can you clarify this comment? I can see this making sense in the serviceStart phase.

          5. I'm wondering it will have some compatibility problem, as previous user don't know they need to handle this error code. On the other side, we will state ATSv1 stable in YARN-3539. Thoughts?
          136 public static final int EXPIRED_ENTITY = 7;

          Can you make a recommendation here? I can see this being an ever increasing list of errors that client are free to log.

          Show
          jeagles Jonathan Eagles added a comment - Thanks for taking another look, Zhijie Shen . I have a few question below, can you help me to address these? 1. Why do we need to change hadoop-hdfs-client/pom.xml? removed. that was part of another change I am working on HADOOP-11883 . 2. Why do we need fstConf? 184 <groupId>de.ruedigermoeller</groupId> 185 <artifactId>fst</artifactId> This is a new dependency introduced for serializing/deserializing otherInfos and primaryValues into the db. 3. We import * in RollingLevelDBTimelineStore. Expanded these imports 4. Move it into serviceInit phase? 309 if (conf.getBoolean(TIMELINE_SERVICE_TTL_ENABLE, true )) { 310 deletionThread = new EntityDeletionThread(conf); 311 deletionThread.start(); 312 } Can you clarify this comment? I can see this making sense in the serviceStart phase. 5. I'm wondering it will have some compatibility problem, as previous user don't know they need to handle this error code. On the other side, we will state ATSv1 stable in YARN-3539 . Thoughts? 136 public static final int EXPIRED_ENTITY = 7; Can you make a recommendation here? I can see this being an ever increasing list of errors that client are free to log.
          Hide
          zjshen Zhijie Shen added a comment -

          This is a new dependency introduced for serializing/deserializing otherInfos and primaryValues into the db.

          I'm curious about the reason why we choose it over GenericObjectMapper, which we used to ser/des those fields in leveldb timeline store.

          Can you clarify this comment? I can see this making sense in the serviceStart phase.

          It's not something critical, but I think it's more elegant the service and its back ground thread don't start running until we call start of it.

          Can you make a recommendation here? I can see this being an ever increasing list of errors that client are free to log.

          I just thought it out loudly. This is the new error code generated by the new store implementation. It should affect the existing usage. If users want to move on with the new implementation, it is probably reasonable to have them handle the new error code. One problem is that YARN-3539 is targeting 2.7.1. Once that jira gets committed, we need to update the documentation again to declare this error code too (perhaps also mentioning this implementation).

          Show
          zjshen Zhijie Shen added a comment - This is a new dependency introduced for serializing/deserializing otherInfos and primaryValues into the db. I'm curious about the reason why we choose it over GenericObjectMapper, which we used to ser/des those fields in leveldb timeline store. Can you clarify this comment? I can see this making sense in the serviceStart phase. It's not something critical, but I think it's more elegant the service and its back ground thread don't start running until we call start of it. Can you make a recommendation here? I can see this being an ever increasing list of errors that client are free to log. I just thought it out loudly. This is the new error code generated by the new store implementation. It should affect the existing usage. If users want to move on with the new implementation, it is probably reasonable to have them handle the new error code. One problem is that YARN-3539 is targeting 2.7.1. Once that jira gets committed, we need to update the documentation again to declare this error code too (perhaps also mentioning this implementation).
          Hide
          jeagles Jonathan Eagles added a comment -

          I'm curious about the reason why we choose it over GenericObjectMapper, which we used to ser/des those fields in leveldb timeline store.

          GenericObjectMapper is quite slow consuming 7-10% of the processing time on my example data. In practice the otherInfo can be much larger and will consume even more time.

          It's not something critical, but I think it's more elegant the service and its back ground thread don't start running until we call start of it.

          Makes sense

          I just thought it out loudly. This is the new error code generated by the new store implementation. It should affect the existing usage. If users want to move on with the new implementation, it is probably reasonable to have them handle the new error code. One problem is that YARN-3539 is targeting 2.7.1. Once that jira gets committed, we need to update the documentation again to declare this error code too (perhaps also mentioning this implementation).

          Should I leave this new error in?

          Show
          jeagles Jonathan Eagles added a comment - I'm curious about the reason why we choose it over GenericObjectMapper, which we used to ser/des those fields in leveldb timeline store. GenericObjectMapper is quite slow consuming 7-10% of the processing time on my example data. In practice the otherInfo can be much larger and will consume even more time. It's not something critical, but I think it's more elegant the service and its back ground thread don't start running until we call start of it. Makes sense I just thought it out loudly. This is the new error code generated by the new store implementation. It should affect the existing usage. If users want to move on with the new implementation, it is probably reasonable to have them handle the new error code. One problem is that YARN-3539 is targeting 2.7.1. Once that jira gets committed, we need to update the documentation again to declare this error code too (perhaps also mentioning this implementation). Should I leave this new error in?
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 1s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 9m 51s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 24s There were no new checkstyle issues.
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 11s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 3m 9s Tests passed in hadoop-yarn-server-applicationhistoryservice.
              42m 27s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730980/YARN-3448.17.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b725078
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7742/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7742/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7742/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7742/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 9m 51s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 24s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 11s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 3m 9s Tests passed in hadoop-yarn-server-applicationhistoryservice.     42m 27s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730980/YARN-3448.17.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b725078 hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7742/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-applicationhistoryservice test log https://builds.apache.org/job/PreCommit-YARN-Build/7742/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7742/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7742/console This message was automatically generated.
          Hide
          zjshen Zhijie Shen added a comment -

          GenericObjectMapper is quite slow consuming 7-10% of the processing time on my example data.

          Thanks, Jonathan! It's useful information. We may want to exploit it in v2 backend implementation too.

          Should I leave this new error in?

          I'm fine with keeping it.

          +1 for the last patch. I'll hold the patch until tomorrow for commit, giving other folks the chance to take a look again.

          Show
          zjshen Zhijie Shen added a comment - GenericObjectMapper is quite slow consuming 7-10% of the processing time on my example data. Thanks, Jonathan! It's useful information. We may want to exploit it in v2 backend implementation too. Should I leave this new error in? I'm fine with keeping it. +1 for the last patch. I'll hold the patch until tomorrow for commit, giving other folks the chance to take a look again.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7759 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7759/)
          YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7759 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7759/ ) YARN-3448 . Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          zjshen Zhijie Shen added a comment -

          Committed the patch to trunk and branch-2. Thanks, Jonathan! And thanks for review, Jason and Mit!

          Show
          zjshen Zhijie Shen added a comment - Committed the patch to trunk and branch-2. Thanks, Jonathan! And thanks for review, Jason and Mit!
          Hide
          jeagles Jonathan Eagles added a comment -

          Thanks so much, everyone!

          Show
          jeagles Jonathan Eagles added a comment - Thanks so much, everyone!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #188 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/188/)
          YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #188 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/188/ ) YARN-3448 . Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #188 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/188/)
          YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #188 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/188/ ) YARN-3448 . Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #178 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/178/)
          YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #178 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/178/ ) YARN-3448 . Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/921/)
          YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/921/ ) YARN-3448 . Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2119 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2119/)
          YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2119 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2119/ ) YARN-3448 . Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2137 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2137/)
          YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2137 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2137/ ) YARN-3448 . Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles. (zjshen: rev daf3e4ef8bf73cbe4a799d51b4765809cd81089f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/util/LeveldbUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDB.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/timeline/TimelinePutResponse.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDB.java
          Hide
          guoshiwei Shiwei Guo added a comment -

          Jonathan Eagles, I noticed that in patch.12, have following changes compared to patch.10:
          1.

          RollingLevelDBTimelineStore#putEntities
            // write entity marker
            // ... ignore code to prepare markerKey
            byte[] markerValue = writeReverseOrderedLong(startAndInsertTime
                .insertTime);
            writeBatch.put(markerKey, markerValue);
          

          is changed to

             // write entity marker
             // ... ignore code to prepare markerKey
             writeBatch.put(markerKey, EMPTY_BYTES);
          

          and 2.

          TestRollingLevelDBTimelineStore.java
            @Test
            public void testGetEntitiesWithFromTs() throws IOException {
              super.testGetEntitiesWithFromTs();
            }
          

          is changed to

            @Test
            public void testGetEntitiesWithFromTs() throws IOException {
              // feature not supported
            }
          

          What's the reason to make this change? Cause I found that if we keep the patch.10 version of these code, plus a small bug fix code(as following code), the testGetEntitiesWithFromTs test case can pass, so we can support the GetEntitiesWithFromTs feature. Maybe I'm missing some other things.

          BUGFIX code in getEntityByTime

          RollingLevelDBTimelineStore#getEntityByTime
          if (fromTs != null) {
                      long insertTime = readReverseOrderedLong(iterator.peekNext()
                          .getValue(), 0);
                      if (insertTime > fromTs) {
                        byte[] firstKey = key;
                        while (iterator.hasNext()) {
                          key = iterator.peekNext().getKey();
          
                       // BUGFIX code block
                         iterator.next();
                          if (!prefixMatches(firstKey, kp.getOffset(), key)) {
                            break;
                          }
                        }
                        continue;
                      }
                    }
          

          change to

          if (fromTs != null) {
                      long insertTime = readReverseOrderedLong(iterator.peekNext()
                          .getValue(), 0);
                      if (insertTime > fromTs) {
                        byte[] firstKey = key;
                        while (iterator.hasNext()) {
                          key = iterator.peekNext().getKey();
                         
                          // BUGFIX code block
                          if (!prefixMatches(firstKey, kp.getOffset(), key)) {
                            break;
                          }
                          iterator.next();
                        }
                        continue;
                      }
                    }
          
          Show
          guoshiwei Shiwei Guo added a comment - Jonathan Eagles , I noticed that in patch.12, have following changes compared to patch.10: 1. RollingLevelDBTimelineStore#putEntities // write entity marker // ... ignore code to prepare markerKey byte [] markerValue = writeReverseOrderedLong(startAndInsertTime .insertTime); writeBatch.put(markerKey, markerValue); is changed to // write entity marker // ... ignore code to prepare markerKey writeBatch.put(markerKey, EMPTY_BYTES); and 2. TestRollingLevelDBTimelineStore.java @Test public void testGetEntitiesWithFromTs() throws IOException { super .testGetEntitiesWithFromTs(); } is changed to @Test public void testGetEntitiesWithFromTs() throws IOException { // feature not supported } What's the reason to make this change? Cause I found that if we keep the patch.10 version of these code, plus a small bug fix code(as following code), the testGetEntitiesWithFromTs test case can pass, so we can support the GetEntitiesWithFromTs feature. Maybe I'm missing some other things. BUGFIX code in getEntityByTime RollingLevelDBTimelineStore#getEntityByTime if (fromTs != null ) { long insertTime = readReverseOrderedLong(iterator.peekNext() .getValue(), 0); if (insertTime > fromTs) { byte [] firstKey = key; while (iterator.hasNext()) { key = iterator.peekNext().getKey(); // BUGFIX code block iterator.next(); if (!prefixMatches(firstKey, kp.getOffset(), key)) { break ; } } continue ; } } change to if (fromTs != null ) { long insertTime = readReverseOrderedLong(iterator.peekNext() .getValue(), 0); if (insertTime > fromTs) { byte [] firstKey = key; while (iterator.hasNext()) { key = iterator.peekNext().getKey(); // BUGFIX code block if (!prefixMatches(firstKey, kp.getOffset(), key)) { break ; } iterator.next(); } continue ; } }
          Hide
          guoshiwei Shiwei Guo added a comment -

          I may found the answer that the insertTime is removed in patch.12. Will it be better to throw a exception when fromTs != null, cause this feature is not supported yet?

          Show
          guoshiwei Shiwei Guo added a comment - I may found the answer that the insertTime is removed in patch.12. Will it be better to throw a exception when fromTs != null, cause this feature is not supported yet?
          Hide
          jeagles Jonathan Eagles added a comment -

          The fromTs implementation does not work at scale due to design limitations. All matching entity types are stored in sorted order by start time. In order to ask the question fromTs you must do a full db scan to check matching entries. Entries would have to be sorted by insert time to make this query work efficiently. Throwing an exception that this functionality isn't support makes a lot of sense.

          Show
          jeagles Jonathan Eagles added a comment - The fromTs implementation does not work at scale due to design limitations. All matching entity types are stored in sorted order by start time. In order to ask the question fromTs you must do a full db scan to check matching entries. Entries would have to be sorted by insert time to make this query work efficiently. Throwing an exception that this functionality isn't support makes a lot of sense.
          Hide
          raviprak Ravi Prakash added a comment -

          Thanks for the awesome design and fix Jon! I've opened YARN-6311 for adding documentation for this store.

          Show
          raviprak Ravi Prakash added a comment - Thanks for the awesome design and fix Jon! I've opened YARN-6311 for adding documentation for this store.

            People

            • Assignee:
              jeagles Jonathan Eagles
              Reporter:
              jeagles Jonathan Eagles
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development