Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: HCatalog, Metastore, WebHCat
    • Labels:
      None
    1. HIVE-4196.v2.patch
      2.51 MB
      Roshan Naik
    2. HIVE-5138.v1.patch
      2.52 MB
      Roshan Naik

      Activity

      Hide
      Roshan Naik added a comment -

      We should try to eliminate the need of intermediate staging area while rolling on new partitions. Seems like there should not be any gotchas while moving data from streaming dir to partition dir directly.

      Thanks. That change is already part of the patch.

      We should make thrift apis in metastore forward compatible. One way to do that is to use struct (which contains all parameters) instead of passing in list of arguments.

      Hate it .. but Ok.

      We should try to leave TBLS table untouched in backend db.

      Sure. Will move them to a new table.

      Show
      Roshan Naik added a comment - We should try to eliminate the need of intermediate staging area while rolling on new partitions. Seems like there should not be any gotchas while moving data from streaming dir to partition dir directly. Thanks. That change is already part of the patch. We should make thrift apis in metastore forward compatible. One way to do that is to use struct (which contains all parameters) instead of passing in list of arguments. Hate it .. but Ok. We should try to leave TBLS table untouched in backend db. Sure. Will move them to a new table.
      Hide
      Roshan Naik added a comment -

      Capturing API related comments from Ashutosh Chauhan noted here in HIVE-4196

      We should try to eliminate the need of intermediate staging area while rolling on new partitions. Seems like there should not be any gotchas while moving data from streaming dir to partition dir directly.
      We should make thrift apis in metastore forward compatible. One way to do that is to use struct (which contains all parameters) instead of passing in list of arguments.
      We should try to leave TBLS table untouched in backend db. That will simplify upgrade story. One way to do that is to have all new columns in a new table and than add constraints for this new table.

      Show
      Roshan Naik added a comment - Capturing API related comments from Ashutosh Chauhan noted here in HIVE-4196 We should try to eliminate the need of intermediate staging area while rolling on new partitions. Seems like there should not be any gotchas while moving data from streaming dir to partition dir directly. We should make thrift apis in metastore forward compatible. One way to do that is to use struct (which contains all parameters) instead of passing in list of arguments. We should try to leave TBLS table untouched in backend db. That will simplify upgrade story. One way to do that is to have all new columns in a new table and than add constraints for this new table.
      Hide
      Roshan Naik added a comment -

      Patch address comments from Eugene, additional unit test, some additional checks in partitionRoll for better error reporting.

      Show
      Roshan Naik added a comment - Patch address comments from Eugene, additional unit test, some additional checks in partitionRoll for better error reporting.
      Hide
      Eugene Koifman added a comment -

      OK, makes sense. It would be useful to add some javadoc about concurrency (or rather why it's not an issue)

      Show
      Eugene Koifman added a comment - OK, makes sense. It would be useful to add some javadoc about concurrency (or rather why it's not an issue)
      Hide
      Roshan Naik added a comment -

      Thanks Eugene Koifman for the comments:

      On Pt 1.

      Thanks. I need to take a closer look at this.

      On Pt 2.

      I think you mean 'safe to invoke concurrently' instead of 'atomic', since the intermediate states are going to be visible when an operation spans both file system and meta store. Here is a summary of the reasons why each operation is concurrency safe:

      • streamingStatus : Readonly metastore operation
      • chunkGet : This is an atomic metastore operation
      • chunkAbort : Just deletes a file. So no concurrency issues here.
      • chunkCommit : Just renames a file. So only one of concurrent operations will succeed.
      • disableStreaming : This is an atomic metastore operation
      • enableStreaming : Does a couple of mkdirs (for setup) followed by an atomic metastore operation. mkdirs() is idempotent, so all concurrent calls succeed. All concurrent invocations enter a transaction to do the metastore update atomically...only one should update metastore.
      • partitionRoll : Creates empty dir for the new current partition & then atomically updates metastore as follows:
        1. Make note of this new current partition dir
        2. Do an addPartition() on the previous current partition.
      • If concurrent partitionRoll() invocations use same arguments, the addPartition() step will fail on all but one. If arguments are not same in concurrent invocations, they all succeed and updates made by the last invocation to exit the metastore transaction would override the others.
      Show
      Roshan Naik added a comment - Thanks Eugene Koifman for the comments: On Pt 1. Thanks. I need to take a closer look at this. On Pt 2. I think you mean 'safe to invoke concurrently' instead of 'atomic', since the intermediate states are going to be visible when an operation spans both file system and meta store. Here is a summary of the reasons why each operation is concurrency safe: streamingStatus : Readonly metastore operation chunkGet : This is an atomic metastore operation chunkAbort : Just deletes a file. So no concurrency issues here. chunkCommit : Just renames a file. So only one of concurrent operations will succeed. disableStreaming : This is an atomic metastore operation enableStreaming : Does a couple of mkdirs (for setup) followed by an atomic metastore operation. mkdirs() is idempotent, so all concurrent calls succeed. All concurrent invocations enter a transaction to do the metastore update atomically...only one should update metastore. partitionRoll : Creates empty dir for the new current partition & then atomically updates metastore as follows: Make note of this new current partition dir Do an addPartition() on the previous current partition. If concurrent partitionRoll() invocations use same arguments, the addPartition() step will fail on all but one. If arguments are not same in concurrent invocations, they all succeed and updates made by the last invocation to exit the metastore transaction would override the others.
      Hide
      Eugene Koifman added a comment -

      Roshan Naik A couple of comments on this patch:
      1. All delegators in WebHCat take the 'user' as determined by Server.java and use that to make secure calls to JobTrakcer, HDFS etc. HCatStreamingDelegator ignores it. Why is that?
      2. Most operations in HCatStreamingDelegator do multiple things (like modify metadata, create some HDFS file, etc.). It sounds like every one of these operations should be atomic. For example, say for some reason 2 identical calls to partitionRoll() happen at the same time. How is this atomicity achieved?

      Show
      Eugene Koifman added a comment - Roshan Naik A couple of comments on this patch: 1. All delegators in WebHCat take the 'user' as determined by Server.java and use that to make secure calls to JobTrakcer, HDFS etc. HCatStreamingDelegator ignores it. Why is that? 2. Most operations in HCatStreamingDelegator do multiple things (like modify metadata, create some HDFS file, etc.). It sounds like every one of these operations should be atomic. For example, say for some reason 2 identical calls to partitionRoll() happen at the same time. How is this atomicity achieved?
      Show
      Roshan Naik added a comment - Patch v2 addresses the review comments from https://issues.apache.org/jira/browse/HIVE-4196?focusedCommentId=13714235&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13714235
      Hide
      Roshan Naik added a comment -

      Patch v2 is based on git commit version 9e9e711

      Show
      Roshan Naik added a comment - Patch v2 is based on git commit version 9e9e711
      Hide
      Roshan Naik added a comment -

      Implement Webhcat API to:

      1) Enable and Disable streaming on a table

      2) Check streaming status

      3) Transaction Support:

      • Get a Chunk File
      • Commit a Chunk File
      • Abort the chunk

      4) Roll Partition: To roll the committed chunks from streaming partition to a new standard partition

      Show
      Roshan Naik added a comment - Implement Webhcat API to: 1) Enable and Disable streaming on a table 2) Check streaming status 3) Transaction Support: Get a Chunk File Commit a Chunk File Abort the chunk 4) Roll Partition: To roll the committed chunks from streaming partition to a new standard partition

        People

        • Assignee:
          Roshan Naik
          Reporter:
          Roshan Naik
        • Votes:
          0 Vote for this issue
          Watchers:
          4 Start watching this issue

          Dates

          • Created:
            Updated:

            Development