Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2393

Allow table drop in hcat prepare

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 4.1.0, 4.2.0
    • Fix Version/s: 5.0.0b1
    • Component/s: core
    • Labels:

      Description

      The hcat prepare only allows to drop partitions. It would be nice to also allow dropping of table, depending on the URL. The current format of the URL is

      hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]

      where at least one partition must be provided, otherwise the prepare step fails with the fololwing exception.

      Starting the execution of prepare actions
      Creating HCatClient for user=ehsan.haq (auth:SIMPLE) and server=thrift://datavault-prod-app2.internal.machines:9083
      Prepare execution in the Launcher Mapper has failed
      Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exception invoking main(), Error trying to drop hcat://datavault-prod-app2.internal.machines:9083/test_rdbms_import_2015110600/test
      org.apache.oozie.action.hadoop.LauncherException: Error trying to drop hcat://datavault-prod-app2.internal.machines:9083/test_rdbms_import_2015110600/test
      at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:178)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      Caused by: org.apache.oozie.action.hadoop.LauncherException: Error trying to drop hcat://datavault-prod-app2.internal.machines:9083/test_rdbms_import_2015110600/test
      at org.apache.oozie.action.hadoop.HCatLauncherURIHandler.delete(HCatLauncherURIHandler.java:64)
      at org.apache.oozie.action.hadoop.PrepareActionsDriver.execute(PrepareActionsDriver.java:89)
      at org.apache.oozie.action.hadoop.PrepareActionsDriver.doOperations(PrepareActionsDriver.java:67)
      at org.apache.oozie.action.hadoop.LauncherMapper.executePrepare(LauncherMapper.java:446)
      at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:174)
      ... 8 more
      Caused by: java.net.URISyntaxException: URI path is not in expected format: hcat://datavault-prod-app2.internal.machines:9083/test_rdbms_import_2015110600/test
      at org.apache.oozie.util.HCatURI.parse(HCatURI.java:66)
      at org.apache.oozie.util.HCatURI.<init>(HCatURI.java:52)
      at org.apache.oozie.util.HCatURI.<init>(HCatURI.java:48)
      at org.apache.oozie.action.hadoop.HCatLauncherURIHandler.delete(HCatLauncherURIHandler.java:52)
      ... 12 more
      
      Oozie Launcher failed, finishing Hadoop job gracefully
      

      Suggestion

      In the URL if the partition is not provided then it should delete the entire table.

        Attachments

        1. OOZIE-2393-00.patch
          14 kB
          Abhishek Bafna
        2. OOZIE-2393-01.patch
          30 kB
          Abhishek Bafna

          Issue Links

            Activity

              People

              • Assignee:
                abhishekbafna Abhishek Bafna
                Reporter:
                ehsan Muhammad Ehsan ul Haque
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: