Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4531

[WebHCat] Collecting task logs to hdfs

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.12.0
    • HCatalog, WebHCat
    • None
    • Hide
      In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, webhcat will create a $statusdir/logs directory after task finish. The attempts here include completed attempts and failed attempts. The layout for logs directory is:

      logs/$job_id (directory for $job_id)
      logs/$job_id/job.xml.html
      logs/$job_id/$attempt_id (directory for $attempt_id)
      logs/$job_id/$attempt_id/stderr
      logs/$job_id/$attempt_id/stdout
      logs/$job_id/$attempt_id/syslog

      NO PRECOMMIT TESTS
      Show
      In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, webhcat will create a $statusdir/logs directory after task finish. The attempts here include completed attempts and failed attempts. The layout for logs directory is: logs/$job_id (directory for $job_id) logs/$job_id/job.xml.html logs/$job_id/$attempt_id (directory for $attempt_id) logs/$job_id/$attempt_id/stderr logs/$job_id/$attempt_id/stdout logs/$job_id/$attempt_id/syslog NO PRECOMMIT TESTS

    Description

      It would be nice we collect task logs after job finish. This is similar to what Amazon EMR does.

      Attachments

        1. samplestatusdirwithlist.tar.gz
          109 kB
          Daniel Dai
        2. HIVE-4531-9.patch
          65 kB
          Daniel Dai
        3. HIVE-4531-8.patch
          59 kB
          Daniel Dai
        4. HIVE-4531-7.patch
          60 kB
          Daniel Dai
        5. HIVE-4531-6.patch
          60 kB
          Daniel Dai
        6. HIVE-4531-5.patch
          62 kB
          Daniel Dai
        7. HIVE-4531-4.patch
          59 kB
          Daniel Dai
        8. HIVE-4531-3.patch
          55 kB
          Daniel Dai
        9. HIVE-4531-2.patch
          54 kB
          Daniel Dai
        10. HIVE-4531-11.patch
          67 kB
          Daniel Dai
        11. HIVE-4531-10.patch
          67 kB
          Daniel Dai
        12. HIVE-4531-1.patch
          53 kB
          Daniel Dai

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6671 WebHCat Job Submission API 'enablelog' parameter is only supported with Hadoop 1

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HIVE-6063 [WebHCat] Collect log for Hadoop 2

          • Major - Major loss of function.
          • Open

          Activity

            People

              daijy Daniel Dai
              daijy Daniel Dai
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: