Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4531

[WebHCat] Collecting task logs to hdfs

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.12.0
    • HCatalog, WebHCat
    • None
    • Hide
      In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, webhcat will create a $statusdir/logs directory after task finish. The attempts here include completed attempts and failed attempts. The layout for logs directory is:

      logs/$job_id (directory for $job_id)
      logs/$job_id/job.xml.html
      logs/$job_id/$attempt_id (directory for $attempt_id)
      logs/$job_id/$attempt_id/stderr
      logs/$job_id/$attempt_id/stdout
      logs/$job_id/$attempt_id/syslog

      NO PRECOMMIT TESTS
      Show
      In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, webhcat will create a $statusdir/logs directory after task finish. The attempts here include completed attempts and failed attempts. The layout for logs directory is: logs/$job_id (directory for $job_id) logs/$job_id/job.xml.html logs/$job_id/$attempt_id (directory for $attempt_id) logs/$job_id/$attempt_id/stderr logs/$job_id/$attempt_id/stdout logs/$job_id/$attempt_id/syslog NO PRECOMMIT TESTS

    Description

      It would be nice we collect task logs after job finish. This is similar to what Amazon EMR does.

      Attachments

        1. HIVE-4531-1.patch
          53 kB
          Daniel Dai
        2. HIVE-4531-2.patch
          54 kB
          Daniel Dai
        3. HIVE-4531-3.patch
          55 kB
          Daniel Dai
        4. HIVE-4531-4.patch
          59 kB
          Daniel Dai
        5. HIVE-4531-5.patch
          62 kB
          Daniel Dai
        6. samplestatusdirwithlist.tar.gz
          109 kB
          Daniel Dai
        7. HIVE-4531-6.patch
          60 kB
          Daniel Dai
        8. HIVE-4531-7.patch
          60 kB
          Daniel Dai
        9. HIVE-4531-8.patch
          59 kB
          Daniel Dai
        10. HIVE-4531-9.patch
          65 kB
          Daniel Dai
        11. HIVE-4531-10.patch
          67 kB
          Daniel Dai
        12. HIVE-4531-11.patch
          67 kB
          Daniel Dai

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6671 WebHCat Job Submission API 'enablelog' parameter is only supported with Hadoop 1

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HIVE-6063 [WebHCat] Collect log for Hadoop 2

          • Major - Major loss of function.
          • Open

          Activity

            People

              daijy Daniel Dai
              daijy Daniel Dai
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: