Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4531

[WebHCat] Collecting task logs to hdfs

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: HCatalog, WebHCat
    • Labels:
      None
    • Release Note:
      Hide
      In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, webhcat will create a $statusdir/logs directory after task finish. The attempts here include completed attempts and failed attempts. The layout for logs directory is:

      logs/$job_id (directory for $job_id)
      logs/$job_id/job.xml.html
      logs/$job_id/$attempt_id (directory for $attempt_id)
      logs/$job_id/$attempt_id/stderr
      logs/$job_id/$attempt_id/stdout
      logs/$job_id/$attempt_id/syslog

      NO PRECOMMIT TESTS
      Show
      In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, webhcat will create a $statusdir/logs directory after task finish. The attempts here include completed attempts and failed attempts. The layout for logs directory is: logs/$job_id (directory for $job_id) logs/$job_id/job.xml.html logs/$job_id/$attempt_id (directory for $attempt_id) logs/$job_id/$attempt_id/stderr logs/$job_id/$attempt_id/stdout logs/$job_id/$attempt_id/syslog NO PRECOMMIT TESTS

      Description

      It would be nice we collect task logs after job finish. This is similar to what Amazon EMR does.

        Attachments

        1. HIVE-4531-1.patch
          53 kB
          Jianyong Dai
        2. HIVE-4531-10.patch
          67 kB
          Jianyong Dai
        3. HIVE-4531-11.patch
          67 kB
          Jianyong Dai
        4. HIVE-4531-2.patch
          54 kB
          Jianyong Dai
        5. HIVE-4531-3.patch
          55 kB
          Jianyong Dai
        6. HIVE-4531-4.patch
          59 kB
          Jianyong Dai
        7. HIVE-4531-5.patch
          62 kB
          Jianyong Dai
        8. HIVE-4531-6.patch
          60 kB
          Jianyong Dai
        9. HIVE-4531-7.patch
          60 kB
          Jianyong Dai
        10. HIVE-4531-8.patch
          59 kB
          Jianyong Dai
        11. HIVE-4531-9.patch
          65 kB
          Jianyong Dai
        12. samplestatusdirwithlist.tar.gz
          109 kB
          Jianyong Dai

          Issue Links

            Activity

              People

              • Assignee:
                daijy Jianyong Dai
                Reporter:
                daijy Jianyong Dai
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: