Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-952

init-hdfs.sh is dog slow. Let's replace it with a direct HDFS API calls and better layout management

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 0.5.0
    • 0.8.0
    • deployment
    • None

    Description

      As has been proposed in this patch by rvs there's a very efficient way of creating layout in HDFS using a tarfile and Groovy script with direct call into DFS APIs.

      Let's making it happen.

      Attachments

        1. BIGTOP-952.patch
          13 kB
          jay vyas
        2. BIGTOP-952.patch
          13 kB
          jay vyas
        3. BIGTOP-952-tested.patch
          5 kB
          jay vyas
        4. BIGTOP-952-tested-refined.patch
          6 kB
          jay vyas
        5. provision2.groovy
          1 kB
          jay vyas
        6. untar.groovy
          2 kB
          Konstantin I Boudnik

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. BIGTOP-1097 introduce bigtop-groovy package

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BIGTOP-1200 Implement Generic Text File to define HCFS filesystem semantics

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue BIGTOP-852 Improve HDFS helper script

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. BIGTOP-922 Existing Puppet recipes are failing to initialize HDFS dictories on the first run

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BIGTOP-1235 Speed-up init-hdfs.sh for complete HCFS compliant provisioning

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. BIGTOP-1255 init-hcfs.json needs to encode privileges as strings, not numbers.

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              jayunit100 jay vyas
              cos Konstantin I Boudnik
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: