Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1235

Speed-up init-hdfs.sh for complete HCFS compliant provisioning

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 0.7.0
    • 1.0.0
    • deployment, general
    • None

    Description

      BIGTOP-1200 provides file schema in JSON format, for any HCFS related tool to read.

      BIGTOP-952 meanwhile give us fast provisioning by running in a single JVM (without multiple haoop fs -... commands_).

      Thus, when those two are finished, we can replace init-hdfs.sh (with the exception of a few extra shell commands in bottom of that script which still will need to be called).

      So this JIRA will:

      • create a new init-hcfs.groovy script which calls provision.groovy in BIGTOP-952, with init-hcfs.json as input.
      • add in the remaining straggler commands from init-hdfs.sh which dont fit into the generic json format as shell commands.
      • delete init-hdfs.sh entirely
      • update puppet scripts to use init-hcfs.groovy, calling them with the "hdfs" argument.

      Attachments

        1. BIGTOP-1235.first.patch
          23 kB
          jay vyas
        2. BIGTOP-1235.patch
          26 kB
          jay vyas
        3. 0001-BIGTOP-1235.-Combine-BIGTOP-1200-and-BIGTOP-952-to-s.patch
          31 kB
          Konstantin I Boudnik
        4. 0001-BIGTOP-1235.-Combine-BIGTOP-1200-and-BIGTOP-952-to-s.patch
          31 kB
          Konstantin I Boudnik
        5. 0001-BIGTOP-1235.-Combine-BIGTOP-1200-and-BIGTOP-952-to-s.patch
          32 kB
          Konstantin I Boudnik
        6. 0001-BIGTOP-1235.-Combine-BIGTOP-1200-and-BIGTOP-952-to-s.patch
          32 kB
          Konstantin I Boudnik
        7. 0001-BIGTOP-1235.-Combine-BIGTOP-1200-and-BIGTOP-952-to-s.patch
          32 kB
          Konstantin I Boudnik
        8. 0001-BIGTOP-1235.-Combine-BIGTOP-1200-and-BIGTOP-952-to-s.patch
          32 kB
          Konstantin I Boudnik
        9. 0001-BIGTOP-1235.-Combine-BIGTOP-1200-and-BIGTOP-952-to-s.patch
          33 kB
          Konstantin I Boudnik
        10. 0001-BIGTOP-1235.-Speed-up-init-hdfs.sh-for-complete-HCFS.patch
          33 kB
          Konstantin I Boudnik

        Issue Links

          Activity

            People

              cos Konstantin I Boudnik
              jayunit100 jay vyas
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: