Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-645

Ensure Crunch works with Hadoop 3.0.0-alpha2

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Make sure crunch works Hadoop 3.

      To this end, Apache Hadoop 3.0.0-alpha2 could be used as a first step.

      1. CRUNCH-645_v1.patch
        0.5 kB
        Gergő Pásztor

        Issue Links

          Activity

          Hide
          pairg Gergő Pásztor added a comment - - edited

          For the first try I found just one issue: FileUtil.copyMerge in crunch-spark is deprecated.

          Show
          pairg Gergő Pásztor added a comment - - edited For the first try I found just one issue: FileUtil.copyMerge in crunch-spark is deprecated.
          Hide
          pairg Gergő Pásztor added a comment -

          Newer Hive and HBase needed for this. HBase (and Spark) is working with the HBase 2 patch in CRUNCH-619. Hive is in progress...

          Show
          pairg Gergő Pásztor added a comment - Newer Hive and HBase needed for this. HBase (and Spark) is working with the HBase 2 patch in CRUNCH-619 . Hive is in progress...
          Hide
          pairg Gergő Pásztor added a comment -

          Hive current version is not support Hadoop 3 yet and we don't have even a snapshot where it is supporting, so I think I will create a Jira for this for future works.

          Show
          pairg Gergő Pásztor added a comment - Hive current version is not support Hadoop 3 yet and we don't have even a snapshot where it is supporting, so I think I will create a Jira for this for future works.
          Hide
          pairg Gergő Pásztor added a comment -

          I uploaded a patch here that is updating the hadoop version in the root pom, so we can apply this patch when everything else is commited for this. The uploaded patch is based on "CRUNCH-619_v5_hbase2.patch".

          Show
          pairg Gergő Pásztor added a comment - I uploaded a patch here that is updating the hadoop version in the root pom, so we can apply this patch when everything else is commited for this. The uploaded patch is based on " CRUNCH-619 _v5_hbase2.patch".
          Hide
          asasvari Attila Sasvari added a comment -

          Thanks Gergő Pásztor. It looks good to me. We need to wait until someone solves the Hive issue Gergő mentioned above (CRUNCH-648).

          Show
          asasvari Attila Sasvari added a comment - Thanks Gergő Pásztor . It looks good to me. We need to wait until someone solves the Hive issue Gergő mentioned above ( CRUNCH-648 ).

            People

            • Assignee:
              pairg Gergő Pásztor
              Reporter:
              asasvari Attila Sasvari
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development