Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-4260

Document usage for hcatalog 1.1

    XMLWordPrintableJSON

    Details

      Description

      The HCatalogIO does not work with environments providing Hive Server 1.x which is in widespread use - as an example the latest Cloudera (5.14.2) provides 1.1.x

      The HCatalogIO marks it's Hive dependencies as provided, so I believe the intention was to be open to multiple versions.

      The issues come from the following:

      • use of HCatUtil.getHiveMetastoreClient(hiveConf) while previous versions used the now deprecated getHiveClient(HiveConf hiveConf)
      • Changes to the signature of RetryingMetaStoreClient.getProxy(...)

      Given this doesn't work in a major Hadoop distro, and will not until the next CDH release later in 2018 (i.e. widespread adoption only expected in 2019) I think it would be worthwhile providing a fix/workaround.

      I think building for 2.3 and relocating in your own app might be a workaround although I'm still testing it. If that is successful I'd propose adding it to the project README or in a separate markdown file linked from the README.

      Does that sound like a reasonable approach please?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                timrobertson100 Tim Robertson
                Reporter:
                timrobertson100 Tim Robertson
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h
                  3h