Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1685

HCatalog integration is not working on JDK8

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.5
    • Fix Version/s: 1.4.6
    • Component/s: None
    • Labels:
      None

      Description

      I was looking into JDK8 compatibility in Sqoop 1 and I found out that HCatalog tests are broken, failing on exceptions such as this one:

      14/11/05 16:29:06 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Projected column id not in list of columns from database
      	at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.initDBColumnInfo(SqoopHCatUtilities.java:483)
      	at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:314)
      	at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:753)
      

      Looking deeper into our HCatalog code, we do have class LCKeyMap that is suppose to store all it's keys in lower case form and provide case independent lookup. Indeed we're overriding methods put and get and lowercasing the key parameter to achieve desired behavior.

      Now to the actual problem. We are in the code calling method putAll to bulk insert values from another Map. This worked in JDK7 as the underlaying implementation of putAll method is calling put method that we're overriding. It seems that this has changed in JDK8 where the putAll method is no longer calling put and hence when we call putAll all the keys are stored with their original case and subsequent lookup fails

        Attachments

        1. SQOOP-1685.patch
          1 kB
          Jarek Jarcec Cecho

          Issue Links

            Activity

              People

              • Assignee:
                jarcec Jarek Jarcec Cecho
                Reporter:
                jarcec Jarek Jarcec Cecho
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: