Hive
  1. Hive
  2. HIVE-2738

NPE in ExprNodeGenericFuncEvaluator

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Here is the query:

      SELECT t.lid, '2011-12-12', s_map2json(s_maxmap(UNION_MAP(t.categoryCount), 100)) FROM ( SELECT theme_lid AS theme_lid, MAP(s_host(referer), COUNT( * )) AS categoryCount FROM PageViewEvent WHERE day >= '20130104' AND day <= '20130112' AND date_ >= '2012-01-04' AND date_ < '2012-01-13' AND lid IS NOT NULL GROUP BY lid, s_host(referer) ) t GROUP BY t.lid

      Removing the call s_map2json make it work but not by removing s_maxmap, but I don't understand what could be wrong with the implementation of my udf. And I don't know how to debug remote hadoop jobs.

      1. HIVE-2738-r1237763.patch
        2 kB
        Nicolas Lalevée
      2. 750c8966-6402-465a-b011-903469fe56da.xml
        30 kB
        Nicolas Lalevée
      3. MapToJsonUDF.java
        4 kB
        Nicolas Lalevée
      4. MapMaxUDF.java
        5 kB
        Nicolas Lalevée
      5. hive_job_logs.txt
        14 kB
        Nicolas Lalevée

        Activity

        Hide
        Nicolas Lalevée added a comment -

        Find attached the log of the hadoop Job, and the implementations of my custom udfs.

        Show
        Nicolas Lalevée added a comment - Find attached the log of the hadoop Job, and the implementations of my custom udfs.
        Hide
        Nicolas Lalevée added a comment -

        For what it's worth, here is the plan of the second hadoop job which is been executed. The two generic udf seems to be declared on ExprNodeGenericFuncDesc (l. 490-538)

        Show
        Nicolas Lalevée added a comment - For what it's worth, here is the plan of the second hadoop job which is been executed. The two generic udf seems to be declared on ExprNodeGenericFuncDesc (l. 490-538)
        Hide
        Nicolas Lalevée added a comment -

        Actually there was some classpath issues, the version of jackson use by hadoop and mine are conflicting. The result is that my genericUDF failed to be instanciated, and the XMLDecoder, rather than reporting the error, returns null. Hence the NPE.

        I suggest the XMLDecoder should always fail on error. See the attached patch.

        Show
        Nicolas Lalevée added a comment - Actually there was some classpath issues, the version of jackson use by hadoop and mine are conflicting. The result is that my genericUDF failed to be instanciated, and the XMLDecoder, rather than reporting the error, returns null. Hence the NPE. I suggest the XMLDecoder should always fail on error. See the attached patch.
        Hide
        Ashutosh Chauhan added a comment -

        +1 will commit if tests pass.

        Show
        Ashutosh Chauhan added a comment - +1 will commit if tests pass.
        Hide
        Ashutosh Chauhan added a comment -

        Patch fails ant test -Dtestcase=TestNegativeCliDriver -Dqfile=udfnull.q

        Show
        Ashutosh Chauhan added a comment - Patch fails ant test -Dtestcase=TestNegativeCliDriver -Dqfile=udfnull.q

          People

          • Assignee:
            Nicolas Lalevée
            Reporter:
            Nicolas Lalevée
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development