Hive
  1. Hive
  2. HIVE-2738

NPE in ExprNodeGenericFuncEvaluator

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Here is the query:

      SELECT t.lid, '2011-12-12', s_map2json(s_maxmap(UNION_MAP(t.categoryCount), 100)) FROM ( SELECT theme_lid AS theme_lid, MAP(s_host(referer), COUNT( * )) AS categoryCount FROM PageViewEvent WHERE day >= '20130104' AND day <= '20130112' AND date_ >= '2012-01-04' AND date_ < '2012-01-13' AND lid IS NOT NULL GROUP BY lid, s_host(referer) ) t GROUP BY t.lid

      Removing the call s_map2json make it work but not by removing s_maxmap, but I don't understand what could be wrong with the implementation of my udf. And I don't know how to debug remote hadoop jobs.

      1. HIVE-2738-r1237763.patch
        2 kB
        Nicolas Lalevée
      2. 750c8966-6402-465a-b011-903469fe56da.xml
        30 kB
        Nicolas Lalevée
      3. MapToJsonUDF.java
        4 kB
        Nicolas Lalevée
      4. MapMaxUDF.java
        5 kB
        Nicolas Lalevée
      5. hive_job_logs.txt
        14 kB
        Nicolas Lalevée

        Activity

        Nicolas Lalevée created issue -
        Hide
        Nicolas Lalevée added a comment -

        Find attached the log of the hadoop Job, and the implementations of my custom udfs.

        Show
        Nicolas Lalevée added a comment - Find attached the log of the hadoop Job, and the implementations of my custom udfs.
        Nicolas Lalevée made changes -
        Field Original Value New Value
        Attachment hive_job_logs.txt [ 12511515 ]
        Attachment MapMaxUDF.java [ 12511516 ]
        Attachment MapToJsonUDF.java [ 12511517 ]
        Hide
        Nicolas Lalevée added a comment -

        For what it's worth, here is the plan of the second hadoop job which is been executed. The two generic udf seems to be declared on ExprNodeGenericFuncDesc (l. 490-538)

        Show
        Nicolas Lalevée added a comment - For what it's worth, here is the plan of the second hadoop job which is been executed. The two generic udf seems to be declared on ExprNodeGenericFuncDesc (l. 490-538)
        Nicolas Lalevée made changes -
        Hide
        Nicolas Lalevée added a comment -

        Actually there was some classpath issues, the version of jackson use by hadoop and mine are conflicting. The result is that my genericUDF failed to be instanciated, and the XMLDecoder, rather than reporting the error, returns null. Hence the NPE.

        I suggest the XMLDecoder should always fail on error. See the attached patch.

        Show
        Nicolas Lalevée added a comment - Actually there was some classpath issues, the version of jackson use by hadoop and mine are conflicting. The result is that my genericUDF failed to be instanciated, and the XMLDecoder, rather than reporting the error, returns null. Hence the NPE. I suggest the XMLDecoder should always fail on error. See the attached patch.
        Nicolas Lalevée made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Nicolas Lalevée made changes -
        Attachment HIVE-2738-r1237763.patch [ 12512429 ]
        Ashutosh Chauhan made changes -
        Assignee Nicolas Lalevée [ hibou ]
        Hide
        Ashutosh Chauhan added a comment -

        +1 will commit if tests pass.

        Show
        Ashutosh Chauhan added a comment - +1 will commit if tests pass.
        Hide
        Ashutosh Chauhan added a comment -

        Patch fails ant test -Dtestcase=TestNegativeCliDriver -Dqfile=udfnull.q

        Show
        Ashutosh Chauhan added a comment - Patch fails ant test -Dtestcase=TestNegativeCliDriver -Dqfile=udfnull.q
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        6d 23h 24m 1 Nicolas Lalevée 30/Jan/12 16:46
        Patch Available Patch Available Open Open
        63d 16h 21m 1 Ashutosh Chauhan 03/Apr/12 10:08

          People

          • Assignee:
            Nicolas Lalevée
            Reporter:
            Nicolas Lalevée
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development