Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-2641

Livy interpreter doesn't support Chinese encoding

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.7.1, 0.8.0
    • Fix Version/s: 0.8.0, 0.7.4
    • Component/s: livy-interpreter
    • Labels:
      None

      Description

      When using Livy to interact with spark. Spark sql with Chinese character always return empty result.

      The reason is because Zeppelin use restTemplate.java from spring frame work. When sending restful request to Livy, the headers is defined as Content-Type:application/json. This setting doesn't specify the encoding of the charset and spring will use ISO-8895-1 by default. But ISO-8895-1 isn't compatible with Chinese. The code Livy receive from Zeppelin contain invalid Chinese encoding and will be interpret by spark as ?.

      Change header from application/json to MediaType.APPLICATION_JSON_UTF8_VALUE in BaseLivyInterpreter.java and rebuild can solve this problem.

      It seems encoding problem affect more than Chinese since ISO-8895-1's coverage is restricted. I think update the encoding to UTF-8 would be helpful.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                why heyang wang
                Reporter:
                why heyang wang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: