Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3499

Backend cannot write catalog-update topic more than 2GB to jByteArray

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      I've encountered a issue after impalad restarts. Logging shows:

      E0509 11:44:08.168478 72547 impala-server.cc:1325] There was an error processing the impalad catalog update. Requesting a full topic update to recover: NegativeArraySizeException: null

      This may be related to size of catalog-update topic in statestored exceeding 2GB.
      In updateCatalogCache, serialized data passed through JNI are represented as byte[], which cannot exceed that limit.

      When I restart catalogd (that is, reset catalog-update topic), everything goes fine.

      I'm assuming that metadata larger than 2GB can be a common case in massive scale usage.
      Is there any possible approach to fix this?
      Thanks.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              HuaisiXu Huaisi Xu
              Reporter:
              He Tianyi He Tianyi

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment