Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-3649

higher memory consumption on RS leading to OOM/abort on immutable index creation with multiple regions on single RS

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 4.9.0
    • 4.9.1, 4.10.0
    • None
    • None

    Description

      Configuration
      hbase-0.98.23 standalone
      Heap 5GB

      When
      Verified that this happens after PHOENIX-3271 Distribute UPSERT SELECT across cluster. https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=accd4a276d1085e5d1069caf93798d8f301e4ed6

      To repro

      CREATE TABLE INDEXED_TABLE (HOST CHAR(2) NOT NULL,DOMAIN VARCHAR NOT NULL, FEATURE VARCHAR NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB BIGINT,STATS.ACTIVE_VISITOR INTEGER CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, FEATURE, DATE)) IMMUTABLE_ROWS=true,MAX_FILESIZE=30485760
      

      Upsert 2M rows (CSV is available at https://goo.gl/OsTSKB) that will create ~4 regions on a single RS and then create index with data present

      CREATE INDEX idx5 ON INDEXED_TABLE (CORE) INCLUDE (DB,ACTIVE_VISITOR)
      

      From RS log

      2017-02-02 13:29:06,899 WARN  [rs,51371,1486070044538-HeapMemoryChore] regionserver.HeapMemoryManager: heapOccupancyPercent 0.97875696 is above heap occupancy alarm watermark (0.95)
      2017-02-02 13:29:18,198 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x15a00ad4f300001, timeout of 10000ms exceeded
      2017-02-02 13:29:18,231 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 10581ms
      GC pool 'ParNew' had collection(s): count=4 time=139ms
      2017-02-02 13:29:19,669 FATAL [RS:0;rs:51371-EventThread] regionserver.HRegionServer: ABORTING region server rs,51371,1486070044538: regionserver:51371-0x15a00ad4f300001, quorum=localhost:2181, baseZNode=/hbase regionserver:51371-0x15a00ad4f300001 received expired from ZooKeeper, aborting
      

      Prior to the change index creation succeeds with as little as 2GB heap.

      ankit@apache.org

      Attachments

        1. PHOENIX-3649.patch
          8 kB
          Ankit Singhal
        2. PHOENIX-3649_v3.patch
          15 kB
          Ankit Singhal
        3. PHOENIX-3649_v2.patch
          14 kB
          Ankit Singhal
        4. PHOENIX-3649_v1.patch
          10 kB
          Ankit Singhal
        5. PHOENIX-3649_4.9_branch.patch
          1 kB
          Ankit Singhal

        Activity

          People

            ankit@apache.org Ankit Singhal
            mujtabachohan Mujtaba Chohan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: