Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4874

Remove schema tuple reference overhead for replicate join hashmap

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently even if pig.schematuple is set to false which is the default, the usage of TupleToMapKey and TuplesToSchemaTupleList instead of plain HashMap<Object, ArrayList<Tuple>> costs a lot of memory. Also key is currently converted to a tuple which is unnecessary.

      1. PIG-4874-2.patch
        23 kB
        Rohini Palaniswamy
      2. PIG-4874-1.patch
        23 kB
        Rohini Palaniswamy

        Issue Links

          Activity

          Hide
          rohini Rohini Palaniswamy added a comment -

          We really take up a lot of space in memory to store the replicate table.

          BufferedOutputStream bout = new BufferedOutputStream(new FileOutputStream("/tmp/data"));
                  for (int i = 0; i < 100; i++) {
                      bout.write(new String(i + "\t" + i + "\n").getBytes());
                  }
                  bout.close();
          
          A = LOAD '/tmp/data' as (x:int, y:int);
          B = LOAD '/tmp/data' as (x:int, y:int);
          C = JOIN A by (x,y), B by (x,y) using 'replicated';
          STORE C into '/tmp/tezout';
          

          The 100 entries from 0 to 99 in above test take up 508 bytes on disk. Retained sizes in Yourkit for the replicates map are

          pig.schematuple=false
          Before patch : 46096
          After patch : 37232
          pig.schematuple=true
          17808

          This patch optimizes schema tuple for the case of primitive keys as well. Currently TestSchemaTuple is only ExecType.LOCAL. Found that SchemaTuple does not work with Tez as it requires some stuff shipped through DistributedCache and is not done. Will create a separate jira to fix SchemaTuple for Tez and also make it more robust so that we can try and make it default for replicate join with such huge memory savings.

          Show
          rohini Rohini Palaniswamy added a comment - We really take up a lot of space in memory to store the replicate table. BufferedOutputStream bout = new BufferedOutputStream( new FileOutputStream( "/tmp/data" )); for ( int i = 0; i < 100; i++) { bout.write( new String (i + "\t" + i + "\n" ).getBytes()); } bout.close(); A = LOAD '/tmp/data' as (x: int , y: int ); B = LOAD '/tmp/data' as (x: int , y: int ); C = JOIN A by (x,y), B by (x,y) using 'replicated'; STORE C into '/tmp/tezout'; The 100 entries from 0 to 99 in above test take up 508 bytes on disk. Retained sizes in Yourkit for the replicates map are pig.schematuple=false Before patch : 46096 After patch : 37232 pig.schematuple=true 17808 This patch optimizes schema tuple for the case of primitive keys as well. Currently TestSchemaTuple is only ExecType.LOCAL. Found that SchemaTuple does not work with Tez as it requires some stuff shipped through DistributedCache and is not done. Will create a separate jira to fix SchemaTuple for Tez and also make it more robust so that we can try and make it default for replicate join with such huge memory savings.
          Hide
          knoguchi Koji Noguchi added a comment -

          I won't be able to review this patch well, but one very minor thing I did notice

          TestSchemaTuple.java
          651         List<Tuple> bar2 = data.get("bar1");
          

          This should be "bar2".

          Show
          knoguchi Koji Noguchi added a comment - I won't be able to review this patch well, but one very minor thing I did notice TestSchemaTuple.java 651 List<Tuple> bar2 = data.get( "bar1" ); This should be "bar2".
          Hide
          rohini Rohini Palaniswamy added a comment - - edited

          Thanks Koji Noguchi for catching that. Updated patch.

          A production job had one vertex doing two replicate joins.
          Replicate input 1 - 870424 records
          Replicate input 2 - 847854 records

          Retained Size of replicate tables in memory

          Before PIG-4874 patch:
          input 1 hashmap - 389,467,984
          input 2 hashmap - 333,995,048

          After PIG-4874 patch:
          input 1 hashmap - 312,870,672
          input 2 hashmap - 198,338,416 (More savings here because the join key is just String whereas input 1 had tuple for join key)

          Show
          rohini Rohini Palaniswamy added a comment - - edited Thanks Koji Noguchi for catching that. Updated patch. A production job had one vertex doing two replicate joins. Replicate input 1 - 870424 records Replicate input 2 - 847854 records Retained Size of replicate tables in memory Before PIG-4874 patch: input 1 hashmap - 389,467,984 input 2 hashmap - 333,995,048 After PIG-4874 patch: input 1 hashmap - 312,870,672 input 2 hashmap - 198,338,416 (More savings here because the join key is just String whereas input 1 had tuple for join key)
          Hide
          daijy Daniel Dai added a comment -

          One question I have initially is why we change replicates from array to List. Rohini Palaniswamy clarified with me offline this is because Java array does not work with generics. By changing to List, we can make better use of generic types.

          +1.

          Show
          daijy Daniel Dai added a comment - One question I have initially is why we change replicates from array to List. Rohini Palaniswamy clarified with me offline this is because Java array does not work with generics. By changing to List, we can make better use of generic types. +1.
          Hide
          rohini Rohini Palaniswamy added a comment -

          Committed to trunk. Thanks for the review Daniel.

          Show
          rohini Rohini Palaniswamy added a comment - Committed to trunk. Thanks for the review Daniel.

            People

            • Assignee:
              rohini Rohini Palaniswamy
              Reporter:
              rohini Rohini Palaniswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development