Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0
    • Component/s: None
    • Labels:
      None

      Description

      After incorporating v11 of the 2856 fix, we discovered that we are still having some ACID violations.

      This time, however, the problem is not about including "newer" updates; but, about missing older updates
      that should be including.

      Here is what seems to be happening.

      There is a race condition in the StoreScanner.getScanners()

      private List<KeyValueScanner> getScanners(Scan scan,
      final NavigableSet<byte[]> columns) throws IOException {
      // First the store file scanners
      List<StoreFileScanner> sfScanners = StoreFileScanner
      .getScannersForStoreFiles(store.getStorefiles(), cacheBlocks,
      isGet, false);
      List<KeyValueScanner> scanners =
      new ArrayList<KeyValueScanner>(sfScanners.size()+1);

      // include only those scan files which pass all filters
      for (StoreFileScanner sfs : sfScanners) {
      if (sfs.shouldSeek(scan, columns))

      { scanners.add(sfs); }

      }

      // Then the memstore scanners
      if (this.store.memstore.shouldSeek(scan))

      { scanners.addAll(this.store.memstore.getScanners()); }

      return scanners;
      }

      If for example there is a call to Store.updateStorefiles() that happens between
      the store.getStorefiles() and this.store.memstore.getScanners(); then
      it is possible that there was a new HFile created, that is not seen by the
      StoreScanner, and the data is not present in the Memstore.snapshot either.

      1. 4485-v1.diff
        0.9 kB
        Amitanand Aiyer
      2. repro_bug-4485.diff
        1.0 kB
        Amitanand Aiyer
      3. 4485-v2.diff
        6 kB
        Amitanand Aiyer
      4. 4485-v3.diff
        6 kB
        Amitanand Aiyer
      5. 4485-v4.diff
        9 kB
        Amitanand Aiyer
      6. 4485-v5.diff
        11 kB
        Amitanand Aiyer
      7. 4485-v6.diff
        11 kB
        Amitanand Aiyer

        Issue Links

          Activity

          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Nicolas Spiegelberg made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Amitanand Aiyer made changes -
          Attachment 4485-v6.diff [ 12504158 ]
          Amitanand Aiyer made changes -
          Attachment 4485-v5.diff [ 12504133 ]
          Jonathan Hsieh made changes -
          Link This issue is related to HBASE-4570 [ HBASE-4570 ]
          Amitanand Aiyer made changes -
          Attachment 4485-v4.diff [ 12496695 ]
          Amitanand Aiyer made changes -
          Attachment 4485-v3.diff [ 12496692 ]
          Amitanand Aiyer made changes -
          Attachment 4485-v2.diff [ 12496691 ]
          Amitanand Aiyer made changes -
          Attachment repro_bug-4485.diff [ 12496676 ]
          Amitanand Aiyer made changes -
          Description After incorporating v11 of the 2856 fix, we discovered that we are still having some ACID violations.

          This time, however, the problem is not about including "newer" updates; but, about missing older updates
          that should be including.

          Here is what seems to be happing.


          0 - Scanner starts scanning.

          0 - MemStore.snapshot is called.

              Scanner has access to kvHeap and snapshot

          1- Flush takes place.
               1.1 KV's in the snapshot are written to the disk.
               1.2 HFile is ready.

          2 Store.updateStoreFiles() deletes the old snapshot.
              
               2.1 updateReaders will not be called until the end of the columnFamily seek.

          3 For a brief window of time, scanner does not have access to certain KeyValues.
             a) Scanner has no longer access to the snapshot because it is flushed to the
          disk.
             b) It does not yet have access to the HFile because the updateReaders was
          not called yet.
                  
          After incorporating v11 of the 2856 fix, we discovered that we are still having some ACID violations.

          This time, however, the problem is not about including "newer" updates; but, about missing older updates
          that should be including.

          Here is what seems to be happening.

          There is a race condition in the StoreScanner.getScanners()

            private List<KeyValueScanner> getScanners(Scan scan,
                final NavigableSet<byte[]> columns) throws IOException {
              // First the store file scanners
              List<StoreFileScanner> sfScanners = StoreFileScanner
                .getScannersForStoreFiles(store.getStorefiles(), cacheBlocks,
                                          isGet, false);
              List<KeyValueScanner> scanners =
                new ArrayList<KeyValueScanner>(sfScanners.size()+1);

              // include only those scan files which pass all filters
              for (StoreFileScanner sfs : sfScanners) {
                if (sfs.shouldSeek(scan, columns)) {
                  scanners.add(sfs);
                }
              }

              // Then the memstore scanners
              if (this.store.memstore.shouldSeek(scan)) {
                scanners.addAll(this.store.memstore.getScanners());
              }
              return scanners;
            }


          If for example there is a call to Store.updateStorefiles() that happens between
          the store.getStorefiles() and this.store.memstore.getScanners(); then
          it is possible that there was a new HFile created, that is not seen by the
          StoreScanner, and the data is not present in the Memstore.snapshot either.
          Amitanand Aiyer made changes -
          Attachment 4485-v1.diff [ 12496496 ]
          Amitanand Aiyer made changes -
          Field Original Value New Value
          Fix Version/s 0.94.0 [ 12316419 ]
          Amitanand Aiyer created issue -

            People

            • Assignee:
              Amitanand Aiyer
              Reporter:
              Amitanand Aiyer
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development