Accumulo
  1. Accumulo
  2. ACCUMULO-2092

Seeing spurious error message about namespace that does not exist

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      db746960fbcafb1651c15ec2e5493d56acb5065c

      Description

      In the conditional random walk test the following spurious error message occurred. The test did not fail, it was running fine.

      24 17:43:45,065 [conditional.Setup] DEBUG: created table banks
      24 17:43:45,101 [conditional.Setup] DEBUG: set table.cache.block.enable false
      24 17:43:45,148 [impl.Tables] ERROR: Table (d) contains reference to namespace () that doesn't exist
      24 17:43:47,012 [conditional.Init] DEBUG: Added splits [b100, b200, b300, b400, b500, b600, b700, b800, b900]
      24 17:44:00,394 [conditional.Init] DEBUG: Added bank b259 9
      

        Activity

        Keith Turner created issue -
        Keith Turner made changes -
        Field Original Value New Value
        Fix Version/s 1.6.0 [ 12322468 ]
        Hide
        John Vines added a comment -

        So in the clone code, it did the clone ZK and then set the namespace after the majority of the ZK clone work occured. I fixed this by making namespace addition PART of that code. I wonder if create() suffers from a similar race issue.

        Show
        John Vines added a comment - So in the clone code, it did the clone ZK and then set the namespace after the majority of the ZK clone work occured. I fixed this by making namespace addition PART of that code. I wonder if create() suffers from a similar race issue.
        Hide
        Christopher Tubbs added a comment -

        No, we believe the problem is that createTable fails because the table exists, but it's Tables cache isn't updated, so it doesn't actually see that the table exists when it checks zookeeper itself. A related problem is ACCUMULO-2093. What needs to happen is that the Tables cache needs to be cleared after every table/fate operation.

        Show
        Christopher Tubbs added a comment - No, we believe the problem is that createTable fails because the table exists, but it's Tables cache isn't updated, so it doesn't actually see that the table exists when it checks zookeeper itself. A related problem is ACCUMULO-2093 . What needs to happen is that the Tables cache needs to be cleared after every table/fate operation.
        Hide
        John Vines added a comment -

        This ticket isn't about a createTable failing...

        Show
        John Vines added a comment - This ticket isn't about a createTable failing...
        Hide
        Keith Turner added a comment -

        Christopher Tubbs and I were looking into this. We think the following happened

        1. Walker 1 is creating table banks
        2. Walker 2 attempted to create table banks, but got a Table Exist exception
        3. Walker 2 performs another table operation, but the info in zoocache is not fully updated and the error is emitted

        In step 2 above the doTableOperation code could clear the client side table cache in the case of exceptions. Currently doTableOperation only clears the cache when the table operation is successful.

        Show
        Keith Turner added a comment - Christopher Tubbs and I were looking into this. We think the following happened Walker 1 is creating table banks Walker 2 attempted to create table banks, but got a Table Exist exception Walker 2 performs another table operation, but the info in zoocache is not fully updated and the error is emitted In step 2 above the doTableOperation code could clear the client side table cache in the case of exceptions. Currently doTableOperation only clears the cache when the table operation is successful.
        Hide
        Keith Turner added a comment -

        I am not sure if clearing the cache is enough because the table info is stored in multiple nodes. In step 3 the client could retry if it sees incomplete information with the expectation that the table information will eventually converge to deleted or complete.

        Show
        Keith Turner added a comment - I am not sure if clearing the cache is enough because the table info is stored in multiple nodes. In step 3 the client could retry if it sees incomplete information with the expectation that the table information will eventually converge to deleted or complete.
        Hide
        Christopher Tubbs added a comment -

        John Vines Right, but it seems to be happening because the client has an inconsistent view of the table's existence after the Master reports that a create table failed, due to it already existing.

        Show
        Christopher Tubbs added a comment - John Vines Right, but it seems to be happening because the client has an inconsistent view of the table's existence after the Master reports that a create table failed, due to it already existing.
        Christopher Tubbs made changes -
        Assignee Christopher Tubbs [ ctubbsii ]
        Christopher Tubbs made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Hide
        John Vines added a comment -

        Or we make it part of the client API that there is a propagation delay, there is a similar behavior for some of the other permissions.

        Show
        John Vines added a comment - Or we make it part of the client API that there is a propagation delay, there is a similar behavior for some of the other permissions.
        Hide
        ASF subversion and git services added a comment -

        Commit b0b2c49925735ca02019773e31c0ebee0e4a4260 in branch refs/heads/1.6.0-SNAPSHOT from Christopher Tubbs
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=b0b2c49 ]

        ACCUMULO-2092 Make client view of table existence consistent

        Clear the cache after exceptions, instead of just successful fate
        operations. This ensures that a failure as the result of a
        TableNotFoundException or TableExistsException will be consistent with
        the client's view, if they check it right after the failure.

        Show
        ASF subversion and git services added a comment - Commit b0b2c49925735ca02019773e31c0ebee0e4a4260 in branch refs/heads/1.6.0-SNAPSHOT from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=b0b2c49 ] ACCUMULO-2092 Make client view of table existence consistent Clear the cache after exceptions, instead of just successful fate operations. This ensures that a failure as the result of a TableNotFoundException or TableExistsException will be consistent with the client's view, if they check it right after the failure.
        Hide
        Christopher Tubbs added a comment -

        Fixed by a combination of ACCUMULO-2093 and this ticket.

        Show
        Christopher Tubbs added a comment - Fixed by a combination of ACCUMULO-2093 and this ticket.
        Christopher Tubbs made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        ASF subversion and git services added a comment -

        Commit b0b2c49925735ca02019773e31c0ebee0e4a4260 in branch refs/heads/master from Christopher Tubbs
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=b0b2c49 ]

        ACCUMULO-2092 Make client view of table existence consistent

        Clear the cache after exceptions, instead of just successful fate
        operations. This ensures that a failure as the result of a
        TableNotFoundException or TableExistsException will be consistent with
        the client's view, if they check it right after the failure.

        Show
        ASF subversion and git services added a comment - Commit b0b2c49925735ca02019773e31c0ebee0e4a4260 in branch refs/heads/master from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=b0b2c49 ] ACCUMULO-2092 Make client view of table existence consistent Clear the cache after exceptions, instead of just successful fate operations. This ensures that a failure as the result of a TableNotFoundException or TableExistsException will be consistent with the client's view, if they check it right after the failure.
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open In Progress In Progress
        15d 41m 1 Christopher Tubbs 08/Jan/14 20:13
        In Progress In Progress Resolved Resolved
        28d 4h 4m 1 Christopher Tubbs 06/Feb/14 00:17

          People

          • Assignee:
            Christopher Tubbs
            Reporter:
            Keith Turner
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development