Uploaded image for project: 'Apache HAWQ'
  1. Apache HAWQ
  2. HAWQ-347

Dead Lock Issue when concurrent use catalog and persistent table.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0.0-incubating
    • Transaction
    • None

    Description

      ------
      prepare
      ------

      create tablespace myts filespace dfs_system;
      

      ----------
      Process 1 : use ctrl_c to abort the create transaction
      -----------

      create FILESPACE fsinvalid ON hdfs ('localhost:10086/hawq/fsinvalid');
      ctrl_C
      

      ---------
      Process 2 : during the ctrl_c period, drop a tablespace
      ---------

      drop TABLESPACE myts;
      

      ************
      The lock state:
      ****************
      1.abort (create Filespace)
      get lock : Persistent table lock
      need lock: pg_filespace lock

      2.commit(drop tablespace)
      get lock: pg_filespace lock
      need lock: Persistent table lock

      ***********
      Debug information
      ***********
      1.debug information for abort (create Filespace) process
      (lldb) bt

      • thread #1: tid = 0x9781a3, 0x99364f7a libsystem_kernel.dylib`semop + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
      • frame #0: 0x99364f7a libsystem_kernel.dylib`semop + 10
        frame #1: 0x003eb14a postgres`PGSemaphoreLock(sema=0x18548cf8, interruptOK='\0') + 301 at pg_sema.c:437
        frame #2: 0x0045972c postgres`LWLockAcquire(lockid=PersistentObjLock, mode=LW_EXCLUSIVE) + 799 at lwlock.c:557
        frame #3: 0x006cd3a9 postgres`PersistentFilespace_Dropped(fsObjName=0x048f6598, persistentTid=0x048f65bc, persistentSerialNum=3) + 411 at cdbpersistentfilespace.c:1079
        frame #4: 0x006e03b4 postgres`PersistentFileSysObj_DropObject(fsObjName=0x048f6598, relStorageMgr=PersistentFileSysRelStorageMgr_None, relationName=0x00000000, persistentTid=0x048f65bc, persistentSerialNum=3, ignoreNonExistence='\x01', debugPrint='\0', debugPrintLevel=14) + 2170 at cdbpersistentfilesysobj.c:2090
        frame #5: 0x006e04b8 postgres`PersistentFileSysObj_EndXactDrop(fsObjName=0x048f6598, relStorageMgr=PersistentFileSysRelStorageMgr_None, relationName=0x00000000, persistentTid=0x048f65bc, persistentSerialNum=3, ignoreNonExistence='\x01') + 113 at cdbpersistentfilesysobj.c:2123
        frame #6: 0x004603c2 postgres`smgrDoDeleteActions(list=0x00b40844, listCount=0x00b40848, forCommit='\0') + 1444 at smgr.c:1527
        frame #7: 0x00461258 postgres`AtEOXact_smgr(forCommit='\0') + 226 at smgr.c:2111
        frame #8: 0x00072abf postgres`AbortTransaction + 774 at xact.c:2867
        frame #9: 0x0007343b postgres`AbortCurrentTransaction + 183 at xact.c:3289
        frame #10: 0x0046ade6 postgres`PostgresMain(argc=4, argv=0x0484b480, username=0x0484baa8) + 6294 at postgres.c:4487
        frame #11: 0x0040c1e9 postgres`BackendRun(port=0x03835c30) + 1008 at postmaster.c:5875
        frame #12: 0x0040b583 postgres`BackendStartup(port=0x03835c30) + 381 at postmaster.c:5468
        frame #13: 0x0040463d postgres`ServerLoop + 1111 at postmaster.c:2147
        frame #14: 0x004033ca postgres`PostmasterMain(argc=9, argv=0x03834620) + 5127 at postmaster.c:1439
        frame #15: 0x0031aa6b postgres`main(argc=9, argv=0x03834620) + 1007 at main.c:226
        frame #16: 0x0000201d postgres`_start + 212
        frame #17: 0x00001f48 postgres`start + 40

      --------------------------------------------
      2.debug information for commit(drop tablespace) process

      • thread #1: tid = 0x97829a, 0x99364f7a libsystem_kernel.dylib`semop + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
      • frame #0: 0x99364f7a libsystem_kernel.dylib`semop + 10
        frame #1: 0x003eb14a postgres`PGSemaphoreLock(sema=0x185492e0, interruptOK='\x01') + 301 at pg_sema.c:437
        frame #2: 0x00456615 postgres`ProcSleep(locallock=0x04847cc8, lockMethodTable=0x0097c804) + 736 at proc.c:983
        frame #3: 0x00451f4b postgres`WaitOnLock(locallock=0x04847cc8, owner=0x0484d15c) + 428 at lock.c:1351
        frame #4: 0x004515a5 postgres`LockAcquire(locktag=0xbfffe788, lockmode=1, sessionLock='\0', dontWait='\0') + 6297 at lock.c:995
        frame #5: 0x0044e98f postgres`LockRelationOid(relid=5009, lockmode=1) + 58 at lmgr.c:102
        frame #6: 0x00039686 postgres`relation_open(relationId=5009, lockmode=1) + 105 at heapam.c:880
        frame #7: 0x0003a01f postgres`heap_open(relationId=5009, lockmode=1) + 31 at heapam.c:1286
        frame #8: 0x0020444c postgres`is_filespace_shared(fsoid=16384) + 29 at filespace.c:737
        frame #9: 0x0009b637 postgres`emit_mmxlog_fs_record(type=MM_OBJ_TABLESPACE, filespace=16384, tablespace=204983, database=0, relfilenode=0, persistentTid=0x0509564c, persistentSerialNum=2, segnum=0, flags=' ', beginLoc=0xbfffed90) + 475 at xlog_mm.c:498
        frame #10: 0x0009ba23 postgres`mmxlog_log_remove_tablespace(tablespace=204983, persistentTid=0x0509564c, persistentSerialNum=2) + 189 at xlog_mm.c:584
        frame #11: 0x006dc13a postgres`PersistentTablespace_DroppedVerifiedActionCallback(fsObjName=0x05095628, persistentTid=0x0509564c, persistentSerialNum=2, verifyExpectedResult=PersistentFileSysObjVerifyExpectedResult_StateChangeNeeded) + 97 at cdbpersistenttablespace.c:1079
        frame #12: 0x006def9a postgres`PersistentFileSysObj_StateChange(fsObjName=0x05095628, persistentTid=0x0509564c, persistentSerialNum=2, nextState=PersistentFileSysState_Free, retryPossible='\0', flushToXLog='\0', oldState=0xbfffef0c, verifiedActionCallback=0x006dc0d9) + 920 at cdbpersistentfilesysobj.c:1432
        frame #13: 0x006dc485 postgres`PersistentTablespace_Dropped(fsObjName=0x05095628, persistentTid=0x0509564c, persistentSerialNum=2) + 771 at cdbpersistenttablespace.c:1142
        frame #14: 0x006e02da postgres`PersistentFileSysObj_DropObject(fsObjName=0x05095628, relStorageMgr=PersistentFileSysRelStorageMgr_None, relationName=0x00000000, persistentTid=0x0509564c, persistentSerialNum=2, ignoreNonExistence='\0', debugPrint='\0', debugPrintLevel=14) + 1952 at cdbpersistentfilesysobj.c:2064
        frame #15: 0x006e04b8 postgres`PersistentFileSysObj_EndXactDrop(fsObjName=0x05095628, relStorageMgr=PersistentFileSysRelStorageMgr_None, relationName=0x00000000, persistentTid=0x0509564c, persistentSerialNum=2, ignoreNonExistence='\0') + 113 at cdbpersistentfilesysobj.c:2123
        frame #16: 0x004603c2 postgres`smgrDoDeleteActions(list=0x00b40844, listCount=0x00b40848, forCommit='\x01') + 1444 at smgr.c:1527
        frame #17: 0x00461258 postgres`AtEOXact_smgr(forCommit='\x01') + 226 at smgr.c:2111
        frame #18: 0x000720f6 postgres`CommitTransaction + 749 at xact.c:2427
        frame #19: 0x00072fb6 postgres`CommitTransactionCommand + 310 at xact.c:3056
        frame #20: 0x004686b8 postgres`finish_xact_command + 116 at postgres.c:3123
        frame #21: 0x00465b6a postgres`exec_simple_query(query_string=0x0507c42c, seqServerHost=0x00000000, seqServerPort=-1) + 2137 at postgres.c:1780
        frame #22: 0x0046b46c postgres`PostgresMain(argc=4, argv=0x0484b480, username=0x0484baa8) + 7964 at postgres.c:4711
        frame #23: 0x0040c1e9 postgres`BackendRun(port=0x03911ea0) + 1008 at postmaster.c:5875
        frame #24: 0x0040b583 postgres`BackendStartup(port=0x03911ea0) + 381 at postmaster.c:5468
        frame #25: 0x0040463d postgres`ServerLoop + 1111 at postmaster.c:2147
        frame #26: 0x004033ca postgres`PostmasterMain(argc=9, argv=0x03834620) + 5127 at postmaster.c:1439
        frame #27: 0x0031aa6b postgres`main(argc=9, argv=0x03834620) + 1007 at main.c:226
        frame #28: 0x0000201d postgres`_start + 212
        frame #29: 0x00001f48 postgres`start + 40

      -----------------------
      Other information
      -----------------------
      intern=# select * from pg_stat_activity ;
      datid | datname | procpid | sess_id | usesysid | usename | current_query | waiting | query_start | backend_start | client_addr | client_port | application_name | xact_start | waiting_resource
      -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      16387 | postgres | 83974 | 65 | 10 | intern | drop TABLESPACE myts; | t | 2016-01-15 18:48:18.910256+08 | 2016-01-15 18:47:57.794331+08 | | -1 | psql | 2016-01-15 18:48:18.910256+08 | f
      205198 | dbindifffs | 83863 | 64 | 10 | intern | create FILESPACE fsinvalid ON hdfs ('localhost:10086/hawq/fsinvalid'); | f | 2016-01-15 18:47:26.849302+08 | 2016-01-15 18:47:26.790389+08 | | -1 | psql | 2016-01-15 18:47:26.849302+08 | f
      16534 | intern | 89599 | 66 | 10 | intern | select * from pg_stat_activity ; | f | 2016-01-15 19:25:14.432401+08 | 2016-01-15 19:13:54.205517+08 | | -1 | psql | 2016-01-15 19:25:14.432401+08 | f
      (3 rows)

      intern=# select * from pg_locks ;
      locktype | database | relation | page | tuple | transactionid | classid | objid | objsubid | transaction | pid | mode | granted | mppsessionid | mppiswriter | gp_segment_id
      -----------------------------------------------------------------------------------------------------------------------------------------+--------------
      relation | 0 | 5009 | | | | | | | 0 | 83863 | AccessExclusiveLock | t | 64 | f | -10000
      transactionid | | | | | 2133 | | | | 2133 | 89599 | ExclusiveLock | t | 66 | f | -10000
      relation | 0 | 1213 | | | | | | | 0 | 83974 | RowExclusiveLock | t | 65 | f | -10000
      transactionid | | | | | 2118 | | | | 0 | 83863 | ExclusiveLock | t | 64 | f | -10000
      relation | 0 | 5009 | | | | | | | 0 | 83974 | AccessShareLock | f | 65 | f | -10000
      relation | 16534 | 10336 | | | | | | | 2133 | 89599 | AccessShareLock | t | 66 | f | -10000
      transactionid | | | | | 2124 | | | | 0 | 83974 | ExclusiveLock | t | 65 | f | -10000
      (7 rows)

      intern=# select * from pg_class where oid=5009;
      relname | relnamespace | reltype | relowner | relam | relfilenode | reltablespace | relpages | reltuples | reltoastrelid | reltoastidxid | relaosegrelid | relaosegidxid | relhasindex | relisshared | relkind | relstorage | relnatts | relchecks | reltriggers | relukeys | relfkeys | relrefs | relhasoids | relhaspkey | relhasrules | relhassubclass | relfrozenxid | relacl | reloptions
      --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------
      pg_filespace | 11 | 6438 | 10 | 0 | 5009 | 1664 | 1 | 1 | 0 | 0 | 0 | 0 | t | t | r | h | 4 | 0 | 0 | 0 | 0 | 0 | t | f | f | f | 903 |

      {=r/intern}

      |
      (1 row)

      Attachments

        Activity

          People

            mli Ming Li
            doli Dong Li
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: