[PHOENIX-3230] Upgrade code running concurrently on different JVMs could make clients unusuable

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.8.1
• Assignee: Samarth Jain
• Reporter: Mujtaba Chohan
• Created: 31/Aug/16 18:55
• Updated: 28/Sep/16 05:15
• Resolved: 09/Sep/16 03:21

Description

If two separate Phoenix connections try to upgrade Phoenix from v4.7 to 4.8.1 then second connection fails with the following exception. This happens even if second connection is couple of seconds apart but within upgrade window. This is likely to happen in situation where pool of client machines all get upgraded to latest Phoenix version. After this exception, all clients will cease to work with undefined column exception due to restore/aborted upgrade.

WARN query.ConnectionQueryServicesImpl: Table already modified at this timestamp, so assuming add of these columns already done: IS_NAMESPACE_MAPPED BOOLEAN
WARN query.ConnectionQueryServicesImpl: Table already modified at this timestamp, so assuming add of these columns already done: AUTO_PARTITION_SEQ VARCHAR
WARN query.ConnectionQueryServicesImpl: Table already modified at this timestamp, so assuming add of these columns already done: APPEND_ONLY_SCHEMA BOOLEAN
WARN query.ConnectionQueryServicesImpl: Starting restore of SYSTEM.CATALOG using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700 because upgrade failed
16/08/31 11:41:05 WARN query.ConnectionQueryServicesImpl: Successfully restored SYSTEM.CATALOG using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
16/08/31 11:41:09 WARN query.ConnectionQueryServicesImpl: Successfully restored and enabled SYSTEM.CATALOG using snapshot SNAPSHOT_SYSTEM.CATALOG_4.7.x_TO_4.8.0_20160831114048-0700
Error: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED (state=42703,code=504)
org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703): Undefined column. columnName=IS_NAMESPACE_MAPPED
	at org.apache.phoenix.schema.PTableImpl.getColumn(PTableImpl.java:693)
	at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.resolveColumn(FromCompiler.java:449)
	at org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:418)
	at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:590)
	at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:578)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:333)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
	at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
	at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:247)
	at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:172)
	at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:177)
	at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2275)
	at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:920)
	at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:193)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:340)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:328)
	at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:326)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1369)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2486)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2282)
	at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2282)
	at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:231)
	at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:144)
	at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
	at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
	at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)