Description
I was tried to test the implementation (https://github.com/apache/samza/pull/1197) of Samza on K8s on AKS (using the latest Samza (master branch)), and got the error below. The job was trying to store logs and the state in Azure file.
(P.S. Samza 1.1 does not hit this issue, and the job built on Samza 1.1 works fine on AKS.)
2019-11-04 00:56:00.558 [main] SamzaContainer [INFO] Entering run loop. 2019-11-04 00:56:00.559 [main] ClusterBasedProcessorLifecycleListener [INFO] Container Started 2019-11-04 00:56:57.106 [kafka-producer-network-thread | kafka_producer-wikipedia_application-1] Metadata [INFO] Cluster ID: YNWIzBLFSa2Lg5wehzo7ZA 2019-11-04 00:57:03.920 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata for: [SystemStreamPartition [kafka, wikipedia-stats-changelog, 7]] 2019-11-04 00:57:03.923 [main] KafkaSystemAdmin [INFO] Fetching SSP metadata for: [SystemStreamPartition [kafka, wikipedia-application-1-window-statsWindow, 7]] 2019-11-04 00:57:04.222 [main] RunLoop [ERROR] Task Partition 7 commit failed org.rocksdb.RocksDBException: while link file to /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7-1572829023927.tmp/000013.sst: /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7/000013.sst: Operation not supported at org.rocksdb.Checkpoint.createCheckpoint(Native Method) at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51) at org.apache.samza.storage.kv.RocksDbKeyValueStore.checkpoint(RocksDbKeyValueStore.scala:245) at org.apache.samza.storage.kv.LoggedStore.checkpoint(LoggedStore.scala:125) at org.apache.samza.storage.kv.SerializedKeyValueStore.checkpoint(SerializedKeyValueStore.scala:170) at org.apache.samza.storage.kv.CachedStore.checkpoint(CachedStore.scala:297) at org.apache.samza.storage.kv.NullSafeKeyValueStore.checkpoint(NullSafeKeyValueStore.scala:103) at org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:215) at org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$checkpoint$1.apply(KeyValueStorageEngine.scala:212) at org.apache.samza.util.TimerUtil$class.updateTimer(TimerUtil.scala:37) at org.apache.samza.storage.kv.KeyValueStorageEngine.updateTimer(KeyValueStorageEngine.scala:37) at org.apache.samza.storage.kv.KeyValueStorageEngine.checkpoint(KeyValueStorageEngine.scala:212) at org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:67) at org.apache.samza.storage.TransactionalStateTaskStorageManager$$anonfun$2.apply(TransactionalStateTaskStorageManager.scala:66) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) at org.apache.samza.storage.TransactionalStateTaskStorageManager.checkpoint(TransactionalStateTaskStorageManager.scala:66) at org.apache.samza.container.TaskInstance.commit(TaskInstance.scala:267) at org.apache.samza.container.RunLoop$AsyncTaskWorker$5.run(RunLoop.java:547) at org.apache.samza.container.RunLoop$AsyncTaskWorker.commit(RunLoop.java:566) at org.apache.samza.container.RunLoop$AsyncTaskWorker.run(RunLoop.java:432) at org.apache.samza.container.RunLoop$AsyncTaskWorker.access$300(RunLoop.java:357) at org.apache.samza.container.RunLoop.runTasks(RunLoop.java:244) at org.apache.samza.container.RunLoop.run(RunLoop.java:176) at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:768) at org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:139) at org.apache.samza.runtime.ContainerLaunchUtil.run(ContainerLaunchUtil.java:82) at org.apache.samza.runtime.LocalContainerRunner.main(LocalContainerRunner.java:80) 2019-11-04 00:57:04.223 [main] RunLoop [ERROR] Caught throwable and stopping run loop org.rocksdb.RocksDBException: while link file to /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7-1572829023927.tmp/000013.sst: /tmp/log/wikipedia-application-1/wikipedia-stats/Partition_7/000013.sst: Operation not supported at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
Attachments
Issue Links
- relates to
-
SAMZA-2067 Samza on Kubernetes
- Open