Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Not A Problem
-
0.90.6
-
None
-
None
-
linux x86_64
Description
It appears that the region servers continue to parse the xml config files as a part of their normal operation (and not just on startup). I realize this might be coming from hadoop config parsing etc, but it is still a major problem should you happen to push out a bad xml config.
Here is the stack trace from the problem in our environment.
13/02/05 13:46:12 INFO regionserver.HRegion: Starting compaction on region tsdb,\x00\x00\x0FP\xFEU\x10\x00\x00\x01\x00 \xEF\x00\x00\x0B\x00\x00\x0F\x00\x00\x0C\x00\x00\x19,1359827642230.aadcc5a9ef4d4f16fb8937c9e93763a1.
13/02/05 13:46:12 INFO regionserver.Store: Started compaction of 3 file(s) in cf=t into hdfs://nn-blah:8020/hbase/tsdb/aadcc5a9ef4d4f16fb8937c9e93763a1/.tmp, seqid=1704265883, totalSize=141.6m
13/02/05 13:47:57 INFO regionserver.StoreFile: Bloom added to HFile (hdfs://nn-blah:8020/hbase/tsdb/aadcc5a9ef4d4f16fb8937c9e93763a1/.tmp/7838002095040213865): 793.2k, 656192/677964 (97%)
13/02/05 13:47:57 INFO regionserver.StoreFile$Reader: Loaded row bloom filter metadata for hdfs://nn-blah:8020/hbase/tsdb/aadcc5a9ef4d4f16fb8937c9e93763a1/t/8826149174883990976
13/02/05 13:47:57 INFO regionserver.Store: Completed compaction of 3 file(s), new file=hdfs://nn-blah:8020/hbase/tsdb/aadcc5a9ef4d4f16fb8937c9e93763a1/t/8826149174883990976, size=141.5m; total size for store is 2.4g
13/02/05 13:47:57 INFO regionserver.HRegion: completed compaction on region tsdb,\x00\x00\x0FP\xFEU\x10\x00\x00\x01\x00 \xEF\x00\x00\x0B\x00\x00\x0F\x00\x00\x0C\x00\x00\x19,1359827642230.aadcc5a9ef4d4f16fb8937c9e93763a1. after 1mins, 44sec
[Fatal Error] mapred-site.xml:173:13: The string "--" is not permitted within comments.
13/02/05 13:55:33 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: The string "--" is not permitted within comments.
13/02/05 13:55:33 INFO regionserver.StoreFile: Bloom added to HFile (hdfs://nn-blah:8020/hbase/tsdb/74a4a785bc317da7282c331f577918a0/.tmp/4658027967280663602): 5.3k, 1/4519 (0%)
[Fatal Error] mapred-site.xml:173:13: The string "--" is not permitted within comments.
13/02/05 13:55:33 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: The string "--" is not permitted within comments.
13/02/05 13:55:33 FATAL regionserver.HRegionServer: ABORTING region server serverName=hbrs-blah,60020,1360021443595, load=(requests=5434, regions=73, usedHeap=4085, maxHeap=15979): Replay of HLog required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region: tsdb,\x00\x060P\xF4\x1Dp\x00\x00\x01\x00\x07\xDB\x00\x00\x02\x00\x02\x02\x00\x00\x87\x00\xB0Y,1359421493144.74a4a785bc317da7282c331f577918a0.
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1054)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:954)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:902)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:394)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:368)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:242)
Caused by: java.lang.RuntimeException: org.xml.sax.SAXParseException: The string "--" is not permitted within comments.
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1393)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1251)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1192)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:493)
at com.hadoop.compression.lzo.LzoCodec.getCompressionStrategy(LzoCodec.java:205)
at com.hadoop.compression.lzo.LzoCompressor.reinit(LzoCompressor.java:204)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:105)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:236)
at org.apache.hadoop.hbase.io.hfile.HFile$Writer.getCompressingStream(HFile.java:397)
at org.apache.hadoop.hbase.io.hfile.HFile$Writer.close(HFile.java:621)
at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:877)
at org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:495)
at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:450)
at org.apache.hadoop.hbase.regionserver.Store.access$100(Store.java:82)
at org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:1529)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1032)
... 5 more
Caused by: org.xml.sax.SAXParseException: The string "--" is not permitted within comments.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:249)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1300)
... 21 more
13/02/05 13:55:33 INFO regionserver.HRegionServer: Dump of metrics: requests=1736, regions=73, stores=73, storefiles=113, storefileIndexSize=584, memstoreSize=3707, compactionQueueSize=0, flushQueueSize=0, usedHeap=4096, maxHeap=15979, blockCacheSize=1109279496, blockCacheFree=2241812216, blockCacheCount=13037, blockCacheHitCount=518453, blockCacheMissCount=1899789, blockCacheEvictedCount=0, blockCacheHitRatio=21, blockCacheHitCachingRatio=97
13/02/05 13:55:33 INFO regionserver.HRegionServer: STOPPED: Replay of HLog required. Forcing server shutdown
13/02/05 13:55:33 INFO regionserver.MemStoreFlusher: regionserver60020.cacheFlusher exiting
13/02/05 13:55:34 INFO ipc.HBaseServer: Stopping server on 60020
13/02/05 13:55:34 INFO ipc.HBaseServer: IPC Server handler 2 on 60020: exiting
13/02/05 13:55:34 INFO ipc.HBaseServer: IPC Server handler 0 on 60020: exiting
13/02/05 13:55:34 INFO ipc.HBaseServer: IPC Server handler 9 on 60020: exiting