Uploaded image for project: 'Apache Celeborn'
  1. Apache Celeborn
  2. CELEBORN-1743

celeborn metrics data interruption

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.5.0
    • 0.6.0

    Description

       

       

      When metrics monitoring data is interrupted, we will find that many flink jobs occasionally fail with errors.

       

      Calc[2] (1/8) (f850bd9e605fe5954e27883e87d596b7_0a448493b4782967b150582570326227_0_0) switched from RUNNING to FAILED on flink-359611-1732523832539-taskmanager-1-3 @ 10.58.29.133 (dataPort=1).
      java.io.IOException: Client /10.xxx.80:9093 is lost, notify related stream 1870268177773    at org.apache.celeborn.plugin.flink.RemoteBufferStreamReader.errorReceived(RemoteBufferStreamReader.java:146)    at org.apache.celeborn.plugin.flink.RemoteBufferStreamReader.lambda$new$0(RemoteBufferStreamReader.java:77)    at org.apache.celeborn.plugin.flink.network.ReadClientHandler.processMessageInternal(ReadClientHandler.java:64)    at org.apache.celeborn.plugin.flink.network.ReadClientHandler.lambda$channelInactive$0(ReadClientHandler.java:145)    at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597)    at org.apache.celeborn.plugin.flink.network.ReadClientHandler.channelInactive(ReadClientHandler.java:136)    at org.apache.celeborn.common.network.server.TransportRequestHandler.channelInactive(TransportRequestHandler.java:74)    at org.apache.celeborn.common.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:141)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)    at org.apache.celeborn.shaded.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)    at org.apache.celeborn.shaded.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:280)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)    at org.apache.celeborn.shaded.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)    at org.apache.celeborn.plugin.flink.network.TransportFrameDecoderWithBufferSupplier.channelInactive(TransportFrameDecoderWithBufferSupplier.java:207)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)    at org.apache.celeborn.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)    at org.apache.celeborn.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)    at org.apache.celeborn.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)    at org.apache.celeborn.shaded.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)    at org.apache.celeborn.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)    at org.apache.celeborn.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)    at org.apache.celeborn.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)    at org.apache.celeborn.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)    at org.apache.celeborn.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)    at org.apache.celeborn.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)    at java.lang.Thread.run(Thread.java:750) 

       

      the worker metrics count:

      $ curl 10.xxx:9096/metrics/prometheus
      metrics_ActiveConnectionCount_Count{applicationId="1732529494224-00acb8438f1c4c176e40e863017ad806",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528283408-449c95b1cbe2a8c740562e04cd965a6f",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528803184-dd11f77814d04bca2121140182ad7c0f",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529148502-3d72447b9773d89e941bcb83059424b7",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529499296-5facd993325e193ce54c8217066dfe26",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530155191-19068d5eba8a23b42a1e581b20ab0e5a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527089786-9fc4a0bfe4c3cb49adf8a87919976a4f",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526536555-c8f0aaacf7dd92bd00a03bbe61c7216a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527096128-5f416a4784a5d5b416dd4db471a51f44",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530177724-eeecda14217d8990ae0536eae83d3e3b",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527117022-769334f90b981c54280195da5d871351",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528325685-129d44a4392d98a830157333d9955558",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528362512-e08d0070fea4d15b81513d169eae5ae7",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526465146-022a70bf95cdffc776d31a22b2b6462a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527422901-c1d2ffb0afe1fd98cb55630c250a3eba",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527429997-79b52000e6c51e2fff7c781f034ace7b",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529486998-4b3ed70849450a6eb5f9f294ec89ec11",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528285266-7781ba29cffa90fc0a6016656b3229e1",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530084006-0bc176290a630a619dc849cda22a0211",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530158062-d50dc238b0bc6c14f8031551b7d54210",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527875483-def670084ebe6873cc1954e8bea348be",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529521002-9aa93df9f5f90f1c75c69f5023df4d52",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527346195-41d2474b451f65ae02713d9b5210317b",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526470930-c8d54e14ab2f2714b4bad3bb37def4ef",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529672387-6d2dd60fa124dce274ba223d0cbbb319",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528321618-627e50a3a191686d8a60295da4f63692",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732525942152-dc553f6a5ec371dcf6cb68d0883e1bf8",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530262651-1aaa92efd1c04f989fa1387bc62deae1",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528910598-d5fb0aea765e74dd8afc18d496043bf0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526468309-03a72b98bd25893bfa4fa2eb4a86d173",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529179140-0434888ea201e3186b723b67294a91f9",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527703564-3f00277ec18409c59acc1a0d291dde88",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528285855-619bc165511acb172e249a5a91a38cfb",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528322716-ef267d9082e53114f16b7e3750145afb",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527131889-684bd3f944be3f930efd16924dd6c652",role="Worker"} 0 1732530713886
      metrics_FetchChunkFailCount_Count{role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527145206-c3dbd2de5b80c97effcf98efb55dc271",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526232655-14bf25d25f3fcac2627bb0281a02a42b",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529487660-0a5799559cd63d030e80c0f6daa6c6d0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530084411-ad717bdef0a4d0e5f9280ad2f36eabf5",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528299992-4c273d802a62d27c41d098cbc3e82458",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530090248-e0402be1c591222680518d7f7fe0f93d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530165457-91710546b2d2116e2a4fbd413e60ebe7",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527745288-26359c60e03fb6f5a173887dd42515d3",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529329955-e6577292ae74b4b00749d6f4efb5a08d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528365917-7ce827d42bb22bed266479e112cafd2f",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528297614-c2b63d243853b803ecd4cd8f80fd65b2",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530094364-11bcf8af8a8b2f245e816c68c96db4f8",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526380819-23e3a3122f7a9bfda5d87e780b1e53c6",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527742498-e32a5e80a436e2f010dc6e04f4f4a813",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528364694-a0eaadb414b7582064228941c165d224",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528881826-3e8b2d594e08336f978cdaaba39f78ee",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528240954-db77318e3b3af2cf3da9fc848b003bca",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530166236-be417e4798e7c94b74008cd81db04cb8",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527699790-fb7528b2ddd44148066c49fea1c206b0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529740309-aa7628e14e414b7d08eea50f0c82af55",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527115009-f2545faefd2be521dbaa73c65458734a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528936077-66b45b663ef19a2a203e3730a64457a0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529492178-029514deacced874d9c5d6960060179b",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529738959-c824e6d11808bb89c25245f995cb4952",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526463151-68769f0d34e17a021540018dd0b8cedf",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527111231-cdd0e5922a2f799d5c4cbf236b9e218e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528141637-bcd2a05d9e86bc087f88ce6d16558ecc",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529331513-a84c257890a76fbd63b25a8e8080414f",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527135217-236e2ff82d6222216aecdb27a5958063",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527693193-e972a9e12ca68781af6b96f2738e109e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526192712-5394fd5265796e43c6f06902a9dcc04f",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528925759-d7cb071ecde4f7a07df94b8ec7f75465",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528324517-8e337fd9c19fcfed406eecc2ad4641ab",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528763104-00c169c618a2e252adb0a662b34ca362",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529485451-77382c0eacf71662b03d95f59f3e1f63",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526193688-c9b7465489868c05a36c8a7f3a4b83b0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529522133-93f61f63575b18833b496e82edf0aa7e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530081502-ab3f0bbff8b9a5c849ad32850c4b31db",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528896327-702b15c4ba5f256b41e72241449489cf",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530158137-45cdfe31502fdf8834ac0aba390b3c91",role="Worker"} 0 1732530713886
      metrics_SlotsAllocated_Count{role="Worker"} 17167 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527134411-db1b88fb9582b0199080d1f92b73f0e2",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527309196-e9a1fa5faddc77e434febce343dd8724",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527710896-10496aa5d6dbd5ee3a562c220f9d12a9",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529485279-b7a2e2cae4d257e54b5f572d17b99b8e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526713759-bef01a8b928fdb8e756e64728c31adf0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527689478-de737c28661ced59fc07b876810f0b32",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529668092-4dcae9fea9910f5a764808c606063f9e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530080407-d86129a9c3800332cb13a4cc74c6cab9",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528886346-285207d5b6406b01fb451285268e819a",role="Worker"} 0 1732530713886
      metrics_ReplicateDataConnectionExceptionCount_Count{role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526720390-b6feb2ca18ad9664f50a4f86ca42708d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528018297-653a2d4d5e679e35d4b4e86778eb6e9d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528902958-54e91d8e796b583f751a1c2bd00edb1b",role="Worker"} 10 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530092746-0985304c2bbec3f271ee93051501991c",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528293597-deb595176475e0d2654633c5ea256226",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527929353-a3d023b0b7f29c61c230bfa355cc8a7d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527727282-c459c362f006f87802c8ef0b3ae267c6",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529218806-8649ebb1b690f7d7bb11ac4a12a4a935",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528931026-6b3632b7773bfc708c366f8f3afb6fd0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530521102-5436423d3080d4f6c4ad84be8194b9f7",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526157523-5809d0bf61adf3c42eedcede91bac70a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528322790-fd31745a1d80f3a65df22c804e6f3c31",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732525858831-de35098ed0263a90f1fc9f178e1f18f0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527437726-9d0ae829877d65cfae7a57fc4da685ac",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527695697-593689f22775eb5f999480b5979a6a63",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732525866022-f333104c53dac03123255b1253f3e1ad",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530083232-d981b2db7030fa4d7456caa63a0fc478",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527143470-dd50d09e21eb673a744cdf9fda9171de",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526576085-0c3a27c1a1c9b8580e3844e37b10b40d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526538455-eec97a576c9da87857fa09c19d25ea62",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529105220-5bbade8eccc1edf958a88f459a42cda6",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526460349-76d82f2ddc69ac78fcc83e48c80d8dc6",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526579848-31251a3f1cfd051e3eecfc87ea69a614",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528467476-b42dd64697d3d5cb29164054a996c714",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530334699-54c544790588426cbb73b6720b58838a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528933205-4126e2c39ef12dccc4d276e149e99da7",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529496109-2cb4b676c21e5963235d7d682ede0e10",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529740126-353319e6c641ec62729cb30ff81e7072",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527168134-3377dd6825a49a11dc13ab4aa0655d9c",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527691426-4fb7a171b015c6ede16ca281e84a9bff",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527644608-e0c92654fc08a8221173a2a7a30d8959",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530337383-b5a7f79592ea4a5487932724fe2cb074",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527144568-36e3dc0aa040ea8e0c30d229ffb2cdee",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530332794-1a6d1645639bdebf1a7ad30cdaf5b303",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528923395-e00fe77897eac8f98df962847c5a8572",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526462798-6cb935438afc21c505b979f8d3c90307",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528897059-b1fb616b0e2f23929eb9aa85c19aa899",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530100000-690abe7861ad2b8db2bd8136b1342690",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528900611-dd30db863454ecf6cefaf944982278a6",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526585247-6d7e70186814667c6996b83dc2ea7aee",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529068693-c18ed21505b3dc0c0e6e87319809ed50",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529673908-4a87244f2b7775193c80ed9ef7ae06a0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527742937-837f019c5cb5e42e865ca8fe816f86c2",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530263922-50731cd9e795672035617a0d4359011c",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528712933-1c1a0540b1b8dc190278da6e58423d8b",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527109340-4f162cfb9112eec61f9b7f00f73dab30",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528933722-a7a632a0b3d96423d6d596448d58236a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529178869-9f5abec0aed6c434cd7fed68f6a6b70e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527099395-06335798dd9b507c056ea4ae688c4dfb",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526010111-948733db6c30116d379bac8363691e4c",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529093144-ef2d85061a2c63cd8941309c01e297ef",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526153222-1abbaae943ee3cbcd1e96b235db96384",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528712452-02bb510f181399d369868715442d2b67",role="Worker"} 4 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528132318-c62d888e9ab2c721e7532b6f3daca51b",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530170180-28294775516a061e1d4122a06447caee",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526395116-1f083f70deecd6aeef7be20fb3979b1d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529490167-21d65836df9c75a65cac532897b601f0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527605679-9c8e6c1c2a8ee66ada3c5a79c46ef52c",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528901745-cb299b680769207a8306782d337923ab",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528327693-26957c88c34ac1462ed917f4cccdb286",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530589011-83ecef1cc1f47c9eb44f9146cf3e065b",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527109658-d7522ead8bf795bd5d5eed33b02b7342",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529528501-9a64fcd06c17b1715b0a5c47895776de",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529219223-96d0c8ffa0f0007ec6fb0d5f018d063d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529664693-a1163d9d985194ef76c4c5f0e44d502c",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528711168-df25ba6faffec5b4de5b7aac77ef52ca",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528203809-c63f0042e4a6208c7a50ec0f157cac29",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526382831-6a00846c6d2d95aa5e4ad7eb1a465b06",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527736905-4729567ec3cc1f5bedae84928996119d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526866018-e192564e1656a24ab2a5d4c91e09bc5c",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529503217-c004054f5b7271bb45862b55e8be3dcb",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526462627-2d6dbcb98317aa3d3ef7dd1095e54286",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527573562-e1f4404c5056a137ba038f72232d9566",role="Worker"} 0 1732530713886
      metrics_RegionFinishFailCount_Count{role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527745628-c868e6c0080142448385b62de9a84e92",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529501644-ff456235bff1c7187f2fcb06d84c9eca",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527108115-6724aabacaa3860c4aa87302b19a8a4e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528899101-66d4b2148b697184c0435ea98a935fc5",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526572398-0c44e0c1107e4c8d20112ec9abe7749a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527748209-bafb49bad8baf2bf4369dc176531009e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527284685-39f31cd6de0d0969d70d1111f1752853",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530003496-db56e5dc8fe29280c72075a850c8c97d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530177913-691065940f6ad658e402e595bc8c3e9a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527729588-22257112fe2882886f2391ab04fdb05c",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528020183-917743ecb7f4109d905b898af00388a7",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528128375-c172e2fdd9af16c1599ffcc8be854f9b",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528130302-93c414c3745732868012b0eff9297ea0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529558230-ab3c058baf21eb99717b716eb3015956",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528652484-9fb51582dd0be95f4687d256f58fef53",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528894401-5d863737c577954a7c32921d630d11ed",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527105382-c87dbcd40f2d7bae39a995e0fefe21bf",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528320680-e7f2ef190ab65d80699b3ef1ff29279e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529480099-3e289dc97ac57e9be5ee09cec0396447",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530171155-b86259f17acec54b590d25cb1b7e4956",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527109288-78dc2def4c0ce4be6e828eb7cc350b92",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526573363-b6e67d9ba0399c569123f9ede8639a98",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528287379-12e7e032dc3cccce9de9709490639d5d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527699146-d27e72a8417cb007d31255e4396fcb96",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527693984-7917af908cf816b9a214ff21c0a5234a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528923401-f8c76f5f04578a65fc67c3aa33ca119d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529402468-b2eca7ebec0c0347731dea2a8d042f71",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529742504-76a57a77b2d0b74673ab945b03f6be34",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527092734-e078657e9774b03b52ebef5ebb35c8ea",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527282010-a83203aa5feeb8f5e3adb139481fc137",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527737914-5869f86af685a7396d3df37492345e29",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529502924-b481b1bc55f0012928f474b7a5588786",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528578054-b7e3cddb7d52c99109d1f1c2d2707f21",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527747711-c48773f5f8628729ee23256ae08e3ae0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528292871-9e9d9af9d1c3b054004b19bfdb61d026",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526936792-d9a69ca9887ee35af180e4e69c1d2d05",role="Worker"} 0 1732530713886
      metrics_RegionStartFailCount_Count{role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528892586-97b005268583f157b1d677571b7222f8",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527130614-2d0981e3b094687391d3d7aacc7474d4",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529442005-6fdd20c9408f2c3942fe92052f0a6c47",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527705678-60d57edccdb5d089b5ae3700a1c4ca7f",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530084952-8f24ce537152ce228a30e907b8e0e11e",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528297433-ea5d762bacba9ede14ae0aee5baf28f9",role="Worker"} 1382 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526459238-d5466c7447f4ec52ced1deb06ff0f593",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528284794-355cd88af703cd32eb17a48ec1b31bb4",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526053949-2bda677724b1423b4c183038fd5af220",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529559518-4ea082031e8b1d6f4de3dd1d5e581763",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527725979-a3a540204d9919a027596604181c1e6a",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526538554-e0c6a5abe5c3856d111a2871e5a747d6",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527096874-56fe857b1f4699bd9265d2c0dd2eaa93",role="Worker"} 0 1732530713886
      metrics_FetchChunkSuccessCount_Count{role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528886145-b8cdda39e3fac56bcf8ebedb1faddba0",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528939008-d75897bb10893f65bb162b700c9e0061",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732526470677-302531f52c12203e913fb34b9275cec9",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527049346-53e0296a87879d353649afd5200d4b92",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529508534-5df0f5b605c8301a649d3ff13933669d",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732527146188-426147d30599320f71c7fcfca0ddbeda",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732529493092-8330177b674cb1c679192b78d6d3f0da",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732530168098-9c077ef168369d5d207fb3e28659cc22",role="Worker"} 0 1732530713886
      metrics_ActiveConnectionCount_Count{applicationId="1732528924202-d82ec4ec8d8b17d46bdd91c24845bd5f",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528933888-711649c94239fb156b4d199d5a195618",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527875024-648fd0719db9fdb016908d09b9106f2e",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528711475-4288ac232bd0ca846c87b87f688bc5b2",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527423426-c7dd2f2adae9428c4296a2401f9f4d2a",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526535130-e6e7085aa16e2f7f7f3396078d477b84",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526534099-5864c580a6e3bdc1abb63f01b67d382e",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530010315-0bb86daa789b42b0ded2609fcf8057ed",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529519685-6a11deb87d577a1da4822d9d41274d66",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529526192-127dc38b63d6f096a3a24f8f59f92230",role="Worker"} 68 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527705283-c4f723dd37bf54a7623de4167a7ed0c0",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530174965-a7fd29b269ec2c29c01ed9c8d43f7e71",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529529890-c7b61d2a6df80c83b2ab855abedbf9d9",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527106005-539b79baf3d024ba93dea3a7bcd3e0b8",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529219703-37e052bc66589ce81e02f8ab3f636ea5",role="Worker"} 41 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527703199-577c800bfffec5295ca1efbaa0003dec",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528577579-68f221491d463ec26a0f0f9d8ab8a526",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529080108-40f80c5ed008cff208309b01e64e44ac",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528479523-3d3dce84ab293406ae908cd4765b53ff",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529484787-59ddd1960acc7df689b60d18b839cf9c",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530165881-401cedae9eda58642b04df62f369aaab",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530081010-5a9ed5d4f34f911ab69f415953ecbf8e",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527097091-a64dc25cb7fe231bbdd1c6c79c7a066d",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528289097-9f58be9d1fccb7573a290ed6ebc6ad5d",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528284814-2f84dfef74caf886cb7223a8f6f3731f",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528289670-c32026ac6e92cc10a8c14a4f8b0a353f",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526379501-dc028889cc0604129f9f65501511b336",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527731034-ab7d40bf1e49420f6facf2b4bbb2be33",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528922581-42de7d9ff0dbd1076677403a447a05e3",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530519311-94f9a6c39cd03a35e83b12d4c00c6cf2",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528935522-c0c2707c4fefa54dd4db80720b9f2a38",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530179445-0c56c6c6adc270154c6e6501d67cc3ea",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528962897-18f792ad8d6dec39f31059ab48bc4474",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527093455-463917f8568022a4bb9a8be42413aaee",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529219100-8e84cf63846263abf3e3c4d738f2d197",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528292493-d64cc6a69570e6b5fc45c34522a5d4ba",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526466986-def6096f8a6aeed3cddc5122328f6273",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526675633-627c5366e18d5a14d603e6e530e50460",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732525913605-90a8d7f511bd0be9a061f2326d4df854",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528018288-4167caf27e8cbe856246051f6c2ff6f1",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528282292-2f9349f09d73e9aac3f4fc51981332fb",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527107293-38dee61fa8d1bffbb3015c4ff2156c0a",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528319823-3a9862ae42aa5c0e8bd49aaca240b5e5",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530223981-c4094ae2c1df82279d97f0006e449653",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528934969-e19b0f0d4183e6f6de5179fcfda5fd44",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527744817-9608a6d8a44d42dc6b61f751ad9edfda",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527111688-693869d490a658298837f76464b22238",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529667493-f44a1eb52fdae3d646f500507d0cf4fc",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527098024-0a8b323a2e86b90ddbbe275f443f7374",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527727584-6edea777f32cd2f600069c4518118006",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529520999-2ab217f89de6fd5eb1a47bd70dd614f2",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529534323-c7c9016453255cd81c89b5a0eed527e4",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528365399-3146a220d07d54855ffe51b434e3170f",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528361272-4fca52711d38b552b6b54a507a9318fc",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528018107-cb311e2136a0b2eb3ae0ad86824e1c96",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530001297-8388f16398dd35e4342405d9aca399cc",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529484007-63f9681ec471cb9620413823e0d466ce",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526087463-64e3e98db41df0e95097b5bce3273cc6",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527690086-2a7cd263717fc1db86badb0d50d87a4f",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526469849-00d37dcddb608238509d04310f8cc1cb",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528886352-e1d397cb1c7ef2d02aa6ab8d3fb84c48",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529998107-5566b7eb3070492432e848ea74d38353",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732525603819-dafccd7320cdbec26538e55654e2d09b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527427801-c58d1fdef19097a08fa2e9ecea19bafa",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529494254-773df449105880c0ec3de6894eb684a8",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527098215-72bf1a677e23a5d83057098403c1999f",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527732880-4aaaadf66f00b8b0ba77b0db5c5dc597",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530175017-3a0979ba26722d412736144a53a6acb8",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528324382-740f37cdceb2fbc656e21848e0bf213b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529077757-49edd7ca19e892abb7e927344d7ac382",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530078497-8dbafb8d6cdbcc0b89bbff27cfd0a03d",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527136678-51a60a99bc05a9bc49d81dda6d5122f1",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526474595-7a4248e5f038d07e5ddb67929ec42716",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526500612-3a171dc5d9b5db230243e1af4466c9c5",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527424598-5bd604b237039deda1d49f76616304cf",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529326078-7e59f9f5d376ad36e25801d3bf1a6b70",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528936146-6fe14900fcabcfa4cc1712aa58212125",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527684806-0c3a37165eb79d43e9b8fe8597b917d6",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528578970-bb1bd0897a7279010a6f343b4aa44311",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528355070-78dc9da4ab38ddbaf6d85143c76f8543",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526604507-3f1954d037d4255dc0268dd273787bc7",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526583763-965fc346f0f07246643fa4a02896e494",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528287405-b4401a038ea1eeaf87e35e64e7933666",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527133631-c93b14315fffc2782002fd7a384ee380",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529506142-154ffd4565ebab3679bd339a34c6fc95",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527980299-a6139ba1d7387ff7cd8167ba8a9c40c2",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528302906-8f90f37cc2595d1efb5affa3989aacf4",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528360880-9487ee2970b8900a326226a6288aec2c",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529335114-d97d5700952786b3878a8793c0584a1b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527701035-04ac26ec8903fda91330acb78d2fb674",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528941566-c9f92077596acc088bbea7b3b851423c",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527101408-d4d45dc3a8b163c9234b875e55703fe3",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527691793-f5b8e87e98660896c6a1803598993d0b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526193186-730e0f186c9324e022cb525b6caa2677",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526584634-0bb4376a1d1a686f5e91e77ef610584d",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529493074-cb28da1de22743bad6783256bf91eb97",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530176935-397e0ef1430e4f955e89a11e7b9c1acb",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528883881-52f1a06bc801a8bd77368ffa1ba56bb7",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527146093-8167c0e97cee288c391cf96eff64d222",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528319081-600bf6f818143d598aa104baea016f08",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527698244-5a72858196b00b9742ba0f2a2203d18d",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527687698-1f907d5acd78ca0ade72b9ec8080e096",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530075682-1086befd7b9f5858d7c17eec19030541",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527139450-fd2268a82fd78cf8cbbf2b5ad258c98b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529220689-05ec05eb3888acd498c2a9ccef590418",role="Worker"} 0 1732530713887
      metrics_ReplicateDataFailCount_Count{role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526791448-f292d5072bb2601dab5b72469a772fd6",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527730898-b4dfe76786ea74b3a6297370829f1800",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{role="Worker"} 53 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528578891-28b7910376fa38c89d267101ab832f21",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528712228-2c8d237a96c2cbfd62c88260e6b34585",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528504743-bc0abd4ab476ed5f02762db397cd9ff0",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530302298-d4c2e2059ccbbcb09edfb55addbe5038",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528283479-7a2211e0757f2e826a8cfec8f125a70f",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526229882-ecc25a28fe7c34252f5462d7b7cf9ce1",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527111476-734752a5915c8f335702fd313c621147",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528923214-58680ceece1f97e1e4815bfd29874c65",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530340111-a55bd74b732a7e73f198415e3286f923",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526471887-2d21db0beccc2965bc314f81665c03b5",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528901278-9bdbdc986840554e9b05ca9da91402c1",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527692481-661cba9133508871a1508c54df1c44a1",role="Worker"} 10 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528885381-0a349a7d579ab163ba09167eecf68e2c",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529217902-c5d71a64bbb6d2837e15cfd876ae22ce",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530166524-9c749a88013562f5d6d618c4f90c13e2",role="Worker"} 6 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526464087-1bccb8ebde1854eaadbc21b1bf2cb277",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527134585-7601be68b692ba3e8dbf66ea7265a506",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529921087-cd0a1ae5e9da01d116f443f07139c5c1",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528884300-bf5622bb015a330f227bc2f00ec5a1cf",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527692440-05d87823b9c8e7b949d8862570ea605b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526540335-0319113c0d0b287a2e8db1648a19f2c3",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528132324-6afc9c8c2647f34b6fe19605d35c537b",role="Worker"} 24 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528289097-9d9cc203795e14565de6509b41850123",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530159568-3e01533b46ba9e61ac87c5c5c730682c",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527134959-f49801f6acdbdf71f37cfed9ac3dd6d8",role="Worker"} 0 1732530713887
      metrics_ReplicateDataTimeoutCount_Count{role="Worker"} 0 1732530713887
      metrics_PushDataHandshakeFailCount_Count{role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530094103-fc08d954fed415f9dfb7ee523efec21d",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526536703-2fbcf08a96058639be39d45fd1a2fb03",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527748895-1221b55bfc1f09e25d1a3fb881e7832c",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528895604-b6208a2526e00645bd739d9419539129",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528891209-615567b680fff9e44384d5e3ba82750d",role="Worker"} 50 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528503106-3788857dbc1dca8d91dcc49d51db2e6a",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530090274-1d2236d59feabd82f659faa98989c493",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528885605-90af7d08c3347915f3683f243c04cd0e",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526345349-ad59d8bce3ebcc23eee5370845280fab",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527106595-63be0a49864c06a37fde6b711cc1fd45",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527761274-6745d4dd42d00294d857cfc8675bd7f5",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529558866-6aef1fa3a1865c4f91db9816c1986c8d",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529490452-2f527ec12e051552bcef5469ebf58dac",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526860996-bb28b0151d422297ec9cd733d105f070",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528901165-dfc1f06c7488640bfb2342057a3af097",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527731184-157e3306792ffa1204f93b26af88c996",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528296135-1a0d8be58b70125a1bcbaba900a6c45b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529504816-31f3c45d07a0b0dc8b68aa4ee612843e",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530157332-461b2ee24d5cfbfcc26b1fc1810cd1d6",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530155722-fab7fc88965f05e9477adce780b6b42a",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526864801-8f93eeba2fd7d7e165e6bfadcc2c6111",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732525874902-68ac7aab3c05c68c36411037cd765751",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526977496-90572804ca7bbb5b93ca425d619594c1",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527929197-3c18e30a7dc25fee939e2ec230f9c95b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732525875100-8a9ac5bae6d8801cc7c691bbd386c110",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527099105-3f5a5e54dbbf76671f9f1f74505640de",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528897335-4b0fa2cc32a8989fb6121b7cfc0618b8",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528650887-1857a431e433c999f6722289c3a581ac",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530373275-6789a808648fcd842b2a8dc8efacf5b8",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528319579-dc7f675382aead96f06801d022731b08",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528391653-3fc27d72a37e04ccc562384fbfbdc195",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528657795-e5a6d53b8012090ad02a406164b90a3b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528294693-422390b730395a2e6cdf7c0160c3a85d",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526464912-858af560136bc7273079252494431630",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527875148-689088acaa4508838a7bf8941b20bf6c",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527726290-b81e835e2d8d27e8de26c895e8c10546",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528888133-5859024377149491803c796928235e9a",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526470891-0c2cd75c07832fc15a21f4b4e647d21b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530166703-0ce88cf942316c72b63bfe2c1bde080c",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732525875099-fdcf4a7e85a7c703f8bbc3949c05b5d1",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527699493-65b66a688adfdb5b34936993ec4032b5",role="Worker"} 67 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529535135-6a932fc9f208394a15a7ca6dcc14e372",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526682324-ee23cc59ddd029f575d96156448fa880",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526719142-04e263db8233d3c136585352eca217ee",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526577893-15bf38f86033ddb9d696e09043eedc40",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529330361-94cf1fecbe620c78baa491998330184e",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530154898-2fa138625d6a1d9f71ac3e02de1655f7",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526501795-e5fdc46162bbcaba622fefca6b563cdf",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528020687-e34a4dc728522a309c085e9db5f2f56b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526575881-bef4c42a9bcb98c3770d62c63f022daf",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526714007-dde940b8e73fc340282f4358b88b2b12",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526229198-07625b56ebe3ac9a77c22a9bf861aa95",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528361980-625d796f61f2f40ef7f1e8f52de74d88",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528896033-a7e2c63b2172893353e9368573c9b97c",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527429519-7c9eda370f58d0003ac9d011a6d1e79f",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527738297-25360a0bbcd3f1273a8a24cbf7e0979d",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732525565280-0001cc0815156ca1f1278d6d15904469",role="Worker"} 24 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528367203-918e5aea05bcb8f6fbe58bbbb2b78fc2",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528941790-4514c5344d7ac0eb302e99c6f0ec42e6",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528967215-c000adfbb92b5c81ad78c8a25d68c610",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526502997-03f207b1df50637c936ed6be9dfdac7b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526466251-b376c179ac62674cb6dc40e2d2ac2af9",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526473889-d6435ffc371feb4716707cefb49b01b5",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529527606-025369de1ff2a7943227320b93640649",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529589753-10860e8a0d12ac424201cc099e9eab21",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527427559-e0de181398dd33bc472d0f269c710e9b",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527700501-2964ed6259dc984807637a8cc4dedb0a",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530520407-5fbe04da1fc75db8077e4919456acf96",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732529530321-64a1801df7cafec3299b6f59f924e171",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528966574-f1a2d9173a07d3882fae6fbc25258be6",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526533614-1344f2cbfdcd2c3979492f8a21117483",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527111085-671e48da8b7a733850bef3d803165de8",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528136244-6ce663daa5aeeccce4afaffda4fd14a5",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732526648102-b6305bfe53af026bcd16a15864730428",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528933202-d8a45df2cd3079dc29081da46cb9ec06",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732530081480-e134dcc6fe59631ddda7a6f2632a9a29",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732527694499-b5d6bb9ca210e49f2d1d035c1afaa690",role="Worker"} 0 1732530713887
      metrics_ActiveConnectionCount_Count{applicationId="1732528469952-0ff6b36ace6295717a08acbd01dac468",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528899340-285d6f62e1b32e00ab66165b2b8d2f18",role="Worker"} 0 1732530713888
      metrics_WriteDataSuccessCount_Count{role="Worker"} 9346903 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529999431-4ef5204c7ab2661fadb8ba8eee664fc6",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529507949-0f4314a79fe0871a222a5ea767f68fa6",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528895783-ee75fac6bf1264204e41851633d852d5",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529501498-f3bd0c29ca942ba9ecd18cc964b1261c",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528964504-e489e58bd2bc5a80116bc1985d53a437",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528928623-1b5d5d5eaf59f34f78ad2fc4788119ef",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527696822-44bd50738f2dae4c76849da830f5800b",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527133652-bf9e3ec7dbdd3675f5e56764afa140fa",role="Worker"} 18 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527871385-4a69f7e9f68d34b4a0b2b646330cb824",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530101210-767d7fe322ff68fe274ea9538b56fbfd",role="Worker"} 0 1732530713888
      metrics_WriteDataHardSplitCount_Count{role="Worker"} 0 1732530713888
      metrics_ReplicateDataWriteFailCount_Count{role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527700468-d91659f1d7d87d3443bad1d5fdafbbef",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527110874-7a384a7fa3ac98b0d7c79aeaa81186a2",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529148315-1bce39e4da694052e310dfb20488e885",role="Worker"} 0 1732530713888
      metrics_ReplicateDataCreateConnectionFailCount_Count{role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527131486-16804496aba4b90df875612e34e36bed",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527872525-5ca09a01cba1495a558fc3dec5feb784",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526538219-74f0dc331dffad72c4043944f97edbbd",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530156947-7851486d3e82c350daec7c6579182610",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529520487-52e3e263099723378272a4bb8d988f74",role="Worker"} 0 1732530713888
      metrics_OpenStreamSuccessCount_Count{role="Worker"} 223349 1732530713888
      metrics_ReplicateDataFailNonCriticalCauseCount_Count{role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528898107-7527ffd6cd3f1d61d38b4a511f776967",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530087892-1ef4be0d08091ff67a1e9bb1980cada3",role="Worker"} 5 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529488915-3abb1aeafd7c6dc4c52fed8c5612d97d",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528882890-b04cd2c939c54e90876133e73cbca30e",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527110348-2cf91339a710bcc07a91e3aed2d15238",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527687080-4abe78811f5811593796237017125fc0",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526156606-307f545ae365a867809fc2084ce39195",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526463311-3a0e2e16601369f74ddbcceb7c7e4dd5",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526052305-f888d4395b0113e323735a5664254d72",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526585635-3e67e5c539def20ea0a0b15539a0f9ad",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528650497-303748551260338fe1ac212b6a8d587a",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526123839-4bff56f9ab2d5a7126f366f2af83f7be",role="Worker"} 148 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528577767-e4f84bcb0b22635d5d5b5b0ff035f24a",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528905508-e6565b9d89156adac80e49e911bcd3fd",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530339876-7e5a3b6a9c77326017809d9a017f649f",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530530411-0a16768de2be4350aafe79f8757f1581",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526538255-1a95f0f8b46951374084672f5ecd8cce",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526717588-2d0e13e6673e8356517a71c0bf47a589",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526084620-4b3f01c2af9382ba7462e642d86f65c6",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526380195-d1f0d4185b10ca0348e9136864e7c403",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528888963-197b3f413f1b6f1add78ec9cb015eca9",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528899171-c43881a88d901d6018533603f3ba22b9",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527132692-63ed9755e9749ec3d3d50de3e58959b7",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526157626-a807c3e759027d497e5f2f4ea7a27319",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527275059-47b04c22d348b691d194e31c3d5a1a9a",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528300591-3cdfa03bd869fad5b15553ce697c5e71",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529216296-b7b414d4b4090a9ea82b05784777c38f",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529494188-fbfbf8791affa6dce65a9f97dbda3ed4",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530085081-da941f10fd3035dd0024b324214cdee0",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529737124-f5538bbefce7b038cf2c3746cdbdd81d",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526158920-1da2a9284cb997213b168739e5e25da7",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528933542-15c36c91d1296fc528e5968c8710316d",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527427386-ccc9e0da7f8e5193a8fa0e118f201a7e",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526195600-b6fdfd32d519264d9a7f42a291835b72",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530520829-32c7d05550b9479be950882976b583ef",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530161971-f2d9daf890b528f49b99cf8203f0e13f",role="Worker"} 65 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526713233-f5413311f1f362ade9a6f4d9b2302efa",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527097533-f34dd616983fc55efe69680d174659ae",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526648799-3eee6967ebc65afd9542dab4bb9029df",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527730087-91bf5a0c8eb29b7164cb99e7c158bead",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529292891-c3077caae31103869e71c70a43f22456",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527276464-69268f7c76773f357dfd390f7912a29f",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529478492-4cb6f955fd8dd74f65847c036a1c13aa",role="Worker"} 0 1732530713888
      metrics_WriteDataFailCount_Count{role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526869902-90942a330cb5a04b69779a5ef8d7319c",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529491930-cd104c40fd95bc516703e230a705d698",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526467134-c3393330ed0db1227152f8b75776d2a8",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529530313-6e3fe15699afe127109983e8e6f9876c",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527692998-5569f20887aa6f1badd2c89135a5fc22",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529070171-babffc3ebb7a7ee869900f941ce365e2",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526463164-2fc6bdba94122f669c1a595417e7ffec",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526537493-ca0950a52d62c336c1a60fd2c13977dd",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530630175-7915e2a3dcba869db69403e1df856753",role="Worker"} 6 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527693235-fa32f087e28cd22bab12898acb04bfca",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528934595-1568874cde929fb4109787edc6f4e722",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528503669-bfd5e7ee758a7ba842d98025d57910a6",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526500199-a1e5feb97e69b32fc1c3370a9f20e5e0",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526538913-56f88a218b2a1bc2c506721b88df3ab2",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530080608-85c0c3d9fd221812dba1cdf0a9162976",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528929529-5e8d8d4af32db078bfd277472800523f",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530087820-8fcd8139936805a70f498051391fb913",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528359636-cd646b58923bd9566e0f96be5f2df34b",role="Worker"} 0 1732530713888
      metrics_OpenStreamFailCount_Count{role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527538204-f5de3c138bf3f96259444e15975d7366",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527015578-fb2827538808791e34be8abea1833443",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527144313-5b6d9196ad0ec34eb95d187c17c29e7a",role="Worker"} 72 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530077404-0d6651dd6f7cb75d083df6a78d178187",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732527101220-67ddde19fd8fa0d7b986e6614bc6613e",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529327912-eb69220853ce353ce3779cdb9ee932e4",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530304192-9c165a0139c3b931f42a29b6700aeb0e",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529494302-6f882f3ad76a37b16faca9de15709ef2",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528891803-b35f191eb0e63844ba8d6ce5dced63c9",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530336672-5e61e598bf765493f23045adefd2549a",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732528929930-80dda13dc3376df0b43db0d8195519a8",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529528793-d0170632266b28ffcb4bb2d77bd93bec",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732529531567-cef877867babfdb2310eb1244a0a0656",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732526232896-93d45d9f8f98cc8f73290157a647ccfb",role="Worker"} 0 1732530713888
      metrics_ActiveConnectionCount_Count{applicationId="1732530161794-10323eb0ebbc255f0e59fd00d32071ed",role="Worker"} 0 1732530713888
      metrics_DeviceOSTotalBytes_Value{device="vde",role="Worker"} 11904850821120 1732530713888
      metrics_DeviceOSFreeBytes_Value{device="vde",role="Worker"} 11304668966912 1732530713888
      metrics_DeviceCelebornTotalBytes_Value{device="vde",role="Worker"} 11304788492023 1732530713889
      metrics_DeviceCelebornFreeBytes_Value{device="vde",role="Worker"} 11304661364736 1732530713889
      metrics_DeviceOSTotalBytes_Value{device="vdh",role="Worker"} 11904850821120 1732530713889
      metrics_DeviceOSFreeBytes_Value{device="vdh",role="Worker"} 11304789196800 1732530713889
      metrics_DeviceCelebornTotalBytes_Value{device="vdh",role="Worker"} 11304789255210 1732530713889
      metrics_DeviceCelebornFreeBytes_Value{device="vdh",role="Worker"} 11304786681856 1732530713889
      metrics_DeviceOSTotalBytes_Value{device="vdk",role="Worker"} 11904850821120 1732530713889
      metrics_DeviceOSFreeBytes_Value{device="vdk",role="Worker"} 11304610414592 1732530713889
      metrics_DeviceCelebornTotalBytes_Value{device="vdk",role="Worker"} 11304787482389 1732530713889
      metrics_DeviceCelebornFreeBytes_Value{device="vdk",role="Worker"} 11304603041792 1732530713889
      metrics_DeviceOSTotalBytes_Value{device="vdd",role="Worker"} 11904850821120 1732530713889
      metrics_DeviceOSFreeBytes_Value{device="vdd",role="Worker"} 11304587526144 1732530713889
      metrics_DeviceCelebornTotalBytes_Value{device="vdd",role="Worker"} 11304788226417 1732530713889
      metrics_DeviceCelebornFreeBytes_Value{device="vdd",role="Worker"} 11304579891200 1732530713889
      metrics_DeviceOSTotalBytes_Value{device="vdi",role="Worker"} 11904850821120 1732530713889
      metrics_DeviceOSFreeBytes_Value{device="vdi",role="Worker"} 11304789274624 1732530713890
      metrics_DeviceCelebornTotalBytes_Value{device="vdi",role="Worker"} 11304789323373 1732530713890
      metrics_DeviceCelebornFreeBytes_Value{device="vdi",role="Worker"} 11304786763776 1732530713890
      metrics_DeviceOSTotalBytes_Value{device="vdl",role="Worker"} 11904850821120 1732530713890
      metrics_DeviceOSFreeBytes_Value{device="vdl",role="Worker"} 11304593068032 1732530713890
      metrics_DeviceCelebornTotalBytes_Value{device="vdl",role="Worker"} 11304788004534 1732530713890
      metrics_DeviceCelebornFreeBytes_Value{device="vdl",role="Worker"} 11304585437184 1732530713890
      metrics_DeviceOSTotalBytes_Value{device="vdc",role="Worker"} 11904850821120 1732530713890
      metrics_DeviceOSFreeBytes_Value{device="vdc",role="Worker"} 11304789196800 1732530713890
      metrics_DeviceCelebornTotalBytes_Value{device="vdc",role="Worker"} 11304789250148 1732530713890
      metrics_DeviceCelebornFreeBytes_Value{device="vdc",role="Worker"} 11304786694144 1732530713890
      metrics_DeviceOSTotalBytes_Value{device="vdf",role="Worker"} 11904850821120 1732530713890
      metrics_DeviceOSFreeBytes_Value{device="vdf",role="Worker"} 11304789299200 1732530713890
      metrics_DeviceCelebornTotalBytes_Value{device="vdf",role="Worker"} 11304789334419 1732530713890
      metrics_DeviceCelebornFreeBytes_Value{device="vdf",role="Worker"} 11304786796544 1732530713890
      metrics_DeviceOSTotalBytes_Value{device="vdg",role="Worker"} 11904850821120 1732530713890
      metrics_DeviceOSFreeBytes_Value{device="vdg",role="Worker"} 11304789372928 1732530713891
      metrics_DeviceCelebornTotalBytes_Value{device="vdg",role="Worker"} 11304789373447 1732530713891
      metrics_DeviceCelebornFreeBytes_Value{device="vdg",role="Worker"} 11304786870272 1732530713891
      metrics_DeviceOSTotalBytes_Value{device="vdj",role="Worker"} 11904850821120 1732530713891
      metrics_DeviceOSFreeBytes_Value{device="vdj",role="Worker"} 11304789757952 1732530713891
      metrics_DeviceCelebornTotalBytes_Value{device="vdj",role="Worker"} 11304789801911 1732530713891
      metrics_DeviceCelebornFreeBytes_Value{device="vdj",role="Worker"} 11304789757952 1732530713891
      metrics_DeviceOSTotalBytes_Value{device="vdb",role="Worker"} 11904850821120 1732530713891
      metrics_DeviceOSFreeBytes_Value{device="vdb",role="Worker"} 11304634548224 1732530713891
      metrics_DeviceCelebornTotalBytes_Value{device="vdb",role="Worker"} 11304788897617 1732530713891
      metrics_DeviceCelebornFreeBytes_Value{device="vdb",role="Worker"} 11304632037376 1732530713891
      metrics_RegisteredShuffleCount_Value{role="Worker"} 30 1732530713891
      metrics_RunningApplicationCount_Value{role="Worker"} 3 1732530713891
      metrics_SortMemory_Value{role="Worker"} 0 1732530713891
      metrics_SortingFiles_Value{role="Worker"} 0 1732530713891
      metrics_SortedFiles_Value{role="Worker"} 0 1732530713891
      metrics_SortedFileSize_Value{role="Worker"} 0 1732530713891
      metrics_DiskBuffer_Value{role="Worker"} 0 1732530713891
      metrics_NettyMemory_Value{role="Worker"} 612368384 1732530713891
      metrics_BufferStreamReadBuffer_Value{role="Worker"} 167772160 1732530713891
      metrics_ReadBufferDispatcherRequestsLength_Value{role="Worker"} 0 1732530713891
      metrics_ReadBufferAllocatedCount_Value{role="Worker"} 5120 1732530713891
      metrics_MemoryFileStorageSize_Value{role="Worker"} 0 1732530713891
      metrics_DirectMemoryUsageRatio_Value{role="Worker"} 0.0038020833333333335 1732530713891
      metrics_EvictedFileCount_Value{role="Worker"} 0 1732530713891
      metrics_MemoryStorageFileCount_Value{role="Worker"} 0 1732530713891
      metrics_ActiveShuffleSize_Value{role="Worker"} 1585715026 1732530713891
      metrics_ActiveShuffleFileCount_Value{role="Worker"} 242 1732530713892
      metrics_PausePushDataTime_Value{role="Worker"} 0 1732530713892
      metrics_PausePushDataAndReplicateTime_Value{role="Worker"} 0 1732530713892
      metrics_PausePushData_Value{role="Worker"} 0 1732530713892
      metrics_PausePushDataAndReplicate_Value{role="Worker"} 0 1732530713892
      metrics_ActiveSlotsCount_Value{role="Worker"} 0 1732530713892
      metrics_ActiveChunkStreamCount_Value{role="Worker"} 0 1732530713892
      metrics_ActiveCreditStreamCount_Value{role="Worker"} 5 1732530713892
      metrics_ActiveMapPartitionCount_Value{role="Worker"} 5 1732530713892
      metrics_FlushDataTime_Count{role="Worker"} 257945 1732530713892
      metrics_FlushDataTime_Max{role="Worker"} 0.0 1732530713892
      metrics_FlushDataTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_FlushDataTime_Min{role="Worker"} 0.0 1732530713892
      metrics_FlushDataTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FlushDataTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FlushDataTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FlushDataTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FlushDataTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FlushDataTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataHandshakeTime_Count{role="Worker"} 17158 1732530713892
      metrics_PrimaryPushDataHandshakeTime_Max{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataHandshakeTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataHandshakeTime_Min{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataHandshakeTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataHandshakeTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataHandshakeTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataHandshakeTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataHandshakeTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataHandshakeTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CleanExpiredShuffleKeysTime_Count{role="Worker"} 154 1732530713892
      metrics_CleanExpiredShuffleKeysTime_Max{role="Worker"} 0.0 1732530713892
      metrics_CleanExpiredShuffleKeysTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_CleanExpiredShuffleKeysTime_Min{role="Worker"} 0.0 1732530713892
      metrics_CleanExpiredShuffleKeysTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CleanExpiredShuffleKeysTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CleanExpiredShuffleKeysTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CleanExpiredShuffleKeysTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CleanExpiredShuffleKeysTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CleanExpiredShuffleKeysTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_Count{role="Worker"} 0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_Max{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_Min{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataHandshakeTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionFinishTime_Count{role="Worker"} 0 1732530713892
      metrics_ReplicaRegionFinishTime_Max{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionFinishTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionFinishTime_Min{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionFinishTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionFinishTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionFinishTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionFinishTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionFinishTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionFinishTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReserveSlotsTime_Count{role="Worker"} 3438 1732530713892
      metrics_ReserveSlotsTime_Max{role="Worker"} 0.0 1732530713892
      metrics_ReserveSlotsTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_ReserveSlotsTime_Min{role="Worker"} 0.0 1732530713892
      metrics_ReserveSlotsTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReserveSlotsTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReserveSlotsTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReserveSlotsTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReserveSlotsTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReserveSlotsTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionFinishTime_Count{role="Worker"} 64721 1732530713892
      metrics_PrimaryRegionFinishTime_Max{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionFinishTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionFinishTime_Min{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionFinishTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionFinishTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionFinishTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionFinishTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionFinishTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionFinishTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CommitFilesTime_Count{role="Worker"} 12343 1732530713892
      metrics_CommitFilesTime_Max{role="Worker"} 0.0 1732530713892
      metrics_CommitFilesTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_CommitFilesTime_Min{role="Worker"} 0.0 1732530713892
      metrics_CommitFilesTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CommitFilesTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CommitFilesTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CommitFilesTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CommitFilesTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_CommitFilesTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionStartTime_Count{role="Worker"} 0 1732530713892
      metrics_ReplicaRegionStartTime_Max{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionStartTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionStartTime_Min{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionStartTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionStartTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionStartTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionStartTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionStartTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaRegionStartTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataTime_Count{role="Worker"} 0 1732530713892
      metrics_ReplicaPushDataTime_Max{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataTime_Min{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_ReplicaPushDataTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionStartTime_Count{role="Worker"} 64722 1732530713892
      metrics_PrimaryRegionStartTime_Max{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionStartTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionStartTime_Min{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionStartTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionStartTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionStartTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionStartTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionStartTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryRegionStartTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FetchChunkTime_Count{role="Worker"} 0 1732530713892
      metrics_FetchChunkTime_Max{role="Worker"} 0.0 1732530713892
      metrics_FetchChunkTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_FetchChunkTime_Min{role="Worker"} 0.0 1732530713892
      metrics_FetchChunkTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FetchChunkTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FetchChunkTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FetchChunkTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FetchChunkTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_FetchChunkTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_OpenStreamTime_Count{role="Worker"} 201437 1732530713892
      metrics_OpenStreamTime_Max{role="Worker"} 0.0 1732530713892
      metrics_OpenStreamTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_OpenStreamTime_Min{role="Worker"} 0.0 1732530713892
      metrics_OpenStreamTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_OpenStreamTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_OpenStreamTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_OpenStreamTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_OpenStreamTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_OpenStreamTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_TakeBufferTime_Count{role="Worker"} 0 1732530713892
      metrics_TakeBufferTime_Max{role="Worker"} 0.0 1732530713892
      metrics_TakeBufferTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_TakeBufferTime_Min{role="Worker"} 0.0 1732530713892
      metrics_TakeBufferTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_TakeBufferTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_TakeBufferTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_TakeBufferTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_TakeBufferTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_TakeBufferTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataTime_Count{role="Worker"} 9346875 1732530713892
      metrics_PrimaryPushDataTime_Max{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataTime_Min{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_PrimaryPushDataTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_SortTime_Count{role="Worker"} 0 1732530713892
      metrics_SortTime_Max{role="Worker"} 0.0 1732530713892
      metrics_SortTime_Mean{role="Worker"} 0.0 1732530713892
      metrics_SortTime_Min{role="Worker"} 0.0 1732530713892
      metrics_SortTime_50thPercentile{role="Worker"} 0.0 1732530713892
      metrics_SortTime_75thPercentile{role="Worker"} 0.0 1732530713892
      metrics_SortTime_95thPercentile{role="Worker"} 0.0 1732530713892
      metrics_SortTime_98thPercentile{role="Worker"} 0.0 1732530713892
      metrics_SortTime_99thPercentile{role="Worker"} 0.0 1732530713892
      metrics_SortTime_999thPercentile{role="Worker"} 0.0 1732530713892
      metrics_active_thread_count_Value{role="Worker",threadPool="celeborn-dispatcher"} 64 1732530713892
      metrics_pending_task_count_Value{role="Worker",threadPool="celeborn-dispatcher"} 0 1732530713892
      metrics_pool_size_Value{role="Worker",threadPool="celeborn-dispatcher"} 64 1732530713892
      metrics_core_pool_size_Value{role="Worker",threadPool="celeborn-dispatcher"} 64 1732530713892
      metrics_maximum_pool_size_Value{role="Worker",threadPool="celeborn-dispatcher"} 64 1732530713892
      metrics_largest_pool_size_Value{role="Worker",threadPool="celeborn-dispatcher"} 64 1732530713892
      metrics_is_terminating_Value{role="Worker",threadPool="celeborn-dispatcher"} 0 1732530713892
      metrics_is_terminated_Value{role="Worker",threadPool="celeborn-dispatcher"} 0 1732530713892
      metrics_is_shutdown_Value{role="Worker",threadPool="celeborn-dispatcher"} 0 1732530713892
      metrics_active_thread_count_Value{role="Worker",threadPool="celeborn-netty-rpc-connection-executor"} 0 1732530713892
      metrics_pending_task_count_Value{role="Worker",threadPool="celeborn-netty-rpc-connection-executor"} 0 1732530713892
      metrics_pool_size_Value{role="Worker",threadPool="celeborn-netty-rpc-connection-executor"} 0 1732530713892
      metrics_core_pool_size_Value{role="Worker",threadPool="celeborn-netty-rpc-connection-executor"} 64 1732530713892
      metrics_maximum_pool_size_Value{role="Worker",threadPool="celeborn-netty-rpc-connection-executor"} 64 1732530713892
      metrics_largest_pool_size_Value{role="Worker",threadPool="celeborn-netty-rpc-connection-executor"} 2 1732530713892
      metrics_is_terminating_Value{role="Worker",threadPool="celeborn-netty-rpc-connection-executor"} 0 1732530713892
      metrics_is_terminated_Value{role="Worker",threadPool="celeborn-netty-rpc-connection-executor"} 0 1732530713892
      metrics_is_shutdown_Value{role="Worker",threadPool="celeborn-netty-rpc-connection-executor"} 0 1732530713892
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data9-cleaner"} 0 1732530713892
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data9-cleaner"} 0 1732530713892
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data9-cleaner"} 0 1732530713892
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data9-cleaner"} 4 1732530713892
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data9-cleaner"} 4 1732530713892
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data9-cleaner"} 4 1732530713892
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data9-cleaner"} 0 1732530713892
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data9-cleaner"} 0 1732530713892
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data9-cleaner"} 0 1732530713892
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data3-cleaner"} 0 1732530713892
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data3-cleaner"} 0 1732530713892
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data3-cleaner"} 4 1732530713892
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data3-cleaner"} 4 1732530713892
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data3-cleaner"} 4 1732530713892
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data3-cleaner"} 4 1732530713892
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data3-cleaner"} 0 1732530713892
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data3-cleaner"} 0 1732530713892
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data3-cleaner"} 0 1732530713892
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data4-cleaner"} 0 1732530713892
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data4-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data4-cleaner"} 4 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data4-cleaner"} 4 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data4-cleaner"} 4 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data4-cleaner"} 4 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data4-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data4-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data4-cleaner"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data1-cleaner"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data1-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data1-cleaner"} 4 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data1-cleaner"} 4 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data1-cleaner"} 4 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data1-cleaner"} 4 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data1-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data1-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data1-cleaner"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data2-cleaner"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data2-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data2-cleaner"} 4 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data2-cleaner"} 4 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data2-cleaner"} 4 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data2-cleaner"} 4 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data2-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data2-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data2-cleaner"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data7-cleaner"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data7-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data7-cleaner"} 4 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data7-cleaner"} 4 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data7-cleaner"} 4 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data7-cleaner"} 4 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data7-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data7-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data7-cleaner"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data8-cleaner"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data8-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data8-cleaner"} 4 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data8-cleaner"} 4 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data8-cleaner"} 4 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data8-cleaner"} 4 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data8-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data8-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data8-cleaner"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data10-cleaner"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data10-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data10-cleaner"} 4 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data10-cleaner"} 4 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data10-cleaner"} 4 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data10-cleaner"} 4 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data10-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data10-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data10-cleaner"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data5-cleaner"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data5-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data5-cleaner"} 4 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data5-cleaner"} 4 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data5-cleaner"} 4 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data5-cleaner"} 4 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data5-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data5-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data5-cleaner"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data11-cleaner"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data11-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data11-cleaner"} 4 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data11-cleaner"} 4 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data11-cleaner"} 4 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data11-cleaner"} 4 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data11-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data11-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data11-cleaner"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-disk-/data6-cleaner"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-disk-/data6-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-disk-/data6-cleaner"} 4 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-disk-/data6-cleaner"} 4 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-disk-/data6-cleaner"} 4 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-disk-/data6-cleaner"} 4 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-disk-/data6-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-disk-/data6-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-disk-/data6-cleaner"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-device-checker"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-device-checker"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-device-checker"} 5 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-device-checker"} 5 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-device-checker"} 5 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-device-checker"} 5 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-device-checker"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-device-checker"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-device-checker"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@1032568028-/data9"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@1032568028-/data9"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@1032568028-/data9"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@901689694-/data3"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@901689694-/data3"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@901689694-/data3"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@1523084197-/data4"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@1523084197-/data4"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@1523084197-/data4"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@747870628-/data1"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@747870628-/data1"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@747870628-/data1"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@9844218-/data2"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@9844218-/data2"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@9844218-/data2"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@1673916398-/data7"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@1673916398-/data7"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@1673916398-/data7"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@479734028-/data8"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@479734028-/data8"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@479734028-/data8"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@1541046463-/data10"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@1541046463-/data10"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@1541046463-/data10"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@753426788-/data5"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@753426788-/data5"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@753426788-/data5"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@1373577787-/data11"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@1373577787-/data11"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@1373577787-/data11"} 0 1732530713893
      metrics_thread_count_Value{role="Worker",threadPool="LocalFlusher@1624817884-/data6"} 1 1732530713893
      metrics_thread_is_terminated_count_Value{role="Worker",threadPool="LocalFlusher@1624817884-/data6"} 0 1732530713893
      metrics_thread_is_shutdown_count_Value{role="Worker",threadPool="LocalFlusher@1624817884-/data6"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-file-sorter-executor"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-file-sorter-executor"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-file-sorter-executor"} 0 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-file-sorter-executor"} 76 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-file-sorter-executor"} 76 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-file-sorter-executor"} 0 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-file-sorter-executor"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-file-sorter-executor"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-file-sorter-executor"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-data-replicator"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-data-replicator"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-data-replicator"} 0 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-data-replicator"} 64 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-data-replicator"} 64 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-data-replicator"} 0 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-data-replicator"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-data-replicator"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-data-replicator"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-files-committer"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-files-committer"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-files-committer"} 25 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-files-committer"} 32 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-files-committer"} 32 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-files-committer"} 32 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-files-committer"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-files-committer"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-files-committer"} 0 1732530713893
      metrics_active_thread_count_Value{role="Worker",threadPool="worker-expired-shuffle-cleaner"} 0 1732530713893
      metrics_pending_task_count_Value{role="Worker",threadPool="worker-expired-shuffle-cleaner"} 0 1732530713893
      metrics_pool_size_Value{role="Worker",threadPool="worker-expired-shuffle-cleaner"} 2 1732530713893
      metrics_core_pool_size_Value{role="Worker",threadPool="worker-expired-shuffle-cleaner"} 64 1732530713893
      metrics_maximum_pool_size_Value{role="Worker",threadPool="worker-expired-shuffle-cleaner"} 64 1732530713893
      metrics_largest_pool_size_Value{role="Worker",threadPool="worker-expired-shuffle-cleaner"} 3 1732530713893
      metrics_is_terminating_Value{role="Worker",threadPool="worker-expired-shuffle-cleaner"} 0 1732530713893
      metrics_is_terminated_Value{role="Worker",threadPool="worker-expired-shuffle-cleaner"} 0 1732530713893
      metrics_is_shutdown_Value{role="Worker",threadPool="worker-expired-shuffle-cleaner"} 0 1732530713893
      metrics_diskFileCount_Value{name="default",role="Worker",tenantId="default"} 350 1732530713893
      metrics_diskBytesWritten_Value{name="default",role="Worker",tenantId="default"} 1856701493 1732530713893
      metrics_hdfsFileCount_Value{name="default",role="Worker",tenantId="default"} 0 1732530713893
      metrics_hdfsBytesWritten_Value{name="default",role="Worker",tenantId="default"} 0 1732530713893
      metrics_jvm_gc_time_Value{name="G1-Old-Generation",role="Worker"} 0 1732530713893
      metrics_jvm_gc_count_Value{name="G1-Young-Generation",role="Worker"} 279 1732530713893
      metrics_jvm_gc_time_Value{name="G1-Young-Generation",role="Worker"} 3602 1732530713893
      metrics_jvm_gc_count_Value{name="G1-Old-Generation",role="Worker"} 0 1732530713893
      metrics_jvm_memory_heap_committed_Value{role="Worker"} 2147483648 1732530713893
      metrics_jvm_memory_non_heap_used_Value{role="Worker"} 104772976 1732530713893
      metrics_jvm_memory_pools_used_Value{name="Code-Cache",role="Worker"} 36316224 1732530713893
      metrics_jvm_memory_pools_committed_Value{name="Code-Cache",role="Worker"} 36700160 1732530713893
      metrics_jvm_memory_heap_used_Value{role="Worker"} 873787808 1732530713893
      metrics_jvm_memory_pools_init_Value{name="G1-Eden-Space",role="Worker"} 113246208 1732530713893
      metrics_jvm_memory_pools_max_Value{name="G1-Old-Gen",role="Worker"} 21474836480 1732530713893
      metrics_jvm_memory_total_committed_Value{role="Worker"} 2256535552 1732530713893
      metrics_jvm_memory_total_init_Value{role="Worker"} 2150039552 1732530713893
      metrics_jvm_memory_non_heap_max_Value{role="Worker"} 3472883712 1732530713893
      metrics_jvm_memory_pools_used_after_gc_Value{name="G1-Survivor-Space",role="Worker"} 8388608 1732530713893
      metrics_jvm_memory_pools_used_Value{name="Compressed-Class-Space",role="Worker"} 7803000 1732530713893
      metrics_jvm_memory_pools_usage_Value{name="Metaspace",role="Worker"} 0.028244104236364365 1732530713893
      metrics_jvm_memory_pools_max_Value{name="Metaspace",role="Worker"} 2147483648 1732530713893
      metrics_jvm_memory_pools_init_Value{name="G1-Survivor-Space",role="Worker"} 0 1732530713893
      metrics_jvm_memory_pools_committed_Value{name="G1-Old-Gen",role="Worker"} 897581056 1732530713893
      metrics_jvm_memory_pools_committed_Value{name="G1-Survivor-Space",role="Worker"} 8388608 1732530713893
      metrics_jvm_memory_pools_committed_Value{name="G1-Eden-Space",role="Worker"} 1241513984 1732530713893
      metrics_jvm_memory_pools_max_Value{name="G1-Survivor-Space",role="Worker"} -1 1732530713893
      metrics_jvm_memory_pools_usage_Value{name="Compressed-Class-Space",role="Worker"} 0.007267110049724579 1732530713893
      metrics_jvm_memory_heap_usage_Value{role="Worker"} 0.04068891555070877 1732530713893
      metrics_jvm_memory_pools_usage_Value{name="G1-Survivor-Space",role="Worker"} 1.0 1732530713893
      metrics_jvm_memory_pools_max_Value{name="Code-Cache",role="Worker"} 251658240 1732530713893
      metrics_jvm_memory_pools_init_Value{name="Compressed-Class-Space",role="Worker"} 0 1732530713893
      metrics_jvm_memory_pools_usage_Value{name="G1-Eden-Space",role="Worker"} 0.3614864864864865 1732530713893
      metrics_jvm_memory_pools_init_Value{name="Metaspace",role="Worker"} 0 1732530713893
      metrics_jvm_memory_pools_max_Value{name="G1-Eden-Space",role="Worker"} -1 1732530713893
      metrics_jvm_memory_pools_usage_Value{name="Code-Cache",role="Worker"} 0.14430770874023438 1732530713893
      metrics_jvm_memory_total_used_Value{role="Worker"} 978560784 1732530713893
      metrics_jvm_memory_pools_used_after_gc_Value{name="G1-Eden-Space",role="Worker"} 0 1732530713893
      metrics_jvm_memory_pools_init_Value{name="Code-Cache",role="Worker"} 2555904 1732530713893
      metrics_jvm_memory_pools_used_Value{name="G1-Survivor-Space",role="Worker"} 8388608 1732530713893
      metrics_jvm_memory_heap_init_Value{role="Worker"} 2147483648 1732530713893
      metrics_jvm_memory_non_heap_committed_Value{role="Worker"} 109051904 1732530713893
      metrics_jvm_memory_pools_committed_Value{name="Compressed-Class-Space",role="Worker"} 8257536 1732530713893
      metrics_jvm_memory_non_heap_init_Value{role="Worker"} 2555904 1732530713893
      metrics_jvm_memory_pools_used_Value{name="G1-Eden-Space",role="Worker"} 448790528 1732530713893
      metrics_jvm_memory_pools_used_Value{name="G1-Old-Gen",role="Worker"} 416608672 1732530713893
      metrics_jvm_memory_pools_usage_Value{name="G1-Old-Gen",role="Worker"} 0.019399853050708772 1732530713893
      metrics_jvm_memory_non_heap_usage_Value{role="Worker"} 0.03016886964512332 1732530713893
      metrics_jvm_memory_pools_used_after_gc_Value{name="G1-Old-Gen",role="Worker"} 0 1732530713893
      metrics_jvm_memory_heap_max_Value{role="Worker"} 21474836480 1732530713893
      metrics_jvm_memory_pools_committed_Value{name="Metaspace",role="Worker"} 64094208 1732530713893
      metrics_jvm_memory_pools_init_Value{name="G1-Old-Gen",role="Worker"} 2034237440 1732530713893
      metrics_jvm_memory_total_max_Value{role="Worker"} 24947720192 1732530713893
      metrics_jvm_memory_pools_used_Value{name="Metaspace",role="Worker"} 60653752 1732530713893
      metrics_jvm_memory_pools_max_Value{name="Compressed-Class-Space",role="Worker"} 1073741824 1732530713893
      metrics_jvm_direct_count_Value{role="Worker"} 4280 1732530713893
      metrics_jvm_mapped_used_Value{role="Worker"} 0 1732530713893
      metrics_jvm_mapped_capacity_Value{role="Worker"} 0 1732530713893
      metrics_jvm_direct_capacity_Value{role="Worker"} 4070771 1732530713893
      metrics_jvm_direct_used_Value{role="Worker"} 4070772 1732530713893
      metrics_jvm_mapped_count_Value{role="Worker"} 0 1732530713893
      metrics_jvm_thread_total_started_count_Value{role="Worker"} 2075 1732530713893
      metrics_jvm_thread_timed_waiting_count_Value{role="Worker"} 91 1732530713893
      metrics_jvm_thread_peak_count_Value{role="Worker"} 307 1732530713894
      metrics_jvm_thread_deadlock_count_Value{role="Worker"} 0 1732530713894
      metrics_jvm_thread_count_Value{role="Worker"} 294 1732530713895
      metrics_jvm_thread_waiting_count_Value{role="Worker"} 93 1732530713895
      metrics_jvm_thread_daemon_count_Value{role="Worker"} 283 1732530713896
      metrics_jvm_thread_new_count_Value{role="Worker"} 0 1732530713896
      metrics_jvm_thread_blocked_count_Value{role="Worker"} 3 1732530713897
      metrics_jvm_thread_runnable_count_Value{role="Worker"} 107 1732530713898
      metrics_jvm_thread_terminated_count_Value{role="Worker"} 0 1732530713899
      metrics_jvm_classloader_loaded_Value{role="Worker"} 9157 1732530713900
      metrics_jvm_classloader_unloaded_Value{role="Worker"} 0 1732530713900
      metrics_JVMCPUTime_Value{role="Worker"} 2342310000000 1732530713900
      metrics_LastMinuteSystemLoad_Value{role="Worker"} 0.11 1732530713900
      metrics_AvailableProcessors_Value{role="Worker"} 76 1732530713900 

      the worker jstack is  celeborn-worker-jstack.txt

       

       

       

      Attachments

        1. image-2024-11-25-18-40-20-363.png
          331 kB
          zhisheng
        2. image-2024-11-25-18-00-32-990.png
          105 kB
          zhisheng
        3. image-2024-11-25-18-00-16-585.png
          101 kB
          zhisheng
        4. celeborn-worker-jstack.txt
          390 kB
          zhisheng

        Activity

          People

            Unassigned Unassigned
            zhisheng zhisheng
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4h 40m
                4h 40m