| [2025-01-20 21:38:51,309 I 4260 4260] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed | |
| [2025-01-20 21:38:51,309 I 4260 4260] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service. | |
| [2025-01-20 21:38:51,317 I 4260 4260] (gcs_server) event.cc:493: Ray Event initialized for GCS | |
| [2025-01-20 21:38:51,317 I 4260 4260] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE | |
| [2025-01-20 21:38:51,317 I 4260 4260] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR | |
| [2025-01-20 21:38:51,317 I 4260 4260] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB | |
| [2025-01-20 21:38:51,317 I 4260 4260] (gcs_server) event.cc:324: Set ray event level to warning | |
| [2025-01-20 21:38:51,322 I 4260 4260] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:42: Loading job table data. | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:54: Loading node table data. | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:80: Loading actor table data. | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data. | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:66: Loading placement group table data. | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0 | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0 | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0 | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0 | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0 | |
| [2025-01-20 21:38:51,324 I 4260 4260] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 0ccd8c1df7bb8ce132ed2cf66bf7ee637c9681f8a93eeb491671cdcf | |
| [2025-01-20 21:38:51,325 I 4260 4260] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0 | |
| [2025-01-20 21:38:51,328 I 4260 4260] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 64855. | |
| [2025-01-20 21:38:51,579 I 4260 4260] (gcs_server) gcs_server.cc:245: Gcs Debug state: | |
| GcsNodeManager: | |
| - RegisterNode request count: 0 | |
| - DrainNode request count: 0 | |
| - GetAllNodeInfo request count: 0 | |
| GcsActorManager: | |
| - RegisterActor request count: 0 | |
| - CreateActor request count: 0 | |
| - GetActorInfo request count: 0 | |
| - GetNamedActorInfo request count: 0 | |
| - GetAllActorInfo request count: 0 | |
| - KillActor request count: 0 | |
| - ListNamedActors request count: 0 | |
| - Registered actors count: 0 | |
| - Destroyed actors count: 0 | |
| - Named actors count: 0 | |
| - Unresolved actors count: 0 | |
| - Pending actors count: 0 | |
| - Created actors count: 0 | |
| - owners_: 0 | |
| - actor_to_register_callbacks_: 0 | |
| - actor_to_restart_callbacks_: 0 | |
| - actor_to_create_callbacks_: 0 | |
| - sorted_destroyed_actor_list_: 0 | |
| GcsResourceManager: | |
| - GetAllAvailableResources request count: 0 | |
| - GetAllTotalResources request count: 0 | |
| - GetAllResourceUsage request count: 0 | |
| GcsPlacementGroupManager: | |
| - CreatePlacementGroup request count: 0 | |
| - RemovePlacementGroup request count: 0 | |
| - GetPlacementGroup request count: 0 | |
| - GetAllPlacementGroup request count: 0 | |
| - WaitPlacementGroupUntilReady request count: 0 | |
| - GetNamedPlacementGroup request count: 0 | |
| - Scheduling pending placement group count: 0 | |
| - Registered placement groups count: 0 | |
| - Named placement group count: 0 | |
| - Pending placement groups count: 0 | |
| - Infeasible placement groups count: 0 | |
| Publisher: | |
| [runtime env manager] ID to URIs table: | |
| [runtime env manager] URIs reference table: | |
| GcsTaskManager: | |
| -Total num task events reported: 0 | |
| -Total num status task events dropped: 0 | |
| -Total num profile events dropped: 0 | |
| -Current num of task events stored: 0 | |
| -Total num of actor creation tasks: 0 | |
| -Total num of actor tasks: 0 | |
| -Total num of normal tasks: 0 | |
| -Total num of driver tasks: 0 | |
| GcsAutoscalerStateManager: | |
| - last_seen_autoscaler_state_version_: 0 | |
| - last_cluster_resource_state_version_: 0 | |
| - pending demands: | |
| [2025-01-20 21:38:51,579 I 4260 4260] (gcs_server) gcs_server.cc:843: Main service Event stats: | |
| Global stats: 25 total (6 active) | |
| Queueing time: mean = 91.677 ms, max = 254.030 ms, min = 4.945 us, total = 2.292 s | |
| Execution time: mean = 10.170 ms, total = 254.247 ms | |
| Event stats: | |
| GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 28.230 ms, total = 254.070 ms, Queueing time: mean = 196.499 ms, max = 253.124 ms, min = 4.945 us, total = 1.768 s | |
| GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 19.692 us, total = 98.462 us, Queueing time: mean = 116.875 us, max = 130.379 us, min = 104.087 us, total = 584.375 us | |
| PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 4.210 us, total = 16.841 us, Queueing time: mean = 126.960 ms, max = 254.030 ms, min = 253.811 ms, total = 507.842 ms | |
| event_loop_lag_probe - 2 total (0 active), Execution time: mean = 16.389 us, total = 32.777 us, Queueing time: mean = 7.503 ms, max = 14.636 ms, min = 369.568 us, total = 15.005 ms | |
| RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 28.872 us, total = 28.872 us, Queueing time: mean = 6.898 us, max = 6.898 us, min = 6.898 us, total = 6.898 us | |
| [2025-01-20 21:38:51,579 I 4260 4260] (gcs_server) gcs_server.cc:847: task_io_context Event stats: | |
| Global stats: 5 total (1 active) | |
| Queueing time: mean = 1.222 ms, max = 5.962 ms, min = 6.682 us, total = 6.111 ms | |
| Execution time: mean = 46.365 us, total = 231.824 us | |
| Event stats: | |
| event_loop_lag_probe - 3 total (0 active), Execution time: mean = 73.327 us, total = 219.981 us, Queueing time: mean = 1.999 ms, max = 5.962 ms, min = 6.682 us, total = 5.996 ms | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.843 us, total = 11.843 us, Queueing time: mean = 115.061 us, max = 115.061 us, min = 115.061 us, total = 115.061 us | |
| GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| [2025-01-20 21:38:51,579 I 4260 4260] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: | |
| Global stats: 5 total (1 active) | |
| Queueing time: mean = 380.370 us, max = 1.696 ms, min = 11.463 us, total = 1.902 ms | |
| Execution time: mean = 237.938 us, total = 1.190 ms | |
| Event stats: | |
| event_loop_lag_probe - 3 total (0 active), Execution time: mean = 391.279 us, total = 1.174 ms, Queueing time: mean = 591.435 us, max = 1.696 ms, min = 11.463 us, total = 1.774 ms | |
| Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.853 us, total = 15.853 us, Queueing time: mean = 127.546 us, max = 127.546 us, min = 127.546 us, total = 127.546 us | |
| [2025-01-20 21:38:51,579 I 4260 4260] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: | |
| Global stats: 5 total (0 active) | |
| Queueing time: mean = 1.112 ms, max = 5.285 ms, min = 9.096 us, total = 5.558 ms | |
| Execution time: mean = 88.510 us, total = 442.551 us | |
| Event stats: | |
| event_loop_lag_probe - 3 total (0 active), Execution time: mean = 146.749 us, total = 440.247 us, Queueing time: mean = 1.788 ms, max = 5.285 ms, min = 9.096 us, total = 5.365 ms | |
| RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.152 us, total = 2.304 us, Queueing time: mean = 96.198 us, max = 98.821 us, min = 93.576 us, total = 192.397 us | |
| [2025-01-20 21:38:53,855 I 4260 4260] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c | |
| [2025-01-20 21:38:53,856 I 4260 4260] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c | |
| [2025-01-20 21:38:53,856 I 4260 4260] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c registered, will try to reschedule all the infeasible placement groups. | |
| [2025-01-20 21:38:53,863 I 4260 4338] (gcs_server) ray_syncer.cc:377: Get connection node_id=1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c | |
| [2025-01-20 21:38:54,681 I 4260 4260] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 3836 | |
| [2025-01-20 21:38:54,681 I 4260 4260] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 3836 | |
| [2025-01-20 21:39:01,331 W 4260 4283] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50426: Failed to connect to remote host: Connection refused; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster. | |
| [2025-01-20 21:39:51,580 I 4260 4260] (gcs_server) gcs_server.cc:245: Gcs Debug state: | |
| GcsNodeManager: | |
| - RegisterNode request count: 1 | |
| - DrainNode request count: 0 | |
| - GetAllNodeInfo request count: 34 | |
| GcsActorManager: | |
| - RegisterActor request count: 0 | |
| - CreateActor request count: 0 | |
| - GetActorInfo request count: 0 | |
| - GetNamedActorInfo request count: 0 | |
| - GetAllActorInfo request count: 0 | |
| - KillActor request count: 0 | |
| - ListNamedActors request count: 0 | |
| - Registered actors count: 0 | |
| - Destroyed actors count: 0 | |
| - Named actors count: 0 | |
| - Unresolved actors count: 0 | |
| - Pending actors count: 0 | |
| - Created actors count: 0 | |
| - owners_: 0 | |
| - actor_to_register_callbacks_: 0 | |
| - actor_to_restart_callbacks_: 0 | |
| - actor_to_create_callbacks_: 0 | |
| - sorted_destroyed_actor_list_: 0 | |
| GcsResourceManager: | |
| - GetAllAvailableResources request count: 0 | |
| - GetAllTotalResources request count: 0 | |
| - GetAllResourceUsage request count: 12 | |
| GcsPlacementGroupManager: | |
| - CreatePlacementGroup request count: 0 | |
| - RemovePlacementGroup request count: 0 | |
| - GetPlacementGroup request count: 0 | |
| - GetAllPlacementGroup request count: 0 | |
| - WaitPlacementGroupUntilReady request count: 0 | |
| - GetNamedPlacementGroup request count: 0 | |
| - Scheduling pending placement group count: 0 | |
| - Registered placement groups count: 0 | |
| - Named placement group count: 0 | |
| - Pending placement groups count: 0 | |
| - Infeasible placement groups count: 0 | |
| Publisher: | |
| GCS_NODE_INFO_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 534 | |
| - current buffered bytes: 0 | |
| GCS_JOB_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 256 | |
| - current buffered bytes: 256 | |
| RAY_LOG_CHANNEL | |
| - cumulative published messages: 23 | |
| - cumulative published bytes: 13675 | |
| - current buffered bytes: 818 | |
| [runtime env manager] ID to URIs table: | |
| [runtime env manager] URIs reference table: | |
| GcsTaskManager: | |
| -Total num task events reported: 41 | |
| -Total num status task events dropped: 0 | |
| -Total num profile events dropped: 0 | |
| -Current num of task events stored: 11 | |
| -Total num of actor creation tasks: 0 | |
| -Total num of actor tasks: 0 | |
| -Total num of normal tasks: 10 | |
| -Total num of driver tasks: 1 | |
| GcsAutoscalerStateManager: | |
| - last_seen_autoscaler_state_version_: 0 | |
| - last_cluster_resource_state_version_: 0 | |
| - pending demands: | |
| [2025-01-20 21:39:51,580 I 4260 4260] (gcs_server) gcs_server.cc:843: Main service Event stats: | |
| Global stats: 1152 total (4 active) | |
| Queueing time: mean = 2.045 ms, max = 254.030 ms, min = 378.000 ns, total = 2.356 s | |
| Execution time: mean = 521.859 us, total = 601.182 ms | |
| Event stats: | |
| event_loop_lag_probe - 242 total (0 active), Execution time: mean = 23.969 us, total = 5.801 ms, Queueing time: mean = 87.704 us, max = 14.636 ms, min = 2.726 us, total = 21.224 ms | |
| GcsInMemoryStore.Put - 143 total (0 active), Execution time: mean = 1.791 ms, total = 256.084 ms, Queueing time: mean = 12.409 ms, max = 253.124 ms, min = 4.502 us, total = 1.774 s | |
| GcsInMemoryStore.Get - 81 total (0 active), Execution time: mean = 18.463 us, total = 1.496 ms, Queueing time: mean = 15.459 us, max = 92.727 us, min = 2.235 us, total = 1.252 ms | |
| InternalKVGcsService.grpc_server.InternalKVGet - 80 total (0 active), Execution time: mean = 545.176 us, total = 43.614 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 80 total (0 active), Execution time: mean = 33.601 us, total = 2.688 ms, Queueing time: mean = 102.072 us, max = 263.957 us, min = 3.438 us, total = 8.166 ms | |
| RayletLoadPulled - 60 total (1 active), Execution time: mean = 273.624 us, total = 16.417 ms, Queueing time: mean = 86.168 us, max = 872.028 us, min = 14.226 us, total = 5.170 ms | |
| NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 57 total (0 active), Execution time: mean = 81.617 us, total = 4.652 ms, Queueing time: mean = 104.090 us, max = 223.449 us, min = 19.613 us, total = 5.933 ms | |
| NodeManagerService.grpc_client.GetResourceLoad - 57 total (0 active), Execution time: mean = 2.544 ms, total = 145.004 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 39 total (0 active), Execution time: mean = 43.094 us, total = 1.681 ms, Queueing time: mean = 105.299 us, max = 178.057 us, min = 16.468 us, total = 4.107 ms | |
| InternalKVGcsService.grpc_server.InternalKVPut - 39 total (0 active), Execution time: mean = 601.574 us, total = 23.461 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 25.393 us, total = 863.353 us, Queueing time: mean = 104.993 us, max = 248.874 us, min = 6.954 us, total = 3.570 ms | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 427.723 us, total = 14.543 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 38.269 us, total = 918.448 us, Queueing time: mean = 100.919 us, max = 186.815 us, min = 16.537 us, total = 2.422 ms | |
| NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 579.804 us, total = 13.915 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 18.783 us, total = 394.435 us, Queueing time: mean = 560.184 us, max = 2.600 ms, min = 16.168 us, total = 11.764 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.408 ms, total = 29.562 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ClusterResourceManager.ResetRemoteNodeView - 20 total (1 active), Execution time: mean = 11.102 us, total = 222.048 us, Queueing time: mean = 62.655 us, max = 130.349 us, min = 43.490 us, total = 1.253 ms | |
| HealthCheck - 18 total (0 active), Execution time: mean = 7.531 us, total = 135.560 us, Queueing time: mean = 104.972 us, max = 198.980 us, min = 19.852 us, total = 1.889 ms | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 89.918 us, total = 1.079 ms, Queueing time: mean = 100.813 us, max = 146.059 us, min = 30.354 us, total = 1.210 ms | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 681.610 us, total = 8.179 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 98.825 us, total = 988.251 us, Queueing time: mean = 119.914 us, max = 147.745 us, min = 48.942 us, total = 1.199 ms | |
| GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 20.125 us, total = 120.749 us, Queueing time: mean = 99.161 us, max = 130.379 us, min = 10.591 us, total = 594.966 us | |
| GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.280 ms, total = 7.677 ms, Queueing time: mean = 62.711 us, max = 87.202 us, min = 50.750 us, total = 376.266 us | |
| NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 837.877 us, total = 4.189 ms, Queueing time: mean = 388.123 us, max = 1.421 ms, min = 110.920 us, total = 1.941 ms | |
| NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.164 ms, total = 10.819 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 349.685 us, total = 1.399 ms, Queueing time: mean = 127.220 ms, max = 254.030 ms, min = 149.272 us, total = 508.880 ms | |
| InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 41.998 us, total = 41.998 us, Queueing time: mean = 119.689 us, max = 119.689 us, min = 119.689 us, total = 119.689 us | |
| GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 797.189 us, total = 797.189 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 1.060 ms, total = 1.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 374.932 us, total = 374.932 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 15.338 us, total = 15.338 us, Queueing time: mean = 103.670 us, max = 103.670 us, min = 103.670 us, total = 103.670 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 601.341 us, total = 601.341 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 22.599 us, total = 22.599 us, Queueing time: mean = 12.193 us, max = 12.193 us, min = 12.193 us, total = 12.193 us | |
| InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 71.823 us, total = 71.823 us, Queueing time: mean = 103.468 us, max = 103.468 us, min = 103.468 us, total = 103.468 us | |
| JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 576.698 us, total = 576.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 859.817 us, total = 859.817 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 546.537 us, total = 546.537 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 138.394 us, total = 138.394 us, Queueing time: mean = 172.415 us, max = 172.415 us, min = 172.415 us, total = 172.415 us | |
| GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.642 us, total = 16.642 us, Queueing time: mean = 10.027 us, max = 10.027 us, min = 10.027 us, total = 10.027 us | |
| GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 8.131 us, total = 8.131 us, Queueing time: mean = 378.000 ns, max = 378.000 ns, min = 378.000 ns, total = 378.000 ns | |
| JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.440 us, total = 22.440 us, Queueing time: mean = 87.206 us, max = 87.206 us, min = 87.206 us, total = 87.206 us | |
| NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 101.666 us, total = 101.666 us, Queueing time: mean = 137.157 us, max = 137.157 us, min = 137.157 us, total = 137.157 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.884 us, total = 22.884 us, Queueing time: mean = 111.600 us, max = 111.600 us, min = 111.600 us, total = 111.600 us | |
| [2025-01-20 21:39:51,580 I 4260 4260] (gcs_server) gcs_server.cc:847: task_io_context Event stats: | |
| Global stats: 2650 total (1 active) | |
| Queueing time: mean = 42.339 us, max = 5.962 ms, min = 2.628 us, total = 112.198 ms | |
| Execution time: mean = 224.182 us, total = 594.082 ms | |
| Event stats: | |
| TaskInfoGcsService.grpc_server.AddTaskEventData - 1197 total (0 active), Execution time: mean = 465.852 us, total = 557.625 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1197 total (0 active), Execution time: mean = 24.558 us, total = 29.396 ms, Queueing time: mean = 85.928 us, max = 483.438 us, min = 5.512 us, total = 102.856 ms | |
| event_loop_lag_probe - 242 total (0 active), Execution time: mean = 28.556 us, total = 6.911 ms, Queueing time: mean = 34.357 us, max = 5.962 ms, min = 2.628 us, total = 8.315 ms | |
| GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 10.622 us, total = 138.080 us, Queueing time: mean = 70.159 us, max = 104.750 us, min = 46.206 us, total = 912.065 us | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.843 us, total = 11.843 us, Queueing time: mean = 115.061 us, max = 115.061 us, min = 115.061 us, total = 115.061 us | |
| [2025-01-20 21:39:51,581 I 4260 4260] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: | |
| Global stats: 434 total (25 active) | |
| Queueing time: mean = 43.866 us, max = 1.696 ms, min = 3.882 us, total = 19.038 ms | |
| Execution time: mean = 123.689 ms, total = 53.681 s | |
| Event stats: | |
| event_loop_lag_probe - 242 total (0 active), Execution time: mean = 30.292 us, total = 7.331 ms, Queueing time: mean = 16.630 us, max = 1.696 ms, min = 3.882 us, total = 4.024 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 47 total (24 active), Execution time: mean = 1.141 s, total = 53.638 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 47 total (0 active), Execution time: mean = 28.713 us, total = 1.350 ms, Queueing time: mean = 175.660 us, max = 875.211 us, min = 4.372 us, total = 8.256 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 72.872 us, total = 1.822 ms, Queueing time: mean = 174.047 us, max = 859.424 us, min = 11.586 us, total = 4.351 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 725.208 us, total = 18.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsPublish - 23 total (0 active), Execution time: mean = 548.575 us, total = 12.617 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 23 total (0 active), Execution time: mean = 73.833 us, total = 1.698 ms, Queueing time: mean = 99.079 us, max = 177.463 us, min = 17.173 us, total = 2.279 ms | |
| Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.853 us, total = 15.853 us, Queueing time: mean = 127.546 us, max = 127.546 us, min = 127.546 us, total = 127.546 us | |
| [2025-01-20 21:39:51,581 I 4260 4260] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: | |
| Global stats: 264 total (0 active) | |
| Queueing time: mean = 32.819 us, max = 5.285 ms, min = 674.000 ns, total = 8.664 ms | |
| Execution time: mean = 26.362 us, total = 6.960 ms | |
| Event stats: | |
| event_loop_lag_probe - 242 total (0 active), Execution time: mean = 23.041 us, total = 5.576 ms, Queueing time: mean = 29.951 us, max = 5.285 ms, min = 2.112 us, total = 7.248 ms | |
| RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 27.952 us, total = 279.516 us, Queueing time: mean = 1.136 us, max = 1.840 us, min = 674.000 ns, total = 11.362 us | |
| - 10 total (0 active), Execution time: mean = 110.186 us, total = 1.102 ms, Queueing time: mean = 121.224 us, max = 180.946 us, min = 96.930 us, total = 1.212 ms | |
| RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.152 us, total = 2.304 us, Queueing time: mean = 96.198 us, max = 98.821 us, min = 93.576 us, total = 192.397 us | |
| [2025-01-20 21:40:51,581 I 4260 4260] (gcs_server) gcs_server.cc:245: Gcs Debug state: | |
| GcsNodeManager: | |
| - RegisterNode request count: 1 | |
| - DrainNode request count: 0 | |
| - GetAllNodeInfo request count: 34 | |
| GcsActorManager: | |
| - RegisterActor request count: 0 | |
| - CreateActor request count: 0 | |
| - GetActorInfo request count: 0 | |
| - GetNamedActorInfo request count: 0 | |
| - GetAllActorInfo request count: 0 | |
| - KillActor request count: 0 | |
| - ListNamedActors request count: 0 | |
| - Registered actors count: 0 | |
| - Destroyed actors count: 0 | |
| - Named actors count: 0 | |
| - Unresolved actors count: 0 | |
| - Pending actors count: 0 | |
| - Created actors count: 0 | |
| - owners_: 0 | |
| - actor_to_register_callbacks_: 0 | |
| - actor_to_restart_callbacks_: 0 | |
| - actor_to_create_callbacks_: 0 | |
| - sorted_destroyed_actor_list_: 0 | |
| GcsResourceManager: | |
| - GetAllAvailableResources request count: 0 | |
| - GetAllTotalResources request count: 0 | |
| - GetAllResourceUsage request count: 24 | |
| GcsPlacementGroupManager: | |
| - CreatePlacementGroup request count: 0 | |
| - RemovePlacementGroup request count: 0 | |
| - GetPlacementGroup request count: 0 | |
| - GetAllPlacementGroup request count: 0 | |
| - WaitPlacementGroupUntilReady request count: 0 | |
| - GetNamedPlacementGroup request count: 0 | |
| - Scheduling pending placement group count: 0 | |
| - Registered placement groups count: 0 | |
| - Named placement group count: 0 | |
| - Pending placement groups count: 0 | |
| - Infeasible placement groups count: 0 | |
| Publisher: | |
| GCS_NODE_INFO_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 534 | |
| - current buffered bytes: 0 | |
| GCS_JOB_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 256 | |
| - current buffered bytes: 256 | |
| RAY_LOG_CHANNEL | |
| - cumulative published messages: 36 | |
| - cumulative published bytes: 23524 | |
| - current buffered bytes: 818 | |
| [runtime env manager] ID to URIs table: | |
| [runtime env manager] URIs reference table: | |
| GcsTaskManager: | |
| -Total num task events reported: 41 | |
| -Total num status task events dropped: 0 | |
| -Total num profile events dropped: 0 | |
| -Current num of task events stored: 11 | |
| -Total num of actor creation tasks: 0 | |
| -Total num of actor tasks: 0 | |
| -Total num of normal tasks: 10 | |
| -Total num of driver tasks: 1 | |
| GcsAutoscalerStateManager: | |
| - last_seen_autoscaler_state_version_: 0 | |
| - last_cluster_resource_state_version_: 0 | |
| - pending demands: | |
| [2025-01-20 21:40:51,582 I 4260 4260] (gcs_server) gcs_server.cc:843: Main service Event stats: | |
| Global stats: 1871 total (4 active) | |
| Queueing time: mean = 1.277 ms, max = 254.030 ms, min = 378.000 ns, total = 2.389 s | |
| Execution time: mean = 421.158 us, total = 787.987 ms | |
| Event stats: | |
| event_loop_lag_probe - 482 total (0 active), Execution time: mean = 23.920 us, total = 11.529 ms, Queueing time: mean = 48.553 us, max = 14.636 ms, min = 2.726 us, total = 23.403 ms | |
| GcsInMemoryStore.Put - 239 total (0 active), Execution time: mean = 1.074 ms, total = 256.635 ms, Queueing time: mean = 7.443 ms, max = 253.124 ms, min = 4.502 us, total = 1.779 s | |
| RayletLoadPulled - 120 total (1 active), Execution time: mean = 276.681 us, total = 33.202 ms, Queueing time: mean = 85.600 us, max = 872.028 us, min = 14.226 us, total = 10.272 ms | |
| NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 117 total (0 active), Execution time: mean = 82.386 us, total = 9.639 ms, Queueing time: mean = 108.431 us, max = 223.449 us, min = 19.613 us, total = 12.686 ms | |
| NodeManagerService.grpc_client.GetResourceLoad - 117 total (0 active), Execution time: mean = 2.091 ms, total = 244.639 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Get - 93 total (0 active), Execution time: mean = 19.235 us, total = 1.789 ms, Queueing time: mean = 15.339 us, max = 92.727 us, min = 2.235 us, total = 1.426 ms | |
| InternalKVGcsService.grpc_server.InternalKVGet - 92 total (0 active), Execution time: mean = 557.360 us, total = 51.277 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 92 total (0 active), Execution time: mean = 36.624 us, total = 3.369 ms, Queueing time: mean = 103.823 us, max = 263.957 us, min = 3.438 us, total = 9.552 ms | |
| InternalKVGcsService.grpc_server.InternalKVPut - 63 total (0 active), Execution time: mean = 604.267 us, total = 38.069 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 63 total (0 active), Execution time: mean = 43.111 us, total = 2.716 ms, Queueing time: mean = 110.425 us, max = 178.057 us, min = 16.468 us, total = 6.957 ms | |
| NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 48 total (0 active), Execution time: mean = 36.161 us, total = 1.736 ms, Queueing time: mean = 115.767 us, max = 221.795 us, min = 16.537 us, total = 5.557 ms | |
| NodeInfoGcsService.grpc_server.CheckAlive - 48 total (0 active), Execution time: mean = 600.220 us, total = 28.811 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ClusterResourceManager.ResetRemoteNodeView - 40 total (1 active), Execution time: mean = 11.115 us, total = 444.596 us, Queueing time: mean = 94.748 us, max = 662.152 us, min = 43.490 us, total = 3.790 ms | |
| HealthCheck - 38 total (0 active), Execution time: mean = 8.210 us, total = 311.977 us, Queueing time: mean = 96.998 us, max = 198.980 us, min = 19.852 us, total = 3.686 ms | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 427.723 us, total = 14.543 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 25.393 us, total = 863.353 us, Queueing time: mean = 104.993 us, max = 248.874 us, min = 6.954 us, total = 3.570 ms | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 24 total (0 active), Execution time: mean = 689.940 us, total = 16.559 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 24 total (0 active), Execution time: mean = 91.618 us, total = 2.199 ms, Queueing time: mean = 115.184 us, max = 158.639 us, min = 30.354 us, total = 2.764 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 18.783 us, total = 394.435 us, Queueing time: mean = 560.184 us, max = 2.600 ms, min = 16.168 us, total = 11.764 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.408 ms, total = 29.562 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GCSServer.deadline_timer.debug_state_dump - 12 total (1 active), Execution time: mean = 1.325 ms, total = 15.900 ms, Queueing time: mean = 76.767 us, max = 149.454 us, min = 50.750 us, total = 921.208 us | |
| GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 98.825 us, total = 988.251 us, Queueing time: mean = 119.914 us, max = 147.745 us, min = 48.942 us, total = 1.199 ms | |
| GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 20.125 us, total = 120.749 us, Queueing time: mean = 99.161 us, max = 130.379 us, min = 10.591 us, total = 594.966 us | |
| NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 837.877 us, total = 4.189 ms, Queueing time: mean = 388.123 us, max = 1.421 ms, min = 110.920 us, total = 1.941 ms | |
| NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.164 ms, total = 10.819 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 349.685 us, total = 1.399 ms, Queueing time: mean = 127.220 ms, max = 254.030 ms, min = 149.272 us, total = 508.880 ms | |
| GCSServer.deadline_timer.debug_state_event_stats_print - 2 total (1 active, 1 running), Execution time: mean = 502.736 us, total = 1.005 ms, Queueing time: mean = 97.528 us, max = 195.056 us, min = 195.056 us, total = 195.056 us | |
| InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 41.998 us, total = 41.998 us, Queueing time: mean = 119.689 us, max = 119.689 us, min = 119.689 us, total = 119.689 us | |
| InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 797.189 us, total = 797.189 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 1.060 ms, total = 1.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 374.932 us, total = 374.932 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 15.338 us, total = 15.338 us, Queueing time: mean = 103.670 us, max = 103.670 us, min = 103.670 us, total = 103.670 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 601.341 us, total = 601.341 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 22.599 us, total = 22.599 us, Queueing time: mean = 12.193 us, max = 12.193 us, min = 12.193 us, total = 12.193 us | |
| InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 71.823 us, total = 71.823 us, Queueing time: mean = 103.468 us, max = 103.468 us, min = 103.468 us, total = 103.468 us | |
| JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 576.698 us, total = 576.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 859.817 us, total = 859.817 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 546.537 us, total = 546.537 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 138.394 us, total = 138.394 us, Queueing time: mean = 172.415 us, max = 172.415 us, min = 172.415 us, total = 172.415 us | |
| GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.642 us, total = 16.642 us, Queueing time: mean = 10.027 us, max = 10.027 us, min = 10.027 us, total = 10.027 us | |
| GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 8.131 us, total = 8.131 us, Queueing time: mean = 378.000 ns, max = 378.000 ns, min = 378.000 ns, total = 378.000 ns | |
| JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.440 us, total = 22.440 us, Queueing time: mean = 87.206 us, max = 87.206 us, min = 87.206 us, total = 87.206 us | |
| NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 101.666 us, total = 101.666 us, Queueing time: mean = 137.157 us, max = 137.157 us, min = 137.157 us, total = 137.157 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.884 us, total = 22.884 us, Queueing time: mean = 111.600 us, max = 111.600 us, min = 111.600 us, total = 111.600 us | |
| [2025-01-20 21:40:51,582 I 4260 4260] (gcs_server) gcs_server.cc:847: task_io_context Event stats: | |
| Global stats: 5422 total (1 active) | |
| Queueing time: mean = 47.143 us, max = 5.962 ms, min = 2.628 us, total = 255.609 ms | |
| Execution time: mean = 242.010 us, total = 1.312 s | |
| Event stats: | |
| TaskInfoGcsService.grpc_server.AddTaskEventData - 2457 total (0 active), Execution time: mean = 502.291 us, total = 1.234 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 2457 total (0 active), Execution time: mean = 26.104 us, total = 64.137 ms, Queueing time: mean = 98.873 us, max = 537.156 us, min = 5.512 us, total = 242.930 ms | |
| event_loop_lag_probe - 482 total (0 active), Execution time: mean = 28.231 us, total = 13.607 ms, Queueing time: mean = 21.863 us, max = 5.962 ms, min = 2.628 us, total = 10.538 ms | |
| GcsTaskManager.GcJobSummary - 25 total (1 active), Execution time: mean = 11.675 us, total = 291.875 us, Queueing time: mean = 81.065 us, max = 150.092 us, min = 46.206 us, total = 2.027 ms | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.843 us, total = 11.843 us, Queueing time: mean = 115.061 us, max = 115.061 us, min = 115.061 us, total = 115.061 us | |
| [2025-01-20 21:40:51,582 I 4260 4260] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: | |
| Global stats: 726 total (25 active) | |
| Queueing time: mean = 34.832 us, max = 1.696 ms, min = 3.882 us, total = 25.288 ms | |
| Execution time: mean = 156.832 ms, total = 113.860 s | |
| Event stats: | |
| event_loop_lag_probe - 482 total (0 active), Execution time: mean = 31.202 us, total = 15.039 ms, Queueing time: mean = 14.427 us, max = 1.696 ms, min = 3.882 us, total = 6.954 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 60 total (24 active), Execution time: mean = 1.897 s, total = 113.798 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 60 total (0 active), Execution time: mean = 33.899 us, total = 2.034 ms, Queueing time: mean = 162.647 us, max = 875.211 us, min = 4.372 us, total = 9.759 ms | |
| InternalPubSubGcsService.grpc_server.GcsPublish - 36 total (0 active), Execution time: mean = 612.221 us, total = 22.040 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 36 total (0 active), Execution time: mean = 86.769 us, total = 3.124 ms, Queueing time: mean = 113.787 us, max = 199.575 us, min = 17.173 us, total = 4.096 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 72.872 us, total = 1.822 ms, Queueing time: mean = 174.047 us, max = 859.424 us, min = 11.586 us, total = 4.351 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 725.208 us, total = 18.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.853 us, total = 15.853 us, Queueing time: mean = 127.546 us, max = 127.546 us, min = 127.546 us, total = 127.546 us | |
| [2025-01-20 21:40:51,582 I 4260 4260] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: | |
| Global stats: 504 total (0 active) | |
| Queueing time: mean = 21.883 us, max = 5.285 ms, min = 674.000 ns, total = 11.029 ms | |
| Execution time: mean = 26.437 us, total = 13.324 ms | |
| Event stats: | |
| event_loop_lag_probe - 482 total (0 active), Execution time: mean = 24.773 us, total = 11.941 ms, Queueing time: mean = 19.944 us, max = 5.285 ms, min = 2.112 us, total = 9.613 ms | |
| RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 27.952 us, total = 279.516 us, Queueing time: mean = 1.136 us, max = 1.840 us, min = 674.000 ns, total = 11.362 us | |
| - 10 total (0 active), Execution time: mean = 110.186 us, total = 1.102 ms, Queueing time: mean = 121.224 us, max = 180.946 us, min = 96.930 us, total = 1.212 ms | |
| RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.152 us, total = 2.304 us, Queueing time: mean = 96.198 us, max = 98.821 us, min = 93.576 us, total = 192.397 us | |
| [2025-01-20 21:41:51,583 I 4260 4260] (gcs_server) gcs_server.cc:245: Gcs Debug state: | |
| GcsNodeManager: | |
| - RegisterNode request count: 1 | |
| - DrainNode request count: 0 | |
| - GetAllNodeInfo request count: 34 | |
| GcsActorManager: | |
| - RegisterActor request count: 0 | |
| - CreateActor request count: 0 | |
| - GetActorInfo request count: 0 | |
| - GetNamedActorInfo request count: 0 | |
| - GetAllActorInfo request count: 0 | |
| - KillActor request count: 0 | |
| - ListNamedActors request count: 0 | |
| - Registered actors count: 0 | |
| - Destroyed actors count: 0 | |
| - Named actors count: 0 | |
| - Unresolved actors count: 0 | |
| - Pending actors count: 0 | |
| - Created actors count: 0 | |
| - owners_: 0 | |
| - actor_to_register_callbacks_: 0 | |
| - actor_to_restart_callbacks_: 0 | |
| - actor_to_create_callbacks_: 0 | |
| - sorted_destroyed_actor_list_: 0 | |
| GcsResourceManager: | |
| - GetAllAvailableResources request count: 0 | |
| - GetAllTotalResources request count: 0 | |
| - GetAllResourceUsage request count: 36 | |
| GcsPlacementGroupManager: | |
| - CreatePlacementGroup request count: 0 | |
| - RemovePlacementGroup request count: 0 | |
| - GetPlacementGroup request count: 0 | |
| - GetAllPlacementGroup request count: 0 | |
| - WaitPlacementGroupUntilReady request count: 0 | |
| - GetNamedPlacementGroup request count: 0 | |
| - Scheduling pending placement group count: 0 | |
| - Registered placement groups count: 0 | |
| - Named placement group count: 0 | |
| - Pending placement groups count: 0 | |
| - Infeasible placement groups count: 0 | |
| Publisher: | |
| GCS_NODE_INFO_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 534 | |
| - current buffered bytes: 0 | |
| GCS_JOB_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 256 | |
| - current buffered bytes: 256 | |
| RAY_LOG_CHANNEL | |
| - cumulative published messages: 49 | |
| - cumulative published bytes: 33373 | |
| - current buffered bytes: 818 | |
| [runtime env manager] ID to URIs table: | |
| [runtime env manager] URIs reference table: | |
| GcsTaskManager: | |
| -Total num task events reported: 41 | |
| -Total num status task events dropped: 0 | |
| -Total num profile events dropped: 0 | |
| -Current num of task events stored: 11 | |
| -Total num of actor creation tasks: 0 | |
| -Total num of actor tasks: 0 | |
| -Total num of normal tasks: 10 | |
| -Total num of driver tasks: 1 | |
| GcsAutoscalerStateManager: | |
| - last_seen_autoscaler_state_version_: 0 | |
| - last_cluster_resource_state_version_: 0 | |
| - pending demands: | |
| [2025-01-20 21:41:51,584 I 4260 4260] (gcs_server) gcs_server.cc:843: Main service Event stats: | |
| Global stats: 2590 total (4 active) | |
| Queueing time: mean = 934.452 us, max = 254.030 ms, min = 378.000 ns, total = 2.420 s | |
| Execution time: mean = 377.795 us, total = 978.489 ms | |
| Event stats: | |
| event_loop_lag_probe - 722 total (0 active), Execution time: mean = 23.945 us, total = 17.289 ms, Queueing time: mean = 35.479 us, max = 14.636 ms, min = 2.726 us, total = 25.616 ms | |
| GcsInMemoryStore.Put - 335 total (0 active), Execution time: mean = 767.731 us, total = 257.190 ms, Queueing time: mean = 5.323 ms, max = 253.124 ms, min = 4.502 us, total = 1.783 s | |
| RayletLoadPulled - 180 total (1 active), Execution time: mean = 284.859 us, total = 51.275 ms, Queueing time: mean = 84.480 us, max = 872.028 us, min = 14.226 us, total = 15.206 ms | |
| NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 177 total (0 active), Execution time: mean = 82.695 us, total = 14.637 ms, Queueing time: mean = 111.899 us, max = 223.449 us, min = 19.613 us, total = 19.806 ms | |
| NodeManagerService.grpc_client.GetResourceLoad - 177 total (0 active), Execution time: mean = 1.960 ms, total = 346.968 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Get - 105 total (0 active), Execution time: mean = 20.245 us, total = 2.126 ms, Queueing time: mean = 15.084 us, max = 92.727 us, min = 2.235 us, total = 1.584 ms | |
| InternalKVGcsService.grpc_server.InternalKVGet - 104 total (0 active), Execution time: mean = 564.612 us, total = 58.720 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 104 total (0 active), Execution time: mean = 38.806 us, total = 4.036 ms, Queueing time: mean = 104.504 us, max = 263.957 us, min = 3.438 us, total = 10.868 ms | |
| InternalKVGcsService.grpc_server.InternalKVPut - 87 total (0 active), Execution time: mean = 595.247 us, total = 51.787 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 87 total (0 active), Execution time: mean = 44.025 us, total = 3.830 ms, Queueing time: mean = 109.879 us, max = 178.057 us, min = 16.468 us, total = 9.560 ms | |
| NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 72 total (0 active), Execution time: mean = 35.181 us, total = 2.533 ms, Queueing time: mean = 120.768 us, max = 221.795 us, min = 16.537 us, total = 8.695 ms | |
| NodeInfoGcsService.grpc_server.CheckAlive - 72 total (0 active), Execution time: mean = 613.565 us, total = 44.177 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ClusterResourceManager.ResetRemoteNodeView - 60 total (1 active), Execution time: mean = 10.773 us, total = 646.369 us, Queueing time: mean = 87.475 us, max = 662.152 us, min = 22.301 us, total = 5.249 ms | |
| HealthCheck - 58 total (0 active), Execution time: mean = 8.119 us, total = 470.896 us, Queueing time: mean = 100.114 us, max = 198.980 us, min = 19.852 us, total = 5.807 ms | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 36 total (0 active), Execution time: mean = 691.036 us, total = 24.877 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 36 total (0 active), Execution time: mean = 89.123 us, total = 3.208 ms, Queueing time: mean = 114.357 us, max = 158.639 us, min = 30.354 us, total = 4.117 ms | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 427.723 us, total = 14.543 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 25.393 us, total = 863.353 us, Queueing time: mean = 104.993 us, max = 248.874 us, min = 6.954 us, total = 3.570 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 18.783 us, total = 394.435 us, Queueing time: mean = 560.184 us, max = 2.600 ms, min = 16.168 us, total = 11.764 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.408 ms, total = 29.562 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GCSServer.deadline_timer.debug_state_dump - 18 total (1 active), Execution time: mean = 1.319 ms, total = 23.747 ms, Queueing time: mean = 77.682 us, max = 149.454 us, min = 36.186 us, total = 1.398 ms | |
| GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 98.825 us, total = 988.251 us, Queueing time: mean = 119.914 us, max = 147.745 us, min = 48.942 us, total = 1.199 ms | |
| GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 20.125 us, total = 120.749 us, Queueing time: mean = 99.161 us, max = 130.379 us, min = 10.591 us, total = 594.966 us | |
| NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 837.877 us, total = 4.189 ms, Queueing time: mean = 388.123 us, max = 1.421 ms, min = 110.920 us, total = 1.941 ms | |
| NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.164 ms, total = 10.819 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 349.685 us, total = 1.399 ms, Queueing time: mean = 127.220 ms, max = 254.030 ms, min = 149.272 us, total = 508.880 ms | |
| GCSServer.deadline_timer.debug_state_event_stats_print - 3 total (1 active, 1 running), Execution time: mean = 938.670 us, total = 2.816 ms, Queueing time: mean = 93.139 us, max = 195.056 us, min = 84.360 us, total = 279.416 us | |
| InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 41.998 us, total = 41.998 us, Queueing time: mean = 119.689 us, max = 119.689 us, min = 119.689 us, total = 119.689 us | |
| InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 797.189 us, total = 797.189 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 1.060 ms, total = 1.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 374.932 us, total = 374.932 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 15.338 us, total = 15.338 us, Queueing time: mean = 103.670 us, max = 103.670 us, min = 103.670 us, total = 103.670 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 601.341 us, total = 601.341 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 22.599 us, total = 22.599 us, Queueing time: mean = 12.193 us, max = 12.193 us, min = 12.193 us, total = 12.193 us | |
| InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 71.823 us, total = 71.823 us, Queueing time: mean = 103.468 us, max = 103.468 us, min = 103.468 us, total = 103.468 us | |
| JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 576.698 us, total = 576.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 859.817 us, total = 859.817 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 546.537 us, total = 546.537 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 138.394 us, total = 138.394 us, Queueing time: mean = 172.415 us, max = 172.415 us, min = 172.415 us, total = 172.415 us | |
| GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.642 us, total = 16.642 us, Queueing time: mean = 10.027 us, max = 10.027 us, min = 10.027 us, total = 10.027 us | |
| GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 8.131 us, total = 8.131 us, Queueing time: mean = 378.000 ns, max = 378.000 ns, min = 378.000 ns, total = 378.000 ns | |
| JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.440 us, total = 22.440 us, Queueing time: mean = 87.206 us, max = 87.206 us, min = 87.206 us, total = 87.206 us | |
| NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 101.666 us, total = 101.666 us, Queueing time: mean = 137.157 us, max = 137.157 us, min = 137.157 us, total = 137.157 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.884 us, total = 22.884 us, Queueing time: mean = 111.600 us, max = 111.600 us, min = 111.600 us, total = 111.600 us | |
| [2025-01-20 21:41:51,584 I 4260 4260] (gcs_server) gcs_server.cc:847: task_io_context Event stats: | |
| Global stats: 8194 total (1 active) | |
| Queueing time: mean = 47.834 us, max = 5.962 ms, min = 2.628 us, total = 391.949 ms | |
| Execution time: mean = 250.084 us, total = 2.049 s | |
| Event stats: | |
| TaskInfoGcsService.grpc_server.AddTaskEventData - 3717 total (0 active), Execution time: mean = 519.360 us, total = 1.930 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 3717 total (0 active), Execution time: mean = 26.315 us, total = 97.814 ms, Queueing time: mean = 101.124 us, max = 537.156 us, min = 5.109 us, total = 375.877 ms | |
| event_loop_lag_probe - 722 total (0 active), Execution time: mean = 28.346 us, total = 20.466 ms, Queueing time: mean = 17.921 us, max = 5.962 ms, min = 2.628 us, total = 12.939 ms | |
| GcsTaskManager.GcJobSummary - 37 total (1 active), Execution time: mean = 11.809 us, total = 436.926 us, Queueing time: mean = 81.542 us, max = 150.092 us, min = 46.206 us, total = 3.017 ms | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.843 us, total = 11.843 us, Queueing time: mean = 115.061 us, max = 115.061 us, min = 115.061 us, total = 115.061 us | |
| [2025-01-20 21:41:51,584 I 4260 4260] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: | |
| Global stats: 1018 total (25 active) | |
| Queueing time: mean = 29.856 us, max = 1.696 ms, min = 2.904 us, total = 30.394 ms | |
| Execution time: mean = 170.937 ms, total = 174.014 s | |
| Event stats: | |
| event_loop_lag_probe - 722 total (0 active), Execution time: mean = 28.371 us, total = 20.484 ms, Queueing time: mean = 12.597 us, max = 1.696 ms, min = 2.904 us, total = 9.095 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 73 total (24 active), Execution time: mean = 2.383 s, total = 173.934 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 73 total (0 active), Execution time: mean = 36.210 us, total = 2.643 ms, Queueing time: mean = 153.094 us, max = 875.211 us, min = 4.372 us, total = 11.176 ms | |
| InternalPubSubGcsService.grpc_server.GcsPublish - 49 total (0 active), Execution time: mean = 641.204 us, total = 31.419 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 49 total (0 active), Execution time: mean = 103.784 us, total = 5.085 ms, Queueing time: mean = 115.186 us, max = 199.575 us, min = 17.173 us, total = 5.644 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 72.872 us, total = 1.822 ms, Queueing time: mean = 174.047 us, max = 859.424 us, min = 11.586 us, total = 4.351 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 725.208 us, total = 18.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.853 us, total = 15.853 us, Queueing time: mean = 127.546 us, max = 127.546 us, min = 127.546 us, total = 127.546 us | |
| [2025-01-20 21:41:51,584 I 4260 4260] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: | |
| Global stats: 744 total (0 active) | |
| Queueing time: mean = 17.949 us, max = 5.285 ms, min = 674.000 ns, total = 13.354 ms | |
| Execution time: mean = 26.223 us, total = 19.510 ms | |
| Event stats: | |
| event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.106 us, total = 18.126 ms, Queueing time: mean = 16.535 us, max = 5.285 ms, min = 2.112 us, total = 11.938 ms | |
| RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 27.952 us, total = 279.516 us, Queueing time: mean = 1.136 us, max = 1.840 us, min = 674.000 ns, total = 11.362 us | |
| - 10 total (0 active), Execution time: mean = 110.186 us, total = 1.102 ms, Queueing time: mean = 121.224 us, max = 180.946 us, min = 96.930 us, total = 1.212 ms | |
| RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.152 us, total = 2.304 us, Queueing time: mean = 96.198 us, max = 98.821 us, min = 93.576 us, total = 192.397 us | |
| [2025-01-20 21:42:51,585 I 4260 4260] (gcs_server) gcs_server.cc:245: Gcs Debug state: | |
| GcsNodeManager: | |
| - RegisterNode request count: 1 | |
| - DrainNode request count: 0 | |
| - GetAllNodeInfo request count: 34 | |
| GcsActorManager: | |
| - RegisterActor request count: 0 | |
| - CreateActor request count: 0 | |
| - GetActorInfo request count: 0 | |
| - GetNamedActorInfo request count: 0 | |
| - GetAllActorInfo request count: 0 | |
| - KillActor request count: 0 | |
| - ListNamedActors request count: 0 | |
| - Registered actors count: 0 | |
| - Destroyed actors count: 0 | |
| - Named actors count: 0 | |
| - Unresolved actors count: 0 | |
| - Pending actors count: 0 | |
| - Created actors count: 0 | |
| - owners_: 0 | |
| - actor_to_register_callbacks_: 0 | |
| - actor_to_restart_callbacks_: 0 | |
| - actor_to_create_callbacks_: 0 | |
| - sorted_destroyed_actor_list_: 0 | |
| GcsResourceManager: | |
| - GetAllAvailableResources request count: 0 | |
| - GetAllTotalResources request count: 0 | |
| - GetAllResourceUsage request count: 48 | |
| GcsPlacementGroupManager: | |
| - CreatePlacementGroup request count: 0 | |
| - RemovePlacementGroup request count: 0 | |
| - GetPlacementGroup request count: 0 | |
| - GetAllPlacementGroup request count: 0 | |
| - WaitPlacementGroupUntilReady request count: 0 | |
| - GetNamedPlacementGroup request count: 0 | |
| - Scheduling pending placement group count: 0 | |
| - Registered placement groups count: 0 | |
| - Named placement group count: 0 | |
| - Pending placement groups count: 0 | |
| - Infeasible placement groups count: 0 | |
| Publisher: | |
| GCS_NODE_INFO_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 534 | |
| - current buffered bytes: 0 | |
| GCS_JOB_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 256 | |
| - current buffered bytes: 256 | |
| RAY_LOG_CHANNEL | |
| - cumulative published messages: 61 | |
| - cumulative published bytes: 43189 | |
| - current buffered bytes: 818 | |
| [runtime env manager] ID to URIs table: | |
| [runtime env manager] URIs reference table: | |
| GcsTaskManager: | |
| -Total num task events reported: 41 | |
| -Total num status task events dropped: 0 | |
| -Total num profile events dropped: 0 | |
| -Current num of task events stored: 11 | |
| -Total num of actor creation tasks: 0 | |
| -Total num of actor tasks: 0 | |
| -Total num of normal tasks: 10 | |
| -Total num of driver tasks: 1 | |
| GcsAutoscalerStateManager: | |
| - last_seen_autoscaler_state_version_: 0 | |
| - last_cluster_resource_state_version_: 0 | |
| - pending demands: | |
| [2025-01-20 21:42:51,586 I 4260 4260] (gcs_server) gcs_server.cc:843: Main service Event stats: | |
| Global stats: 3309 total (4 active) | |
| Queueing time: mean = 741.631 us, max = 254.030 ms, min = 378.000 ns, total = 2.454 s | |
| Execution time: mean = 351.573 us, total = 1.163 s | |
| Event stats: | |
| event_loop_lag_probe - 962 total (0 active), Execution time: mean = 24.327 us, total = 23.403 ms, Queueing time: mean = 29.081 us, max = 14.636 ms, min = 2.726 us, total = 27.976 ms | |
| GcsInMemoryStore.Put - 431 total (0 active), Execution time: mean = 598.027 us, total = 257.750 ms, Queueing time: mean = 4.154 ms, max = 253.124 ms, min = 3.250 us, total = 1.791 s | |
| RayletLoadPulled - 240 total (1 active), Execution time: mean = 282.552 us, total = 67.812 ms, Queueing time: mean = 83.501 us, max = 872.028 us, min = 14.226 us, total = 20.040 ms | |
| NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 237 total (0 active), Execution time: mean = 82.419 us, total = 19.533 ms, Queueing time: mean = 111.257 us, max = 223.449 us, min = 19.613 us, total = 26.368 ms | |
| NodeManagerService.grpc_client.GetResourceLoad - 237 total (0 active), Execution time: mean = 1.881 ms, total = 445.686 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Get - 117 total (0 active), Execution time: mean = 20.611 us, total = 2.411 ms, Queueing time: mean = 14.859 us, max = 92.727 us, min = 2.235 us, total = 1.738 ms | |
| InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 116 total (0 active), Execution time: mean = 40.187 us, total = 4.662 ms, Queueing time: mean = 105.679 us, max = 263.957 us, min = 3.438 us, total = 12.259 ms | |
| InternalKVGcsService.grpc_server.InternalKVGet - 116 total (0 active), Execution time: mean = 567.967 us, total = 65.884 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVPut - 111 total (0 active), Execution time: mean = 590.963 us, total = 65.597 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 111 total (0 active), Execution time: mean = 43.239 us, total = 4.800 ms, Queueing time: mean = 108.199 us, max = 178.057 us, min = 16.468 us, total = 12.010 ms | |
| NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 96 total (0 active), Execution time: mean = 34.475 us, total = 3.310 ms, Queueing time: mean = 120.945 us, max = 222.930 us, min = 16.537 us, total = 11.611 ms | |
| NodeInfoGcsService.grpc_server.CheckAlive - 96 total (0 active), Execution time: mean = 607.933 us, total = 58.362 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ClusterResourceManager.ResetRemoteNodeView - 80 total (1 active), Execution time: mean = 10.632 us, total = 850.597 us, Queueing time: mean = 86.355 us, max = 662.152 us, min = 22.301 us, total = 6.908 ms | |
| HealthCheck - 78 total (0 active), Execution time: mean = 8.018 us, total = 625.385 us, Queueing time: mean = 102.233 us, max = 198.989 us, min = 19.852 us, total = 7.974 ms | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 48 total (0 active), Execution time: mean = 690.934 us, total = 33.165 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 48 total (0 active), Execution time: mean = 88.098 us, total = 4.229 ms, Queueing time: mean = 116.443 us, max = 169.117 us, min = 30.354 us, total = 5.589 ms | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 427.723 us, total = 14.543 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 25.393 us, total = 863.353 us, Queueing time: mean = 104.993 us, max = 248.874 us, min = 6.954 us, total = 3.570 ms | |
| GCSServer.deadline_timer.debug_state_dump - 24 total (1 active), Execution time: mean = 1.358 ms, total = 32.591 ms, Queueing time: mean = 78.353 us, max = 149.454 us, min = 36.186 us, total = 1.880 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 18.783 us, total = 394.435 us, Queueing time: mean = 560.184 us, max = 2.600 ms, min = 16.168 us, total = 11.764 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.408 ms, total = 29.562 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 98.825 us, total = 988.251 us, Queueing time: mean = 119.914 us, max = 147.745 us, min = 48.942 us, total = 1.199 ms | |
| GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 20.125 us, total = 120.749 us, Queueing time: mean = 99.161 us, max = 130.379 us, min = 10.591 us, total = 594.966 us | |
| NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 837.877 us, total = 4.189 ms, Queueing time: mean = 388.123 us, max = 1.421 ms, min = 110.920 us, total = 1.941 ms | |
| NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.164 ms, total = 10.819 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GCSServer.deadline_timer.debug_state_event_stats_print - 4 total (1 active, 1 running), Execution time: mean = 1.132 ms, total = 4.530 ms, Queueing time: mean = 95.246 us, max = 195.056 us, min = 84.360 us, total = 380.986 us | |
| PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 349.685 us, total = 1.399 ms, Queueing time: mean = 127.220 ms, max = 254.030 ms, min = 149.272 us, total = 508.880 ms | |
| InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 41.998 us, total = 41.998 us, Queueing time: mean = 119.689 us, max = 119.689 us, min = 119.689 us, total = 119.689 us | |
| InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 797.189 us, total = 797.189 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 1.060 ms, total = 1.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 374.932 us, total = 374.932 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 15.338 us, total = 15.338 us, Queueing time: mean = 103.670 us, max = 103.670 us, min = 103.670 us, total = 103.670 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 601.341 us, total = 601.341 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 22.599 us, total = 22.599 us, Queueing time: mean = 12.193 us, max = 12.193 us, min = 12.193 us, total = 12.193 us | |
| InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 71.823 us, total = 71.823 us, Queueing time: mean = 103.468 us, max = 103.468 us, min = 103.468 us, total = 103.468 us | |
| JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 576.698 us, total = 576.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 859.817 us, total = 859.817 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 546.537 us, total = 546.537 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 138.394 us, total = 138.394 us, Queueing time: mean = 172.415 us, max = 172.415 us, min = 172.415 us, total = 172.415 us | |
| GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.642 us, total = 16.642 us, Queueing time: mean = 10.027 us, max = 10.027 us, min = 10.027 us, total = 10.027 us | |
| GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 8.131 us, total = 8.131 us, Queueing time: mean = 378.000 ns, max = 378.000 ns, min = 378.000 ns, total = 378.000 ns | |
| JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.440 us, total = 22.440 us, Queueing time: mean = 87.206 us, max = 87.206 us, min = 87.206 us, total = 87.206 us | |
| NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 101.666 us, total = 101.666 us, Queueing time: mean = 137.157 us, max = 137.157 us, min = 137.157 us, total = 137.157 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.884 us, total = 22.884 us, Queueing time: mean = 111.600 us, max = 111.600 us, min = 111.600 us, total = 111.600 us | |
| [2025-01-20 21:42:51,586 I 4260 4260] (gcs_server) gcs_server.cc:847: task_io_context Event stats: | |
| Global stats: 10966 total (1 active) | |
| Queueing time: mean = 48.249 us, max = 5.962 ms, min = 2.628 us, total = 529.103 ms | |
| Execution time: mean = 251.751 us, total = 2.761 s | |
| Event stats: | |
| TaskInfoGcsService.grpc_server.AddTaskEventData - 4977 total (0 active), Execution time: mean = 523.337 us, total = 2.605 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 4977 total (0 active), Execution time: mean = 25.672 us, total = 127.769 ms, Queueing time: mean = 102.405 us, max = 537.156 us, min = 4.793 us, total = 509.667 ms | |
| event_loop_lag_probe - 962 total (0 active), Execution time: mean = 28.796 us, total = 27.702 ms, Queueing time: mean = 15.997 us, max = 5.962 ms, min = 2.628 us, total = 15.389 ms | |
| GcsTaskManager.GcJobSummary - 49 total (1 active), Execution time: mean = 11.704 us, total = 573.494 us, Queueing time: mean = 80.239 us, max = 150.092 us, min = 46.206 us, total = 3.932 ms | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.843 us, total = 11.843 us, Queueing time: mean = 115.061 us, max = 115.061 us, min = 115.061 us, total = 115.061 us | |
| [2025-01-20 21:42:51,587 I 4260 4260] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: | |
| Global stats: 1306 total (25 active) | |
| Queueing time: mean = 27.248 us, max = 1.696 ms, min = 2.904 us, total = 35.585 ms | |
| Execution time: mean = 179.333 ms, total = 234.209 s | |
| Event stats: | |
| event_loop_lag_probe - 962 total (0 active), Execution time: mean = 27.051 us, total = 26.023 ms, Queueing time: mean = 11.714 us, max = 1.696 ms, min = 2.904 us, total = 11.269 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 85 total (24 active), Execution time: mean = 2.754 s, total = 234.114 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 85 total (0 active), Execution time: mean = 37.113 us, total = 3.155 ms, Queueing time: mean = 148.899 us, max = 875.211 us, min = 4.372 us, total = 12.656 ms | |
| InternalPubSubGcsService.grpc_server.GcsPublish - 61 total (0 active), Execution time: mean = 644.533 us, total = 39.317 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 61 total (0 active), Execution time: mean = 102.702 us, total = 6.265 ms, Queueing time: mean = 117.725 us, max = 199.575 us, min = 17.173 us, total = 7.181 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 72.872 us, total = 1.822 ms, Queueing time: mean = 174.047 us, max = 859.424 us, min = 11.586 us, total = 4.351 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 725.208 us, total = 18.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.853 us, total = 15.853 us, Queueing time: mean = 127.546 us, max = 127.546 us, min = 127.546 us, total = 127.546 us | |
| [2025-01-20 21:42:51,587 I 4260 4260] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: | |
| Global stats: 984 total (0 active) | |
| Queueing time: mean = 15.973 us, max = 5.285 ms, min = 674.000 ns, total = 15.718 ms | |
| Execution time: mean = 26.370 us, total = 25.948 ms | |
| Event stats: | |
| event_loop_lag_probe - 962 total (0 active), Execution time: mean = 25.534 us, total = 24.564 ms, Queueing time: mean = 14.867 us, max = 5.285 ms, min = 2.112 us, total = 14.302 ms | |
| RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 27.952 us, total = 279.516 us, Queueing time: mean = 1.136 us, max = 1.840 us, min = 674.000 ns, total = 11.362 us | |
| - 10 total (0 active), Execution time: mean = 110.186 us, total = 1.102 ms, Queueing time: mean = 121.224 us, max = 180.946 us, min = 96.930 us, total = 1.212 ms | |
| RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.152 us, total = 2.304 us, Queueing time: mean = 96.198 us, max = 98.821 us, min = 93.576 us, total = 192.397 us | |
| [2025-01-20 21:43:51,587 I 4260 4260] (gcs_server) gcs_server.cc:245: Gcs Debug state: | |
| GcsNodeManager: | |
| - RegisterNode request count: 1 | |
| - DrainNode request count: 0 | |
| - GetAllNodeInfo request count: 34 | |
| GcsActorManager: | |
| - RegisterActor request count: 0 | |
| - CreateActor request count: 0 | |
| - GetActorInfo request count: 0 | |
| - GetNamedActorInfo request count: 0 | |
| - GetAllActorInfo request count: 0 | |
| - KillActor request count: 0 | |
| - ListNamedActors request count: 0 | |
| - Registered actors count: 0 | |
| - Destroyed actors count: 0 | |
| - Named actors count: 0 | |
| - Unresolved actors count: 0 | |
| - Pending actors count: 0 | |
| - Created actors count: 0 | |
| - owners_: 0 | |
| - actor_to_register_callbacks_: 0 | |
| - actor_to_restart_callbacks_: 0 | |
| - actor_to_create_callbacks_: 0 | |
| - sorted_destroyed_actor_list_: 0 | |
| GcsResourceManager: | |
| - GetAllAvailableResources request count: 0 | |
| - GetAllTotalResources request count: 0 | |
| - GetAllResourceUsage request count: 60 | |
| GcsPlacementGroupManager: | |
| - CreatePlacementGroup request count: 0 | |
| - RemovePlacementGroup request count: 0 | |
| - GetPlacementGroup request count: 0 | |
| - GetAllPlacementGroup request count: 0 | |
| - WaitPlacementGroupUntilReady request count: 0 | |
| - GetNamedPlacementGroup request count: 0 | |
| - Scheduling pending placement group count: 0 | |
| - Registered placement groups count: 0 | |
| - Named placement group count: 0 | |
| - Pending placement groups count: 0 | |
| - Infeasible placement groups count: 0 | |
| Publisher: | |
| GCS_NODE_INFO_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 534 | |
| - current buffered bytes: 0 | |
| GCS_JOB_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 256 | |
| - current buffered bytes: 256 | |
| RAY_LOG_CHANNEL | |
| - cumulative published messages: 73 | |
| - cumulative published bytes: 53005 | |
| - current buffered bytes: 818 | |
| [runtime env manager] ID to URIs table: | |
| [runtime env manager] URIs reference table: | |
| GcsTaskManager: | |
| -Total num task events reported: 41 | |
| -Total num status task events dropped: 0 | |
| -Total num profile events dropped: 0 | |
| -Current num of task events stored: 11 | |
| -Total num of actor creation tasks: 0 | |
| -Total num of actor tasks: 0 | |
| -Total num of normal tasks: 10 | |
| -Total num of driver tasks: 1 | |
| GcsAutoscalerStateManager: | |
| - last_seen_autoscaler_state_version_: 0 | |
| - last_cluster_resource_state_version_: 0 | |
| - pending demands: | |
| [2025-01-20 21:43:51,588 I 4260 4260] (gcs_server) gcs_server.cc:843: Main service Event stats: | |
| Global stats: 4028 total (4 active) | |
| Queueing time: mean = 617.227 us, max = 254.030 ms, min = 378.000 ns, total = 2.486 s | |
| Execution time: mean = 335.378 us, total = 1.351 s | |
| Event stats: | |
| event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 23.959 us, total = 28.798 ms, Queueing time: mean = 25.035 us, max = 14.636 ms, min = 2.691 us, total = 30.092 ms | |
| GcsInMemoryStore.Put - 527 total (0 active), Execution time: mean = 490.236 us, total = 258.354 ms, Queueing time: mean = 3.406 ms, max = 253.124 ms, min = 3.250 us, total = 1.795 s | |
| RayletLoadPulled - 300 total (1 active), Execution time: mean = 278.548 us, total = 83.565 ms, Queueing time: mean = 82.540 us, max = 872.028 us, min = 14.226 us, total = 24.762 ms | |
| NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 297 total (0 active), Execution time: mean = 81.580 us, total = 24.229 ms, Queueing time: mean = 110.705 us, max = 223.449 us, min = 19.613 us, total = 32.879 ms | |
| NodeManagerService.grpc_client.GetResourceLoad - 297 total (0 active), Execution time: mean = 1.844 ms, total = 547.530 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVPut - 135 total (0 active), Execution time: mean = 591.531 us, total = 79.857 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 135 total (0 active), Execution time: mean = 42.808 us, total = 5.779 ms, Queueing time: mean = 109.555 us, max = 187.137 us, min = 16.468 us, total = 14.790 ms | |
| GcsInMemoryStore.Get - 129 total (0 active), Execution time: mean = 20.823 us, total = 2.686 ms, Queueing time: mean = 14.689 us, max = 92.727 us, min = 2.235 us, total = 1.895 ms | |
| InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 128 total (0 active), Execution time: mean = 41.256 us, total = 5.281 ms, Queueing time: mean = 107.532 us, max = 263.957 us, min = 3.438 us, total = 13.764 ms | |
| InternalKVGcsService.grpc_server.InternalKVGet - 128 total (0 active), Execution time: mean = 574.914 us, total = 73.589 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 120 total (0 active), Execution time: mean = 33.811 us, total = 4.057 ms, Queueing time: mean = 122.930 us, max = 222.930 us, min = 16.537 us, total = 14.752 ms | |
| NodeInfoGcsService.grpc_server.CheckAlive - 120 total (0 active), Execution time: mean = 602.270 us, total = 72.272 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ClusterResourceManager.ResetRemoteNodeView - 100 total (1 active), Execution time: mean = 10.427 us, total = 1.043 ms, Queueing time: mean = 92.297 us, max = 915.090 us, min = 22.301 us, total = 9.230 ms | |
| HealthCheck - 98 total (0 active), Execution time: mean = 7.899 us, total = 774.063 us, Queueing time: mean = 104.105 us, max = 198.989 us, min = 19.852 us, total = 10.202 ms | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 60 total (0 active), Execution time: mean = 689.864 us, total = 41.392 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 60 total (0 active), Execution time: mean = 87.537 us, total = 5.252 ms, Queueing time: mean = 117.240 us, max = 196.259 us, min = 30.354 us, total = 7.034 ms | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 427.723 us, total = 14.543 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 25.393 us, total = 863.353 us, Queueing time: mean = 104.993 us, max = 248.874 us, min = 6.954 us, total = 3.570 ms | |
| GCSServer.deadline_timer.debug_state_dump - 30 total (1 active), Execution time: mean = 1.374 ms, total = 41.211 ms, Queueing time: mean = 79.002 us, max = 149.454 us, min = 36.186 us, total = 2.370 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 18.783 us, total = 394.435 us, Queueing time: mean = 560.184 us, max = 2.600 ms, min = 16.168 us, total = 11.764 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.408 ms, total = 29.562 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 98.825 us, total = 988.251 us, Queueing time: mean = 119.914 us, max = 147.745 us, min = 48.942 us, total = 1.199 ms | |
| GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 20.125 us, total = 120.749 us, Queueing time: mean = 99.161 us, max = 130.379 us, min = 10.591 us, total = 594.966 us | |
| NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 837.877 us, total = 4.189 ms, Queueing time: mean = 388.123 us, max = 1.421 ms, min = 110.920 us, total = 1.941 ms | |
| NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.164 ms, total = 10.819 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GCSServer.deadline_timer.debug_state_event_stats_print - 5 total (1 active, 1 running), Execution time: mean = 1.415 ms, total = 7.076 ms, Queueing time: mean = 89.161 us, max = 195.056 us, min = 64.820 us, total = 445.806 us | |
| PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 349.685 us, total = 1.399 ms, Queueing time: mean = 127.220 ms, max = 254.030 ms, min = 149.272 us, total = 508.880 ms | |
| InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 41.998 us, total = 41.998 us, Queueing time: mean = 119.689 us, max = 119.689 us, min = 119.689 us, total = 119.689 us | |
| InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 797.189 us, total = 797.189 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 1.060 ms, total = 1.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 374.932 us, total = 374.932 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 15.338 us, total = 15.338 us, Queueing time: mean = 103.670 us, max = 103.670 us, min = 103.670 us, total = 103.670 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 601.341 us, total = 601.341 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 22.599 us, total = 22.599 us, Queueing time: mean = 12.193 us, max = 12.193 us, min = 12.193 us, total = 12.193 us | |
| InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 71.823 us, total = 71.823 us, Queueing time: mean = 103.468 us, max = 103.468 us, min = 103.468 us, total = 103.468 us | |
| JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 576.698 us, total = 576.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 859.817 us, total = 859.817 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 546.537 us, total = 546.537 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 138.394 us, total = 138.394 us, Queueing time: mean = 172.415 us, max = 172.415 us, min = 172.415 us, total = 172.415 us | |
| GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.642 us, total = 16.642 us, Queueing time: mean = 10.027 us, max = 10.027 us, min = 10.027 us, total = 10.027 us | |
| GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 8.131 us, total = 8.131 us, Queueing time: mean = 378.000 ns, max = 378.000 ns, min = 378.000 ns, total = 378.000 ns | |
| JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.440 us, total = 22.440 us, Queueing time: mean = 87.206 us, max = 87.206 us, min = 87.206 us, total = 87.206 us | |
| NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 101.666 us, total = 101.666 us, Queueing time: mean = 137.157 us, max = 137.157 us, min = 137.157 us, total = 137.157 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.884 us, total = 22.884 us, Queueing time: mean = 111.600 us, max = 111.600 us, min = 111.600 us, total = 111.600 us | |
| [2025-01-20 21:43:51,588 I 4260 4260] (gcs_server) gcs_server.cc:847: task_io_context Event stats: | |
| Global stats: 13738 total (1 active) | |
| Queueing time: mean = 48.901 us, max = 5.962 ms, min = 2.628 us, total = 671.799 ms | |
| Execution time: mean = 253.382 us, total = 3.481 s | |
| Event stats: | |
| TaskInfoGcsService.grpc_server.AddTaskEventData - 6237 total (0 active), Execution time: mean = 526.525 us, total = 3.284 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 6237 total (0 active), Execution time: mean = 25.945 us, total = 161.821 ms, Queueing time: mean = 104.062 us, max = 537.156 us, min = 4.793 us, total = 649.034 ms | |
| event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 28.687 us, total = 34.481 ms, Queueing time: mean = 14.736 us, max = 5.962 ms, min = 2.628 us, total = 17.712 ms | |
| GcsTaskManager.GcJobSummary - 61 total (1 active), Execution time: mean = 11.630 us, total = 709.457 us, Queueing time: mean = 80.935 us, max = 151.149 us, min = 46.206 us, total = 4.937 ms | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.843 us, total = 11.843 us, Queueing time: mean = 115.061 us, max = 115.061 us, min = 115.061 us, total = 115.061 us | |
| [2025-01-20 21:43:51,589 I 4260 4260] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: | |
| Global stats: 1595 total (25 active) | |
| Queueing time: mean = 26.292 us, max = 1.696 ms, min = 2.744 us, total = 41.936 ms | |
| Execution time: mean = 184.532 ms, total = 294.328 s | |
| Event stats: | |
| event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 27.918 us, total = 33.557 ms, Queueing time: mean = 12.055 us, max = 1.696 ms, min = 2.744 us, total = 14.490 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 97 total (24 active), Execution time: mean = 3.033 s, total = 294.215 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 97 total (0 active), Execution time: mean = 38.151 us, total = 3.701 ms, Queueing time: mean = 144.478 us, max = 875.211 us, min = 4.372 us, total = 14.014 ms | |
| InternalPubSubGcsService.grpc_server.GcsPublish - 73 total (0 active), Execution time: mean = 659.311 us, total = 48.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 73 total (0 active), Execution time: mean = 105.448 us, total = 7.698 ms, Queueing time: mean = 121.298 us, max = 199.575 us, min = 17.173 us, total = 8.855 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 72.872 us, total = 1.822 ms, Queueing time: mean = 174.047 us, max = 859.424 us, min = 11.586 us, total = 4.351 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 725.208 us, total = 18.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 12.428 us, total = 24.857 us, Queueing time: mean = 49.191 us, max = 98.382 us, min = 98.382 us, total = 98.382 us | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.853 us, total = 15.853 us, Queueing time: mean = 127.546 us, max = 127.546 us, min = 127.546 us, total = 127.546 us | |
| [2025-01-20 21:43:51,589 I 4260 4260] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: | |
| Global stats: 1224 total (0 active) | |
| Queueing time: mean = 14.627 us, max = 5.285 ms, min = 674.000 ns, total = 17.903 ms | |
| Execution time: mean = 25.982 us, total = 31.802 ms | |
| Event stats: | |
| event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 25.306 us, total = 30.418 ms, Queueing time: mean = 13.717 us, max = 5.285 ms, min = 2.112 us, total = 16.487 ms | |
| RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 27.952 us, total = 279.516 us, Queueing time: mean = 1.136 us, max = 1.840 us, min = 674.000 ns, total = 11.362 us | |
| - 10 total (0 active), Execution time: mean = 110.186 us, total = 1.102 ms, Queueing time: mean = 121.224 us, max = 180.946 us, min = 96.930 us, total = 1.212 ms | |
| RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.152 us, total = 2.304 us, Queueing time: mean = 96.198 us, max = 98.821 us, min = 93.576 us, total = 192.397 us | |
| [2025-01-20 21:44:51,589 I 4260 4260] (gcs_server) gcs_server.cc:245: Gcs Debug state: | |
| GcsNodeManager: | |
| - RegisterNode request count: 1 | |
| - DrainNode request count: 0 | |
| - GetAllNodeInfo request count: 34 | |
| GcsActorManager: | |
| - RegisterActor request count: 0 | |
| - CreateActor request count: 0 | |
| - GetActorInfo request count: 0 | |
| - GetNamedActorInfo request count: 0 | |
| - GetAllActorInfo request count: 0 | |
| - KillActor request count: 0 | |
| - ListNamedActors request count: 0 | |
| - Registered actors count: 0 | |
| - Destroyed actors count: 0 | |
| - Named actors count: 0 | |
| - Unresolved actors count: 0 | |
| - Pending actors count: 0 | |
| - Created actors count: 0 | |
| - owners_: 0 | |
| - actor_to_register_callbacks_: 0 | |
| - actor_to_restart_callbacks_: 0 | |
| - actor_to_create_callbacks_: 0 | |
| - sorted_destroyed_actor_list_: 0 | |
| GcsResourceManager: | |
| - GetAllAvailableResources request count: 0 | |
| - GetAllTotalResources request count: 0 | |
| - GetAllResourceUsage request count: 72 | |
| GcsPlacementGroupManager: | |
| - CreatePlacementGroup request count: 0 | |
| - RemovePlacementGroup request count: 0 | |
| - GetPlacementGroup request count: 0 | |
| - GetAllPlacementGroup request count: 0 | |
| - WaitPlacementGroupUntilReady request count: 0 | |
| - GetNamedPlacementGroup request count: 0 | |
| - Scheduling pending placement group count: 0 | |
| - Registered placement groups count: 0 | |
| - Named placement group count: 0 | |
| - Pending placement groups count: 0 | |
| - Infeasible placement groups count: 0 | |
| Publisher: | |
| GCS_NODE_INFO_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 534 | |
| - current buffered bytes: 0 | |
| GCS_JOB_CHANNEL | |
| - cumulative published messages: 1 | |
| - cumulative published bytes: 256 | |
| - current buffered bytes: 256 | |
| RAY_LOG_CHANNEL | |
| - cumulative published messages: 86 | |
| - cumulative published bytes: 62854 | |
| - current buffered bytes: 818 | |
| [runtime env manager] ID to URIs table: | |
| [runtime env manager] URIs reference table: | |
| GcsTaskManager: | |
| -Total num task events reported: 41 | |
| -Total num status task events dropped: 0 | |
| -Total num profile events dropped: 0 | |
| -Current num of task events stored: 11 | |
| -Total num of actor creation tasks: 0 | |
| -Total num of actor tasks: 0 | |
| -Total num of normal tasks: 10 | |
| -Total num of driver tasks: 1 | |
| GcsAutoscalerStateManager: | |
| - last_seen_autoscaler_state_version_: 0 | |
| - last_cluster_resource_state_version_: 0 | |
| - pending demands: | |
| [2025-01-20 21:44:51,590 I 4260 4260] (gcs_server) gcs_server.cc:843: Main service Event stats: | |
| Global stats: 4747 total (4 active) | |
| Queueing time: mean = 530.233 us, max = 254.030 ms, min = 378.000 ns, total = 2.517 s | |
| Execution time: mean = 321.869 us, total = 1.528 s | |
| Event stats: | |
| event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 23.593 us, total = 34.021 ms, Queueing time: mean = 22.288 us, max = 14.636 ms, min = 2.691 us, total = 32.139 ms | |
| GcsInMemoryStore.Put - 623 total (0 active), Execution time: mean = 415.638 us, total = 258.942 ms, Queueing time: mean = 2.889 ms, max = 253.124 ms, min = 3.250 us, total = 1.800 s | |
| RayletLoadPulled - 360 total (1 active), Execution time: mean = 272.925 us, total = 98.253 ms, Queueing time: mean = 81.601 us, max = 872.028 us, min = 14.226 us, total = 29.376 ms | |
| NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 357 total (0 active), Execution time: mean = 80.548 us, total = 28.756 ms, Queueing time: mean = 110.376 us, max = 223.449 us, min = 19.613 us, total = 39.404 ms | |
| NodeManagerService.grpc_client.GetResourceLoad - 357 total (0 active), Execution time: mean = 1.798 ms, total = 641.730 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVPut - 159 total (0 active), Execution time: mean = 590.051 us, total = 93.818 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 159 total (0 active), Execution time: mean = 42.394 us, total = 6.741 ms, Queueing time: mean = 110.636 us, max = 187.137 us, min = 16.468 us, total = 17.591 ms | |
| NodeInfoGcsService.grpc_server.CheckAlive - 144 total (0 active), Execution time: mean = 597.785 us, total = 86.081 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 144 total (0 active), Execution time: mean = 33.241 us, total = 4.787 ms, Queueing time: mean = 123.748 us, max = 389.621 us, min = 16.537 us, total = 17.820 ms | |
| GcsInMemoryStore.Get - 141 total (0 active), Execution time: mean = 21.109 us, total = 2.976 ms, Queueing time: mean = 14.551 us, max = 92.727 us, min = 2.235 us, total = 2.052 ms | |
| InternalKVGcsService.grpc_server.InternalKVGet - 140 total (0 active), Execution time: mean = 577.907 us, total = 80.907 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 140 total (0 active), Execution time: mean = 42.060 us, total = 5.888 ms, Queueing time: mean = 107.122 us, max = 263.957 us, min = 3.438 us, total = 14.997 ms | |
| ClusterResourceManager.ResetRemoteNodeView - 120 total (1 active), Execution time: mean = 10.219 us, total = 1.226 ms, Queueing time: mean = 91.302 us, max = 915.090 us, min = 22.301 us, total = 10.956 ms | |
| HealthCheck - 118 total (0 active), Execution time: mean = 7.830 us, total = 923.937 us, Queueing time: mean = 106.028 us, max = 198.989 us, min = 19.852 us, total = 12.511 ms | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 72 total (0 active), Execution time: mean = 85.558 us, total = 6.160 ms, Queueing time: mean = 117.831 us, max = 196.259 us, min = 30.354 us, total = 8.484 ms | |
| NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 72 total (0 active), Execution time: mean = 692.129 us, total = 49.833 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GCSServer.deadline_timer.debug_state_dump - 36 total (1 active), Execution time: mean = 1.387 ms, total = 49.917 ms, Queueing time: mean = 76.711 us, max = 149.454 us, min = 36.186 us, total = 2.762 ms | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 25.393 us, total = 863.353 us, Queueing time: mean = 104.993 us, max = 248.874 us, min = 6.954 us, total = 3.570 ms | |
| NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 427.723 us, total = 14.543 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 18.783 us, total = 394.435 us, Queueing time: mean = 560.184 us, max = 2.600 ms, min = 16.168 us, total = 11.764 ms | |
| WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.408 ms, total = 29.562 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 98.825 us, total = 988.251 us, Queueing time: mean = 119.914 us, max = 147.745 us, min = 48.942 us, total = 1.199 ms | |
| GCSServer.deadline_timer.debug_state_event_stats_print - 6 total (1 active, 1 running), Execution time: mean = 1.466 ms, total = 8.797 ms, Queueing time: mean = 84.675 us, max = 195.056 us, min = 62.246 us, total = 508.052 us | |
| GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 20.125 us, total = 120.749 us, Queueing time: mean = 99.161 us, max = 130.379 us, min = 10.591 us, total = 594.966 us | |
| NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 837.877 us, total = 4.189 ms, Queueing time: mean = 388.123 us, max = 1.421 ms, min = 110.920 us, total = 1.941 ms | |
| NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.164 ms, total = 10.819 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 349.685 us, total = 1.399 ms, Queueing time: mean = 127.220 ms, max = 254.030 ms, min = 149.272 us, total = 508.880 ms | |
| InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 41.998 us, total = 41.998 us, Queueing time: mean = 119.689 us, max = 119.689 us, min = 119.689 us, total = 119.689 us | |
| InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 797.189 us, total = 797.189 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 1.060 ms, total = 1.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 374.932 us, total = 374.932 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 15.338 us, total = 15.338 us, Queueing time: mean = 103.670 us, max = 103.670 us, min = 103.670 us, total = 103.670 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 601.341 us, total = 601.341 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 22.599 us, total = 22.599 us, Queueing time: mean = 12.193 us, max = 12.193 us, min = 12.193 us, total = 12.193 us | |
| InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 71.823 us, total = 71.823 us, Queueing time: mean = 103.468 us, max = 103.468 us, min = 103.468 us, total = 103.468 us | |
| JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 576.698 us, total = 576.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 859.817 us, total = 859.817 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 546.537 us, total = 546.537 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 138.394 us, total = 138.394 us, Queueing time: mean = 172.415 us, max = 172.415 us, min = 172.415 us, total = 172.415 us | |
| GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.642 us, total = 16.642 us, Queueing time: mean = 10.027 us, max = 10.027 us, min = 10.027 us, total = 10.027 us | |
| GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 8.131 us, total = 8.131 us, Queueing time: mean = 378.000 ns, max = 378.000 ns, min = 378.000 ns, total = 378.000 ns | |
| JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.440 us, total = 22.440 us, Queueing time: mean = 87.206 us, max = 87.206 us, min = 87.206 us, total = 87.206 us | |
| NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 101.666 us, total = 101.666 us, Queueing time: mean = 137.157 us, max = 137.157 us, min = 137.157 us, total = 137.157 us | |
| InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.884 us, total = 22.884 us, Queueing time: mean = 111.600 us, max = 111.600 us, min = 111.600 us, total = 111.600 us | |
| [2025-01-20 21:44:51,590 I 4260 4260] (gcs_server) gcs_server.cc:847: task_io_context Event stats: | |
| Global stats: 16510 total (1 active) | |
| Queueing time: mean = 49.319 us, max = 5.962 ms, min = 2.628 us, total = 814.261 ms | |
| Execution time: mean = 254.658 us, total = 4.204 s | |
| Event stats: | |
| TaskInfoGcsService.grpc_server.AddTaskEventData - 7497 total (0 active), Execution time: mean = 529.364 us, total = 3.969 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 7497 total (0 active), Execution time: mean = 25.836 us, total = 193.694 ms, Queueing time: mean = 105.146 us, max = 1.610 ms, min = 4.793 us, total = 788.279 ms | |
| event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 28.587 us, total = 41.223 ms, Queueing time: mean = 13.917 us, max = 5.962 ms, min = 2.628 us, total = 20.068 ms | |
| GcsTaskManager.GcJobSummary - 73 total (1 active), Execution time: mean = 11.529 us, total = 841.647 us, Queueing time: mean = 79.430 us, max = 151.149 us, min = 46.206 us, total = 5.798 ms | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.843 us, total = 11.843 us, Queueing time: mean = 115.061 us, max = 115.061 us, min = 115.061 us, total = 115.061 us | |
| [2025-01-20 21:44:51,590 I 4260 4260] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: | |
| Global stats: 1887 total (25 active) | |
| Queueing time: mean = 25.015 us, max = 1.696 ms, min = 2.744 us, total = 47.203 ms | |
| Execution time: mean = 187.853 ms, total = 354.478 s | |
| Event stats: | |
| event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 27.637 us, total = 39.852 ms, Queueing time: mean = 11.694 us, max = 1.696 ms, min = 2.744 us, total = 16.863 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 110 total (24 active), Execution time: mean = 3.221 s, total = 354.349 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 110 total (0 active), Execution time: mean = 38.802 us, total = 4.268 ms, Queueing time: mean = 139.921 us, max = 875.211 us, min = 4.372 us, total = 15.391 ms | |
| InternalPubSubGcsService.grpc_server.GcsPublish - 86 total (0 active), Execution time: mean = 660.345 us, total = 56.790 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 86 total (0 active), Execution time: mean = 104.033 us, total = 8.947 ms, Queueing time: mean = 120.596 us, max = 215.537 us, min = 17.173 us, total = 10.371 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 72.872 us, total = 1.822 ms, Queueing time: mean = 174.047 us, max = 859.424 us, min = 11.586 us, total = 4.351 ms | |
| InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 725.208 us, total = 18.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 12.428 us, total = 24.857 us, Queueing time: mean = 49.191 us, max = 98.382 us, min = 98.382 us, total = 98.382 us | |
| PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.853 us, total = 15.853 us, Queueing time: mean = 127.546 us, max = 127.546 us, min = 127.546 us, total = 127.546 us | |
| [2025-01-20 21:44:51,590 I 4260 4260] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: | |
| Global stats: 1464 total (0 active) | |
| Queueing time: mean = 13.822 us, max = 5.285 ms, min = 674.000 ns, total = 20.236 ms | |
| Execution time: mean = 25.975 us, total = 38.027 ms | |
| Event stats: | |
| event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 25.411 us, total = 36.643 ms, Queueing time: mean = 13.051 us, max = 5.285 ms, min = 2.112 us, total = 18.820 ms | |
| RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 27.952 us, total = 279.516 us, Queueing time: mean = 1.136 us, max = 1.840 us, min = 674.000 ns, total = 11.362 us | |
| - 10 total (0 active), Execution time: mean = 110.186 us, total = 1.102 ms, Queueing time: mean = 121.224 us, max = 180.946 us, min = 96.930 us, total = 1.212 ms | |
| RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.152 us, total = 2.304 us, Queueing time: mean = 96.198 us, max = 98.821 us, min = 93.576 us, total = 192.397 us | |
| [2025-01-20 21:45:03,871 I 4260 4260] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000 | |
| [2025-01-20 21:45:03,912 I 4260 4260] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c | |
| [2025-01-20 21:45:03,912 I 4260 4260] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c | |
| [2025-01-20 21:45:03,912 I 4260 4260] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c | |
| [2025-01-20 21:45:03,912 I 4260 4260] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c | |
| [2025-01-20 21:45:04,040 I 4260 4309] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c | |
| [2025-01-20 21:45:04,041 I 4260 4309] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=1b777e63cfc963b3e23257c9de65d9bed4a8abe223aa6aed7802892c | |
| [2025-01-20 21:45:04,075 I 4260 4260] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down... | |
| [2025-01-20 21:45:04,077 I 4260 4260] (gcs_server) gcs_server.cc:267: Stopping GCS server. | |
| [2025-01-20 21:45:04,175 I 4260 4260] (gcs_server) gcs_server.cc:284: GCS server stopped. | |
| [2025-01-20 21:45:04,175 I 4260 4260] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped. | |
| [2025-01-20 21:45:04,277 I 4260 4260] (gcs_server) stats.h:120: Stats module has shutdown. | |