2025-04-25 12:27:30.605786399   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if config file exists: context deadline exceeded
2025-04-25 12:27:30.892846786   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175151
2025-04-25 12:27:30.892849371   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:30.980826274   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:31.018533767   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:31.018569965   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:31.192756029   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175160) =======
2025-04-25 12:27:31.192759546   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:31.192760918   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:31.192761960   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3268.09m cores, Cores=24
2025-04-25 12:27:31.192763152   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15762.01 MB, Total=128728.99 MB, Usage=12.24%
2025-04-25 12:27:31.192787007   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:31.192788640   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:31.192789772   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:31.392706070   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175156
2025-04-25 12:27:31.392709977   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:31.493160923   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:31.493500200   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0028eca60})
2025-04-25 12:27:31.593493085   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:31.692450944   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:31.692535292   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:31.692582100   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:31.792604981   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:31.893751353   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175161
2025-04-25 12:27:31.893754549   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:31.893824941   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134430) =======
2025-04-25 12:27:31.893825853   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:31.893832235   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:31.893836252   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:31.893841683   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:31.893842354   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:31.893851692   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:31.893852383   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:31.980585038   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:31.993946417   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175162
2025-04-25 12:27:31.993967797   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:32.092797315   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175163
2025-04-25 12:27:32.092799870   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:32.125289848   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:32.125295078   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:32.193348038   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175170) =======
2025-04-25 12:27:32.193350272   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:32.193351153   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:32.193371181   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5243.70m cores, Cores=24
2025-04-25 12:27:32.193371922   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15738.84 MB, Total=128728.99 MB, Usage=12.23%
2025-04-25 12:27:32.193372584   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:32.193373365   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:32.193377373   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:32.292864236   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175165
2025-04-25 12:27:32.292870508   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:32.592758139   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175168
2025-04-25 12:27:32.592761415   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:32.599369093   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": net/http: timeout awaiting response headers
2025-04-25 12:27:32.599372910   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": net/http: timeout awaiting response headers
2025-04-25 12:27:32.892562203   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134440) =======
2025-04-25 12:27:32.892564878   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:32.892566211   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:32.892589284   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:32.892590216   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:32.892591147   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:32.892592390   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:32.892593342   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:32.900025920   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:27:32.900030037   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:27:32.980393022   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:27:33.019372554   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:27:33.019378425   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:33.085994656   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175199, managers: 7
2025-04-25 12:27:33.085997691   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134400
2025-04-25 12:27:33.085999164   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:27:33.086000306   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:27:33.086001248   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175185
2025-04-25 12:27:33.086093972   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175185
2025-04-25 12:27:33.086096737   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175178
2025-04-25 12:27:33.086098140   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:27:33.086099282   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175172
2025-04-25 12:27:33.086100294   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:27:33.086101256   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134441
2025-04-25 12:27:33.086102298   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:27:33.086103320   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134400
2025-04-25 12:27:33.092351021   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175173
2025-04-25 12:27:33.092369376   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:33.191834708   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175180) =======
2025-04-25 12:27:33.191837894   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:33.191839186   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:33.191840208   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3160.49m cores, Cores=24
2025-04-25 12:27:33.191841220   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15754.86 MB, Total=128728.99 MB, Usage=12.24%
2025-04-25 12:27:33.191864865   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:33.191866047   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:33.191867019   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:33.191868091   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:33.192223118   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0001fd520})
2025-04-25 12:27:33.291915757   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:33.392482590   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:33.392612063   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:33.392631139   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:33.492888039   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:33.593037798   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175178
2025-04-25 12:27:33.593072383   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:33.691638586   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175179
2025-04-25 12:27:33.691641982   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:33.793033164   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175180
2025-04-25 12:27:33.793036410   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:33.980552885   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:33.992564167   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175182
2025-04-25 12:27:33.992567874   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:34.112925609   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:34.112931309   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:34.191745473   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175190) =======
2025-04-25 12:27:34.191748057   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:34.191748959   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:34.191750111   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4739.50m cores, Cores=24
2025-04-25 12:27:34.191751273   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15775.13 MB, Total=128728.99 MB, Usage=12.25%
2025-04-25 12:27:34.191752486   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:34.191783003   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:34.191783795   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:34.292686618   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175185
2025-04-25 12:27:34.292690265   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:34.292771939   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134450) =======
2025-04-25 12:27:34.292774764   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:34.292776076   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:34.292777569   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:34.292795803   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:34.292796895   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:34.292798148   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:34.292799090   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:34.299546690   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:34.299552962   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:34.792115260   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175190
2025-04-25 12:27:34.792141569   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:34.893183895   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:34.893573047   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc004ec5400})
2025-04-25 12:27:34.980466433   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:27:34.992770526   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:35.092564906   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:35.092635068   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:35.092662479   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:35.192471938   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175200) =======
2025-04-25 12:27:35.192476386   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:35.192478521   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:35.192507796   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2200.00m cores, Cores=24
2025-04-25 12:27:35.192509318   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15767.79 MB, Total=128728.99 MB, Usage=12.25%
2025-04-25 12:27:35.192510541   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:35.192512064   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:35.192513136   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:35.192600169   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:35.292349925   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175195
2025-04-25 12:27:35.292378038   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:35.393708957   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175196
2025-04-25 12:27:35.393714888   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:35.400771258   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:35.400774344   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:35.492926342   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175197
2025-04-25 12:27:35.492929248   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:35.691512189   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175199
2025-04-25 12:27:35.691516046   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:35.697140586   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134460) =======
2025-04-25 12:27:35.697143121   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:35.697144493   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:35.697160213   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:35.697161114   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:35.697162046   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:35.697168328   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:35.697169290   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:35.980664804   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:35.992355174   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175202
2025-04-25 12:27:35.992357668   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:36.115931717   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:27:36.115939331   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:36.192788371   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175210) =======
2025-04-25 12:27:36.192790996   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:36.192792288   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:36.192793400   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4400.00m cores, Cores=24
2025-04-25 12:27:36.192816494   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15725.46 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:27:36.192817576   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:36.192818738   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:36.192819690   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:36.492780296   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175207
2025-04-25 12:27:36.492782981   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:36.592016277   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:36.592357187   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0028edf20})
2025-04-25 12:27:36.692899112   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:36.793023783   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:36.793100838   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:36.793109404   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:36.893620952   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:36.980761783   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:27:36.993820323   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175212
2025-04-25 12:27:36.993822968   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:37.092645131   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175213
2025-04-25 12:27:37.092648057   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:37.092765457   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134470) =======
2025-04-25 12:27:37.092766870   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:37.092767782   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:37.092768593   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:37.092786827   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:37.092787569   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:37.092788400   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:37.092789052   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:37.193088020   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175220) =======
2025-04-25 12:27:37.193090104   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:37.193090946   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:37.193109831   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2987.55m cores, Cores=24
2025-04-25 12:27:37.193110552   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15738.56 MB, Total=128728.99 MB, Usage=12.23%
2025-04-25 12:27:37.193111194   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:37.193111975   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:37.193116293   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:37.193382544   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175214
2025-04-25 12:27:37.193385449   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:37.392428564   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175216
2025-04-25 12:27:37.392431840   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:37.398893413   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:37.398900386   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:37.691913896   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175219
2025-04-25 12:27:37.691917112   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:37.698802682   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:37.698822739   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:37.980311752   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:38.086143026   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175249, managers: 7
2025-04-25 12:27:38.086145872   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175235
2025-04-25 12:27:38.086147234   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175235
2025-04-25 12:27:38.086171139   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175228
2025-04-25 12:27:38.086172241   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:27:38.086173263   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175222
2025-04-25 12:27:38.086174255   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:27:38.086175197   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134479
2025-04-25 12:27:38.086176209   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:27:38.086177181   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134438
2025-04-25 12:27:38.086178152   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134438
2025-04-25 12:27:38.086184424   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:27:38.086185486   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:27:38.116667300   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134480) =======
2025-04-25 12:27:38.116671838   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:38.116673121   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:38.116674383   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:38.116675265   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:38.116676266   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:38.116677699   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:38.116678751   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:38.124831622   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:27:38.124868682   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:38.191675449   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175230) =======
2025-04-25 12:27:38.191677644   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:38.191678525   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:38.191679216   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4474.58m cores, Cores=24
2025-04-25 12:27:38.191706077   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15730.42 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:27:38.191706808   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:38.191707610   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:38.191708261   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:38.192120055   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175224
2025-04-25 12:27:38.192136496   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:38.293070858   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:38.293416057   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc000f55b20})
2025-04-25 12:27:38.392381188   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:38.492019275   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:38.492085910   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:38.492098904   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:38.592797443   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:38.692192533   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175229
2025-04-25 12:27:38.692195057   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:38.792749916   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175230
2025-04-25 12:27:38.792752641   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:38.892662699   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175231
2025-04-25 12:27:38.892685592   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:38.980513213   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:39.092565588   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175233
2025-04-25 12:27:39.092568013   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:39.119926707   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:39.119932518   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:39.194094477   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175240) =======
2025-04-25 12:27:39.194099707   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:39.194101270   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:39.194102402   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5530.86m cores, Cores=24
2025-04-25 12:27:39.194133801   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15781.22 MB, Total=128728.99 MB, Usage=12.26%
2025-04-25 12:27:39.194135134   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:39.194136546   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:39.194137618   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:39.202115812   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:27:39.202208246   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:39.392592437   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:39.492431631   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134490) =======
2025-04-25 12:27:39.492433855   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:39.492434747   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:39.492435538   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:39.492454714   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:39.492455415   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:39.492456267   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:39.492456908   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:39.892299827   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175241
2025-04-25 12:27:39.892302242   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:39.980681439   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:39.993687571   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:39.994053749   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc006c7c7a0})
2025-04-25 12:27:40.110296460   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:40.192483971   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175250) =======
2025-04-25 12:27:40.192486095   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:40.192486967   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:40.192504500   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2071.07m cores, Cores=24
2025-04-25 12:27:40.192505271   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15694.17 MB, Total=128728.99 MB, Usage=12.19%
2025-04-25 12:27:40.192505922   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:40.192506704   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:40.192507315   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:40.192565514   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:40.192654381   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:40.192670161   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:40.292529633   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:40.392897199   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175246
2025-04-25 12:27:40.392899814   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:40.493331942   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175247
2025-04-25 12:27:40.493334577   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:40.501651747   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:27:40.593140979   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175248
2025-04-25 12:27:40.593144185   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:40.791816492   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:40.893577297   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134500) =======
2025-04-25 12:27:40.893580352   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:40.893581254   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:40.893606732   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:40.893607393   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:40.893608014   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:40.893608856   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:40.893609497   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:40.980239107   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:41.094104719   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175253
2025-04-25 12:27:41.094107865   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:41.139356158   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 25.638341ms remaining, needed 30ms), skipping
2025-04-25 12:27:41.192636165   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175260) =======
2025-04-25 12:27:41.192640834   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:41.192642908   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:41.192644631   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5459.23m cores, Cores=24
2025-04-25 12:27:41.192678314   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15701.47 MB, Total=128728.99 MB, Usage=12.20%
2025-04-25 12:27:41.192680438   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:41.192682813   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:41.192684776   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:41.201864127   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:41.201869336   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:41.592772674   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175258
2025-04-25 12:27:41.592777373   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:41.692309079   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:41.692665839   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002f0af20})
2025-04-25 12:27:41.792134877   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:41.891552177   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:41.891630464   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:41.891653387   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:41.980779869   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.38 seconds ago
2025-04-25 12:27:41.993925854   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:42.092442371   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175263
2025-04-25 12:27:42.092447491   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:42.191606576   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175270) =======
2025-04-25 12:27:42.191609391   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:42.191610684   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:42.191611696   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3762.71m cores, Cores=24
2025-04-25 12:27:42.191637474   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15713.51 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:27:42.191639608   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:42.191641792   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:42.191643375   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:42.192045310   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175264
2025-04-25 12:27:42.192057032   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:42.292526721   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175265
2025-04-25 12:27:42.292568840   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:42.292766321   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134510) =======
2025-04-25 12:27:42.292767704   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:42.292779316   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:42.292780137   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:42.292784345   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:42.292787932   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:42.292801217   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:42.292801928   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:42.305179939   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.877839ms remaining, needed 30ms), skipping
2025-04-25 12:27:42.493476406   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175267
2025-04-25 12:27:42.493479833   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:42.792965613   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175270
2025-04-25 12:27:42.792972035   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:42.801284105   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:27:42.980280084   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:43.085468790   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175299, managers: 7
2025-04-25 12:27:43.085471044   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175285
2025-04-25 12:27:43.085491232   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175285
2025-04-25 12:27:43.085492024   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175278
2025-04-25 12:27:43.085492705   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:27:43.085493386   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175272
2025-04-25 12:27:43.085494067   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:27:43.085494709   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134517
2025-04-25 12:27:43.085499457   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:27:43.085500169   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134476
2025-04-25 12:27:43.085500840   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134476
2025-04-25 12:27:43.085504457   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:27:43.085508004   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:27:43.113809499   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:43.113848783   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:43.193448802   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175280) =======
2025-04-25 12:27:43.193451607   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:43.193452899   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:43.193453891   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3186.72m cores, Cores=24
2025-04-25 12:27:43.193484899   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15703.94 MB, Total=128728.99 MB, Usage=12.20%
2025-04-25 12:27:43.193486032   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:43.193487184   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:43.193488156   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:43.293471870   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175275
2025-04-25 12:27:43.293475477   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:43.293670884   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134520) =======
2025-04-25 12:27:43.293672307   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:43.293673359   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:43.293674431   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:43.293675283   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:43.293676244   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:43.293677417   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:43.293678398   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:43.391829018   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:43.392126588   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001214f00})
2025-04-25 12:27:43.492323723   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:43.591689045   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:43.591740963   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:43.591753426   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:43.692711372   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:43.793090710   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175280
2025-04-25 12:27:43.793093956   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:43.892414885   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175281
2025-04-25 12:27:43.892418332   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:43.980108063   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:43.993345078   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175282
2025-04-25 12:27:43.993348034   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:44.193704584   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175290) =======
2025-04-25 12:27:44.193708331   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:44.193709814   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:44.193732968   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1600.00m cores, Cores=24
2025-04-25 12:27:44.193734330   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15720.59 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:27:44.193735502   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:44.193736835   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:44.193737977   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:44.194117320   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175284
2025-04-25 12:27:44.194127299   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:44.218694106   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:44.218721217   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:44.493488974   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175287
2025-04-25 12:27:44.493513560   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:44.692359792   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134530) =======
2025-04-25 12:27:44.692361986   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:44.692362878   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:44.692380962   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:44.692381603   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:44.692382214   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:44.692383056   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:44.692383697   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:44.980654570   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:44.991972499   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:45.092233114   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:45.092563765   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc006f34220})
2025-04-25 12:27:45.192637509   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175300) =======
2025-04-25 12:27:45.192641346   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:45.192642859   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:45.192667465   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3000.00m cores, Cores=24
2025-04-25 12:27:45.192668838   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15704.04 MB, Total=128728.99 MB, Usage=12.20%
2025-04-25 12:27:45.192669920   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:45.192671202   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:45.192672204   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:45.192730033   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:45.291928892   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:45.292016036   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:45.292033038   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:45.393505510   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:45.493671617   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175297
2025-04-25 12:27:45.493674713   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:45.507458997   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 26.709473ms remaining, needed 30ms), skipping
2025-04-25 12:27:45.593745410   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175298
2025-04-25 12:27:45.593749197   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:45.692755054   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175299
2025-04-25 12:27:45.692760254   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:45.892576408   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175301
2025-04-25 12:27:45.892579293   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:45.980635422   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:46.093111834   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134540) =======
2025-04-25 12:27:46.093115020   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:46.093116402   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:46.093117534   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:46.093142722   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:46.093144495   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:46.093146489   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:46.093148002   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:46.192378891   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175310) =======
2025-04-25 12:27:46.192398718   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:46.192410811   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:46.192417313   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4336.13m cores, Cores=24
2025-04-25 12:27:46.192429145   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15725.71 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:27:46.192435527   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:46.192447530   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:46.192453862   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:46.192680177   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175304
2025-04-25 12:27:46.192877528   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:46.205033672   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.504737ms remaining, needed 30ms), skipping
2025-04-25 12:27:46.693745773   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175309
2025-04-25 12:27:46.693750622   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:46.792040011   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:46.792364381   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc003a6ef80})
2025-04-25 12:27:46.892352333   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:46.980588440   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:27:46.993587439   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:46.993656589   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:46.993676717   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:47.092386536   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:47.192057833   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175320) =======
2025-04-25 12:27:47.192061039   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:47.192062362   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:47.192085836   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2227.85m cores, Cores=24
2025-04-25 12:27:47.192086948   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15702.11 MB, Total=128728.99 MB, Usage=12.20%
2025-04-25 12:27:47.192087940   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:47.192089142   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:47.192090094   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:47.192338360   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175314
2025-04-25 12:27:47.192370942   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:47.199893368   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:47.199925138   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:47.292396043   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: failed to check context: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:47.316458202   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:47.316460807   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:47.393669591   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175316
2025-04-25 12:27:47.393672727   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:47.492474940   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134550) =======
2025-04-25 12:27:47.492478237   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:47.492479539   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:47.492501290   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:47.492502182   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:47.492503103   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:47.492504326   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:47.492505267   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:47.592507165   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175318
2025-04-25 12:27:47.592511353   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:47.892697156   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175321
2025-04-25 12:27:47.892700642   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:47.980411673   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:48.086306495   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175349, managers: 7
2025-04-25 12:27:48.086309380   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175322
2025-04-25 12:27:48.086310833   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:27:48.086311925   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134555
2025-04-25 12:27:48.086334988   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:27:48.086336090   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134514
2025-04-25 12:27:48.086337112   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134514
2025-04-25 12:27:48.086338214   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:27:48.086339236   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:27:48.086340168   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175335
2025-04-25 12:27:48.086341140   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175335
2025-04-25 12:27:48.086342122   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175328
2025-04-25 12:27:48.086344035   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:27:48.119081868   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:27:48.119085194   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:48.192444293   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175330) =======
2025-04-25 12:27:48.192447299   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:48.192448181   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:48.192448812   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3037.97m cores, Cores=24
2025-04-25 12:27:48.192469811   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15706.15 MB, Total=128728.99 MB, Usage=12.20%
2025-04-25 12:27:48.192470503   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:48.192471274   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:48.192471905   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:48.392740318   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175326
2025-04-25 12:27:48.392743274   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:48.492647297   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:48.492945307   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001f22740})
2025-04-25 12:27:48.592048837   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:48.692396364   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:48.692481564   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:48.692535736   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:48.792217022   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:48.892218348   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175331
2025-04-25 12:27:48.892221534   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:48.892416080   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134560) =======
2025-04-25 12:27:48.892416972   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:48.892430106   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:48.892430908   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:48.892434745   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:48.892438181   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:48.892450284   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:48.892450985   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:48.980942202   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:48.993049054   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175332
2025-04-25 12:27:48.993052430   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:49.093186977   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175333
2025-04-25 12:27:49.093235899   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:49.193088646   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175340) =======
2025-04-25 12:27:49.193091502   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:49.193092373   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:49.193113343   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2319.33m cores, Cores=24
2025-04-25 12:27:49.193114084   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15686.62 MB, Total=128728.99 MB, Usage=12.19%
2025-04-25 12:27:49.193114755   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:49.193115567   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:49.193116218   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:49.219003967   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:49.219007374   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:49.293110872   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175335
2025-04-25 12:27:49.293114018   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:49.592752160   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175338
2025-04-25 12:27:49.592755296   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:49.893522279   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134570) =======
2025-04-25 12:27:49.893524774   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:49.893525626   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:49.893547206   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:49.893548428   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:49.893549410   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:49.893550853   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:49.893552095   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:49.900758788   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:49.900762285   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:49.980376049   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:27:50.092978575   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175343
2025-04-25 12:27:50.092982012   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:50.110576035   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.338036ms remaining, needed 30ms), skipping
2025-04-25 12:27:50.193004208   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175350) =======
2025-04-25 12:27:50.193006933   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:50.193007864   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:50.193008596   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2600.00m cores, Cores=24
2025-04-25 12:27:50.193027081   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15734.62 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:27:50.193027812   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:50.193028613   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:50.193029265   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:50.193040756   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:50.193423706   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0002b76a0})
2025-04-25 12:27:50.392159468   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:50.492866891   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:50.492936101   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:50.492946250   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:50.592805089   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:50.692772020   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175349
2025-04-25 12:27:50.692775416   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:50.793203565   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175350
2025-04-25 12:27:50.793206711   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:50.798660710   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:27:50.798672643   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:50.893419084   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175351
2025-04-25 12:27:50.893422230   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:50.980214243   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:51.092282115   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175353
2025-04-25 12:27:51.092304086   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:51.118473054   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:51.118477302   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:51.192203932   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175360) =======
2025-04-25 12:27:51.192206617   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:51.192207929   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:51.192208951   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3112.97m cores, Cores=24
2025-04-25 12:27:51.192209953   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15725.78 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:27:51.192232555   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:51.192233828   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:51.192234820   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:51.292373334   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134580) =======
2025-04-25 12:27:51.292376169   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:51.292377521   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:51.292400755   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:51.292402478   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:51.292404071   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:51.292406095   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:51.292413990   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:51.393458907   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175356
2025-04-25 12:27:51.393462253   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:51.892884175   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175361
2025-04-25 12:27:51.892886951   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:51.981024632   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:51.993455773   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:51.993803887   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc006ecd2a0})
2025-04-25 12:27:52.093371408   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:52.192405747   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175370) =======
2025-04-25 12:27:52.192408402   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:52.192409333   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:52.192410075   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2610.88m cores, Cores=24
2025-04-25 12:27:52.192432577   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15759.00 MB, Total=128728.99 MB, Usage=12.24%
2025-04-25 12:27:52.192433409   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:52.192434240   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:52.192434891   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:52.192485416   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:52.192553875   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:52.192562511   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:52.292300442   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:52.393126949   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175366
2025-04-25 12:27:52.393132198   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:52.406486735   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.094838ms remaining, needed 30ms), skipping
2025-04-25 12:27:52.493153051   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175367
2025-04-25 12:27:52.493155536   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:52.592278060   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175368
2025-04-25 12:27:52.592281367   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:52.692915131   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134590) =======
2025-04-25 12:27:52.692919489   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:52.692921012   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:52.692922405   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:52.692941250   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:52.692941932   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:52.692942753   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:52.692943354   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:52.793140097   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175370
2025-04-25 12:27:52.793143574   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:52.980592324   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:53.086114495   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175399, managers: 7
2025-04-25 12:27:53.086132068   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175372
2025-04-25 12:27:53.086138249   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:27:53.086144521   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134593
2025-04-25 12:27:53.086150352   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:27:53.086156093   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134552
2025-04-25 12:27:53.086161874   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134552
2025-04-25 12:27:53.086167614   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:27:53.086173435   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:27:53.086179006   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175385
2025-04-25 12:27:53.086184867   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175385
2025-04-25 12:27:53.086185738   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175378
2025-04-25 12:27:53.086191389   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:27:53.114474540   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175373
2025-04-25 12:27:53.114477816   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:53.121647550   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:53.121697293   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:53.192960553   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175380) =======
2025-04-25 12:27:53.192962677   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:53.192963549   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:53.192964250   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4826.09m cores, Cores=24
2025-04-25 12:27:53.192983857   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15771.11 MB, Total=128728.99 MB, Usage=12.25%
2025-04-25 12:27:53.192984588   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:53.192985410   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:53.192986081   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:53.593227618   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175378
2025-04-25 12:27:53.593230944   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:53.692713846   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:53.693138083   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc00465c120})
2025-04-25 12:27:53.792941627   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:53.892580512   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:53.892660622   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:53.892676262   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:53.980766434   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.38 seconds ago
2025-04-25 12:27:53.993659523   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:54.093482323   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175383
2025-04-25 12:27:54.093485789   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:54.093720651   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134600) =======
2025-04-25 12:27:54.093722694   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:54.093724448   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:54.093736691   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:54.093738184   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:54.093739787   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:54.093749134   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:54.093750857   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:54.214478843   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175390) =======
2025-04-25 12:27:54.214481809   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:54.214483171   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:54.214484193   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3298.97m cores, Cores=24
2025-04-25 12:27:54.214485215   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15794.43 MB, Total=128728.99 MB, Usage=12.27%
2025-04-25 12:27:54.214509441   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:54.214510633   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:54.214511595   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:54.214774108   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175384
2025-04-25 12:27:54.214888904   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:27:54.292609949   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175385
2025-04-25 12:27:54.292649133   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:54.493171711   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175387
2025-04-25 12:27:54.493174496   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:54.792458460   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175390
2025-04-25 12:27:54.792461716   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:54.919764287   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:54.919796087   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:54.980392161   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:27:55.093225660   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134610) =======
2025-04-25 12:27:55.093229407   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:55.093230910   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:55.093252902   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:55.093254104   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:55.093259815   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:55.093265355   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:55.093272559   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:55.191807719   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175400) =======
2025-04-25 12:27:55.191810123   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:55.191830842   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:55.191831534   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2033.90m cores, Cores=24
2025-04-25 12:27:55.191832225   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15741.63 MB, Total=128728.99 MB, Usage=12.23%
2025-04-25 12:27:55.191832896   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:55.191837174   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:55.191850880   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:55.293230806   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:55.392250817   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:55.392671848   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002becae0})
2025-04-25 12:27:55.492752483   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:55.592576124   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:55.592626879   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:55.592639994   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:55.692626852   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:55.792516558   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:55.893052949   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175401
2025-04-25 12:27:55.893055564   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:55.981099498   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:55.993479242   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175402
2025-04-25 12:27:55.993481797   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:56.018653541   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:56.018679550   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:56.192422322   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175410) =======
2025-04-25 12:27:56.192424876   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:56.192426169   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:56.192427171   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1721.52m cores, Cores=24
2025-04-25 12:27:56.192450104   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15743.86 MB, Total=128728.99 MB, Usage=12.23%
2025-04-25 12:27:56.192451146   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:56.192452328   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:56.192453300   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:56.192950344   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175404
2025-04-25 12:27:56.192963589   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:56.493110864   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175407
2025-04-25 12:27:56.493113819   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:56.493375150   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134620) =======
2025-04-25 12:27:56.493376142   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:56.493383236   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:56.493383967   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:56.493393435   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:56.493394116   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:56.493399486   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:56.493404716   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:56.500330621   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": net/http: timeout awaiting response headers
2025-04-25 12:27:56.500408608   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": net/http: timeout awaiting response headers
2025-04-25 12:27:56.822894011   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:27:56.822928826   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:56.980812936   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:27:56.994123239   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175412
2025-04-25 12:27:56.994125754   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:57.093262474   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:57.093646366   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001472a00})
2025-04-25 12:27:57.192364018   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175420) =======
2025-04-25 12:27:57.192367194   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:57.192368497   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:57.192369538   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2338.98m cores, Cores=24
2025-04-25 12:27:57.192393383   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15751.69 MB, Total=128728.99 MB, Usage=12.24%
2025-04-25 12:27:57.192394455   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:57.192395668   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:57.192396649   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:57.192416567   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:57.292413474   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:57.292486852   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:57.292503092   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:57.392603162   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:57.492797250   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175417
2025-04-25 12:27:57.492799845   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:57.499825628   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:57.499848350   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:57.592749512   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175418
2025-04-25 12:27:57.592752288   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:57.601093924   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:57.601123549   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:27:57.693453077   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175419
2025-04-25 12:27:57.693456634   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:57.893253176   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175421
2025-04-25 12:27:57.893256292   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:57.893530197   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134630) =======
2025-04-25 12:27:57.893531489   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:57.893532391   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:57.893542590   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:57.893543211   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:57.893547259   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:57.893551847   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:57.893552509   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:57.925320338   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:27:57.925349583   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:57.980618135   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:27:58.086297981   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175449, managers: 7
2025-04-25 12:27:58.086300315   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134590
2025-04-25 12:27:58.086301267   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134590
2025-04-25 12:27:58.086302048   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:27:58.086322637   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:27:58.086323348   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175435
2025-04-25 12:27:58.086324019   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175435
2025-04-25 12:27:58.086324671   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175428
2025-04-25 12:27:58.086325342   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:27:58.086325983   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175422
2025-04-25 12:27:58.086326644   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:27:58.086327296   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134631
2025-04-25 12:27:58.086327977   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:27:58.193434033   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175430) =======
2025-04-25 12:27:58.193437189   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:58.193438421   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:58.193439393   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2677.69m cores, Cores=24
2025-04-25 12:27:58.193440365   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15736.59 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:27:58.193463479   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:58.193464631   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:58.193465552   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:58.193740579   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175424
2025-04-25 12:27:58.193748965   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:58.692686570   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175429
2025-04-25 12:27:58.692689746   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:58.721143757   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:27:58.721147995   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:58.792492618   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:27:58.792819392   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc007952600})
2025-04-25 12:27:58.893070596   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:27:58.980446385   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.38 seconds ago
2025-04-25 12:27:58.993286424   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:27:58.993430314   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:27:58.993607147   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:27:59.092615806   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:27:59.192888961   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175440) =======
2025-04-25 12:27:59.192891696   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:59.192892999   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:27:59.192893990   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2008.37m cores, Cores=24
2025-04-25 12:27:59.192894942   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15722.25 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:27:59.192936200   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:27:59.192937522   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:27:59.192938454   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:27:59.193172694   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175434
2025-04-25 12:27:59.193185428   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:59.292564053   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175435
2025-04-25 12:27:59.292566848   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:59.292944067   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134640) =======
2025-04-25 12:27:59.292944989   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:27:59.292951060   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:27:59.292954917   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:27:59.292959325   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:27:59.292962792   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:27:59.292973452   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:27:59.292974805   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:27:59.392932306   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175436
2025-04-25 12:27:59.392935412   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:59.592749397   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175438
2025-04-25 12:27:59.592752553   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:59.818500173   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:27:59.818524488   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:27:59.892881051   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175441
2025-04-25 12:27:59.892884277   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:27:59.980345906   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:00.192866790   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175450) =======
2025-04-25 12:28:00.192869655   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:00.192870978   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:00.192871979   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1606.69m cores, Cores=24
2025-04-25 12:28:00.192895113   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15720.28 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:28:00.192896205   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:00.192897377   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:00.192898349   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:00.293130487   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134650) =======
2025-04-25 12:28:00.293132781   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:00.293133653   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:00.293152298   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:00.293152909   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:00.293153510   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:00.293157788   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:00.293158460   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:00.392850753   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175446
2025-04-25 12:28:00.392853378   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:00.493078192   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:00.493392132   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0051243e0})
2025-04-25 12:28:00.592325329   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:00.693548149   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:00.693614754   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:00.693623431   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:00.792267635   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:00.892433779   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175451
2025-04-25 12:28:00.892437336   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:00.922592453   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:00.922619164   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:00.980631953   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.58 seconds ago
2025-04-25 12:28:00.993706923   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175452
2025-04-25 12:28:00.993710360   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:01.092883708   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175453
2025-04-25 12:28:01.092886683   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:01.191938383   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175460) =======
2025-04-25 12:28:01.191941609   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:01.191942932   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:01.191943954   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2309.62m cores, Cores=24
2025-04-25 12:28:01.191985281   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15711.04 MB, Total=128728.99 MB, Usage=12.20%
2025-04-25 12:28:01.191987395   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:01.191989730   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:01.191991573   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:01.292052900   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175455
2025-04-25 12:28:01.292074100   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:01.299713205   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:01.299718826   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:01.593431305   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175458
2025-04-25 12:28:01.593435153   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:01.692294451   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134660) =======
2025-04-25 12:28:01.692297266   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:01.692298689   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:01.692300051   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:01.692300983   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:01.692302005   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:01.692325609   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:01.692326781   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:01.719265315   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:01.719289199   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:01.981100139   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:02.092264100   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175463
2025-04-25 12:28:02.092267316   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:02.192604421   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175470) =======
2025-04-25 12:28:02.192608689   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:02.192610242   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:02.192634057   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4169.49m cores, Cores=24
2025-04-25 12:28:02.192635530   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15709.16 MB, Total=128728.99 MB, Usage=12.20%
2025-04-25 12:28:02.192636662   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:02.192637994   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:02.192639236   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:02.192655277   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:02.193050309   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002e028e0})
2025-04-25 12:28:02.293503141   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:02.392669586   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:02.392737354   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:02.392744808   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:02.493281348   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:02.592124505   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175468
2025-04-25 12:28:02.592128142   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:02.600141410   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:02.600145829   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:02.692337527   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175469
2025-04-25 12:28:02.692340502   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:02.793797812   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175470
2025-04-25 12:28:02.793804034   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:02.823955215   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:02.823979992   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:02.980593602   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:02.993194842   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175472
2025-04-25 12:28:02.993197497   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:03.086243110   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175499, managers: 7
2025-04-25 12:28:03.086245835   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134669
2025-04-25 12:28:03.086246797   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:03.086247548   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134628
2025-04-25 12:28:03.086248199   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134628
2025-04-25 12:28:03.086272144   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:03.086272886   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:03.086273497   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175485
2025-04-25 12:28:03.086274108   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175485
2025-04-25 12:28:03.086274769   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175478
2025-04-25 12:28:03.086275421   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:03.086276052   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175472
2025-04-25 12:28:03.086276693   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:28:03.093136985   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134670) =======
2025-04-25 12:28:03.093140211   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:03.093141674   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:03.093166651   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:03.093167853   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:03.093168995   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:03.093170588   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:03.093171670   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:03.193168405   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175480) =======
2025-04-25 12:28:03.193171000   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:03.193172312   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:03.193173334   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3585.06m cores, Cores=24
2025-04-25 12:28:03.193196097   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15727.97 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:03.193197249   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:03.193198472   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:03.193199423   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:03.293624804   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175475
2025-04-25 12:28:03.293627820   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:03.613879281   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:03.613882847   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:03.793027268   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175480
2025-04-25 12:28:03.793030644   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:03.893297667   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:03.893706135   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0012ea520})
2025-04-25 12:28:03.980920279   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:28:03.992850348   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:04.092806867   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:04.092861380   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:04.092875145   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:04.192000102   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175490) =======
2025-04-25 12:28:04.192003268   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:04.192004570   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:04.192005612   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2521.01m cores, Cores=24
2025-04-25 12:28:04.192028245   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15729.80 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:04.192030199   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:04.192032413   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:04.192034286   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:04.192101042   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:04.292527865   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175485
2025-04-25 12:28:04.292531262   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:04.392387302   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175486
2025-04-25 12:28:04.392390859   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:04.493430694   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175487
2025-04-25 12:28:04.493433148   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:04.493861032   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134680) =======
2025-04-25 12:28:04.493861954   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:04.493868136   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:04.493872033   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:04.493885268   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:04.493885939   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:04.493890688   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:04.493909854   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:04.693111666   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175489
2025-04-25 12:28:04.693115243   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:04.719964799   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:04.719989055   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:04.981024495   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:04.993619624   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175492
2025-04-25 12:28:04.993623852   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:05.191782574   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175500) =======
2025-04-25 12:28:05.191784758   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:05.191785640   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:05.191786331   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1531.91m cores, Cores=24
2025-04-25 12:28:05.191806920   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15752.11 MB, Total=128728.99 MB, Usage=12.24%
2025-04-25 12:28:05.191807641   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:05.191808442   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:05.191809094   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:05.492756998   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175497
2025-04-25 12:28:05.492760755   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:05.493096586   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134690) =======
2025-04-25 12:28:05.493098670   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:05.493100002   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:05.493109460   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:05.493110412   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:05.493116103   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:05.493117355   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:05.493133535   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:05.524557127   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:05.524560874   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:05.592895359   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:05.593263290   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc007f34d80})
2025-04-25 12:28:05.693608540   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:05.792564489   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:05.792658395   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:05.792659748   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:05.892171260   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:05.980590348   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.58 seconds ago
2025-04-25 12:28:05.993479840   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175502
2025-04-25 12:28:05.993483277   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:06.093284504   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175503
2025-04-25 12:28:06.093287640   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:06.192458252   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175510) =======
2025-04-25 12:28:06.192462280   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:06.192463772   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:06.192486675   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2200.00m cores, Cores=24
2025-04-25 12:28:06.192487968   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15757.12 MB, Total=128728.99 MB, Usage=12.24%
2025-04-25 12:28:06.192489090   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:06.192490673   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:06.192491815   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:06.192660432   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:06.393097204   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175506
2025-04-25 12:28:06.393100400   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:06.615516617   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:06.615520033   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:06.693066389   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175509
2025-04-25 12:28:06.693071368   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:06.700853081   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:06.700858672   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:06.893413432   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134700) =======
2025-04-25 12:28:06.893416207   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:06.893417540   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:06.893418702   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:06.893443829   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:06.893444861   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:06.893446083   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:06.893447005   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:06.980826900   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:07.192272951   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175520) =======
2025-04-25 12:28:07.192276197   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:07.192277830   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:07.192279162   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2912.13m cores, Cores=24
2025-04-25 12:28:07.192301093   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15759.20 MB, Total=128728.99 MB, Usage=12.24%
2025-04-25 12:28:07.192302426   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:07.192303999   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:07.192305231   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:07.192565881   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175514
2025-04-25 12:28:07.192575770   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:07.293404217   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:07.293876154   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0050b9620})
2025-04-25 12:28:07.392288090   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:07.491985913   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:07.492108263   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:07.492109825   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:07.593496973   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:07.692053621   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175519
2025-04-25 12:28:07.692056827   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:07.720806333   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:07.720809469   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:07.793459083   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175520
2025-04-25 12:28:07.793462269   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:07.892170832   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175521
2025-04-25 12:28:07.892174108   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:07.980338618   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:08.086330348   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175549, managers: 7
2025-04-25 12:28:08.086402223   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134666
2025-04-25 12:28:08.086462466   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134666
2025-04-25 12:28:08.086514774   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:08.086549870   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:08.086598902   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175535
2025-04-25 12:28:08.086600225   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175535
2025-04-25 12:28:08.086601297   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175528
2025-04-25 12:28:08.086602279   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:08.086603231   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175522
2025-04-25 12:28:08.086604212   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:28:08.086605164   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134707
2025-04-25 12:28:08.086606136   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:08.093612572   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175523
2025-04-25 12:28:08.093646196   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:08.193313881   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175530) =======
2025-04-25 12:28:08.193324111   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:08.193325052   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:08.193329561   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3966.94m cores, Cores=24
2025-04-25 12:28:08.193330262   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15812.19 MB, Total=128728.99 MB, Usage=12.28%
2025-04-25 12:28:08.193334620   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:08.193335402   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:08.193339720   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:08.201565157   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:08.201572912   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:08.292817910   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134710) =======
2025-04-25 12:28:08.292820675   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:08.292821997   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:08.292823170   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:08.292845952   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:08.292847866   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:08.292850130   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:08.292851663   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:08.392718644   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175526
2025-04-25 12:28:08.392721188   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:08.514458808   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:08.514483084   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:08.892517126   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175531
2025-04-25 12:28:08.892520092   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:08.980481239   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:08.992966120   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:08.993364369   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002c068e0})
2025-04-25 12:28:09.093072922   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:09.193106025   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175540) =======
2025-04-25 12:28:09.193109211   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:09.193110563   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:09.193111595   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2108.79m cores, Cores=24
2025-04-25 12:28:09.193133757   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15738.93 MB, Total=128728.99 MB, Usage=12.23%
2025-04-25 12:28:09.193134839   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:09.193136061   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:09.193137043   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:09.193187217   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:09.193258361   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:09.193265474   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:09.294321328   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:09.393208467   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175536
2025-04-25 12:28:09.393217384   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:09.401090499   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:09.401129633   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:09.493252912   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175537
2025-04-25 12:28:09.493255887   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:09.592725220   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175538
2025-04-25 12:28:09.592728356   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:09.620340976   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:09.620345234   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:09.693869551   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134720) =======
2025-04-25 12:28:09.693873989   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:09.693875322   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:09.693876634   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:09.693877586   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:09.693901471   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:09.693917701   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:09.693918903   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:09.792482605   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:09.980554700   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:10.092630252   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175543
2025-04-25 12:28:10.092632957   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:10.192308477   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175550) =======
2025-04-25 12:28:10.192311723   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:10.192313026   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:10.192314068   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2218.49m cores, Cores=24
2025-04-25 12:28:10.192355926   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15731.74 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:10.192357008   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:10.192358191   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:10.192359162   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:10.419761146   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:10.419764252   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:10.593293909   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175548
2025-04-25 12:28:10.593298006   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:10.693522168   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:10.693867296   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001363f20})
2025-04-25 12:28:10.793309017   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134730) =======
2025-04-25 12:28:10.793311261   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:10.793312153   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:10.793332992   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:10.793333653   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:10.793334295   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:10.793335136   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:10.793335777   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:10.892487714   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:10.980736109   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:28:10.994437757   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:10.994550239   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:10.994551651   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:11.092341548   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:11.193330717   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175560) =======
2025-04-25 12:28:11.193356315   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:11.193368788   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:11.193382424   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2082.64m cores, Cores=24
2025-04-25 12:28:11.193383466   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15730.26 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:11.193384217   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:11.193385039   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:11.193385670   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:11.193533748   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175554
2025-04-25 12:28:11.193542895   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:11.292170466   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175555
2025-04-25 12:28:11.292173442   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:11.299646575   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:11.299697390   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:11.392632183   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175556
2025-04-25 12:28:11.392636121   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:11.400865375   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": net/http: timeout awaiting response headers
2025-04-25 12:28:11.400891464   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": net/http: timeout awaiting response headers
2025-04-25 12:28:11.521460607   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:11.521465947   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:11.592984916   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175558
2025-04-25 12:28:11.592987441   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:11.892918500   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175561
2025-04-25 12:28:11.892922137   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:11.980800859   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:12.092585834   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134740) =======
2025-04-25 12:28:12.092588529   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:12.092589421   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:12.092609228   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:12.092609839   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:12.092610451   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:12.092611272   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:12.092611903   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:12.192504732   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175570) =======
2025-04-25 12:28:12.192507086   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:12.192507978   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:12.192527604   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1815.13m cores, Cores=24
2025-04-25 12:28:12.192528406   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15728.06 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:12.192529077   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:12.192529889   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:12.192530510   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:12.322106358   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:12.322109675   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:12.393397829   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175566
2025-04-25 12:28:12.393401025   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:12.492656495   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:12.493067968   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0045d2500})
2025-04-25 12:28:12.592675891   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:12.693292520   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:12.693371378   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:12.693379142   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:12.792481465   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:12.893172343   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175571
2025-04-25 12:28:12.893174868   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:12.980942640   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:12.995095436   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175572
2025-04-25 12:28:12.995301614   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:28:13.085815056   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175599, managers: 7
2025-04-25 12:28:13.085819294   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175585
2025-04-25 12:28:13.085820757   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175585
2025-04-25 12:28:13.085822039   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175578
2025-04-25 12:28:13.085823291   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:13.085847567   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175572
2025-04-25 12:28:13.085848970   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:28:13.085850162   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134745
2025-04-25 12:28:13.085851354   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:13.085852516   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134704
2025-04-25 12:28:13.085853819   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134704
2025-04-25 12:28:13.085868837   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:13.085870210   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:13.093636775   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175573
2025-04-25 12:28:13.093805842   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:28:13.192962156   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175580) =======
2025-04-25 12:28:13.192964922   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:13.192966294   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:13.192990219   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2510.46m cores, Cores=24
2025-04-25 12:28:13.192991321   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15713.08 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:28:13.192992303   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:13.192993495   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:13.192994457   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:13.293284561   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175575
2025-04-25 12:28:13.293288048   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:13.431379008   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 29.639742ms remaining, needed 30ms), skipping
2025-04-25 12:28:13.493180054   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134750) =======
2025-04-25 12:28:13.493182879   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:13.493184222   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:13.493252350   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:13.493254283   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:13.493255105   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:13.493264643   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:13.493265344   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:13.593409634   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175578
2025-04-25 12:28:13.593413031   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:13.980985735   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:14.093045958   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175583
2025-04-25 12:28:14.093049204   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:14.193499129   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175590) =======
2025-04-25 12:28:14.193502625   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:14.193504068   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:14.193528664   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1991.70m cores, Cores=24
2025-04-25 12:28:14.193529946   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15697.51 MB, Total=128728.99 MB, Usage=12.19%
2025-04-25 12:28:14.193531179   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:14.193532521   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:14.193533643   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:14.193557238   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:14.193899992   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0067546e0})
2025-04-25 12:28:14.293297379   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:14.392321845   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:14.392347012   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:14.392554361   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:14.493207739   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:14.593062054   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175588
2025-04-25 12:28:14.593064759   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:14.608360513   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 25.357724ms remaining, needed 30ms), skipping
2025-04-25 12:28:14.693682569   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175589
2025-04-25 12:28:14.693688250   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:14.792667621   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175590
2025-04-25 12:28:14.792672210   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:14.892483294   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134760) =======
2025-04-25 12:28:14.892485448   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:14.892486340   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:14.892512689   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:14.892513681   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:14.892514623   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:14.892515915   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:14.892516877   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:14.981028608   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:14.994035229   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175592
2025-04-25 12:28:14.994038365   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:15.193196291   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175600) =======
2025-04-25 12:28:15.193199347   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:15.193221127   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:15.193221829   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1915.97m cores, Cores=24
2025-04-25 12:28:15.193222490   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15678.08 MB, Total=128728.99 MB, Usage=12.18%
2025-04-25 12:28:15.193223161   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:15.193223923   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:15.193228060   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:15.310728847   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175595
2025-04-25 12:28:15.310732954   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:15.318717790   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:28:15.318743738   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:28:15.793010395   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175600
2025-04-25 12:28:15.793220520   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:28:15.891948839   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:15.892291824   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0048c0300})
2025-04-25 12:28:15.980273799   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:28:15.993999322   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:16.092754421   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:16.092851454   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:16.092870079   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:16.192761163   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175610) =======
2025-04-25 12:28:16.192763868   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:16.192765210   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:16.192766262   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2479.34m cores, Cores=24
2025-04-25 12:28:16.192767264   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15685.01 MB, Total=128728.99 MB, Usage=12.18%
2025-04-25 12:28:16.192768266   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:16.192790969   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:16.192792001   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:16.192914821   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:16.293826213   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175605
2025-04-25 12:28:16.293829449   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:16.294286959   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134770) =======
2025-04-25 12:28:16.294288351   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:16.294299623   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:16.294300424   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:16.294304512   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:16.294308339   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:16.294313479   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:16.294317075   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:16.323667260   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:16.323670335   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:16.392262333   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175606
2025-04-25 12:28:16.392265439   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:16.493817404   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175607
2025-04-25 12:28:16.493821181   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:16.501360899   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:16.501365488   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:16.692979296   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175609
2025-04-25 12:28:16.692985929   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:16.981122502   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:16.995277341   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175612
2025-04-25 12:28:16.995280527   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:17.120236372   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:17.120259155   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:17.192860346   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175620) =======
2025-04-25 12:28:17.192863001   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:17.192864274   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:17.192865256   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3012.55m cores, Cores=24
2025-04-25 12:28:17.192890523   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15696.46 MB, Total=128728.99 MB, Usage=12.19%
2025-04-25 12:28:17.192891565   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:17.192892717   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:17.192893669   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:17.200136910   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:17.200140888   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:17.294109963   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134780) =======
2025-04-25 12:28:17.294112749   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:17.294114111   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:17.294115253   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:17.294138056   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:17.294139148   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:17.294140381   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:17.294141332   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:17.492547915   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175617
2025-04-25 12:28:17.492580516   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:17.593282134   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:17.593660214   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002698e60})
2025-04-25 12:28:17.692377572   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:17.792224323   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:17.792281340   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:17.792290157   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:17.893303459   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:17.980376496   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:28:17.994333053   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175622
2025-04-25 12:28:17.994336209   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:18.085657047   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175649, managers: 7
2025-04-25 12:28:18.085661566   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134742
2025-04-25 12:28:18.085682495   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:18.085683347   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:18.085683988   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175635
2025-04-25 12:28:18.085684649   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175635
2025-04-25 12:28:18.085685310   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175628
2025-04-25 12:28:18.085736767   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:18.085738160   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175622
2025-04-25 12:28:18.085739132   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:28:18.085739853   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134783
2025-04-25 12:28:18.085964766   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:18.085967240   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134742
2025-04-25 12:28:18.092555681   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175623
2025-04-25 12:28:18.092741390   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:28:18.192140810   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175630) =======
2025-04-25 12:28:18.192143525   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:18.192144848   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:18.192168893   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2108.79m cores, Cores=24
2025-04-25 12:28:18.192169985   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15684.73 MB, Total=128728.99 MB, Usage=12.18%
2025-04-25 12:28:18.192170977   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:18.192172139   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:18.192173111   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:18.192445663   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175624
2025-04-25 12:28:18.192501568   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:18.217888474   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:18.217933920   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:18.394563169   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175626
2025-04-25 12:28:18.394565744   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:18.692770606   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175629
2025-04-25 12:28:18.692774413   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:18.693185696   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134790) =======
2025-04-25 12:28:18.693186938   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:18.693187970   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:18.693195875   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:18.693196747   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:18.693202147   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:18.693208799   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:18.693209791   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:18.980629666   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:19.025538357   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:19.025562593   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:19.192982435   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175640) =======
2025-04-25 12:28:19.192984429   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:19.192985321   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:19.193005789   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1004.18m cores, Cores=24
2025-04-25 12:28:19.193006571   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15690.02 MB, Total=128728.99 MB, Usage=12.19%
2025-04-25 12:28:19.193007252   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:19.193008053   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:19.193008695   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:19.193258033   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175634
2025-04-25 12:28:19.193329397   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:19.293251549   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:19.293731742   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001410320})
2025-04-25 12:28:19.392621313   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:19.493469856   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:19.493528045   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:19.493536612   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:19.592622783   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:19.692531429   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175639
2025-04-25 12:28:19.692534585   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:19.792934084   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175640
2025-04-25 12:28:19.792937340   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:19.892384199   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175641
2025-04-25 12:28:19.892387866   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:19.980632604   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:20.121630254   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175643
2025-04-25 12:28:20.121632949   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:20.122121737   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134800) =======
2025-04-25 12:28:20.122123120   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:20.122135643   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:20.122137277   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:20.122144420   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:20.122145752   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:20.122147486   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:20.122165029   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:20.194169889   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175650) =======
2025-04-25 12:28:20.194172564   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:20.194173466   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:20.194174187   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4827.59m cores, Cores=24
2025-04-25 12:28:20.194174858   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15678.15 MB, Total=128728.99 MB, Usage=12.18%
2025-04-25 12:28:20.194175519   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:20.194176321   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:20.194176962   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:20.201118567   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:20.201121723   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:20.392679033   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175646
2025-04-25 12:28:20.392682239   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:20.399958763   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:20.399963031   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:20.821273239   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:20.821275784   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:20.892557986   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175651
2025-04-25 12:28:20.892560581   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:20.980553455   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:20.993028328   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:20.994661367   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc00049a300})
2025-04-25 12:28:21.093298654   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:21.193035136   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175660) =======
2025-04-25 12:28:21.193037200   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:21.193038052   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:21.193057368   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3589.74m cores, Cores=24
2025-04-25 12:28:21.193058099   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15729.46 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:21.193058751   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:21.193059532   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:21.193063570   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:21.193163618   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:21.193243087   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:21.193253326   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:21.292897896   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:21.392761588   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175656
2025-04-25 12:28:21.392767319   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:21.492850683   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175657
2025-04-25 12:28:21.492855422   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:21.493517426   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134810) =======
2025-04-25 12:28:21.493518849   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:21.493519911   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:21.493529709   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:21.493530631   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:21.493536272   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:21.493537504   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:21.493544437   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:21.501080940   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:21.501085177   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:21.593178958   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175658
2025-04-25 12:28:21.593181713   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:21.793196577   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175660
2025-04-25 12:28:21.793199693   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:21.800754620   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:21.800785277   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:21.914263483   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:21.914286136   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:21.981115540   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:28:22.094211167   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175663
2025-04-25 12:28:22.094226947   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:22.102219947   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:28:22.102277645   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:28:22.192435938   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175670) =======
2025-04-25 12:28:22.192438694   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:22.192440016   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:22.192441078   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3517.24m cores, Cores=24
2025-04-25 12:28:22.192442060   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15732.94 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:22.192465564   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:22.192466796   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:22.192467778   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:22.492954065   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134820) =======
2025-04-25 12:28:22.492956269   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:22.492957161   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:22.492976988   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:22.492977629   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:22.492978261   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:22.492979112   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:22.492979763   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:22.592306957   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175668
2025-04-25 12:28:22.592310844   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:22.693129690   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:22.693449021   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0065ac0c0})
2025-04-25 12:28:22.813387166   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:22.893164311   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:22.893252006   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:22.893267605   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:22.980441170   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.38 seconds ago
2025-04-25 12:28:22.993007845   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:23.086220698   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175699, managers: 7
2025-04-25 12:28:23.086225527   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175672
2025-04-25 12:28:23.086226869   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:28:23.086227921   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134821
2025-04-25 12:28:23.086254712   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:23.086255934   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134780
2025-04-25 12:28:23.086256966   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134780
2025-04-25 12:28:23.086258048   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:23.086259050   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:23.086260022   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175685
2025-04-25 12:28:23.086261024   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175685
2025-04-25 12:28:23.086262106   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175678
2025-04-25 12:28:23.086263188   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:23.094609322   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175673
2025-04-25 12:28:23.094633006   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:23.193008601   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175680) =======
2025-04-25 12:28:23.193010785   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:23.193011657   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:23.193031734   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2553.19m cores, Cores=24
2025-04-25 12:28:23.193032456   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15763.57 MB, Total=128728.99 MB, Usage=12.25%
2025-04-25 12:28:23.193033117   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:23.193033908   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:23.193034540   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:23.193310979   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175674
2025-04-25 12:28:23.193324625   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:23.292444428   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175675
2025-04-25 12:28:23.292465788   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:23.492855465   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175677
2025-04-25 12:28:23.492857990   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:23.720570923   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:23.720596801   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:23.793900462   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175680
2025-04-25 12:28:23.793916141   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:23.893096248   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134830) =======
2025-04-25 12:28:23.893098512   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:23.893099374   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:23.893118370   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:23.893118981   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:23.893123519   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:23.893127306   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:23.893127968   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:23.980389167   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:24.192993477   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175690) =======
2025-04-25 12:28:24.192997234   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:24.192998517   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:24.192999559   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1506.28m cores, Cores=24
2025-04-25 12:28:24.193022692   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15732.30 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:24.193023854   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:24.193025227   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:24.193026299   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:24.293941816   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175685
2025-04-25 12:28:24.293947447   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:24.392939650   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:24.393284689   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc005320820})
2025-04-25 12:28:24.493328268   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:24.592632298   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:24.592690717   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:24.592704493   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:24.693366324   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:24.793434289   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175690
2025-04-25 12:28:24.793437175   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:24.817881620   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:24.817927726   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:24.894000400   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175691
2025-04-25 12:28:24.894027561   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:24.901625158   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:24.901631059   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:24.981061623   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.68 seconds ago
2025-04-25 12:28:24.994000868   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175692
2025-04-25 12:28:24.994014534   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:25.193490063   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175700) =======
2025-04-25 12:28:25.193492718   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:25.193494010   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:25.193495022   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1983.47m cores, Cores=24
2025-04-25 12:28:25.193525109   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15737.96 MB, Total=128728.99 MB, Usage=12.23%
2025-04-25 12:28:25.193527253   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:25.193529617   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:25.193530679   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:25.193836183   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175694
2025-04-25 12:28:25.193846072   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:25.293014266   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134840) =======
2025-04-25 12:28:25.293018694   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:25.293020177   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:25.293049852   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:25.293050975   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:25.293052127   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:25.293053700   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:25.293054892   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:25.300053734   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": net/http: timeout awaiting response headers
2025-04-25 12:28:25.300075936   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": net/http: timeout awaiting response headers
2025-04-25 12:28:25.492937065   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175697
2025-04-25 12:28:25.492958786   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:25.614767786   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:25.614770300   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:25.980676150   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:25.993403106   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175702
2025-04-25 12:28:25.993408065   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:26.092621324   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:26.092979597   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002bceda0})
2025-04-25 12:28:26.192961009   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175710) =======
2025-04-25 12:28:26.192963664   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:26.192965027   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:26.192966189   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2126.58m cores, Cores=24
2025-04-25 12:28:26.192967271   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15717.22 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:28:26.193001756   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:26.193004321   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:26.193006124   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:26.193014349   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:26.292854486   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:26.292914589   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:26.292924728   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:26.393718867   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:26.493436393   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175707
2025-04-25 12:28:26.493549336   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:28:26.592635555   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175708
2025-04-25 12:28:26.592639002   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:26.693172391   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175709
2025-04-25 12:28:26.693175747   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:26.693895390   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134850) =======
2025-04-25 12:28:26.693897433   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:26.693898576   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:26.693927610   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:26.693929544   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:26.693931467   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:26.693943470   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:26.693945554   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:26.715867679   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:26.715911281   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:26.893262765   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175711
2025-04-25 12:28:26.893290016   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:26.980805382   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:27.193540016   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175720) =======
2025-04-25 12:28:27.193542200   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:27.193543072   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:27.193543783   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1807.53m cores, Cores=24
2025-04-25 12:28:27.193561266   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15718.25 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:28:27.193561977   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:27.193562779   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:27.193563430   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:27.193827286   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175714
2025-04-25 12:28:27.193828939   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:27.515470548   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:27.515867063   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:27.692531384   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175719
2025-04-25 12:28:27.692549949   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:27.693157611   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134860) =======
2025-04-25 12:28:27.693158944   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:27.693159825   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:27.693206032   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:27.693210200   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:27.693212254   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:27.693214618   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:27.693216422   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:27.792466139   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:27.792920693   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0013b1320})
2025-04-25 12:28:27.892998946   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:27.980403434   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:28:27.992972744   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:27.993077010   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:27.993130080   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:28.086374822   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175749, managers: 7
2025-04-25 12:28:28.086390101   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134860
2025-04-25 12:28:28.086422562   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:28.086423934   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134819
2025-04-25 12:28:28.086425066   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134819
2025-04-25 12:28:28.086426179   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:28.086427241   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:28.086428182   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175735
2025-04-25 12:28:28.086429164   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175735
2025-04-25 12:28:28.086430136   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175728
2025-04-25 12:28:28.086431068   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:28.086432010   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175722
2025-04-25 12:28:28.086432991   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: creating, desired state: active
2025-04-25 12:28:28.094496354   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:28.192454463   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175730) =======
2025-04-25 12:28:28.192457088   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:28.192458361   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:28.192459363   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2974.36m cores, Cores=24
2025-04-25 12:28:28.192483287   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15718.04 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:28:28.192484360   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:28.192485542   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:28.192486494   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:28.192760338   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175724
2025-04-25 12:28:28.192787139   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:28.292971201   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175725
2025-04-25 12:28:28.292974237   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:28.300304972   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:28.300355257   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:28.394828006   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175726
2025-04-25 12:28:28.394831142   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:28.592271476   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175728
2025-04-25 12:28:28.592274632   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:28.618595493   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:28.618627253   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:28.893611754   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175731
2025-04-25 12:28:28.893634927   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:28.980329391   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:29.094086128   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134870) =======
2025-04-25 12:28:29.094096347   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:29.094101246   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:29.094105795   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:29.094106406   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:29.094107047   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:29.094111917   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:29.094124931   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:29.101469032   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:29.101475935   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:29.193647711   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175740) =======
2025-04-25 12:28:29.193649845   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:29.193650756   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:29.193670674   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3400.00m cores, Cores=24
2025-04-25 12:28:29.193671415   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15727.01 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:29.193672097   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:29.193672898   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:29.193673529   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:29.202855223   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: error reading config file benthos.yaml in service directory /run/service/benthos-dataflow-core-1: context deadline exceeded
2025-04-25 12:28:29.394742741   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175736
2025-04-25 12:28:29.394745386   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:29.442235858   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 29.587814ms remaining, needed 30ms), skipping
2025-04-25 12:28:29.492606674   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:29.493081336   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0082a2580})
2025-04-25 12:28:29.593131276   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:29.692875952   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:29.693092820   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:29.693118438   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:29.793231677   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:29.893349024   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175741
2025-04-25 12:28:29.893381264   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:29.980449921   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:29.994424471   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175742
2025-04-25 12:28:29.994455640   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:30.093856420   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175743
2025-04-25 12:28:30.093859065   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:30.193184163   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175750) =======
2025-04-25 12:28:30.193187499   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:30.193188842   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:30.193189853   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2924.37m cores, Cores=24
2025-04-25 12:28:30.193214700   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15711.59 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:28:30.193215762   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:30.193216974   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:30.193217936   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:30.293086957   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:30.493288528   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134880) =======
2025-04-25 12:28:30.493290772   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:30.493291674   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:30.493311391   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:30.493312042   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:30.493312673   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:30.493313505   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:30.493314146   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:30.520933156   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:30.520954957   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:30.593210607   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175748
2025-04-25 12:28:30.593216228   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:30.980776667   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:31.093460688   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175753
2025-04-25 12:28:31.093466850   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:31.101346346   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:31.101369500   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:31.192790866   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175760) =======
2025-04-25 12:28:31.192794572   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:31.192795464   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:31.192817345   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2553.19m cores, Cores=24
2025-04-25 12:28:31.192818067   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15709.19 MB, Total=128728.99 MB, Usage=12.20%
2025-04-25 12:28:31.192818708   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:31.192819479   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:31.192820100   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:31.192836872   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:31.193253224   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc005752100})
2025-04-25 12:28:31.293565777   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:31.392493549   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:31.392560074   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:31.392571735   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:31.493776676   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:31.613831187   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175758
2025-04-25 12:28:31.613834734   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:31.692960975   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175759
2025-04-25 12:28:31.692963790   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:31.794929621   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175760
2025-04-25 12:28:31.794932606   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:31.893702140   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134890) =======
2025-04-25 12:28:31.893704384   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:31.893705276   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:31.893706047   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:31.893725824   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:31.893727007   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:31.893728429   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:31.893729451   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:31.980461364   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:31.993541514   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175762
2025-04-25 12:28:31.993543979   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:32.191749999   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175770) =======
2025-04-25 12:28:32.191753967   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:32.191756231   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:32.191758064   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1600.00m cores, Cores=24
2025-04-25 12:28:32.191786588   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15733.05 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:32.191788311   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:32.191790285   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:32.191791908   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:32.295153611   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175765
2025-04-25 12:28:32.295156827   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:32.793332337   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175770
2025-04-25 12:28:32.793334702   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:32.801025073   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:32.801104332   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:32.893480542   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:32.893856208   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc003a4d140})
2025-04-25 12:28:32.980670075   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:28:32.993531293   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:33.085901301   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175799, managers: 7
2025-04-25 12:28:33.085915217   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134858
2025-04-25 12:28:33.085916820   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:33.085944853   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:33.085946906   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175785
2025-04-25 12:28:33.085948980   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175785
2025-04-25 12:28:33.085950774   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175778
2025-04-25 12:28:33.085960181   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:33.085962045   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175772
2025-04-25 12:28:33.085964149   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: to_be_created, desired state: active
2025-04-25 12:28:33.085974037   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134899
2025-04-25 12:28:33.085976131   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:33.085978045   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134858
2025-04-25 12:28:33.093079269   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:33.093177954   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:33.093196539   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:33.193159606   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175780) =======
2025-04-25 12:28:33.193161960   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:33.193162852   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:33.193182649   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2510.46m cores, Cores=24
2025-04-25 12:28:33.193183390   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15723.60 MB, Total=128728.99 MB, Usage=12.21%
2025-04-25 12:28:33.193184062   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:33.193184853   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:33.193189041   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:33.193371053   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:33.293296789   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175775
2025-04-25 12:28:33.293301328   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:33.294102423   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134900) =======
2025-04-25 12:28:33.294162716   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:33.294164750   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:33.294180239   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:33.294181231   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:33.294190449   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:33.294193063   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:33.294201239   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:33.393296554   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175776
2025-04-25 12:28:33.393301824   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:33.406240298   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.092083ms remaining, needed 30ms), skipping
2025-04-25 12:28:33.492940150   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175777
2025-04-25 12:28:33.492943567   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:33.692617656   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175779
2025-04-25 12:28:33.692621513   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:33.980316527   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:33.993164270   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175782
2025-04-25 12:28:33.993167295   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:34.113207009   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:34.114924836   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:34.193699947   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175790) =======
2025-04-25 12:28:34.193703865   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:34.193705398   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:34.193729102   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3949.37m cores, Cores=24
2025-04-25 12:28:34.193730485   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15728.85 MB, Total=128728.99 MB, Usage=12.22%
2025-04-25 12:28:34.193731637   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:34.193733049   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:34.193734111   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:34.293332883   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134910) =======
2025-04-25 12:28:34.293336700   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:34.293338323   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:34.293339906   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:34.293341048   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:34.293342350   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:34.293368329   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:34.293369702   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:34.492645542   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175787
2025-04-25 12:28:34.492649429   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:34.593031433   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:34.593528226   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002a6d080})
2025-04-25 12:28:34.693178074   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:34.793585936   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:34.793664674   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:34.793677097   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:34.892984661   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:34.981113519   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:28:34.994899295   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175792
2025-04-25 12:28:34.994911087   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:35.093405523   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175793
2025-04-25 12:28:35.093409069   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:35.192695684   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175800) =======
2025-04-25 12:28:35.192697808   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:35.192698700   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:35.192718487   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3000.00m cores, Cores=24
2025-04-25 12:28:35.192719278   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15782.23 MB, Total=128728.99 MB, Usage=12.26%
2025-04-25 12:28:35.192719950   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:35.192720741   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:35.192721382   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:35.192916619   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:35.244590923   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 25.774147ms remaining, needed 30ms), skipping
2025-04-25 12:28:35.393770504   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175796
2025-04-25 12:28:35.393773119   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:35.401503165   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:35.401537500   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:35.694008948   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175799
2025-04-25 12:28:35.694012545   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:35.694865678   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134920) =======
2025-04-25 12:28:35.694867021   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:35.694875096   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:35.694875887   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:35.694879765   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:35.694883361   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:35.694895594   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:35.694896316   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:35.980267111   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:36.040801454   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 26.186151ms remaining, needed 30ms), skipping
2025-04-25 12:28:36.192664147   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175810) =======
2025-04-25 12:28:36.192666903   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:36.192668225   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:36.192669237   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3605.15m cores, Cores=24
2025-04-25 12:28:36.192694324   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15759.99 MB, Total=128728.99 MB, Usage=12.24%
2025-04-25 12:28:36.192695406   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:36.192696578   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:36.192697550   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:36.192868722   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:36.295035699   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:36.295414791   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0018bb4e0})
2025-04-25 12:28:36.393062606   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:36.494091665   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:36.494226178   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:36.494261704   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:36.592760068   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:36.693472201   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175809
2025-04-25 12:28:36.693498410   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:36.793086141   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175810
2025-04-25 12:28:36.793089297   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:36.801760201   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": net/http: timeout awaiting response headers
2025-04-25 12:28:36.801794956   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": net/http: timeout awaiting response headers
2025-04-25 12:28:36.893473073   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175811
2025-04-25 12:28:36.893477531   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:36.980464663   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:37.113367498   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175813
2025-04-25 12:28:37.113372948   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:37.113966734   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134930) =======
2025-04-25 12:28:37.113969709   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:37.113971042   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:37.113984758   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:37.113985890   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:37.113987062   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:37.113988595   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:37.113989767   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:37.195035293   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175820) =======
2025-04-25 12:28:37.195038920   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:37.195040273   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:37.195041325   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=7138.46m cores, Cores=24
2025-04-25 12:28:37.195042337   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15818.12 MB, Total=128728.99 MB, Usage=12.29%
2025-04-25 12:28:37.195043349   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:37.195044551   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:37.195045533   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:37.203202671   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": net/http: timeout awaiting response headers
2025-04-25 12:28:37.203246634   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": net/http: timeout awaiting response headers
2025-04-25 12:28:37.394620694   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175816
2025-04-25 12:28:37.394642084   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:37.821106144   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:37.821135149   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:37.893459617   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175821
2025-04-25 12:28:37.893464626   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:37.980120035   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:37.993635011   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:37.994051865   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001034660})
2025-04-25 12:28:38.085990040   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175849, managers: 7
2025-04-25 12:28:38.085995511   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 0, tick: 175822
2025-04-25 12:28:38.085997574   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134938
2025-04-25 12:28:38.085999448   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:38.086034584   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134897
2025-04-25 12:28:38.086036708   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134897
2025-04-25 12:28:38.086038762   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:38.086040675   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:38.086042429   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175835
2025-04-25 12:28:38.086044252   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175835
2025-04-25 12:28:38.086046076   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175828
2025-04-25 12:28:38.086047929   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:38.092810066   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:38.192211516   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175830) =======
2025-04-25 12:28:38.192214962   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:38.192216265   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:38.192217237   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2430.38m cores, Cores=24
2025-04-25 12:28:38.192240180   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15795.40 MB, Total=128728.99 MB, Usage=12.27%
2025-04-25 12:28:38.192241222   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:38.192242384   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:38.192243336   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:38.192278502   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:38.192383229   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:38.192397876   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:38.292860470   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:38.393321802   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175826
2025-04-25 12:28:38.393325609   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:38.401439516   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:38.401534004   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:38.493331495   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175827
2025-04-25 12:28:38.493334160   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:38.494004700   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134940) =======
2025-04-25 12:28:38.494006593   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:38.494007996   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:38.494019878   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:38.494020900   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:38.494021952   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:38.494027653   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:38.494033985   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:38.593533188   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175828
2025-04-25 12:28:38.593538268   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:38.792824055   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175830
2025-04-25 12:28:38.792830037   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:38.980263933   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.06 seconds ago
2025-04-25 12:28:39.093543411   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175833
2025-04-25 12:28:39.093546677   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:39.193127354   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175840) =======
2025-04-25 12:28:39.193129568   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:39.193130460   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:39.193131151   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3012.55m cores, Cores=24
2025-04-25 12:28:39.193150588   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15790.52 MB, Total=128728.99 MB, Usage=12.27%
2025-04-25 12:28:39.193151309   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:39.193152100   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:39.193152752   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:39.493557546   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134950) =======
2025-04-25 12:28:39.493561393   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:39.493562726   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:39.493563858   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:39.493564699   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:39.493565631   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:39.493566913   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:39.493567915   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:39.593360610   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175838
2025-04-25 12:28:39.593363596   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:39.616893292   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:39.616896398   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:39.693459150   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:39.693850365   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc007589080})
2025-04-25 12:28:39.792918489   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:39.893411369   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:39.893488925   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:39.893498183   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:39.980691411   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:28:39.994919588   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:40.093675815   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175843
2025-04-25 12:28:40.093680774   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:40.193504387   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175850) =======
2025-04-25 12:28:40.193508205   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:40.193509457   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:40.193510519   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3949.37m cores, Cores=24
2025-04-25 12:28:40.193511591   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15788.18 MB, Total=128728.99 MB, Usage=12.26%
2025-04-25 12:28:40.193567135   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:40.193569209   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:40.193570391   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:40.193753696   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if log file exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:40.202078549   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:40.202082016   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:40.294604028   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175845
2025-04-25 12:28:40.294609037   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:40.304141731   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: context deadline exceeded
2025-04-25 12:28:40.494367893   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175847
2025-04-25 12:28:40.494370498   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:40.793779621   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175850
2025-04-25 12:28:40.793782807   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:40.893810393   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134960) =======
2025-04-25 12:28:40.893813599   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:40.893814892   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:40.893839117   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:40.893840720   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:40.893842213   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:40.893844267   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:40.893845790   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:40.980374229   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:41.193937645   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175860) =======
2025-04-25 12:28:41.193939919   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:41.193940801   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:41.193967170   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4638.66m cores, Cores=24
2025-04-25 12:28:41.193967942   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15819.54 MB, Total=128728.99 MB, Usage=12.29%
2025-04-25 12:28:41.193968603   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:41.193969395   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:41.193970016   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:41.202456182   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:28:41.293336019   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175855
2025-04-25 12:28:41.293339024   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:41.395001392   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:41.395485682   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc005fd6460})
2025-04-25 12:28:41.494011466   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:41.613702032   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:41.613705098   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:41.613727620   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:41.692438559   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:41.793265026   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175860
2025-04-25 12:28:41.793286486   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:41.893060265   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175861
2025-04-25 12:28:41.893107514   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:41.981121976   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:41.993253042   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175862
2025-04-25 12:28:41.993362497   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:28:42.193856424   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175870) =======
2025-04-25 12:28:42.193859690   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:42.193861073   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:42.193862115   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3753.09m cores, Cores=24
2025-04-25 12:28:42.193863136   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15813.20 MB, Total=128728.99 MB, Usage=12.28%
2025-04-25 12:28:42.193887182   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:42.193888414   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:42.193889406   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:42.194145918   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175864
2025-04-25 12:28:42.194152129   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:42.202190294   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:42.202219369   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:42.295184453   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134970) =======
2025-04-25 12:28:42.295189172   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:42.295191206   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:42.295193160   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:42.295194532   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:42.295222605   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:42.295224869   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:42.295226522   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:42.419285917   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:42.419312457   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:42.493874319   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175867
2025-04-25 12:28:42.493877986   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:42.980961404   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:42.993503072   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:43.085606336   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175899, managers: 7
2025-04-25 12:28:43.085631945   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175872
2025-04-25 12:28:43.085644338   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:28:43.085656421   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 134977
2025-04-25 12:28:43.085662953   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:43.085674845   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134936
2025-04-25 12:28:43.085681407   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134936
2025-04-25 12:28:43.085688290   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:43.085700223   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:43.085706715   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175885
2025-04-25 12:28:43.085718918   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175885
2025-04-25 12:28:43.085725350   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175878
2025-04-25 12:28:43.085731762   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:43.093418827   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:43.093869424   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc005321ac0})
2025-04-25 12:28:43.192441855   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175880) =======
2025-04-25 12:28:43.192444029   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:43.192444911   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:43.192445592   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2644.07m cores, Cores=24
2025-04-25 12:28:43.192446253   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15824.37 MB, Total=128728.99 MB, Usage=12.29%
2025-04-25 12:28:43.192446885   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:43.192503551   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:43.192505655   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:43.192507028   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:43.293337111   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:43.293401572   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:43.293413866   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:43.392746766   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:43.523067887   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175877
2025-04-25 12:28:43.523095680   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:43.531851774   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:43.531856723   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:43.593971785   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175878
2025-04-25 12:28:43.593998976   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:43.693873284   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175879
2025-04-25 12:28:43.693875969   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:43.694549204   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134980) =======
2025-04-25 12:28:43.694550497   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:43.694551619   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:43.694559784   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:43.694573199   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:43.694580203   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:43.694585903   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:43.694586915   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:43.893192703   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175881
2025-04-25 12:28:43.893197873   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:43.902093609   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:43.902100352   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:43.980675746   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:28:44.102617882   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:28:44.217282442   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175890) =======
2025-04-25 12:28:44.217284796   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:44.217285718   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:44.217286439   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=6522.65m cores, Cores=24
2025-04-25 12:28:44.217305796   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15819.55 MB, Total=128728.99 MB, Usage=12.29%
2025-04-25 12:28:44.217306557   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:44.217307369   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:44.217308040   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:44.217716918   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175884
2025-04-25 12:28:44.217728540   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:44.225916266   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: error reading config file benthos.yaml in service directory /run/service/benthos-dataflow-core-1: context deadline exceeded
2025-04-25 12:28:44.694239339   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175889
2025-04-25 12:28:44.694262212   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:44.694954042   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 134990) =======
2025-04-25 12:28:44.694956206   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:44.694957368   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:44.694993536   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:44.694996151   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:44.694998125   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:44.695000549   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:44.695002403   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:44.794251305   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:44.794656276   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc004d484a0})
2025-04-25 12:28:44.891808058   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:44.981111651   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:28:44.994980042   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:44.995047098   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:44.995057848   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:45.094124959   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:45.193225031   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175900) =======
2025-04-25 12:28:45.193227506   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:45.193228367   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:45.193248164   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4372.88m cores, Cores=24
2025-04-25 12:28:45.193248896   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15817.79 MB, Total=128728.99 MB, Usage=12.29%
2025-04-25 12:28:45.193249537   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:45.193250318   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:45.193250980   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:45.193608902   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175894
2025-04-25 12:28:45.193618791   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:45.218291604   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context deadline exceeded
2025-04-25 12:28:45.218313836   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context deadline exceeded
2025-04-25 12:28:45.293336384   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175895
2025-04-25 12:28:45.293341072   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:45.301932036   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": net/http: timeout awaiting response headers
2025-04-25 12:28:45.301935643   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": net/http: timeout awaiting response headers
2025-04-25 12:28:45.393335495   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175896
2025-04-25 12:28:45.393338080   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:45.592940440   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175898
2025-04-25 12:28:45.592963303   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if log file exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:45.893156356   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175901
2025-04-25 12:28:45.893158931   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:45.980927701   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:46.093392196   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135000) =======
2025-04-25 12:28:46.093394571   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:46.093395462   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:46.093415440   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:46.093416091   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:46.093416712   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:46.093417554   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:46.093418195   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:46.100293765   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": net/http: timeout awaiting response headers
2025-04-25 12:28:46.100355852   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": net/http: timeout awaiting response headers
2025-04-25 12:28:46.192851544   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175910) =======
2025-04-25 12:28:46.192854239   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:46.192855160   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:46.192876481   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2008.37m cores, Cores=24
2025-04-25 12:28:46.192877232   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15832.76 MB, Total=128728.99 MB, Usage=12.30%
2025-04-25 12:28:46.192877913   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:46.192878705   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:46.192879356   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:46.201490828   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:46.201494755   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:46.393662323   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175906
2025-04-25 12:28:46.393664848   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:46.493307670   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:46.493682705   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc003213940})
2025-04-25 12:28:46.592762790   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:46.693023854   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:46.693160000   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:46.693167704   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:46.793209756   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:46.894241969   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175911
2025-04-25 12:28:46.894244734   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:46.980865987   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:46.993709001   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175912
2025-04-25 12:28:46.993711746   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:47.022253169   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:47.022280290   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:47.094049664   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175913
2025-04-25 12:28:47.094052329   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:47.192100545   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175920) =======
2025-04-25 12:28:47.192105033   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:47.192107207   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:47.192134809   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1655.17m cores, Cores=24
2025-04-25 12:28:47.192136492   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15822.57 MB, Total=128728.99 MB, Usage=12.29%
2025-04-25 12:28:47.192137975   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:47.192139748   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:47.192141131   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:47.293204001   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175915
2025-04-25 12:28:47.293207167   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:47.493532805   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135010) =======
2025-04-25 12:28:47.493537574   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:47.493539789   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:47.493567020   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:47.493568753   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:47.493570606   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:47.493580004   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:47.493582198   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:47.593484738   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175918
2025-04-25 12:28:47.593489137   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:47.805297673   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 29.637537ms remaining, needed 30ms), skipping
2025-04-25 12:28:47.980417816   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:48.086194755   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175949, managers: 7
2025-04-25 12:28:48.086196999   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 134974
2025-04-25 12:28:48.086218309   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 134974
2025-04-25 12:28:48.086219171   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:48.086219892   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:48.086220523   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175935
2025-04-25 12:28:48.086234129   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175935
2025-04-25 12:28:48.086235251   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175928
2025-04-25 12:28:48.086236243   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:48.086243867   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175922
2025-04-25 12:28:48.086244919   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:28:48.086245871   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135015
2025-04-25 12:28:48.086251301   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:48.094682585   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175923
2025-04-25 12:28:48.094686783   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:48.193113950   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175930) =======
2025-04-25 12:28:48.193116645   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:48.193117507   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:48.193137224   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1634.04m cores, Cores=24
2025-04-25 12:28:48.193138466   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15810.96 MB, Total=128728.99 MB, Usage=12.28%
2025-04-25 12:28:48.193139538   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:48.193140810   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:48.193141832   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:48.193149797   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:48.193572762   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0017e3c00})
2025-04-25 12:28:48.293199484   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:48.393882951   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:48.393976787   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:48.393986936   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:48.492667991   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:48.593181298   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175928
2025-04-25 12:28:48.593185165   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:48.694270828   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175929
2025-04-25 12:28:48.694273463   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:48.702091174   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:48.702094781   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:48.795192483   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:48.821142707   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:48.821145502   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:48.892994656   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135020) =======
2025-04-25 12:28:48.892997261   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:48.892998624   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:48.892999796   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:48.893165808   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:48.893168844   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:48.893171619   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:48.893172741   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:48.980939386   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:48.994435617   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175932
2025-04-25 12:28:48.994440486   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:49.192998692   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175940) =======
2025-04-25 12:28:49.193002219   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:49.193003541   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:49.193004563   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1518.99m cores, Cores=24
2025-04-25 12:28:49.193046753   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15784.07 MB, Total=128728.99 MB, Usage=12.26%
2025-04-25 12:28:49.193048716   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:49.193050700   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:49.193052233   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:49.294349173   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175935
2025-04-25 12:28:49.294382095   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:49.302749648   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:49.302754648   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:49.635861328   [WARN]	[control/loop.go:320:320]	[ControlLoop]			Skipping reconcile cycle due to remaining time: 49.679402ms
2025-04-25 12:28:49.702796056   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:49.702803961   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:49.893986929   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175940
2025-04-25 12:28:49.893991948   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:49.902535162   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:28:49.980946757   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:28:49.996060167   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:49.996741327   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001c39a00})
2025-04-25 12:28:50.097526555   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:50.194439527   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175950) =======
2025-04-25 12:28:50.194442723   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:50.194443655   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:50.194444336   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=6782.61m cores, Cores=24
2025-04-25 12:28:50.194478721   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15864.43 MB, Total=128728.99 MB, Usage=12.32%
2025-04-25 12:28:50.194479642   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:50.194480434   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:50.194481095   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:50.194608124   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:50.194788202   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:50.194816165   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:50.298337885   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:50.400865387   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175945
2025-04-25 12:28:50.401476045   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:50.402291417   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135030) =======
2025-04-25 12:28:50.402295004   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:50.402296296   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:50.402298059   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:50.402299222   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:50.402300504   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:50.402302027   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:50.402303139   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:50.412950807   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:50.412964212   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:50.500195080   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:50.507537748   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:50.507700293   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:50.594396887   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175947
2025-04-25 12:28:50.594407026   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:50.721013947   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:50.721213712   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:50.793223477   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:50.980652620   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.07 seconds ago
2025-04-25 12:28:51.006740703   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:51.006819862   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:51.101415258   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175952
2025-04-25 12:28:51.101815240   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:28:51.116312622   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:51.117034429   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:51.198198877   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175960) =======
2025-04-25 12:28:51.198207764   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:51.198209136   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Degraded, CPU=Degraded, Memory=Active, Disk=Active
2025-04-25 12:28:51.198276964   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=22965.52m cores, Cores=24
2025-04-25 12:28:51.198278286   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=17734.14 MB, Total=128728.99 MB, Usage=13.78%
2025-04-25 12:28:51.198279398   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:51.198286261   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:51.198287423   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:51.411152904   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135040) =======
2025-04-25 12:28:51.411156390   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:51.411157683   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:51.411188711   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:51.411189703   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:51.411190695   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:51.411192047   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:51.411193129   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:51.596979936   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175957
2025-04-25 12:28:51.596983813   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:51.605110314   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:51.605131804   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:28:51.693188714   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:51.693714552   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0055a61a0})
2025-04-25 12:28:51.793638432   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:51.893640578   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:51.893798685   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:51.893811990   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:51.980957127   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.38 seconds ago
2025-04-25 12:28:52.015265677   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:52.093360897   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175962
2025-04-25 12:28:52.093505479   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:28:52.096819666   [ERROR]	[fsm/baseFSM.go:376:376]	[dataflow-core-1]			dataflow-core-1 instance dataflow-core-1 is not in a shutdown state, resetting state and removing it
2025-04-25 12:28:52.096821249   [INFO]	[fsm/baseFSM.go:150:150]	[dataflow-core-1]			Setting desired state of FSM dataflow-core-1 to stopped
2025-04-25 12:28:52.201131030   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175970) =======
2025-04-25 12:28:52.201356324   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:52.201357987   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:52.201359109   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=13960.16m cores, Cores=24
2025-04-25 12:28:52.201360271   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16107.96 MB, Total=128728.99 MB, Usage=12.51%
2025-04-25 12:28:52.201361363   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:52.201362706   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:52.201363778   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:52.202939959   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:52.210044460   [INFO]	[fsm/base_manager.go:394:394]	[BenthosManagerDFCServicecore-1]			Updated desired state of instance dataflow-core-1 from stopped to active
2025-04-25 12:28:52.210050801   [INFO]	[fsm/baseFSM.go:150:150]	[dataflow-core-1]			Setting desired state of FSM dataflow-core-1 to active
2025-04-25 12:28:52.294853997   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175964
2025-04-25 12:28:52.294856792   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:52.495936005   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175966
2025-04-25 12:28:52.496240227   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:52.506628257   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:52.507067713   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:52.602287342   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:52.602292311   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:52.793220329   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175969
2025-04-25 12:28:52.793222984   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:52.794010824   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135050) =======
2025-04-25 12:28:52.794012757   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:52.794014230   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:52.794029479   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:52.794030531   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:52.794031633   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:52.794037845   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:52.794039157   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:52.828175824   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:52.828180162   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:52.980481816   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:28:53.086180346   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 175999, managers: 7
2025-04-25 12:28:53.086289992   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 175985
2025-04-25 12:28:53.086292868   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 175978
2025-04-25 12:28:53.086294020   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:53.086295012   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 175971
2025-04-25 12:28:53.086295903   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:28:53.086296765   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135052
2025-04-25 12:28:53.086297657   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:53.086298538   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135011
2025-04-25 12:28:53.086299400   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135011
2025-04-25 12:28:53.086300281   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:53.086301123   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:53.086301915   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 175985
2025-04-25 12:28:53.193668960   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175980) =======
2025-04-25 12:28:53.193673479   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:53.193675562   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:53.193737098   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4925.76m cores, Cores=24
2025-04-25 12:28:53.193740635   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15931.21 MB, Total=128728.99 MB, Usage=12.38%
2025-04-25 12:28:53.193768828   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:53.193769749   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:53.193770451   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:53.202395458   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:28:53.293323042   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175974
2025-04-25 12:28:53.293325937   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:53.402580009   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:53.453520793   [ERROR]	[fsm/baseFSM.go:370:370]	[redpanda]			error force removing redpanda instance redpanda: context deadline exceeded
2025-04-25 12:28:53.453526374   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{failed to force remove the redpanda instance: operation permanently failed after max retries : context deadline exceeded {}})
2025-04-25 12:28:53.499210089   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:53.599746559   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:53.599867206   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:53.599895629   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:53.700111216   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:53.801395401   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175979
2025-04-25 12:28:53.801410390   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:53.811370335   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:53.811379483   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:53.899894363   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175980
2025-04-25 12:28:53.899900504   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:53.960949162   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:53.960965903   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:53.981070091   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.68 seconds ago
2025-04-25 12:28:53.999129317   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:54.013163780   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:54.013699656   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:54.108931268   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:54.108938141   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:54.199131725   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 175990) =======
2025-04-25 12:28:54.199362449   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:54.199364473   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Degraded, CPU=Degraded, Memory=Active, Disk=Active
2025-04-25 12:28:54.199365895   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=23482.76m cores, Cores=24
2025-04-25 12:28:54.199367388   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16996.64 MB, Total=128728.99 MB, Usage=13.20%
2025-04-25 12:28:54.199368781   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:54.199370314   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:54.199371426   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:54.199817203   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:54.201663122   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135060) =======
2025-04-25 12:28:54.201667991   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:54.201669414   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:54.201698038   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:54.201699941   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:54.201925465   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:54.201928050   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:54.201929443   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:54.308733670   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: get metrics: failed to execute request for http://localhost:9000/metrics: Get "http://localhost:9000/metrics": dial tcp [::1]:9000: i/o timeout
2025-04-25 12:28:54.308742768   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9000/metrics: Get "http://localhost:9000/metrics": dial tcp [::1]:9000: i/o timeout
2025-04-25 12:28:54.315959509   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:54.316944550   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:54.501963132   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175986
2025-04-25 12:28:54.501973371   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:54.620875272   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:54.620997982   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:54.950521536   [WARN]	[control/loop.go:320:320]	[ControlLoop]			Skipping reconcile cycle due to remaining time: 35.973536ms
2025-04-25 12:28:54.980591241   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:28:55.099931215   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:55.198147244   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:55.198835397   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc00350e960})
2025-04-25 12:28:55.298192450   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176000) =======
2025-04-25 12:28:55.298198943   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:55.298200315   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Degraded, CPU=Degraded, Memory=Active, Disk=Active
2025-04-25 12:28:55.298201518   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=23900.00m cores, Cores=24
2025-04-25 12:28:55.298236533   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=18043.81 MB, Total=128728.99 MB, Usage=14.02%
2025-04-25 12:28:55.298237806   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:55.298239248   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:55.298240330   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:55.298278953   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:55.397592555   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:55.397840982   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:55.397895023   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:55.500521360   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:55.599313973   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:55.612511212   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:55.614926822   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:55.699114861   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 175997
2025-04-25 12:28:55.699192136   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:55.700745865   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135070) =======
2025-04-25 12:28:55.700778336   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:55.700797833   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:55.700826447   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:55.700848438   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:55.700865029   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:55.700884466   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:55.700899264   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:55.705030747   [ERROR]	[fsm/baseFSM.go:376:376]	[dataflow-core-1]			dataflow-core-1 instance dataflow-core-1 is not in a shutdown state, resetting state and removing it
2025-04-25 12:28:55.705070422   [INFO]	[fsm/baseFSM.go:150:150]	[dataflow-core-1]			Setting desired state of FSM dataflow-core-1 to stopped
2025-04-25 12:28:55.799248694   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:55.804997517   [INFO]	[fsm/base_manager.go:394:394]	[BenthosManagerDFCServicecore-1]			Updated desired state of instance dataflow-core-1 from stopped to active
2025-04-25 12:28:55.805002817   [INFO]	[fsm/baseFSM.go:150:150]	[dataflow-core-1]			Setting desired state of FSM dataflow-core-1 to active
2025-04-25 12:28:55.911454912   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:55.911504826   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:55.980516988   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:28:55.998955546   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:56.010770799   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:56.010775337   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:56.144316045   [WARN]	[control/loop.go:320:320]	[ControlLoop]			Skipping reconcile cycle due to remaining time: 41.926824ms
2025-04-25 12:28:56.210734233   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:56.211339550   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:56.312760061   [ERROR]	[s6/reconcile.go:97:97]	[benthos-dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get S6 service config for benthos-dataflow-core-1-faulty: failed to read config directory: context deadline exceeded
2025-04-25 12:28:56.401135980   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176010) =======
2025-04-25 12:28:56.401140408   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:56.401145778   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Degraded, CPU=Degraded, Memory=Active, Disk=Active
2025-04-25 12:28:56.401146950   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=23186.44m cores, Cores=24
2025-04-25 12:28:56.401148112   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16995.97 MB, Total=128728.99 MB, Usage=13.20%
2025-04-25 12:28:56.401149224   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:56.401150607   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:56.401151689   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:56.401576868   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176003
2025-04-25 12:28:56.402915243   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:56.411088111   [ERROR]	[s6/reconcile.go:97:97]	[benthos-dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get S6 service config for benthos-dataflow-core-1-faulty: failed to read config directory: context deadline exceeded
2025-04-25 12:28:56.417432052   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:56.417505470   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:56.511969569   [ERROR]	[s6/reconcile.go:97:97]	[benthos-dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get S6 service config for benthos-dataflow-core-1-faulty: failed to read config directory: context deadline exceeded
2025-04-25 12:28:56.619367672   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: failed liveness check: failed to execute request for http://localhost:9000/ping: Get "http://localhost:9000/ping": dial tcp [::1]:9000: i/o timeout
2025-04-25 12:28:56.619376850   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9000/ping: Get "http://localhost:9000/ping": dial tcp [::1]:9000: i/o timeout
2025-04-25 12:28:56.709942763   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:56.709950878   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:56.801145734   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135080) =======
2025-04-25 12:28:56.801210816   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:56.801229301   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:56.801280948   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:56.801304362   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:56.801320773   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:56.801343255   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:56.801359816   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:56.900685661   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:56.980959605   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.07 seconds ago
2025-04-25 12:28:57.000937576   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:57.001699378   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc004ab0e00})
2025-04-25 12:28:57.102630779   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:57.198789532   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:57.199022721   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:57.199067765   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:57.299689725   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: creating, desired state: active)
2025-04-25 12:28:57.354393640   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:28:57.354794303   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:57.398559723   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176020) =======
2025-04-25 12:28:57.398563961   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:57.398565334   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Degraded, CPU=Degraded, Memory=Active, Disk=Active
2025-04-25 12:28:57.398566356   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=23593.22m cores, Cores=24
2025-04-25 12:28:57.398567458   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=17855.55 MB, Total=128728.99 MB, Usage=13.87%
2025-04-25 12:28:57.398568490   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:57.398569742   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:57.398570744   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:57.398649832   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:57.499044766   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:57.599467101   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176015
2025-04-25 12:28:57.599472341   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:57.699006607   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176016
2025-04-25 12:28:57.699012999   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:57.710708486   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:57.711920063   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:57.805633120   [ERROR]	[fsm/baseFSM.go:376:376]	[dataflow-core-1]			dataflow-core-1 instance dataflow-core-1 is not in a shutdown state, resetting state and removing it
2025-04-25 12:28:57.805689857   [INFO]	[fsm/baseFSM.go:150:150]	[dataflow-core-1]			Setting desired state of FSM dataflow-core-1 to stopped
2025-04-25 12:28:57.899858390   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176018
2025-04-25 12:28:57.899965291   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:57.911040794   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:57.911937558   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:57.980319987   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.17 seconds ago
2025-04-25 12:28:58.013513351   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:58.013519382   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:58.086797360   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176049, managers: 7
2025-04-25 12:28:58.086892659   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176026
2025-04-25 12:28:58.086932494   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:28:58.086956960   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176019
2025-04-25 12:28:58.088693824   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:28:58.088729821   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135088
2025-04-25 12:28:58.088750871   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:28:58.088774034   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135047
2025-04-25 12:28:58.088813358   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135047
2025-04-25 12:28:58.088836301   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:28:58.088856630   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:28:58.088873571   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176034
2025-04-25 12:28:58.088980933   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176034
2025-04-25 12:28:58.110917654   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:58.110924587   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:58.195765603   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176021
2025-04-25 12:28:58.195773288   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:58.197170984   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135090) =======
2025-04-25 12:28:58.197829301   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:58.197831956   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:58.197833469   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:58.197834571   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:58.197835783   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:58.197837356   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:58.197838588   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:58.200750430   [INFO]	[fsm/base_manager.go:394:394]	[BenthosManagerDFCServicecore-1]			Updated desired state of instance dataflow-core-1 from stopped to active
2025-04-25 12:28:58.200753526   [INFO]	[fsm/baseFSM.go:150:150]	[dataflow-core-1]			Setting desired state of FSM dataflow-core-1 to active
2025-04-25 12:28:58.336017552   [WARN]	[fsm/base_manager.go:507:507]	[DataFlowCompManagerCore]			not enough time left to reconcile instance core-1-faulty (only 49.94447ms remaining, needed 50ms), skipping
2025-04-25 12:28:58.394005097   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176030) =======
2025-04-25 12:28:58.394007792   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:58.394009075   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:58.394010067   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=8923.08m cores, Cores=24
2025-04-25 12:28:58.394033831   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16057.00 MB, Total=128728.99 MB, Usage=12.47%
2025-04-25 12:28:58.394034893   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:58.394036065   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:58.394037027   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:58.409546521   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 25.709106ms remaining, needed 30ms), skipping
2025-04-25 12:28:58.694242577   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176026
2025-04-25 12:28:58.694248328   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:58.792871181   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:28:58.793466069   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002e1d760})
2025-04-25 12:28:58.893796110   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:28:58.981121064   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:28:58.994666037   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:28:58.994781173   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:28:58.994834403   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:28:59.093600215   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:28:59.194184474   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176031
2025-04-25 12:28:59.194188661   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:59.294300503   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176032
2025-04-25 12:28:59.294316322   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:59.325061185   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:59.325065724   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:59.394485010   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176040) =======
2025-04-25 12:28:59.394487094   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:59.394488006   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:28:59.394507022   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=6453.78m cores, Cores=24
2025-04-25 12:28:59.394507783   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16001.78 MB, Total=128728.99 MB, Usage=12.43%
2025-04-25 12:28:59.394508444   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:28:59.394509246   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:28:59.394509897   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:28:59.394766048   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176033
2025-04-25 12:28:59.394776859   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:59.594062418   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176035
2025-04-25 12:28:59.594065033   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:59.594889813   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135100) =======
2025-04-25 12:28:59.594891747   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:28:59.594919879   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:28:59.594921563   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:28:59.594929287   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:28:59.594930570   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:28:59.594936811   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:28:59.594942101   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:28:59.893519682   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176038
2025-04-25 12:28:59.893523649   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:28:59.902745247   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:28:59.902767559   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:28:59.980916940   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:00.200680751   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:00.200683225   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:00.393460925   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176050) =======
2025-04-25 12:29:00.393463260   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:00.393464171   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:00.393494839   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4700.00m cores, Cores=24
2025-04-25 12:29:00.393495991   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15994.30 MB, Total=128728.99 MB, Usage=12.42%
2025-04-25 12:29:00.393497013   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:00.393498195   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:00.393499157   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:00.393862660   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176043
2025-04-25 12:29:00.393871988   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:00.493597142   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:00.494031839   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc00263c240})
2025-04-25 12:29:00.594076253   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:00.694222278   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:00.694290887   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:00.694299013   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:00.793339821   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:00.894650084   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176048
2025-04-25 12:29:00.894652529   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:00.980670065   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:00.994335664   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176049
2025-04-25 12:29:00.994339802   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:00.995146528   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135110) =======
2025-04-25 12:29:00.995147670   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:00.995155455   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:00.995159572   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:00.995164111   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:00.995167728   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:00.995181964   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:00.995183227   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:01.092745797   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176050
2025-04-25 12:29:01.092749514   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:01.119606070   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:01.119629484   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:01.293090576   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176052
2025-04-25 12:29:01.293093802   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:01.393477484   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176060) =======
2025-04-25 12:29:01.393481772   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:01.393483175   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:01.393484287   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4658.23m cores, Cores=24
2025-04-25 12:29:01.393521517   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16111.89 MB, Total=128728.99 MB, Usage=12.52%
2025-04-25 12:29:01.393522318   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:01.393523120   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:01.393523771   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:01.400377580   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:01.400409520   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:01.595132656   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:01.918041927   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:01.918068186   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:01.980267014   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:01.996268312   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135120) =======
2025-04-25 12:29:01.996270557   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:01.996271478   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:01.996289993   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:01.996290654   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:01.996295323   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:01.996299431   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:01.996300102   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:02.093820834   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176060
2025-04-25 12:29:02.093842214   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:02.192784437   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:02.193354798   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001411600})
2025-04-25 12:29:02.294123313   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:02.394529717   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176070) =======
2025-04-25 12:29:02.394532082   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:02.394532953   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:02.394594158   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4551.72m cores, Cores=24
2025-04-25 12:29:02.394595060   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16359.33 MB, Total=128728.99 MB, Usage=12.71%
2025-04-25 12:29:02.394595741   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:02.394596543   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:02.394597154   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:02.394647228   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:02.394850530   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:02.394878954   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:02.493665744   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:02.595196341   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176065
2025-04-25 12:29:02.595199847   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:02.694860661   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176066
2025-04-25 12:29:02.695082999   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:02.703824114   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:02.703875600   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:02.793577099   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176067
2025-04-25 12:29:02.793580856   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:02.803170476   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:02.803202687   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:02.901307195   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:02.901310732   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:02.981001028   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:02.994940672   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:03.012114824   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 24.188339ms remaining, needed 30ms), skipping
2025-04-25 12:29:03.085675773   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176099, managers: 7
2025-04-25 12:29:03.085678628   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176076
2025-04-25 12:29:03.085701241   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:03.085702052   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176069
2025-04-25 12:29:03.085702794   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:03.085703475   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135126
2025-04-25 12:29:03.085704146   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:03.085704817   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135085
2025-04-25 12:29:03.085724434   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135085
2025-04-25 12:29:03.085725226   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:03.085725897   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:03.085726548   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176084
2025-04-25 12:29:03.085727200   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176084
2025-04-25 12:29:03.293796193   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176072
2025-04-25 12:29:03.293798717   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:03.302073126   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:03.302100437   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:03.393304299   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176080) =======
2025-04-25 12:29:03.393306333   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:03.393307254   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:03.393354914   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4537.82m cores, Cores=24
2025-04-25 12:29:03.393362208   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16600.85 MB, Total=128728.99 MB, Usage=12.90%
2025-04-25 12:29:03.393366396   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:03.393377216   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:03.393377897   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:03.394316381   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135130) =======
2025-04-25 12:29:03.394318605   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:03.394319897   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:03.394321130   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:03.394331900   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:03.394333072   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:03.394334435   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:03.394335567   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:03.401790155   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:03.401794874   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:03.729926477   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:03.729953207   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:03.795318725   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176077
2025-04-25 12:29:03.795323665   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:03.893091300   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:03.893566704   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc000e01060})
2025-04-25 12:29:03.980411985   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:03.993392557   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:04.092674468   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:04.092744840   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:04.092751903   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:04.193810813   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: creating, desired state: active)
2025-04-25 12:29:04.294035475   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:04.394114754   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176090) =======
2025-04-25 12:29:04.394117289   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:04.394118191   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:04.394137507   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4600.00m cores, Cores=24
2025-04-25 12:29:04.394138279   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16579.95 MB, Total=128728.99 MB, Usage=12.88%
2025-04-25 12:29:04.394138960   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:04.394139761   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:04.394143949   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:04.394438262   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176083
2025-04-25 12:29:04.394453391   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:04.494254878   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176084
2025-04-25 12:29:04.494278362   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:04.594178735   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176085
2025-04-25 12:29:04.594311815   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:04.603262083   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:04.603267764   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:04.720707866   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:04.720710611   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:04.793419378   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176087
2025-04-25 12:29:04.793423877   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:04.794324830   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135140) =======
2025-04-25 12:29:04.794327194   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:04.794327975   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:04.794338345   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:04.794349706   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:04.794353383   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:04.794364143   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:04.794370345   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:04.980092887   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:05.093105188   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176090
2025-04-25 12:29:05.093111209   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:05.102895605   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:05.102901085   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:05.393681711   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176100) =======
2025-04-25 12:29:05.393685558   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:05.393687071   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:05.393688324   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4429.18m cores, Cores=24
2025-04-25 12:29:05.393689596   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16572.61 MB, Total=128728.99 MB, Usage=12.87%
2025-04-25 12:29:05.393716116   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:05.393717939   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:05.393719101   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:05.424743780   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1]			not enough time left to reconcile instance benthos-dataflow-core-1 (only 13.057472ms remaining, needed 30ms), skipping
2025-04-25 12:29:05.594444184   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176095
2025-04-25 12:29:05.594448051   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:05.604627920   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:05.604634162   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:05.693597092   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:05.694186269   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc00140c700})
2025-04-25 12:29:05.793153278   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:05.894551996   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:05.894621277   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:05.894634932   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:05.980976978   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:29:05.995307195   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:06.094175398   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176100
2025-04-25 12:29:06.094177963   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:06.102701299   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:06.102719043   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:06.194587232   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176101
2025-04-25 12:29:06.194590298   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:06.195606107   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135150) =======
2025-04-25 12:29:06.195609533   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:06.195610455   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:06.195631575   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:06.195632206   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:06.195632847   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:06.195644018   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:06.195645270   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:06.205175208   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:06.205180548   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:06.294144460   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176102
2025-04-25 12:29:06.294149500   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:06.393854846   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176110) =======
2025-04-25 12:29:06.393857070   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:06.393857991   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:06.393858703   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=6305.08m cores, Cores=24
2025-04-25 12:29:06.393879291   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16601.25 MB, Total=128728.99 MB, Usage=12.90%
2025-04-25 12:29:06.393880013   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:06.393880804   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:06.393881466   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:06.400199167   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:06.400218203   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:06.494447670   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176104
2025-04-25 12:29:06.494451196   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:06.638238537   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 29.419288ms remaining, needed 30ms), skipping
2025-04-25 12:29:06.794227104   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176107
2025-04-25 12:29:06.794252482   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:06.980991363   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:07.194375863   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135160) =======
2025-04-25 12:29:07.194378097   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:07.194378969   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:07.194410868   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:07.194412211   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:07.194413022   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:07.194413914   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:07.194414565   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:07.294008161   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176112
2025-04-25 12:29:07.294010796   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:07.323221967   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:07.323238187   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:07.393325759   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176120) =======
2025-04-25 12:29:07.393328003   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:07.393328895   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:07.393329606   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5128.21m cores, Cores=24
2025-04-25 12:29:07.393356246   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16608.73 MB, Total=128728.99 MB, Usage=12.90%
2025-04-25 12:29:07.393356987   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:07.393357779   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:07.393358420   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:07.393364211   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:07.393710391   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc006d4a960})
2025-04-25 12:29:07.494513690   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:07.593230309   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:07.593295020   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:07.593297254   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:07.693463516   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:07.793301932   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176117
2025-04-25 12:29:07.793318694   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:07.802812273   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:07.802839104   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:07.894596425   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176118
2025-04-25 12:29:07.894619358   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:07.903354512   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:29:07.980612978   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.78 seconds ago
2025-04-25 12:29:07.993931254   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176119
2025-04-25 12:29:07.993948186   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:08.086351070   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176149, managers: 7
2025-04-25 12:29:08.086355077   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176134
2025-04-25 12:29:08.086383180   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176134
2025-04-25 12:29:08.086384663   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176126
2025-04-25 12:29:08.086385755   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:08.086386927   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176119
2025-04-25 12:29:08.086388109   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:08.086389191   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135164
2025-04-25 12:29:08.086390304   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:08.086391416   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135123
2025-04-25 12:29:08.086409830   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135123
2025-04-25 12:29:08.086411243   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:08.086412295   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:08.193659473   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176121
2025-04-25 12:29:08.193662548   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:08.202873136   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:08.202877975   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:08.393950418   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176130) =======
2025-04-25 12:29:08.393964685   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:08.393965616   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:08.393970105   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2256.41m cores, Cores=24
2025-04-25 12:29:08.393970816   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16636.85 MB, Total=128728.99 MB, Usage=12.92%
2025-04-25 12:29:08.393975325   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:08.393976166   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:08.393980675   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:08.440884652   [WARN]	[fsm/base_manager.go:507:507]	[DataFlowCompManagerCore]			not enough time left to reconcile instance core-1 (only 45.462008ms remaining, needed 50ms), skipping
2025-04-25 12:29:08.494628462   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176124
2025-04-25 12:29:08.494651174   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:08.503353727   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if service exists: failed to check if S6 service exists: context deadline exceeded
2025-04-25 12:29:08.594092243   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135170) =======
2025-04-25 12:29:08.594095349   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:08.594096211   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:08.594125756   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:08.594127309   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:08.594128301   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:08.594129604   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:08.594130606   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:08.980609576   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:08.995533890   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176129
2025-04-25 12:29:08.995539240   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:09.093781627   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:09.094337852   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc005125460})
2025-04-25 12:29:09.225453170   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:09.225475312   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:09.293427650   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:09.393570538   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176140) =======
2025-04-25 12:29:09.393573945   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:09.393575327   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:09.393576369   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4271.19m cores, Cores=24
2025-04-25 12:29:09.393577381   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16743.32 MB, Total=128728.99 MB, Usage=13.01%
2025-04-25 12:29:09.393600735   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:09.393601987   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:09.393602959   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:09.393698529   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:09.393793918   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:09.393812283   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:09.494103659   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:09.593569164   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176135
2025-04-25 12:29:09.593571849   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:09.602738654   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:29:09.693074504   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176136
2025-04-25 12:29:09.693078411   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:09.793969605   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176137
2025-04-25 12:29:09.794074522   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:09.802453537   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if service exists: failed to check if S6 service exists: context deadline exceeded
2025-04-25 12:29:09.980662128   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:09.994949485   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:09.995940577   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135180) =======
2025-04-25 12:29:09.995990692   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:09.995994198   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:09.996020217   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:09.996021660   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:09.996022972   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:09.996024595   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:09.996025868   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:10.120734937   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:10.120739075   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:10.294444483   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176142
2025-04-25 12:29:10.294447980   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:10.393574137   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176150) =======
2025-04-25 12:29:10.393578194   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:10.393579717   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:10.393581010   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=9682.40m cores, Cores=24
2025-04-25 12:29:10.393612258   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16969.18 MB, Total=128728.99 MB, Usage=13.18%
2025-04-25 12:29:10.393613611   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:10.393615024   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:10.393616065   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:10.402320843   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:10.402365657   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:10.793476579   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176147
2025-04-25 12:29:10.793492609   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:10.802481070   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:10.802484636   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:10.894256574   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:10.894603125   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0064444e0})
2025-04-25 12:29:10.980779809   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:29:11.094489239   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:11.195261480   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:11.195302206   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:11.195355105   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:11.293649850   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:11.394307164   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176160) =======
2025-04-25 12:29:11.394310760   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:11.394312203   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:11.394335697   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4561.98m cores, Cores=24
2025-04-25 12:29:11.394337040   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=18151.93 MB, Total=128728.99 MB, Usage=14.10%
2025-04-25 12:29:11.394338142   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:11.394339454   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:11.394340566   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:11.394611756   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176153
2025-04-25 12:29:11.394622176   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:11.395517759   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135190) =======
2025-04-25 12:29:11.395519382   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:11.395529801   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:11.395531204   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:11.395536354   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:11.395541483   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:11.395548667   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:11.395554077   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:11.402514155   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:11.402519836   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:11.493594584   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176154
2025-04-25 12:29:11.493630461   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:11.594237812   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176155
2025-04-25 12:29:11.594352026   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:11.603916709   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:11.603922750   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:11.795623964   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176157
2025-04-25 12:29:11.795627000   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:11.801788909   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:11.801794048   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:11.908982134   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 26.12638ms remaining, needed 30ms), skipping
2025-04-25 12:29:11.980346033   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:12.095055773   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176160
2025-04-25 12:29:12.095082493   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if log file exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:12.392916589   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176170) =======
2025-04-25 12:29:12.392918954   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:12.392919825   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:12.392966633   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2678.11m cores, Cores=24
2025-04-25 12:29:12.393054037   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=17057.95 MB, Total=128728.99 MB, Usage=13.25%
2025-04-25 12:29:12.393065178   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:12.393066210   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:12.393070358   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:12.394024762   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135200) =======
2025-04-25 12:29:12.394025934   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:12.394026805   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:12.394043397   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:12.394044078   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:12.394044719   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:12.394050019   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:12.394050710   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:12.593499212   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176165
2025-04-25 12:29:12.593502538   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:12.643382056   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 24.89173ms remaining, needed 30ms), skipping
2025-04-25 12:29:12.694368744   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:12.694715095   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc007866b20})
2025-04-25 12:29:12.793773494   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:12.893707379   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:12.893770408   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:12.893776940   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:12.980756101   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:29:12.997777526   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:13.085482501   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176199, managers: 7
2025-04-25 12:29:13.085486699   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176176
2025-04-25 12:29:13.085488172   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:13.085489404   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176169
2025-04-25 12:29:13.085512498   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:13.085513750   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135202
2025-04-25 12:29:13.085514982   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:13.085516184   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135161
2025-04-25 12:29:13.085517377   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135161
2025-04-25 12:29:13.085518619   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:13.085519821   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:13.085520953   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176184
2025-04-25 12:29:13.085522116   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176184
2025-04-25 12:29:13.093871896   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:13.194132534   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176171
2025-04-25 12:29:13.194135950   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:13.293308735   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:13.392927286   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176180) =======
2025-04-25 12:29:13.392932216   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:13.392961270   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:13.392963144   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3025.21m cores, Cores=24
2025-04-25 12:29:13.392965017   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=17172.93 MB, Total=128728.99 MB, Usage=13.34%
2025-04-25 12:29:13.392967001   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:13.392976208   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:13.392977972   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:13.493383923   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176174
2025-04-25 12:29:13.493387800   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:13.501744393   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:13.501768869   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:13.618779051   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:13.618784111   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:13.794740378   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176177
2025-04-25 12:29:13.794744245   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:13.796000265   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135210) =======
2025-04-25 12:29:13.796001608   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:13.796002520   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:13.796015674   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:13.796016806   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:13.796017949   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:13.796069566   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:13.796070908   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:13.901318918   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:13.901321653   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:13.980748954   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:14.104215558   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:14.104220337   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:14.294271730   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176182
2025-04-25 12:29:14.294278723   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:14.331346697   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:14.331371924   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:14.393302868   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176190) =======
2025-04-25 12:29:14.393305312   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:14.393306625   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:14.393307637   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=11540.43m cores, Cores=24
2025-04-25 12:29:14.393327945   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=17076.45 MB, Total=128728.99 MB, Usage=13.27%
2025-04-25 12:29:14.393329057   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:14.393330229   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:14.393331191   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:14.393345668   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:14.393728227   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002bc3260})
2025-04-25 12:29:14.492797276   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:14.593928290   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:14.594147622   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:14.594167088   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:14.698502293   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:14.802938237   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176187
2025-04-25 12:29:14.802944348   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:14.811029150   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:14.811073925   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:14.898317122   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:14.911816368   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:14.911822479   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:14.980701339   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.78 seconds ago
2025-04-25 12:29:14.998668631   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if log file exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:15.009735036   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:15.010460930   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:15.199341943   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176191
2025-04-25 12:29:15.199607942   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:15.200956797   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135220) =======
2025-04-25 12:29:15.200974951   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:15.201008575   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:15.201038811   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:15.201040204   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:15.201041546   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:15.201043410   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:15.201044763   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:15.210433014   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:15.211002283   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:15.395500831   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176200) =======
2025-04-25 12:29:15.395505119   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:15.395506432   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:15.395540005   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=11544.30m cores, Cores=24
2025-04-25 12:29:15.395541267   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16917.65 MB, Total=128728.99 MB, Usage=13.14%
2025-04-25 12:29:15.395542329   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:15.395543672   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:15.395544764   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:15.440810662   [WARN]	[control/loop.go:320:320]	[ControlLoop]			Skipping reconcile cycle due to remaining time: 45.193604ms
2025-04-25 12:29:15.495128249   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176194
2025-04-25 12:29:15.495131065   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:15.515886245   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 20.824828ms remaining, needed 30ms), skipping
2025-04-25 12:29:15.717939961   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: failed readiness check: failed to execute request for http://localhost:9000/ready: Get "http://localhost:9000/ready": dial tcp: lookup localhost: i/o timeout
2025-04-25 12:29:15.717946323   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9000/ready: Get "http://localhost:9000/ready": dial tcp: lookup localhost: i/o timeout
2025-04-25 12:29:15.818419090   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:15.818425622   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:15.911982223   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:15.912060019   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:15.981046881   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.37 seconds ago
2025-04-25 12:29:16.009426408   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176199
2025-04-25 12:29:16.009589134   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:16.017586943   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: failed liveness check: failed to execute request for http://localhost:9000/ping: Get "http://localhost:9000/ping": dial tcp [::1]:9000: i/o timeout
2025-04-25 12:29:16.017594587   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9000/ping: Get "http://localhost:9000/ping": dial tcp [::1]:9000: i/o timeout
2025-04-25 12:29:16.100499181   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:16.102925721   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc00343bcc0})
2025-04-25 12:29:16.201928455   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:16.300071754   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:16.300252304   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:16.300298521   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:16.397014528   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176210) =======
2025-04-25 12:29:16.397057418   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:16.397059192   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Degraded, CPU=Degraded, Memory=Active, Disk=Active
2025-04-25 12:29:16.397065544   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=22017.39m cores, Cores=24
2025-04-25 12:29:16.397071064   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=17547.86 MB, Total=128728.99 MB, Usage=13.63%
2025-04-25 12:29:16.397076414   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:16.397094077   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:16.397095320   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:16.397393610   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:16.502268008   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if log file exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:16.518602922   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:16.518609294   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:16.646565094   [WARN]	[control/loop.go:320:320]	[ControlLoop]			Skipping reconcile cycle due to remaining time: 38.913674ms
2025-04-25 12:29:16.701617081   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176205
2025-04-25 12:29:16.701633291   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:16.704310922   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135230) =======
2025-04-25 12:29:16.704314259   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:16.704315671   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:16.704316964   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:16.704317906   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:16.704319028   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:16.704320390   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:16.704321532   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:16.799781670   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:16.811225824   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:16.812925067   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:16.980405627   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.07 seconds ago
2025-04-25 12:29:16.998537398   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176208
2025-04-25 12:29:16.998541486   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:17.292926027   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176211
2025-04-25 12:29:17.292960983   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:17.427702984   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:17.427728232   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:17.494661454   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176220) =======
2025-04-25 12:29:17.494665582   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:17.494667045   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:17.494694997   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=12400.00m cores, Cores=24
2025-04-25 12:29:17.494696380   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16880.39 MB, Total=128728.99 MB, Usage=13.11%
2025-04-25 12:29:17.494697572   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:17.494698985   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:17.494700057   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:17.506589228   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:17.506594287   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:17.602670041   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:17.602728842   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:17.698203306   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135240) =======
2025-04-25 12:29:17.698214847   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:17.698220067   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:17.698225798   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:17.698230888   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:17.698231589   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:17.698237059   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:17.698242529   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:17.794968105   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176216
2025-04-25 12:29:17.794971160   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:17.894589731   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:17.895223212   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001f23500})
2025-04-25 12:29:17.981108537   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:17.998714751   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:18.086050561   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176249, managers: 7
2025-04-25 12:29:18.086053767   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176225
2025-04-25 12:29:18.086055330   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:18.086056522   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176218
2025-04-25 12:29:18.086057685   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: to_be_created, desired state: active
2025-04-25 12:29:18.086086759   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135241
2025-04-25 12:29:18.086087962   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:18.086089014   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135199
2025-04-25 12:29:18.086090045   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135199
2025-04-25 12:29:18.086096648   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:18.086097810   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:18.086098802   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176233
2025-04-25 12:29:18.086104352   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176233
2025-04-25 12:29:18.095082012   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:18.095152695   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:18.095166841   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:18.193348843   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:18.295253780   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176221
2025-04-25 12:29:18.295257147   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:18.304138566   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if config file exists: context deadline exceeded
2025-04-25 12:29:18.394167838   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176222
2025-04-25 12:29:18.394170994   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:18.494026349   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176230) =======
2025-04-25 12:29:18.494029155   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:18.494030097   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:18.494030808   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=8672.27m cores, Cores=24
2025-04-25 12:29:18.494053951   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16635.14 MB, Total=128728.99 MB, Usage=12.92%
2025-04-25 12:29:18.494054703   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:18.494055554   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:18.494056216   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:18.494340410   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176223
2025-04-25 12:29:18.494343325   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:18.544280711   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.274717ms remaining, needed 30ms), skipping
2025-04-25 12:29:18.695376141   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176225
2025-04-25 12:29:18.695381041   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:18.706921767   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:18.706964126   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:18.804327780   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:18.804372714   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:18.980257506   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.07 seconds ago
2025-04-25 12:29:18.999466200   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176228
2025-04-25 12:29:18.999471180   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:19.095853469   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135250) =======
2025-04-25 12:29:19.095881502   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:19.095883035   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:19.095883876   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:19.095884497   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:19.095885159   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:19.095885980   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:19.095886882   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:19.230217690   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:19.230247546   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:19.494421429   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176240) =======
2025-04-25 12:29:19.494785493   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:19.494790853   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:19.494792265   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=8000.00m cores, Cores=24
2025-04-25 12:29:19.494793628   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16663.68 MB, Total=128728.99 MB, Usage=12.94%
2025-04-25 12:29:19.494794820   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:19.494796333   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:19.494797616   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:19.495320578   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176233
2025-04-25 12:29:19.495387674   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:19.505918613   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:19.505994065   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:19.593250326   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:19.593597207   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002274e80})
2025-04-25 12:29:19.693307932   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:19.793261141   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:19.793395884   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:19.793418537   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:19.893867278   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:19.980302897   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.58 seconds ago
2025-04-25 12:29:19.994150347   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176238
2025-04-25 12:29:19.994154325   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:20.094174290   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176239
2025-04-25 12:29:20.094179610   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:20.194649510   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176240
2025-04-25 12:29:20.194652005   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:20.227445646   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:20.227450526   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:20.394304657   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176242
2025-04-25 12:29:20.394457334   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:20.403415898   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:20.403419795   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed liveness check: failed to execute request for http://localhost:9001/ping: Get "http://localhost:9001/ping": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:20.493739192   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176250) =======
2025-04-25 12:29:20.493742007   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:20.493743370   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:20.493744432   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5106.38m cores, Cores=24
2025-04-25 12:29:20.493769278   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16366.43 MB, Total=128728.99 MB, Usage=12.71%
2025-04-25 12:29:20.493770370   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:20.493771613   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:20.493772615   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:20.494786430   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135260) =======
2025-04-25 12:29:20.494788033   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:20.494788985   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:20.494809784   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:20.494810445   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:20.494811066   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:20.494811918   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:20.494812569   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:20.695973455   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176245
2025-04-25 12:29:20.695976311   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:20.980376660   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:21.027462658   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:29:21.027468649   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:29:21.194220648   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176250
2025-04-25 12:29:21.194264711   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:21.293270059   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:21.293655924   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc000deaae0})
2025-04-25 12:29:21.392924257   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:21.493282197   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176260) =======
2025-04-25 12:29:21.493284601   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:21.493285513   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:21.493286214   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4920.50m cores, Cores=24
2025-04-25 12:29:21.493307574   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16389.89 MB, Total=128728.99 MB, Usage=12.73%
2025-04-25 12:29:21.493308326   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:21.493309127   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:21.493309778   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:21.493364030   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:21.493443299   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:21.493455352   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:21.593303203   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:21.693630135   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176255
2025-04-25 12:29:21.693698013   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:21.793984698   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176256
2025-04-25 12:29:21.793987934   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:21.894699849   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176257
2025-04-25 12:29:21.894710840   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:21.895814845   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135270) =======
2025-04-25 12:29:21.895816748   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:21.895818071   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:21.895832618   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:21.895833821   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:21.895834923   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:21.895836355   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:21.895843489   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:21.980925975   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:22.006619685   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 29.323678ms remaining, needed 30ms), skipping
2025-04-25 12:29:22.094348924   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:22.393963451   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176262
2025-04-25 12:29:22.393965966   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:22.493933633   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176270) =======
2025-04-25 12:29:22.493937790   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:22.493939944   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:22.493941628   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3200.00m cores, Cores=24
2025-04-25 12:29:22.493943421   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16144.35 MB, Total=128728.99 MB, Usage=12.54%
2025-04-25 12:29:22.493945144   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:22.493947148   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:22.493948831   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:22.893853631   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176267
2025-04-25 12:29:22.893857438   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:22.894898724   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135280) =======
2025-04-25 12:29:22.894900047   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:22.894918161   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:22.894919032   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:22.894919633   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:22.894924533   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:22.894925394   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:22.894930223   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:22.980952335   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:22.995148611   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:22.995714004   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001ca44a0})
2025-04-25 12:29:23.086075158   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176299, managers: 7
2025-04-25 12:29:23.086077863   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176283
2025-04-25 12:29:23.086079296   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176275
2025-04-25 12:29:23.086102490   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:23.086103562   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 0, tick: 176268
2025-04-25 12:29:23.086104554   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135280
2025-04-25 12:29:23.086105596   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:23.086106587   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135238
2025-04-25 12:29:23.086107629   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135238
2025-04-25 12:29:23.086108661   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:23.086109673   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:23.086110605   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176283
2025-04-25 12:29:23.094299232   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:23.193850396   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:23.193938862   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:23.193955503   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:23.295859407   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:23.394755350   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176272
2025-04-25 12:29:23.394758987   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:23.405295586   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:23.405300254   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:23.493758694   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176280) =======
2025-04-25 12:29:23.493761970   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:23.493763262   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:23.493764254   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=6489.27m cores, Cores=24
2025-04-25 12:29:23.493765256   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16228.98 MB, Total=128728.99 MB, Usage=12.61%
2025-04-25 12:29:23.493791035   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:23.493792307   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:23.493793269   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:23.494100125   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176273
2025-04-25 12:29:23.494150941   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:23.502942551   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:29:23.502947801   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:29:23.594389466   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176274
2025-04-25 12:29:23.594392151   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:23.723444760   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:23.723470909   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:23.794088745   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176276
2025-04-25 12:29:23.794091891   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:23.981138084   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:24.094017883   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176279
2025-04-25 12:29:24.094024705   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:24.301003733   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135290) =======
2025-04-25 12:29:24.301033348   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:24.301039550   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:24.301045782   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:24.301051703   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:24.301062844   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:24.301069426   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:24.301075678   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:24.305417807   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:24.305422596   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:24.525397414   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176290) =======
2025-04-25 12:29:24.525400800   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:24.525401702   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:24.525402383   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=7094.34m cores, Cores=24
2025-04-25 12:29:24.525455312   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16401.39 MB, Total=128728.99 MB, Usage=12.74%
2025-04-25 12:29:24.525456725   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:24.525457807   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:24.525458538   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:24.535897935   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:24.535940775   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:24.594299616   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176284
2025-04-25 12:29:24.594303693   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:24.693939516   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:24.694329699   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc007f44320})
2025-04-25 12:29:24.793403435   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:24.893081978   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:24.893160906   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:24.893177938   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:24.980223703   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.38 seconds ago
2025-04-25 12:29:24.993755530   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:25.094584074   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176289
2025-04-25 12:29:25.094611215   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:25.104260356   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:25.104264073   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:25.193500534   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176290
2025-04-25 12:29:25.193503740   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:25.201788668   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:25.201821981   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:25.295057497   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176291
2025-04-25 12:29:25.295158687   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:25.304220735   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: error reading config file benthos.yaml in service directory /run/service/benthos-dataflow-core-1: context deadline exceeded
2025-04-25 12:29:25.514452825   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176300) =======
2025-04-25 12:29:25.514455099   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:25.514456011   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:25.514456692   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3333.33m cores, Cores=24
2025-04-25 12:29:25.514478433   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16478.81 MB, Total=128728.99 MB, Usage=12.80%
2025-04-25 12:29:25.514479154   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:25.514479956   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:25.514480597   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:25.514789497   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176293
2025-04-25 12:29:25.514791892   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:25.694727884   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135300) =======
2025-04-25 12:29:25.694730699   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:25.694732022   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:25.694733174   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:25.694755686   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:25.694756698   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:25.694757920   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:25.694758852   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:25.795228601   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176296
2025-04-25 12:29:25.795237939   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:25.804843037   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:25.804853858   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:25.980992233   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:26.127425691   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:26.128528584   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:26.294702144   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176301
2025-04-25 12:29:26.294708226   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:26.304690814   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:26.304695633   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:26.396232697   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:26.396735251   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0012e97e0})
2025-04-25 12:29:26.493081722   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176310) =======
2025-04-25 12:29:26.493085239   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:26.493086141   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:26.493103213   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=8640.00m cores, Cores=24
2025-04-25 12:29:26.493128240   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16172.57 MB, Total=128728.99 MB, Usage=12.56%
2025-04-25 12:29:26.493129132   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:26.493130033   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:26.493130705   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:26.493157104   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:26.595559214   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:26.595639956   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:26.595654704   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:26.694052389   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:26.794709340   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176306
2025-04-25 12:29:26.794731842   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:26.893821468   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176307
2025-04-25 12:29:26.893825606   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:26.980596995   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:26.995366838   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176308
2025-04-25 12:29:26.995372679   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:27.005603263   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:29:27.096075977   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135310) =======
2025-04-25 12:29:27.096083071   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:27.096084062   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:27.096085024   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:27.096111223   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:27.096112446   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:27.096113969   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:27.096115011   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:27.106332530   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:27.106335345   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:27.194150475   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176310
2025-04-25 12:29:27.194155966   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:27.211948910   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 23.9711ms remaining, needed 30ms), skipping
2025-04-25 12:29:27.495002816   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176320) =======
2025-04-25 12:29:27.495007234   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:27.495008607   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:27.495009729   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=8240.34m cores, Cores=24
2025-04-25 12:29:27.495010921   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16160.79 MB, Total=128728.99 MB, Usage=12.55%
2025-04-25 12:29:27.495041719   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:27.495043162   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:27.495044274   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:27.495446480   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176313
2025-04-25 12:29:27.495465836   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:27.504151106   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:27.504161035   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:29:27.701824857   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:27.701872687   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:27.926501339   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:27.926507801   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:27.980498122   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:27.995337286   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176318
2025-04-25 12:29:27.995341934   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:28.085990779   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176349, managers: 7
2025-04-25 12:29:28.085993163   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135277
2025-04-25 12:29:28.085994125   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135277
2025-04-25 12:29:28.086028139   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:28.086028951   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:28.086029602   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176333
2025-04-25 12:29:28.086030243   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176333
2025-04-25 12:29:28.086030894   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176325
2025-04-25 12:29:28.086031576   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:28.086032207   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176318
2025-04-25 12:29:28.086032868   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:28.086033509   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135319
2025-04-25 12:29:28.086034191   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:28.094464141   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:28.094993255   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc000dbc620})
2025-04-25 12:29:28.193146191   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:28.294430961   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:28.294492056   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:28.294504249   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:28.393402635   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:28.494040950   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176330) =======
2025-04-25 12:29:28.494044076   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:28.494045098   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:28.494045779   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5278.01m cores, Cores=24
2025-04-25 12:29:28.494073431   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16320.12 MB, Total=128728.99 MB, Usage=12.68%
2025-04-25 12:29:28.494074193   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:28.494074994   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:28.494075646   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:28.494308704   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:28.495640506   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135320) =======
2025-04-25 12:29:28.495641849   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:28.495654152   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:28.495654973   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:28.495659532   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:28.495660233   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:28.495674420   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:28.495675632   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:28.503947946   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:29:28.594829498   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176324
2025-04-25 12:29:28.594833947   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:28.604652065   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:28.604679116   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:28.694993783   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176325
2025-04-25 12:29:28.694999464   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:28.894452568   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176327
2025-04-25 12:29:28.894457167   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:28.926778010   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:28.926784372   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:28.980500698   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:29.194703167   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176330
2025-04-25 12:29:29.194707225   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:29.203810831   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:29.203813897   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:29.405426439   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:29.405464621   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:29.494198877   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176340) =======
2025-04-25 12:29:29.494202915   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:29.494204368   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:29.494205510   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=10643.48m cores, Cores=24
2025-04-25 12:29:29.494228263   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16405.54 MB, Total=128728.99 MB, Usage=12.74%
2025-04-25 12:29:29.494229164   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:29.494269360   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:29.494270803   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:29.495936032   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135330) =======
2025-04-25 12:29:29.495938947   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:29.495955348   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:29.495956300   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:29.495960568   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:29.495961259   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:29.495974203   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:29.495974945   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:29.504422288   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:29.504427047   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:29.633638984   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:29.633662438   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:29.695378035   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176335
2025-04-25 12:29:29.695383575   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:29.793490344   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:29.793963182   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001197500})
2025-04-25 12:29:29.893386745   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:29.980821470   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:29:29.994588389   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:29.994673669   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:29.994682175   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:30.094033532   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:30.194175785   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176340
2025-04-25 12:29:30.194178540   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:30.294123924   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176341
2025-04-25 12:29:30.294126739   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:30.393464951   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176342
2025-04-25 12:29:30.393468227   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:30.493677576   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176350) =======
2025-04-25 12:29:30.493680081   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:30.493680943   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:30.493702193   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2835.44m cores, Cores=24
2025-04-25 12:29:30.493702914   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16300.91 MB, Total=128728.99 MB, Usage=12.66%
2025-04-25 12:29:30.493703565   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:30.493704347   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:30.493704948   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:30.595501449   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176344
2025-04-25 12:29:30.595505106   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:30.625993275   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:30.626031327   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:30.893859842   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176347
2025-04-25 12:29:30.893865022   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:30.894990176   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135340) =======
2025-04-25 12:29:30.894992180   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:30.895006707   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:30.895008160   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:30.895013870   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:30.895021585   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:30.895027346   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:30.895032896   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:30.980952574   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:31.324593805   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:31.324598945   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:31.394185332   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176352
2025-04-25 12:29:31.394188127   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:31.493390103   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176360) =======
2025-04-25 12:29:31.493394652   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:31.493396124   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:31.493397297   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4085.11m cores, Cores=24
2025-04-25 12:29:31.493456478   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16350.71 MB, Total=128728.99 MB, Usage=12.70%
2025-04-25 12:29:31.493458552   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:31.493459574   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:31.493460315   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:31.493461086   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:31.493897467   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001448f00})
2025-04-25 12:29:31.593664655   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:31.694705666   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:31.694793001   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:31.694807798   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:31.794510034   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:31.894589109   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176357
2025-04-25 12:29:31.894592796   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:31.903683878   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:31.903687705   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:31.980901425   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.58 seconds ago
2025-04-25 12:29:31.995169034   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:32.093834542   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176359
2025-04-25 12:29:32.093839211   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:32.294431014   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176361
2025-04-25 12:29:32.294435272   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:32.295569733   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135350) =======
2025-04-25 12:29:32.295571226   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:32.295581325   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:32.295582107   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:32.295586124   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:32.295586815   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:32.295596363   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:32.295597065   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:32.328615137   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:32.328650213   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:32.495337157   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176370) =======
2025-04-25 12:29:32.495377964   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:32.495380779   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:32.495411347   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4740.74m cores, Cores=24
2025-04-25 12:29:32.495412579   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16320.38 MB, Total=128728.99 MB, Usage=12.68%
2025-04-25 12:29:32.495413571   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:32.495414773   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:32.495415715   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:32.504646300   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:29:32.594714974   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176364
2025-04-25 12:29:32.594722137   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:32.980856985   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:33.085659755   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176399, managers: 7
2025-04-25 12:29:33.085664324   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176375
2025-04-25 12:29:33.085665627   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:33.085699771   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176368
2025-04-25 12:29:33.085700753   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:33.085701604   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135357
2025-04-25 12:29:33.085702526   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:33.085703478   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135315
2025-04-25 12:29:33.085704490   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135315
2025-04-25 12:29:33.085705421   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:33.085706453   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:33.085707285   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176383
2025-04-25 12:29:33.085713246   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176383
2025-04-25 12:29:33.094425426   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176369
2025-04-25 12:29:33.094434363   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:33.108155556   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.715427ms remaining, needed 30ms), skipping
2025-04-25 12:29:33.193767525   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:33.194179720   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc000ae2d00})
2025-04-25 12:29:33.293823687   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:33.394778907   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:33.394866111   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:33.394884485   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:33.494232996   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176380) =======
2025-04-25 12:29:33.494236162   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:33.494237494   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:33.494238556   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2610.88m cores, Cores=24
2025-04-25 12:29:33.494239568   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16202.93 MB, Total=128728.99 MB, Usage=12.59%
2025-04-25 12:29:33.494265216   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:33.494266469   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:33.494267461   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:33.494346930   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:33.594513448   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176374
2025-04-25 12:29:33.594516093   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:33.694032149   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:33.695143978   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135360) =======
2025-04-25 12:29:33.695145221   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:33.695153797   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:33.695154628   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:33.695167212   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:33.695167913   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:33.695168755   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:33.695169396   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:33.794583430   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176376
2025-04-25 12:29:33.794588069   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:33.803633406   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:33.803638004   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:33.981112719   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:33.994503160   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176378
2025-04-25 12:29:33.994505655   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:34.294744178   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176381
2025-04-25 12:29:34.294749118   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:34.304476296   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:34.304510199   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:34.495071935   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176390) =======
2025-04-25 12:29:34.495076653   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:34.495078186   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:34.495106600   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5670.89m cores, Cores=24
2025-04-25 12:29:34.495107952   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16276.91 MB, Total=128728.99 MB, Usage=12.64%
2025-04-25 12:29:34.495109064   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:34.495110457   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:34.495111479   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:34.694670761   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135370) =======
2025-04-25 12:29:34.694674778   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:34.694675991   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:34.694677153   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:34.694721266   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:34.694722588   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:34.694723881   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:34.694724882   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:34.815085245   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176386
2025-04-25 12:29:34.815194320   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:34.894211488   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:34.894586743   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc000a6aa40})
2025-04-25 12:29:34.980340508   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.16 seconds ago
2025-04-25 12:29:34.995944078   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:35.093770809   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:35.093845099   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:35.093898910   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:35.193601747   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:35.293620297   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176391
2025-04-25 12:29:35.293623252   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:35.394206303   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176392
2025-04-25 12:29:35.394209839   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:35.494412956   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176400) =======
2025-04-25 12:29:35.494417795   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:35.494419068   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:35.494420090   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4444.44m cores, Cores=24
2025-04-25 12:29:35.494421182   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16189.74 MB, Total=128728.99 MB, Usage=12.58%
2025-04-25 12:29:35.494477007   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:35.494478570   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:35.494479632   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:35.494758946   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176393
2025-04-25 12:29:35.494767743   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:35.694093786   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176395
2025-04-25 12:29:35.694098124   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:35.719816020   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:35.719837921   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:35.902934188   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:29:35.980261039   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:35.996184420   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176398
2025-04-25 12:29:35.996187696   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:36.004963997   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp: lookup localhost: i/o timeout
2025-04-25 12:29:36.005004002   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp: lookup localhost: i/o timeout
2025-04-25 12:29:36.095640683   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135380) =======
2025-04-25 12:29:36.095643408   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:36.095644300   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:36.095673755   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:36.095674396   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:36.095675037   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:36.095675899   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:36.095676570   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:36.425356935   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:36.425362115   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:36.493505550   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176410) =======
2025-04-25 12:29:36.493507954   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:36.493508846   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:36.493509527   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5974.68m cores, Cores=24
2025-04-25 12:29:36.493527661   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16148.16 MB, Total=128728.99 MB, Usage=12.54%
2025-04-25 12:29:36.493528383   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:36.493529174   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:36.493529826   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:36.493996753   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176403
2025-04-25 12:29:36.494022000   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:36.593041262   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:36.593447095   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001cb5700})
2025-04-25 12:29:36.793297674   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:36.893593885   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:36.893663797   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:36.893680448   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:36.980970660   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:29:36.996210867   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:37.094359944   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176409
2025-04-25 12:29:37.094391133   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if log file exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:37.194924910   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176410
2025-04-25 12:29:37.194957351   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:37.293662683   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176411
2025-04-25 12:29:37.293666110   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:37.419537307   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:37.419539992   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:37.494116936   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176420) =======
2025-04-25 12:29:37.494122076   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:37.494124150   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:37.494156040   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5042.02m cores, Cores=24
2025-04-25 12:29:37.494158123   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16092.93 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:29:37.494159847   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:37.494161921   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:37.494163484   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:37.494615523   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176413
2025-04-25 12:29:37.494627405   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:37.495741348   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135390) =======
2025-04-25 12:29:37.495743032   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:37.495753401   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:37.495754904   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:37.495760735   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:37.495767998   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:37.495773709   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:37.495779220   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:37.794576824   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176416
2025-04-25 12:29:37.794580020   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:37.980400906   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:38.086321205   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176449, managers: 7
2025-04-25 12:29:38.086325563   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176433
2025-04-25 12:29:38.086371109   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176425
2025-04-25 12:29:38.086374105   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:38.086376149   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176418
2025-04-25 12:29:38.086378152   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:38.086380026   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135395
2025-04-25 12:29:38.086381979   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:38.086383873   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135353
2025-04-25 12:29:38.086385767   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135353
2025-04-25 12:29:38.086387760   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:38.086399282   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:38.086400324   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176433
2025-04-25 12:29:38.126153382   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:38.126179661   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:38.293853147   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176421
2025-04-25 12:29:38.293858427   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:38.393783020   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:38.394291044   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001766640})
2025-04-25 12:29:38.493658009   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176430) =======
2025-04-25 12:29:38.493660905   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:38.493661797   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:38.493662498   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3916.32m cores, Cores=24
2025-04-25 12:29:38.493663169   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16096.58 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:29:38.493683988   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:38.493684850   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:38.493685521   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:38.493714766   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:38.593795512   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:38.593883828   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:38.593916339   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:38.694305846   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:38.793465822   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176426
2025-04-25 12:29:38.793471262   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:38.894639171   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176427
2025-04-25 12:29:38.894645242   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:38.895964882   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135400) =======
2025-04-25 12:29:38.895968028   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:38.895980431   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:38.895981794   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:38.895988376   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:38.895995900   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:38.896002332   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:38.896008364   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:38.903819832   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:38.903822948   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:38.981027318   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:38.994525842   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:39.141620380   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.142009ms remaining, needed 30ms), skipping
2025-04-25 12:29:39.194492258   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176430
2025-04-25 12:29:39.194516323   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:39.204768047   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:39.204869257   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:39.494319867   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176440) =======
2025-04-25 12:29:39.494326419   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:39.494327872   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:39.494329104   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4117.15m cores, Cores=24
2025-04-25 12:29:39.494360163   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16018.23 MB, Total=128728.99 MB, Usage=12.44%
2025-04-25 12:29:39.494361435   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:39.494362988   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:39.494364321   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:39.494640249   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:39.895042292   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135410) =======
2025-04-25 12:29:39.895044416   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:39.895045328   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:39.895064193   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:39.895064824   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:39.895065456   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:39.895066277   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:39.895066908   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:39.980770239   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:39.994794921   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:40.092813873   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:40.093235596   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001c52100})
2025-04-25 12:29:40.294376898   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:40.393974897   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:40.394068173   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:40.394154274   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:40.494714621   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176450) =======
2025-04-25 12:29:40.494717787   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:40.494719090   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:40.494720122   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3950.62m cores, Cores=24
2025-04-25 12:29:40.494721103   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16030.81 MB, Total=128728.99 MB, Usage=12.45%
2025-04-25 12:29:40.494746331   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:40.494747603   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:40.494748575   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:40.494859584   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:40.593510042   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176444
2025-04-25 12:29:40.593513438   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:40.694176809   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176445
2025-04-25 12:29:40.694180175   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:40.794171593   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176446
2025-04-25 12:29:40.794200518   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:40.808015066   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.883593ms remaining, needed 30ms), skipping
2025-04-25 12:29:40.980827838   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:40.994417614   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176448
2025-04-25 12:29:40.994466205   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if log file exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:41.294238049   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176451
2025-04-25 12:29:41.294241595   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:41.295451168   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135420) =======
2025-04-25 12:29:41.295453382   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:41.295454715   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:41.295485753   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:41.295486815   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:41.295487747   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:41.295488999   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:41.295489951   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:41.421664227   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:41.421703441   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:41.495200475   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176460) =======
2025-04-25 12:29:41.495204853   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:41.495206196   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:41.495207448   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5491.53m cores, Cores=24
2025-04-25 12:29:41.495233908   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16092.21 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:29:41.495235150   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:41.495236753   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:41.495237955   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:41.793778094   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176456
2025-04-25 12:29:41.793781681   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:41.892867026   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:41.893214028   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002c77f60})
2025-04-25 12:29:41.980423068   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:41.994106038   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:42.094307371   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:42.094374186   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:42.094382472   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:42.193546735   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:42.294015019   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176461
2025-04-25 12:29:42.294018315   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:42.393923651   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176462
2025-04-25 12:29:42.393928430   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:42.420707579   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:42.420730563   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:42.494916951   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176470) =======
2025-04-25 12:29:42.494920278   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:42.494921790   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:42.494922953   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4700.00m cores, Cores=24
2025-04-25 12:29:42.494989167   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16025.98 MB, Total=128728.99 MB, Usage=12.45%
2025-04-25 12:29:42.494991131   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:42.495008163   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:42.495009305   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:42.495386694   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176463
2025-04-25 12:29:42.495397164   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:42.693922851   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:42.695187488   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135430) =======
2025-04-25 12:29:42.695189752   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:42.695191034   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:42.695205562   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:42.695210781   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:42.695215580   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:42.695220690   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:42.695221852   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:42.980587525   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:42.994178493   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176468
2025-04-25 12:29:42.994181248   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:43.085930338   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176499, managers: 7
2025-04-25 12:29:43.085933133   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176483
2025-04-25 12:29:43.085960214   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176483
2025-04-25 12:29:43.085961236   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176475
2025-04-25 12:29:43.085962088   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:43.085962939   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176468
2025-04-25 12:29:43.085963801   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:43.085969241   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135433
2025-04-25 12:29:43.085970153   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:43.085984640   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135391
2025-04-25 12:29:43.085985622   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135391
2025-04-25 12:29:43.085992064   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:43.085996713   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:43.119254255   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:43.119276547   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:43.494034401   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176480) =======
2025-04-25 12:29:43.494037246   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:43.494038569   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:43.494039581   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2410.04m cores, Cores=24
2025-04-25 12:29:43.494040562   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15980.08 MB, Total=128728.99 MB, Usage=12.41%
2025-04-25 12:29:43.494065990   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:43.494067192   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:43.494068154   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:43.494461914   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176473
2025-04-25 12:29:43.494496108   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:43.594235262   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:43.594583837   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001d8faa0})
2025-04-25 12:29:43.693675454   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:43.793967797   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:43.794034412   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:43.794047917   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:43.894044635   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:43.980404839   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:29:43.994434581   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176478
2025-04-25 12:29:43.994437998   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:44.094375363   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176479
2025-04-25 12:29:44.094380162   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:44.096138236   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135440) =======
2025-04-25 12:29:44.096140791   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:44.096142013   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:44.096156039   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:44.096157001   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:44.096157953   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:44.096163634   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:44.096168282   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:44.122351612   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:44.122355870   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:44.194887629   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176480
2025-04-25 12:29:44.194892990   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:44.394253375   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176482
2025-04-25 12:29:44.394258574   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:44.403933594   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:44.403938103   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:44.494434258   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176490) =======
2025-04-25 12:29:44.494437043   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:44.494437935   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:44.494438616   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4800.00m cores, Cores=24
2025-04-25 12:29:44.494457492   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16020.16 MB, Total=128728.99 MB, Usage=12.44%
2025-04-25 12:29:44.494458243   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:44.494459045   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:44.494459706   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:44.694356489   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176485
2025-04-25 12:29:44.694361098   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:44.703179688   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:44.703182273   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:44.825268948   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:44.825285569   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:44.981036521   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:45.094681955   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135450) =======
2025-04-25 12:29:45.094686233   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:45.094688236   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:45.094690120   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:45.094691623   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:45.094740064   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:45.094742318   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:45.094744021   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:45.193729468   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176490
2025-04-25 12:29:45.193732785   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:45.294700917   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:45.295260919   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002c77c40})
2025-04-25 12:29:45.393673189   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:45.494393286   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176500) =======
2025-04-25 12:29:45.494397524   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:45.494399638   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:45.494401341   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3160.49m cores, Cores=24
2025-04-25 12:29:45.494430355   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16080.87 MB, Total=128728.99 MB, Usage=12.49%
2025-04-25 12:29:45.494432229   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:45.494434383   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:45.494436086   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:45.494469258   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:45.494558917   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:45.494569687   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:45.594383981   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:45.694207313   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176495
2025-04-25 12:29:45.694210238   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:45.794388567   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176496
2025-04-25 12:29:45.794509995   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:45.822801726   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:45.822834137   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:45.893921152   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176497
2025-04-25 12:29:45.893924769   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:45.980616535   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:46.094050832   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:46.103161842   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: error reading config file benthos.yaml in service directory /run/service/benthos-dataflow-core-1: context deadline exceeded
2025-04-25 12:29:46.394888646   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176502
2025-04-25 12:29:46.394893205   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:46.493763755   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176510) =======
2025-04-25 12:29:46.493766020   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:46.493766921   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:46.493786248   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3139.24m cores, Cores=24
2025-04-25 12:29:46.493787009   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15979.87 MB, Total=128728.99 MB, Usage=12.41%
2025-04-25 12:29:46.493787660   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:46.493788462   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:46.493792730   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:46.494994558   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135460) =======
2025-04-25 12:29:46.494995470   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:46.495003595   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:46.495004326   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:46.495007763   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:46.495011229   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:46.500620590   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:46.500623555   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:46.521347156   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:46.521379977   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:46.895405924   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176507
2025-04-25 12:29:46.895409320   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:46.980087263   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:46.995016355   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:46.995427548   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002f45500})
2025-04-25 12:29:47.094607991   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:47.195416162   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:47.195495762   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:47.195511241   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:47.293222053   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:47.394121325   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176512
2025-04-25 12:29:47.394124511   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:47.494505971   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176520) =======
2025-04-25 12:29:47.494509238   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:47.494510610   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:47.494511672   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2823.53m cores, Cores=24
2025-04-25 12:29:47.494534896   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15963.95 MB, Total=128728.99 MB, Usage=12.40%
2025-04-25 12:29:47.494536008   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:47.494537200   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:47.494538172   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:47.494951208   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176513
2025-04-25 12:29:47.494961257   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:47.523677907   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:47.523680391   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:47.594920133   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176514
2025-04-25 12:29:47.595189649   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:47.794818858   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176516
2025-04-25 12:29:47.794822445   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:47.896383321   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135470) =======
2025-04-25 12:29:47.896439507   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:47.896441280   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:47.896442482   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:47.896443414   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:47.896444396   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:47.896445608   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:47.896446610   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:47.980353904   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:48.086410348   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176549, managers: 7
2025-04-25 12:29:48.086413303   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176533
2025-04-25 12:29:48.086414796   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176525
2025-04-25 12:29:48.086443280   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:48.086444472   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176518
2025-04-25 12:29:48.086445604   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:48.086446646   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135471
2025-04-25 12:29:48.086447748   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:48.086448790   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135429
2025-04-25 12:29:48.086449832   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135429
2025-04-25 12:29:48.086475741   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:48.086476903   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:48.086477915   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176533
2025-04-25 12:29:48.095411241   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176519
2025-04-25 12:29:48.095414517   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:48.239965062   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.282693ms remaining, needed 30ms), skipping
2025-04-25 12:29:48.494781749   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176530) =======
2025-04-25 12:29:48.494784194   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:48.494785075   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:48.494785767   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2722.69m cores, Cores=24
2025-04-25 12:29:48.494807558   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15979.34 MB, Total=128728.99 MB, Usage=12.41%
2025-04-25 12:29:48.494808279   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:48.494809070   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:48.494809742   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:48.593437356   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176524
2025-04-25 12:29:48.593639816   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:48.694479517   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:48.694820147   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc003143580})
2025-04-25 12:29:48.794674627   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:48.894121110   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:48.894202983   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:48.894215317   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:48.980384621   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.38 seconds ago
2025-04-25 12:29:48.995660235   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:49.093649339   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:49.194816524   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176530
2025-04-25 12:29:49.194820442   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:49.244350480   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.017595ms remaining, needed 30ms), skipping
2025-04-25 12:29:49.294850541   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176531
2025-04-25 12:29:49.294855250   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:49.296211909   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135480) =======
2025-04-25 12:29:49.296215316   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:49.296216728   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:49.296228731   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:49.296234431   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:49.296235684   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:49.296242597   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:49.296243859   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:49.303956902   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:29:49.493782109   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176540) =======
2025-04-25 12:29:49.493785555   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:49.493786717   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:49.493787409   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3815.90m cores, Cores=24
2025-04-25 12:29:49.493812125   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16013.80 MB, Total=128728.99 MB, Usage=12.44%
2025-04-25 12:29:49.493813277   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:49.493814530   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:49.493815502   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:49.494356949   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176533
2025-04-25 12:29:49.494359253   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:49.794067694   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176536
2025-04-25 12:29:49.794072613   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:49.919456042   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:49.919473284   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:49.980115042   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:50.294446369   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176541
2025-04-25 12:29:50.294450647   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:50.295768924   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135490) =======
2025-04-25 12:29:50.295770917   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:50.295772220   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:50.295783721   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:50.295784844   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:50.295786016   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:50.295787679   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:50.295794382   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:50.303433966   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: error reading config file benthos.yaml in service directory /run/service/benthos-dataflow-core-1: context deadline exceeded
2025-04-25 12:29:50.394431754   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:50.394927245   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002f58a20})
2025-04-25 12:29:50.493865292   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176550) =======
2025-04-25 12:29:50.493867957   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:50.493869299   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:50.493870331   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3268.09m cores, Cores=24
2025-04-25 12:29:50.493894947   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16066.09 MB, Total=128728.99 MB, Usage=12.48%
2025-04-25 12:29:50.493896019   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:50.493897242   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:50.493898213   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:50.493924022   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:50.594787617   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:50.594830638   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:50.594849694   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:50.694194075   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:50.794403461   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176546
2025-04-25 12:29:50.794406026   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:50.894375501   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176547
2025-04-25 12:29:50.894377905   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:50.921829478   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:50.921861157   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:50.980898399   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:50.996345024   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176548
2025-04-25 12:29:50.996468075   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:51.194351002   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176550
2025-04-25 12:29:51.194355340   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:51.494438133   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176560) =======
2025-04-25 12:29:51.494440207   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:51.494441079   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:51.494441760   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2309.62m cores, Cores=24
2025-04-25 12:29:51.494549893   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15987.45 MB, Total=128728.99 MB, Usage=12.42%
2025-04-25 12:29:51.494551476   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:51.494552889   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:51.494554071   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:51.494743136   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176553
2025-04-25 12:29:51.494754357   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:51.617582324   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:51.617612060   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:51.694575246   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135500) =======
2025-04-25 12:29:51.694577239   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:51.694578111   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:51.694578903   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:51.694598349   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:51.694599060   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:51.694599882   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:51.694600513   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:51.980759934   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:51.995136538   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176558
2025-04-25 12:29:51.995139904   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:52.093949880   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:52.094295850   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001ecfd80})
2025-04-25 12:29:52.193976552   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:52.295300352   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:52.295357570   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:52.295372598   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:52.393242758   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:52.494499873   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176570) =======
2025-04-25 12:29:52.494502057   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:52.494502948   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:52.494503660   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2677.69m cores, Cores=24
2025-04-25 12:29:52.494504321   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15999.24 MB, Total=128728.99 MB, Usage=12.43%
2025-04-25 12:29:52.494529608   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:52.494530440   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:52.494531101   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:52.494860290   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176563
2025-04-25 12:29:52.494875338   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:52.505035519   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if config file exists: context deadline exceeded
2025-04-25 12:29:52.595603058   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176564
2025-04-25 12:29:52.595630469   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:52.621909358   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:52.621912614   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:52.695211514   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176565
2025-04-25 12:29:52.695338463   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:52.894110039   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176567
2025-04-25 12:29:52.894139685   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:52.902783797   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:29:52.902787815   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:29:52.981148056   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:53.086206643   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176599, managers: 7
2025-04-25 12:29:53.086267828   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135509
2025-04-25 12:29:53.086280191   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:53.086281253   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135467
2025-04-25 12:29:53.086305819   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135467
2025-04-25 12:29:53.086306821   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:53.086307683   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:53.086308484   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176583
2025-04-25 12:29:53.086309316   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176583
2025-04-25 12:29:53.086310157   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176575
2025-04-25 12:29:53.086310999   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:53.086311831   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176568
2025-04-25 12:29:53.086312682   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:53.096162750   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135510) =======
2025-04-25 12:29:53.096165075   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:53.096187657   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:53.096188960   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:53.096189661   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:53.096190322   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:53.096191134   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:53.096191805   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:53.194662133   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176570
2025-04-25 12:29:53.194665870   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:53.323860640   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:53.323884244   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:53.493880661   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176580) =======
2025-04-25 12:29:53.493883677   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:53.493885009   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:53.493886071   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2338.98m cores, Cores=24
2025-04-25 12:29:53.493887093   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15984.88 MB, Total=128728.99 MB, Usage=12.42%
2025-04-25 12:29:53.493925655   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:53.493927128   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:53.493928110   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:53.694369203   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176575
2025-04-25 12:29:53.694471956   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:53.793829902   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:53.794678176   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002f0b340})
2025-04-25 12:29:53.894989843   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:53.980507954   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:29:53.994078473   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:53.994168282   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:53.994184843   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:54.093449513   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:54.195227616   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176580
2025-04-25 12:29:54.195230953   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:54.294271562   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176581
2025-04-25 12:29:54.294274117   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:54.322733172   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:54.322811158   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:54.394868415   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176582
2025-04-25 12:29:54.394872944   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:54.404800828   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:54.404806599   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:54.494773469   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176590) =======
2025-04-25 12:29:54.494776154   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:54.494901419   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:54.494927288   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5873.42m cores, Cores=24
2025-04-25 12:29:54.494931997   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=15990.28 MB, Total=128728.99 MB, Usage=12.42%
2025-04-25 12:29:54.494954519   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:54.494956102   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:54.494957024   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:54.496500003   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135520) =======
2025-04-25 12:29:54.496501325   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:54.496512667   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:54.496513508   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:54.496517676   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:54.496518367   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:54.496523457   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:54.496527154   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:54.595100956   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176584
2025-04-25 12:29:54.595106797   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:54.894063823   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:54.903147121   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:54.903150577   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:54.980766139   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:55.020960284   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:55.020972266   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:55.394254280   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176592
2025-04-25 12:29:55.394256855   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:55.494615250   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176600) =======
2025-04-25 12:29:55.494618867   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:55.494619749   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:55.494647721   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4800.00m cores, Cores=24
2025-04-25 12:29:55.494648493   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16017.75 MB, Total=128728.99 MB, Usage=12.44%
2025-04-25 12:29:55.494649164   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:55.494649965   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:55.494650596   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:55.494663771   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:55.495134917   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002e53ea0})
2025-04-25 12:29:55.594394978   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:55.694358962   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:55.694423023   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:55.694431148   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:55.793661473   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:55.894092726   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176597
2025-04-25 12:29:55.894220796   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:55.895513605   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135530) =======
2025-04-25 12:29:55.895514728   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:55.895522973   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:55.895523825   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:55.895535557   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:55.895536268   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:55.895550775   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:55.895551998   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:55.980659396   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:55.995592375   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176598
2025-04-25 12:29:55.995595842   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:56.042289560   [WARN]	[fsm/base_manager.go:507:507]	[DataFlowCompManagerCore]			not enough time left to reconcile instance core-1 (only 43.460702ms remaining, needed 50ms), skipping
2025-04-25 12:29:56.093318975   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176599
2025-04-25 12:29:56.093324485   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:56.295066913   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176601
2025-04-25 12:29:56.295072143   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:56.494468111   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176610) =======
2025-04-25 12:29:56.494470666   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:56.494471547   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:56.494472219   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2531.65m cores, Cores=24
2025-04-25 12:29:56.494494541   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16025.93 MB, Total=128728.99 MB, Usage=12.45%
2025-04-25 12:29:56.494495242   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:56.494496033   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:56.494496665   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:56.594060867   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176604
2025-04-25 12:29:56.594065566   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:56.895978926   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135540) =======
2025-04-25 12:29:56.895981932   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:56.895983264   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:56.896011558   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:56.896012499   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:56.896013411   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:56.896014653   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:56.896015605   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:56.903678584   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:29:56.903688092   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": net/http: timeout awaiting response headers
2025-04-25 12:29:56.981055797   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:29:57.095231485   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176609
2025-04-25 12:29:57.095235052   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:57.195603976   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:57.195996744   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0038b02e0})
2025-04-25 12:29:57.298761431   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:57.394442186   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:57.394499374   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:57.394506637   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:57.494514884   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176620) =======
2025-04-25 12:29:57.494517459   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:57.494518351   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:57.494540643   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4717.95m cores, Cores=24
2025-04-25 12:29:57.494541384   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16104.94 MB, Total=128728.99 MB, Usage=12.51%
2025-04-25 12:29:57.494542055   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:57.494542867   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:57.494543518   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:57.494748143   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:57.594562435   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176614
2025-04-25 12:29:57.594566092   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:57.625925908   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:29:57.625955323   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:57.694279485   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176615
2025-04-25 12:29:57.694283983   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:57.703993758   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if service exists: failed to check if S6 service exists: context deadline exceeded
2025-04-25 12:29:57.795015399   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176616
2025-04-25 12:29:57.795020368   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:57.980337292   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:29:57.995079583   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176618
2025-04-25 12:29:57.995083841   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:58.085690542   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176649, managers: 7
2025-04-25 12:29:58.085696123   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176625
2025-04-25 12:29:58.085729946   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:29:58.085732180   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176618
2025-04-25 12:29:58.085734154   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:29:58.085736108   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135547
2025-04-25 12:29:58.085738121   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:29:58.085740075   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135505
2025-04-25 12:29:58.085742189   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135505
2025-04-25 12:29:58.085744073   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:29:58.085769430   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:29:58.085770532   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176633
2025-04-25 12:29:58.085772025   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176633
2025-04-25 12:29:58.294988756   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176621
2025-04-25 12:29:58.294993906   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:58.328464207   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135550) =======
2025-04-25 12:29:58.328469156   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:58.328470589   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:58.328500555   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:58.328501627   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:58.328502639   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:58.328504052   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:58.328505164   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:58.339144315   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 29.705025ms remaining, needed 30ms), skipping
2025-04-25 12:29:58.494793602   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176630) =======
2025-04-25 12:29:58.494796397   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:58.494797730   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:58.494798752   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2521.01m cores, Cores=24
2025-04-25 12:29:58.494799794   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16034.71 MB, Total=128728.99 MB, Usage=12.46%
2025-04-25 12:29:58.494823398   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:58.494824691   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:58.494825682   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:58.795096325   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176626
2025-04-25 12:29:58.795098970   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:58.803803035   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": net/http: timeout awaiting response headers
2025-04-25 12:29:58.803842720   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": net/http: timeout awaiting response headers
2025-04-25 12:29:58.893927931   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:29:58.894283289   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc003c25b00})
2025-04-25 12:29:58.980389964   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:29:59.115837251   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:29:59.193878491   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:29:59.194005009   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:29:59.194019547   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:29:59.294159060   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:29:59.393698956   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176632
2025-04-25 12:29:59.393704627   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:59.493514100   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176640) =======
2025-04-25 12:29:59.493517006   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:59.493518288   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:29:59.493519280   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2655.32m cores, Cores=24
2025-04-25 12:29:59.493543766   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16049.28 MB, Total=128728.99 MB, Usage=12.47%
2025-04-25 12:29:59.493544828   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:29:59.493546020   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:29:59.493546962   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:29:59.493804215   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176633
2025-04-25 12:29:59.493933448   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:29:59.593687127   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176634
2025-04-25 12:29:59.593689691   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:59.695455891   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135560) =======
2025-04-25 12:29:59.695458035   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:29:59.695458927   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:29:59.695478875   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:29:59.695479516   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:29:59.695480147   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:29:59.695480978   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:29:59.695481620   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:29:59.794167812   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176636
2025-04-25 12:29:59.794171910   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:29:59.803723497   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:59.803735510   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:59.929015773   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:29:59.929020712   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:29:59.980924891   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:30:00.095324078   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176639
2025-04-25 12:30:00.095328086   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:00.104189807   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: error reading config file benthos.yaml in service directory /run/service/benthos-dataflow-core-1: context deadline exceeded
2025-04-25 12:30:00.494549326   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176650) =======
2025-04-25 12:30:00.494552002   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:00.494552873   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:00.494574123   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2300.00m cores, Cores=24
2025-04-25 12:30:00.494574864   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16035.42 MB, Total=128728.99 MB, Usage=12.46%
2025-04-25 12:30:00.494575516   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:00.494576307   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:00.494590213   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:00.594169864   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176644
2025-04-25 12:30:00.594176056   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:00.639297049   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.025499ms remaining, needed 30ms), skipping
2025-04-25 12:30:00.694949210   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:00.695314506   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc007956020})
2025-04-25 12:30:00.794884850   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:00.895858921   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:00.895954320   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:00.896023080   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:00.980479945   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:30:00.993971205   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:01.096207989   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176649
2025-04-25 12:30:01.096211245   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:01.097587962   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135570) =======
2025-04-25 12:30:01.097593182   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:01.097597931   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:01.097598692   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:01.097603090   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:01.097603762   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:01.097608370   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:01.097642044   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:01.105281629   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:01.105337063   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:01.194368113   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176650
2025-04-25 12:30:01.194371259   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:01.203582777   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if config file exists: context deadline exceeded
2025-04-25 12:30:01.295313570   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176651
2025-04-25 12:30:01.295316826   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:01.304988379   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if config file exists: context deadline exceeded
2025-04-25 12:30:01.493457665   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176660) =======
2025-04-25 12:30:01.493461532   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:01.493462955   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:01.493464067   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3684.65m cores, Cores=24
2025-04-25 12:30:01.493465249   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16059.11 MB, Total=128728.99 MB, Usage=12.48%
2025-04-25 12:30:01.493494534   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:01.493496127   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:01.493497279   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:01.493845213   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176653
2025-04-25 12:30:01.493847517   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:01.502924142   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: context deadline exceeded
2025-04-25 12:30:01.620607901   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:01.620632768   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:01.794328934   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176656
2025-04-25 12:30:01.794332291   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:01.980290538   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:02.095037649   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135580) =======
2025-04-25 12:30:02.095040544   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:02.095041426   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:02.095068957   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:02.095069609   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:02.095070250   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:02.095071061   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:02.095071693   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:02.294098245   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176661
2025-04-25 12:30:02.294102302   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:02.416497163   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:02.416932631   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0062e8c60})
2025-04-25 12:30:02.495535116   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176670) =======
2025-04-25 12:30:02.495539194   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:02.495540516   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:02.495541548   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3850.27m cores, Cores=24
2025-04-25 12:30:02.495566605   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16053.66 MB, Total=128728.99 MB, Usage=12.47%
2025-04-25 12:30:02.495567797   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:02.495569170   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:02.495570272   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:02.593531702   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:02.695479182   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:02.695556327   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:02.695574862   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:02.795118696   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:02.894443838   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176667
2025-04-25 12:30:02.894447766   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:02.903143766   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:02.903174373   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:02.980660650   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:30:02.995441464   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176668
2025-04-25 12:30:02.995444920   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:03.085712473   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176699, managers: 7
2025-04-25 12:30:03.085797493   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176683
2025-04-25 12:30:03.085816739   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176683
2025-04-25 12:30:03.085817941   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176675
2025-04-25 12:30:03.085818833   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:30:03.085819494   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176668
2025-04-25 12:30:03.085820186   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:30:03.085820837   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135585
2025-04-25 12:30:03.085821528   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:30:03.085822179   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135543
2025-04-25 12:30:03.085822851   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135543
2025-04-25 12:30:03.085823572   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:30:03.085824253   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:30:03.095987380   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176669
2025-04-25 12:30:03.095990546   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:03.294594905   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176671
2025-04-25 12:30:03.294598682   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:03.324108721   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:03.324134430   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:03.493923274   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176680) =======
2025-04-25 12:30:03.493926389   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:03.493927271   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:03.493951276   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3025.21m cores, Cores=24
2025-04-25 12:30:03.493952018   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16081.55 MB, Total=128728.99 MB, Usage=12.49%
2025-04-25 12:30:03.493952699   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:03.493953490   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:03.493954132   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:03.495795592   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135590) =======
2025-04-25 12:30:03.495797906   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:03.495812714   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:03.495813556   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:03.495833814   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:03.495838843   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:03.495848431   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:03.495853821   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:03.595659847   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176674
2025-04-25 12:30:03.595677971   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:03.700244022   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:03.700306399   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:03.980397151   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:04.120407441   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176679
2025-04-25 12:30:04.120412360   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:04.129833963   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:04.129837309   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:04.194428978   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:04.194761863   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0010bc700})
2025-04-25 12:30:04.293660284   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:04.394322669   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:04.394414882   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:04.394435130   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:04.495147168   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176690) =======
2025-04-25 12:30:04.495151376   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:04.495153059   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:04.495154282   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5848.74m cores, Cores=24
2025-04-25 12:30:04.495155504   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16096.77 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:30:04.495187304   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:04.495189087   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:04.495190259   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:04.495373343   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:04.594409773   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176684
2025-04-25 12:30:04.594455930   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:04.602611544   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:04.602639647   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:04.694678939   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176685
2025-04-25 12:30:04.694682105   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:04.795219986   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176686
2025-04-25 12:30:04.795224114   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:04.894983192   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135600) =======
2025-04-25 12:30:04.894985546   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:04.894986428   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:04.895006465   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:04.895007076   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:04.895007698   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:04.895008519   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:04.895009160   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:04.980957308   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:04.995175173   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176688
2025-04-25 12:30:04.995178720   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:05.021321622   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:05.021328255   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:05.205070577   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:30:05.293998793   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:05.494884919   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176700) =======
2025-04-25 12:30:05.494888796   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:05.494890128   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:05.494891120   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2924.37m cores, Cores=24
2025-04-25 12:30:05.494892142   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16099.20 MB, Total=128728.99 MB, Usage=12.51%
2025-04-25 12:30:05.494929633   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:05.494930955   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:05.494931927   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:05.738234563   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 29.692542ms remaining, needed 30ms), skipping
2025-04-25 12:30:05.794930505   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176696
2025-04-25 12:30:05.794971592   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:05.895310079   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:05.895717444   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc000e6b120})
2025-04-25 12:30:05.980782952   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:30:05.995515976   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:06.095983544   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:06.096058135   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:06.096065809   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:06.195078063   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:06.295266237   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176701
2025-04-25 12:30:06.295270124   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:06.296849362   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135610) =======
2025-04-25 12:30:06.296851325   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:06.296875200   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:06.296877294   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:06.296885850   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:06.296887213   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:06.296888796   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:06.296894116   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:06.305294129   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:06.305320309   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:06.395472064   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176702
2025-04-25 12:30:06.395583623   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:30:06.404512831   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:06.404519263   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:06.493833024   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176710) =======
2025-04-25 12:30:06.493835559   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:06.493836431   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:06.493859574   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3880.85m cores, Cores=24
2025-04-25 12:30:06.493860376   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16115.28 MB, Total=128728.99 MB, Usage=12.52%
2025-04-25 12:30:06.493861037   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:06.493861828   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:06.493862439   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:06.494612288   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176703
2025-04-25 12:30:06.494623590   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:06.623922963   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:06.623927381   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:06.694071202   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if log file exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:06.980552360   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:06.995129290   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176708
2025-04-25 12:30:06.995133268   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:07.295127467   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135620) =======
2025-04-25 12:30:07.295129901   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:07.295130743   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:07.295162673   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:07.295163324   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:07.295163945   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:07.295164787   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:07.295165438   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:07.323465614   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:07.323470303   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:07.494236783   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176720) =======
2025-04-25 12:30:07.494240690   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:07.494242023   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:07.494243075   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2722.69m cores, Cores=24
2025-04-25 12:30:07.494244187   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16111.45 MB, Total=128728.99 MB, Usage=12.52%
2025-04-25 12:30:07.494270196   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:07.494271578   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:07.494272640   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:07.494586079   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176713
2025-04-25 12:30:07.494595687   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:07.593298249   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:07.593659358   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0001fd700})
2025-04-25 12:30:07.894749356   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:07.980287181   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:30:07.996802093   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:07.996825567   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:07.996850624   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:08.085666739   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176749, managers: 7
2025-04-25 12:30:08.085670015   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176733
2025-04-25 12:30:08.085670967   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176733
2025-04-25 12:30:08.085717274   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176725
2025-04-25 12:30:08.085719088   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:30:08.085720260   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176718
2025-04-25 12:30:08.085721422   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: creating, desired state: active
2025-04-25 12:30:08.085722534   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135624
2025-04-25 12:30:08.085723686   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:30:08.085724698   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135582
2025-04-25 12:30:08.085725770   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135582
2025-04-25 12:30:08.085726962   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:30:08.085727984   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:30:08.094859302   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:08.195020645   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176720
2025-04-25 12:30:08.195025073   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:08.294268241   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176721
2025-04-25 12:30:08.294279392   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:08.340276551   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.103818ms remaining, needed 30ms), skipping
2025-04-25 12:30:08.393981302   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176722
2025-04-25 12:30:08.393986511   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:08.494642284   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176730) =======
2025-04-25 12:30:08.494645409   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:08.494646301   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:08.494647002   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=5412.77m cores, Cores=24
2025-04-25 12:30:08.494667541   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16145.45 MB, Total=128728.99 MB, Usage=12.54%
2025-04-25 12:30:08.494668252   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:08.494669054   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:08.494669705   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:08.595850604   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176724
2025-04-25 12:30:08.595876683   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:08.605478675   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:08.605483394   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:08.697415243   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135630) =======
2025-04-25 12:30:08.697418059   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:08.697419411   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:08.697443767   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:08.697444749   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:08.697445691   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:08.697453215   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:08.697454217   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:08.893581840   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176727
2025-04-25 12:30:08.893584485   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:08.902886654   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:08.902924555   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:08.980700666   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:30:09.022481091   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:09.022508273   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:09.203246439   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:09.203253593   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:09.394633054   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176732
2025-04-25 12:30:09.394655556   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:09.403932648   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:30:09.494080375   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176740) =======
2025-04-25 12:30:09.494085374   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:09.494087428   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:09.494089222   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4307.69m cores, Cores=24
2025-04-25 12:30:09.494091195   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16168.80 MB, Total=128728.99 MB, Usage=12.56%
2025-04-25 12:30:09.494121532   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:09.494123857   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:09.494125600   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:09.494127664   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:09.494497409   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002f44100})
2025-04-25 12:30:09.593462273   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:09.694519479   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:09.694572599   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:09.694581506   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:09.794408541   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:09.894175572   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176737
2025-04-25 12:30:09.894190681   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:09.980446886   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:09.994314854   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176738
2025-04-25 12:30:09.994320204   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:10.045087585   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.461871ms remaining, needed 30ms), skipping
2025-04-25 12:30:10.094522023   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176739
2025-04-25 12:30:10.094525679   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:10.096343846   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135640) =======
2025-04-25 12:30:10.096346310   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:10.096347663   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:10.096363072   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:10.096364004   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:10.096369945   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:10.096375435   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:10.096380585   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:10.103521252   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:10.103530519   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get version: failed to execute request for http://localhost:9001/version: Get "http://localhost:9001/version": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:10.294491083   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176741
2025-04-25 12:30:10.294494099   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:10.493924558   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176750) =======
2025-04-25 12:30:10.493928275   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:10.493929667   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:10.493953662   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2553.19m cores, Cores=24
2025-04-25 12:30:10.493954915   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16136.30 MB, Total=128728.99 MB, Usage=12.54%
2025-04-25 12:30:10.493956057   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:10.493957339   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:10.493958331   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:10.594766049   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176744
2025-04-25 12:30:10.594769234   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:10.723303971   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:10.723325110   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:10.980773044   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:11.094364722   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176749
2025-04-25 12:30:11.094369220   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:11.095809046   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135650) =======
2025-04-25 12:30:11.095811119   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:11.095812412   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:11.095822100   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:11.095828171   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:11.095834844   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:11.095840324   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:11.095845624   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:11.193538328   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:11.193928602   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc00342ce00})
2025-04-25 12:30:11.440473227   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 29.898559ms remaining, needed 30ms), skipping
2025-04-25 12:30:11.494802452   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176760) =======
2025-04-25 12:30:11.494805237   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:11.494806529   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:11.494807561   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4271.19m cores, Cores=24
2025-04-25 12:30:11.494830675   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16102.49 MB, Total=128728.99 MB, Usage=12.51%
2025-04-25 12:30:11.494831757   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:11.494832959   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:11.494833931   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:11.494849790   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:11.594656487   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:11.594783726   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:11.594843428   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:11.694506284   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:11.794666424   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176756
2025-04-25 12:30:11.794671093   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:11.804244030   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:11.804258467   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:11.894994019   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176757
2025-04-25 12:30:11.894996343   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:11.980811459   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:11.996047317   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176758
2025-04-25 12:30:11.996070781   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:12.193623642   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176760
2025-04-25 12:30:12.193647136   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:12.341693323   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.143592ms remaining, needed 30ms), skipping
2025-04-25 12:30:12.494388877   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176770) =======
2025-04-25 12:30:12.494391232   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:12.494392103   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:12.494392795   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2500.00m cores, Cores=24
2025-04-25 12:30:12.494412842   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16091.34 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:30:12.494413534   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:12.494414335   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:12.494414976   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:12.494690223   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176763
2025-04-25 12:30:12.494696615   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:12.496207614   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135660) =======
2025-04-25 12:30:12.496225428   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:12.496231790   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:12.496243522   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:12.496250475   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:12.496257147   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:12.496268789   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:12.496275722   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:12.504053527   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:12.504079576   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:12.980661049   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:12.995718802   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176768
2025-04-25 12:30:12.995722960   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:13.040990457   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.007165ms remaining, needed 30ms), skipping
2025-04-25 12:30:13.085975784   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176799, managers: 7
2025-04-25 12:30:13.085978589   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176768
2025-04-25 12:30:13.086002584   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:30:13.086004227   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135665
2025-04-25 12:30:13.086005850   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:30:13.086007564   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135623
2025-04-25 12:30:13.086009046   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135623
2025-04-25 12:30:13.086010439   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:30:13.086011922   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:30:13.086013184   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176783
2025-04-25 12:30:13.086014587   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176783
2025-04-25 12:30:13.086034103   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176775
2025-04-25 12:30:13.086035887   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:30:13.094820964   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:13.095224693   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc007713120})
2025-04-25 12:30:13.193852563   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:13.294792859   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:13.294872589   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:13.294892887   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:13.394057857   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:13.494760797   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176780) =======
2025-04-25 12:30:13.494762911   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:13.494763833   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:13.494786996   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3200.00m cores, Cores=24
2025-04-25 12:30:13.494787768   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16090.10 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:30:13.494788439   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:13.494789230   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:13.494789872   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:13.494969980   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:13.595260706   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176774
2025-04-25 12:30:13.595263932   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:13.694810980   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176775
2025-04-25 12:30:13.694815258   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:13.895543343   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176777
2025-04-25 12:30:13.895579662   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:13.896995713   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135670) =======
2025-04-25 12:30:13.896997015   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:13.896997857   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:13.897006834   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:13.897010771   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:13.897011442   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:13.897023595   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:13.897024306   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:13.980534271   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:14.193876529   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get logs: failed to check if service exists: failed to check if S6 service exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:14.203985183   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if service exists: failed to check if S6 service exists: context deadline exceeded
2025-04-25 12:30:14.403765898   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:14.403769465   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:14.494066302   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176790) =======
2025-04-25 12:30:14.494069097   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:14.494070419   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:14.494071431   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3037.97m cores, Cores=24
2025-04-25 12:30:14.494072443   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16112.84 MB, Total=128728.99 MB, Usage=12.52%
2025-04-25 12:30:14.494096969   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:14.494098232   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:14.494099214   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:14.624734185   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:14.624761076   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:14.695571528   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176785
2025-04-25 12:30:14.695690622   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:30:14.793817691   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:14.794337208   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0019d1ae0})
2025-04-25 12:30:14.894000564   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:14.980459940   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.28 seconds ago
2025-04-25 12:30:14.994361461   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:14.994456710   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:14.994475936   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:15.093971908   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:15.195680767   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176790
2025-04-25 12:30:15.195686618   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:15.294272450   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176791
2025-04-25 12:30:15.294276998   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:15.295729227   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135680) =======
2025-04-25 12:30:15.295731571   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:15.295732663   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:15.295742331   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:15.295743323   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:15.295748393   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:15.295753252   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:15.295759734   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:15.394217946   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176792
2025-04-25 12:30:15.394223136   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:15.493743934   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176800) =======
2025-04-25 12:30:15.493746810   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:15.493748142   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:15.493749154   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3848.10m cores, Cores=24
2025-04-25 12:30:15.493773620   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16086.66 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:30:15.493774742   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:15.493775954   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:15.493776926   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:15.617685529   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176794
2025-04-25 12:30:15.617690128   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:15.627225464   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:15.627250241   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:15.895117209   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176797
2025-04-25 12:30:15.895122389   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:15.980866051   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:16.222340387   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:16.222367639   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:16.296559234   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135690) =======
2025-04-25 12:30:16.296562139   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:16.296563041   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:16.296563842   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:16.296583309   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:16.296584020   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:16.296584862   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:16.296585503   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:16.303917640   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:16.303921016   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: get metrics: failed to execute request for http://localhost:9001/metrics: Get "http://localhost:9001/metrics": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:16.394724164   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176802
2025-04-25 12:30:16.394729294   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:16.494422115   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176810) =======
2025-04-25 12:30:16.494424379   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:16.494425261   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:16.494425922   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3327.73m cores, Cores=24
2025-04-25 12:30:16.494451190   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16099.89 MB, Total=128728.99 MB, Usage=12.51%
2025-04-25 12:30:16.494452422   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:16.494454005   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:16.494455107   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:16.494461048   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:16.494802390   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0014610e0})
2025-04-25 12:30:16.593920581   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:16.695100858   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:16.695169597   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:16.695191969   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:16.794783580   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:16.894990277   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176807
2025-04-25 12:30:16.894996439   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:16.981039543   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:16.994930163   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176808
2025-04-25 12:30:16.994934060   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:17.094789206   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176809
2025-04-25 12:30:17.094807400   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:17.241155436   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.26532ms remaining, needed 30ms), skipping
2025-04-25 12:30:17.293970775   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176811
2025-04-25 12:30:17.294001854   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to check if log file exists: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:17.495324436   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176820) =======
2025-04-25 12:30:17.495329425   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:17.495330978   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:17.495332120   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2912.13m cores, Cores=24
2025-04-25 12:30:17.495361095   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16090.06 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:30:17.495362527   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:17.495363870   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:17.495364972   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:17.594974877   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176814
2025-04-25 12:30:17.594977392   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:17.695767655   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135700) =======
2025-04-25 12:30:17.695769789   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:17.695770721   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:17.695790358   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:17.695791029   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:17.695791661   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:17.695792482   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:17.695793133   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:17.980356303   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.05 seconds ago
2025-04-25 12:30:18.086506798   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176849, managers: 7
2025-04-25 12:30:18.086510014   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176818
2025-04-25 12:30:18.086511267   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:30:18.086585456   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135703
2025-04-25 12:30:18.086587309   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:30:18.086588462   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135661
2025-04-25 12:30:18.086589494   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135661
2025-04-25 12:30:18.086590415   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:30:18.086591297   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:30:18.086592088   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176833
2025-04-25 12:30:18.086592920   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176833
2025-04-25 12:30:18.086593752   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176825
2025-04-25 12:30:18.086594573   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:30:18.096521095   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176819
2025-04-25 12:30:18.096659144   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:18.193511478   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:18.193854352   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001f8ec20})
2025-04-25 12:30:18.296803953   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:18.395995081   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:18.396057388   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:18.396072366   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:18.494651084   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176830) =======
2025-04-25 12:30:18.494654471   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:18.494655843   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:18.494656865   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3139.24m cores, Cores=24
2025-04-25 12:30:18.494680800   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16090.26 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:30:18.494681882   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:18.494683105   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:18.494684086   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:18.494773214   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:18.594509326   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176824
2025-04-25 12:30:18.594512001   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:18.694407303   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176825
2025-04-25 12:30:18.694410849   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:18.816012918   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176826
2025-04-25 12:30:18.816015713   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:18.980752751   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:18.995130997   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176828
2025-04-25 12:30:18.995134314   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:19.095544442   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135710) =======
2025-04-25 12:30:19.095549572   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:19.095551826   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:19.095553890   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:19.095583967   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:19.095586111   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:19.095587684   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:19.095588635   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:19.203736083   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:19.203774865   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:19.294301313   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176831
2025-04-25 12:30:19.294303838   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:19.440465132   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.354839ms remaining, needed 30ms), skipping
2025-04-25 12:30:19.493990625   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176840) =======
2025-04-25 12:30:19.493992910   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:19.493993801   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:19.493994523   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4697.87m cores, Cores=24
2025-04-25 12:30:19.494015312   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16100.11 MB, Total=128728.99 MB, Usage=12.51%
2025-04-25 12:30:19.494016033   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:19.494016835   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:19.494017486   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:19.503296832   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if service exists: failed to check if S6 service exists: context deadline exceeded
2025-04-25 12:30:19.795601967   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176836
2025-04-25 12:30:19.795605514   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:19.893859882   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:19.894410567   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0030791a0})
2025-04-25 12:30:19.980336951   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:30:19.995407338   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:20.094322297   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:20.094387881   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:20.094389884   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:20.194504077   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:20.294871355   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176841
2025-04-25 12:30:20.294874751   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:20.331272452   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:20.331277942   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:20.394390630   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176842
2025-04-25 12:30:20.394394236   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:20.494030421   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176850) =======
2025-04-25 12:30:20.494033276   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:20.494034629   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:20.494035641   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3327.73m cores, Cores=24
2025-04-25 12:30:20.494062702   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16081.15 MB, Total=128728.99 MB, Usage=12.49%
2025-04-25 12:30:20.494063804   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:20.494064996   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:20.494065938   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:20.494313212   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176843
2025-04-25 12:30:20.494323101   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:20.495851904   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135720) =======
2025-04-25 12:30:20.495853346   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:20.495854448   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:20.495865509   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:20.495866421   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:20.495867393   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:20.495872973   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:20.495879706   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:20.694286738   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176845
2025-04-25 12:30:20.694290636   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:20.704502894   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if config file exists: context deadline exceeded
2025-04-25 12:30:20.981032448   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:20.994478723   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176848
2025-04-25 12:30:20.994481949   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:21.041195783   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.694137ms remaining, needed 30ms), skipping
2025-04-25 12:30:21.493122414   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176860) =======
2025-04-25 12:30:21.493125109   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:21.493126411   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:21.493127433   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3645.57m cores, Cores=24
2025-04-25 12:30:21.493150066   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16095.68 MB, Total=128728.99 MB, Usage=12.50%
2025-04-25 12:30:21.493151118   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:21.493152330   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:21.493153312   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:21.493525682   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176853
2025-04-25 12:30:21.493540970   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:21.495278786   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135730) =======
2025-04-25 12:30:21.495280559   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:21.495291019   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:21.495291820   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:21.495295637   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:21.495296299   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:21.495301098   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:21.495304775   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:21.595095689   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:21.595659108   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc003a6e0a0})
2025-04-25 12:30:21.724014094   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:21.724019464   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:21.794366886   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:21.893967834   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:21.894046973   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:21.894059316   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:21.980326099   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:30:21.995087716   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:22.095673685   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176859
2025-04-25 12:30:22.095701276   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:22.195613158   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176860
2025-04-25 12:30:22.195617216   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:22.293612728   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176861
2025-04-25 12:30:22.293640931   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to get logs: failed to read log file: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:22.495180168   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176870) =======
2025-04-25 12:30:22.495183374   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:22.495184236   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:22.495184917   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=6700.00m cores, Cores=24
2025-04-25 12:30:22.495210615   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16179.00 MB, Total=128728.99 MB, Usage=12.57%
2025-04-25 12:30:22.495211347   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:22.495212158   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:22.495212819   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:22.495521309   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176863
2025-04-25 12:30:22.495524926   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:22.505259888   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:22.505363012   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: failed readiness check: failed to execute request for http://localhost:9001/ready: Get "http://localhost:9001/ready": dial tcp [::1]:9001: i/o timeout
2025-04-25 12:30:22.636873647   [WARN]	[control/loop.go:320:320]	[ControlLoop]			Skipping reconcile cycle due to remaining time: 49.18997ms
2025-04-25 12:30:22.709447742   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.080424ms remaining, needed 30ms), skipping
2025-04-25 12:30:22.795662077   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176866
2025-04-25 12:30:22.795758959   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:30:22.895186280   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135740) =======
2025-04-25 12:30:22.895189577   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:22.895190919   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:22.895192161   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:22.895215736   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:22.895216908   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:22.895218311   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:22.895219383   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:22.903673077   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: context deadline exceeded
2025-04-25 12:30:22.980756455   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:30:23.086224497   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176899, managers: 7
2025-04-25 12:30:23.086228965   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176883
2025-04-25 12:30:23.086230739   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176883
2025-04-25 12:30:23.086232251   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176875
2025-04-25 12:30:23.086260244   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:30:23.086261837   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176868
2025-04-25 12:30:23.086263260   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:30:23.086264612   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135741
2025-04-25 12:30:23.086266205   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:30:23.086267698   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135698
2025-04-25 12:30:23.086269171   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135698
2025-04-25 12:30:23.086270664   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:30:23.086272116   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:30:23.293393663   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176871
2025-04-25 12:30:23.293398171   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:23.316920118   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:23.316949944   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:23.394091491   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:23.394434305   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc006bf9e80})
2025-04-25 12:30:23.493251370   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176880) =======
2025-04-25 12:30:23.493254526   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:23.493255859   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:23.493256861   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1915.97m cores, Cores=24
2025-04-25 12:30:23.493278611   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16144.48 MB, Total=128728.99 MB, Usage=12.54%
2025-04-25 12:30:23.493279683   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:23.493280886   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:23.493281868   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:23.594436124   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:23.694254480   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:23.694354478   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:23.694369677   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:23.794357802   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:23.894069768   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176877
2025-04-25 12:30:23.894074798   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:23.980096250   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:23.995591695   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176878
2025-04-25 12:30:23.995607645   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:24.094313451   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176879
2025-04-25 12:30:24.094316677   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:24.294533559   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176881
2025-04-25 12:30:24.294536465   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:24.325263520   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135750) =======
2025-04-25 12:30:24.325265794   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:24.325266616   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:24.325289569   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:24.325290210   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:24.325290852   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:24.325291703   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:24.325292344   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:24.337297161   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.2757ms remaining, needed 30ms), skipping
2025-04-25 12:30:24.493631960   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176890) =======
2025-04-25 12:30:24.493635126   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:24.493636449   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:24.493637420   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=903.77m cores, Cores=24
2025-04-25 12:30:24.493661235   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16148.62 MB, Total=128728.99 MB, Usage=12.54%
2025-04-25 12:30:24.493662297   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:24.493663449   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:24.493664431   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:24.595425828   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176884
2025-04-25 12:30:24.595429485   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:24.980934699   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:25.093994445   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176889
2025-04-25 12:30:25.094015315   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:25.194137652   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:25.194482029   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0014104c0})
2025-04-25 12:30:25.294359646   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:25.394663124   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:25.394745799   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:25.394768211   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:25.494144577   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176900) =======
2025-04-25 12:30:25.494147292   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:25.494148614   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:25.494149646   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2016.81m cores, Cores=24
2025-04-25 12:30:25.494171357   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16161.92 MB, Total=128728.99 MB, Usage=12.55%
2025-04-25 12:30:25.494172399   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:25.494173581   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:25.494174533   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:25.494247540   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:25.594486687   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176894
2025-04-25 12:30:25.594491355   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:25.694737395   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176895
2025-04-25 12:30:25.694850107   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated (current state: stopped, desired state: active)
2025-04-25 12:30:25.696622928   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135760) =======
2025-04-25 12:30:25.696624300   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:25.696634950   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:25.696635762   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:25.696639770   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:25.696640441   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:25.696652714   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:25.696653415   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:25.793667622   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176896
2025-04-25 12:30:25.793672642   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:25.980317632   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.06 seconds ago
2025-04-25 12:30:25.994435800   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176898
2025-04-25 12:30:25.994440589   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:26.294386319   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176901
2025-04-25 12:30:26.294389064   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:26.493763753   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176910) =======
2025-04-25 12:30:26.493766648   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:26.493767941   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:26.493768953   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2100.00m cores, Cores=24
2025-04-25 12:30:26.493791816   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16166.07 MB, Total=128728.99 MB, Usage=12.56%
2025-04-25 12:30:26.493792888   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:26.493794070   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:26.493795042   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:26.695054973   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135770) =======
2025-04-25 12:30:26.695057719   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:26.695059031   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:26.695060183   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:26.695083627   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:26.695084659   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:26.695085902   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:26.695086823   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:26.702433337   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if service exists: failed to check if S6 service exists: context deadline exceeded
2025-04-25 12:30:26.794034654   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176906
2025-04-25 12:30:26.794037299   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:26.894765414   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:26.895156228   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002de67e0})
2025-04-25 12:30:26.980975320   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:30:26.995984882   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:27.094035085   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:27.094109245   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:27.094118913   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:27.193671248   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:27.295048593   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176911
2025-04-25 12:30:27.295052160   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:27.394428776   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176912
2025-04-25 12:30:27.394433745   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:27.419326628   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:27.419349140   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:27.494096238   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176920) =======
2025-04-25 12:30:27.494098953   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:27.494099864   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:27.494100556   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3645.57m cores, Cores=24
2025-04-25 12:30:27.494101227   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16165.59 MB, Total=128728.99 MB, Usage=12.56%
2025-04-25 12:30:27.494101868   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:27.494102650   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:27.494130843   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:27.494370483   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176913
2025-04-25 12:30:27.494392985   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:27.503882095   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:27.503919004   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:27.694416313   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176915
2025-04-25 12:30:27.694419429   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:27.980343542   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:27.994935870   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176918
2025-04-25 12:30:27.994939567   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:28.086207216   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176949, managers: 7
2025-04-25 12:30:28.086212597   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135736
2025-04-25 12:30:28.086214861   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135736
2025-04-25 12:30:28.086249456   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:30:28.086251480   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:30:28.086253283   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176933
2025-04-25 12:30:28.086255106   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176933
2025-04-25 12:30:28.086257050   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176925
2025-04-25 12:30:28.086258914   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:30:28.086260677   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176918
2025-04-25 12:30:28.086270085   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:30:28.086272058   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135779
2025-04-25 12:30:28.086273952   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:30:28.114064910   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135780) =======
2025-04-25 12:30:28.114067074   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:28.114067986   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:28.114087232   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:28.114088424   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:28.114089446   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:28.114090809   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:28.114091871   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:28.494557844   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176930) =======
2025-04-25 12:30:28.494562173   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:28.494563495   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:28.494564667   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=1493.78m cores, Cores=24
2025-04-25 12:30:28.494597439   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16169.82 MB, Total=128728.99 MB, Usage=12.56%
2025-04-25 12:30:28.494598761   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:28.494600114   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:28.494601156   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:28.494921358   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176923
2025-04-25 12:30:28.494932669   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:28.505735567   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:28.505809616   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:28.595036070   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:28.595415773   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0035a8400})
2025-04-25 12:30:28.894589942   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:28.981170754   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.16 seconds ago
2025-04-25 12:30:28.994384463   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:28.994465174   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:28.994480483   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:29.093411611   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:29.195117323   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176930
2025-04-25 12:30:29.195152970   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:29.295210795   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176931
2025-04-25 12:30:29.295213881   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:29.395305300   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176932
2025-04-25 12:30:29.395308395   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:29.494635478   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176940) =======
2025-04-25 12:30:29.494638163   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:29.494639445   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:29.494640437   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2430.38m cores, Cores=24
2025-04-25 12:30:29.494664502   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16143.61 MB, Total=128728.99 MB, Usage=12.54%
2025-04-25 12:30:29.494665554   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:29.494666716   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:29.494667668   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:29.496182024   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135790) =======
2025-04-25 12:30:29.496183096   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:29.496190089   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:29.496193916   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:29.496197292   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:29.496200679   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:29.496211219   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:29.496211900   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:29.594564040   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176934
2025-04-25 12:30:29.594567407   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:29.622437674   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:29.622442793   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:29.894129083   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176937
2025-04-25 12:30:29.894132359   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:29.980674108   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:30.395706149   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176942
2025-04-25 12:30:30.395709375   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:30.493500080   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176950) =======
2025-04-25 12:30:30.493503687   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:30.493504980   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:30.493506011   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2564.10m cores, Cores=24
2025-04-25 12:30:30.493531900   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16113.93 MB, Total=128728.99 MB, Usage=12.52%
2025-04-25 12:30:30.493532952   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:30.493534164   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:30.493535146   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:30.493536218   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:30.494005290   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0013850e0})
2025-04-25 12:30:30.594766426   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:30.694821897   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:30.694913018   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:30.694934298   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:30.795581260   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:30.894193469   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176947
2025-04-25 12:30:30.894197016   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:30.896150998   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135800) =======
2025-04-25 12:30:30.896152901   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:30.896153993   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:30.896169262   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:30.896170244   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:30.896171226   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:30.896172478   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:30.896198477   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:30.980656681   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:30.994749891   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176948
2025-04-25 12:30:30.994753158   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:31.094101700   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176949
2025-04-25 12:30:31.094105116   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:31.238264436   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.916415ms remaining, needed 30ms), skipping
2025-04-25 12:30:31.296162084   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176951
2025-04-25 12:30:31.296164418   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:31.493766445   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176960) =======
2025-04-25 12:30:31.493768579   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:31.493769451   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:31.493770112   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2734.18m cores, Cores=24
2025-04-25 12:30:31.493770784   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16160.08 MB, Total=128728.99 MB, Usage=12.55%
2025-04-25 12:30:31.493771465   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:31.493772306   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:31.493772978   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:31.594371569   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176954
2025-04-25 12:30:31.594374113   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:31.896235650   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135810) =======
2025-04-25 12:30:31.896238476   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:31.896239818   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:31.896240950   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:31.896274824   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:31.896276407   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:31.896277649   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:31.896278611   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:31.908543426   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 27.927115ms remaining, needed 30ms), skipping
2025-04-25 12:30:31.981027311   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:30:32.094445680   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176959
2025-04-25 12:30:32.094450669   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:32.194468519   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:32.194835178   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc001eba640})
2025-04-25 12:30:32.293570358   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:32.393751194   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:32.393841854   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:32.393851943   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:32.493799721   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176970) =======
2025-04-25 12:30:32.493802436   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:32.493803839   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:32.493804911   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=3400.00m cores, Cores=24
2025-04-25 12:30:32.493805913   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16123.82 MB, Total=128728.99 MB, Usage=12.53%
2025-04-25 12:30:32.493828205   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:32.493829477   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:32.493830479   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:32.493955153   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:32.595065866   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176964
2025-04-25 12:30:32.595069794   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:32.694685708   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176965
2025-04-25 12:30:32.694689385   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:32.794514833   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176966
2025-04-25 12:30:32.794518159   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:32.820601549   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1-faulty]			error updating observed state for dataflow-core-1-faulty: failed to get health check: context deadline exceeded
2025-04-25 12:30:32.820626707   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1-faulty]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:32.981025199   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:32.994762191   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176968
2025-04-25 12:30:32.994765256   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:33.085603908   [INFO]	[cmd/main.go:148:148]	[SnapshotLogger]			System snapshot at tick 176999, managers: 7
2025-04-25 12:30:33.085606694   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: S6ManagerCore, instances: 0, tick: 176983
2025-04-25 12:30:33.085607986   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: BenthosManagerCore, instances: 0, tick: 176983
2025-04-25 12:30:33.085609138   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: ContainerManager_Core, instances: 1, tick: 176975
2025-04-25 12:30:33.085641349   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: Core, current state: active, desired state: active
2025-04-25 12:30:33.085642421   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: RedpandaManagerCore, instances: 1, tick: 176968
2025-04-25 12:30:33.085643433   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: redpanda, current state: stopped, desired state: active
2025-04-25 12:30:33.085644395   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: AgentManager_Core, instances: 1, tick: 135817
2025-04-25 12:30:33.085645346   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: agent, current state: active, desired state: active
2025-04-25 12:30:33.085646298   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: NmapCompManager_Core, instances: 0, tick: 135774
2025-04-25 12:30:33.085647260   [INFO]	[cmd/main.go:154:154]	[SnapshotLogger]			Manager: DataFlowCompManagerCore, instances: 2, tick: 135774
2025-04-25 12:30:33.085648222   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1, current state: active, desired state: active
2025-04-25 12:30:33.085649244   [INFO]	[cmd/main.go:159:159]	[SnapshotLogger]			Instance: core-1-faulty, current state: starting, desired state: active
2025-04-25 12:30:33.295427038   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176971
2025-04-25 12:30:33.295434412   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:33.297825174   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135820) =======
2025-04-25 12:30:33.297826927   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:33.297827679   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:33.297848297   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:33.297848939   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:33.297849560   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:33.297850421   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:33.297854539   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:33.514749917   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176980) =======
2025-04-25 12:30:33.514753754   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:33.514755107   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:33.514756149   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2916.67m cores, Cores=24
2025-04-25 12:30:33.514757161   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16151.90 MB, Total=128728.99 MB, Usage=12.55%
2025-04-25 12:30:33.514781897   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:33.514783140   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:33.514784101   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:33.794546484   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176976
2025-04-25 12:30:33.794549820   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:33.894506555   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:33.894867443   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc002de8d00})
2025-04-25 12:30:33.980823581   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.18 seconds ago
2025-04-25 12:30:33.995817424   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:34.094783037   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:34.094843290   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:34.094857066   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:34.195453793   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:34.295222985   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176981
2025-04-25 12:30:34.295225861   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:34.393643784   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176982
2025-04-25 12:30:34.393647541   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:34.440631632   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 28.852805ms remaining, needed 30ms), skipping
2025-04-25 12:30:34.494371257   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 176990) =======
2025-04-25 12:30:34.494373391   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:34.494374312   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:34.494375014   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=4700.00m cores, Cores=24
2025-04-25 12:30:34.494395943   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16151.32 MB, Total=128728.99 MB, Usage=12.55%
2025-04-25 12:30:34.494396664   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:34.494397466   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:34.494398117   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:34.494813988   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176983
2025-04-25 12:30:34.494824087   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:34.693596072   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176985
2025-04-25 12:30:34.693599088   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:34.695219272   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135830) =======
2025-04-25 12:30:34.695221647   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:34.695223159   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:34.695249198   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:34.695250511   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:34.695266651   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:34.695268965   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:34.695270348   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:34.702706751   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:34.702711159   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:34.980554766   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.08 seconds ago
2025-04-25 12:30:34.994572745   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176988
2025-04-25 12:30:34.994576322   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:35.024491682   [ERROR]	[benthos/actions.go:145:145]	[dataflow-core-1]			error updating observed state for dataflow-core-1: failed to get health check: context deadline exceeded
2025-04-25 12:30:35.024521207   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get health check: context deadline exceeded
2025-04-25 12:30:35.494398527   [INFO]	[container/reconcile.go:143:143]	[Core]			======= Container Instance State: Core (tick: 177000) =======
2025-04-25 12:30:35.494401903   [INFO]	[container/reconcile.go:144:144]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:35.494403276   [INFO]	[container/reconcile.go:149:149]	[Core]			Health: Overall=Active, CPU=Active, Memory=Active, Disk=Active
2025-04-25 12:30:35.494404308   [INFO]	[container/reconcile.go:156:156]	[Core]			CPU: Usage=2621.85m cores, Cores=24
2025-04-25 12:30:35.494405319   [INFO]	[container/reconcile.go:166:166]	[Core]			Memory: Used=16145.63 MB, Total=128728.99 MB, Usage=12.54%
2025-04-25 12:30:35.494406301   [INFO]	[container/reconcile.go:176:176]	[Core]			Disk: Used=400.08 GB, Total=457.38 GB, Usage=87.47%
2025-04-25 12:30:35.494443571   [INFO]	[container/reconcile.go:179:179]	[Core]			Architecture: amd64, HWID: ffbdc22aa503b7eaacea32c9a484c62786f9a4b403a21c805d6299b0ef0c8d1b
2025-04-25 12:30:35.494445635   [INFO]	[container/reconcile.go:181:181]	[Core]			=================================================
2025-04-25 12:30:35.494827613   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176993
2025-04-25 12:30:35.494843373   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:35.594753319   [ERROR]	[fsm/baseFSM.go:362:362]	[redpanda]			redpanda instance redpanda is already in a shutdown state, force removing it
2025-04-25 12:30:35.595128925   [ERROR]	[redpanda/reconcile.go:47:47]	[redpanda]			error reconciling Redpanda instance redpanda: %!w(*fmt.wrapError=&{operation permanently failed after max retries: failed to update observed state: failed to get observed Redpanda config: failed to get redpanda status: failed to parse metrics: could not parse redpanda metrics/configuration: no sections found. This can happen when the redpanda service is not running, or the logs where rotated 0xc0037643a0})
2025-04-25 12:30:35.694097543   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to active
2025-04-25 12:30:35.794228935   [WARN]	[redpanda/redpanda.go:489:489]	[RedpandaServiceredpanda]			baseDir is empty, using default value /data
2025-04-25 12:30:35.794275703   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda]			Setting desired state of FSM redpanda to stopped
2025-04-25 12:30:35.794282196   [INFO]	[fsm/baseFSM.go:150:150]	[redpanda-monitor]			Setting desired state of FSM redpanda-monitor to running
2025-04-25 12:30:35.893700949   [INFO]	[redpanda/fsm_callbacks.go:36:36]	[redpanda]			Entering stopped state for redpanda
2025-04-25 12:30:35.980923437   [INFO]	[starvationchecker/starvation.go:102:102]	[StarveCheck]			Control loop is healthy, last reconcile was 0.48 seconds ago
2025-04-25 12:30:35.994821651   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176998
2025-04-25 12:30:35.994824917   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:36.039428665   [WARN]	[fsm/base_manager.go:507:507]	[S6ManagerBenthosServicedataflow-core-1-faulty]			not enough time left to reconcile instance benthos-dataflow-core-1-faulty (only 29.987437ms remaining, needed 30ms), skipping
2025-04-25 12:30:36.095411124   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 176999
2025-04-25 12:30:36.095414851   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:36.097275046   [INFO]	[agent_monitor/reconcile.go:132:132]	[Core]			======= Agent Instance State: Core (tick: 135840) =======
2025-04-25 12:30:36.097276689   [INFO]	[agent_monitor/reconcile.go:133:133]	[Core]			FSM States: Current=active, Desired=active
2025-04-25 12:30:36.097277471   [INFO]	[agent_monitor/reconcile.go:138:138]	[Core]			Health: Overall=Active, Latency=Neutral, Release=Neutral
2025-04-25 12:30:36.097290515   [INFO]	[agent_monitor/reconcile.go:144:144]	[Core]			Location: map[0:UMH-Systems-GmbH---Dev-Team 1:a 2:b 3:c 4: 5: 6:]
2025-04-25 12:30:36.097291146   [INFO]	[agent_monitor/reconcile.go:148:148]	[Core]			Latency: &{0 0 0 0 0}
2025-04-25 12:30:36.097291778   [INFO]	[agent_monitor/reconcile.go:152:152]	[Core]			Release: Channel=enterprise, Version=132fa35
2025-04-25 12:30:36.097296186   [INFO]	[agent_monitor/reconcile.go:154:154]	[Core]			Component Versions: [{UMH Core 132fa35} {S6 Overlay 3.2.0.2} {Benthos 0.6.2} {Redpanda 24.3.8}]
2025-04-25 12:30:36.097299773   [INFO]	[agent_monitor/reconcile.go:158:158]	[Core]			=================================================
2025-04-25 12:30:36.104741626   [ERROR]	[benthos/reconcile.go:95:95]	[dataflow-core-1]			error reconciling external changes: failed to update observed state: failed to get observed Benthos config: failed to get benthos config file for service benthos-dataflow-core-1: failed to check if config file exists: context deadline exceeded
2025-04-25 12:30:36.193739649   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 177000
2025-04-25 12:30:36.193743166   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
2025-04-25 12:30:36.393800294   [INFO]	[redpanda/redpanda.go:367:367]	[RedpandaServiceredpanda]			Getting health check and metrics for tick 177002
2025-04-25 12:30:36.393805083   [ERROR]	[redpanda/actions.go:165:165]	[redpanda]			error updating observed state for redpanda: failed to get health check: context canceled (current state: stopped, desired state: active)
