Kibana: Elastic did not load properly. Check the server output for more information.

[atbohmer]

Hello,

Running for almost a year now on a VM the T-Pot iso installation. Over the weekend something died?! There is more then enough disk space. Any advise how to debug other then a fresh install appriciated!

cat /opt/tpot/version

22.04.0

/opt/tpot/bin/dps.sh

[ ========| System |======== ]
DATE: Mon 12 Jun 2023 02:58:46 PM UTC
UPTIME: 14:58:46 up 2:39, 2 users, load average: 0.27, 0.23, 0.19
T-POT: ACTIVE
BLACKHOLE: DISABLED

NAME STATUS PORTS
adbhoney Up 2 hours (healthy) 0.0.0.0:5555->5555/tcp
ciscoasa Up 2 hours 0.0.0.0:5000->5000/udp, 0.0.0.0:8443->8443/tcp
citrixhoneypot Up 2 hours 0.0.0.0:443->443/tcp
conpot_guardian_ast Up 2 hours (healthy) 0.0.0.0:10001->10001/tcp
conpot_iec104 Up 2 hours (healthy) 0.0.0.0:161->161/udp, 0.0.0.0:2404->2404/tcp
conpot_ipmi Up 2 hours (healthy) 0.0.0.0:623->623/udp
conpot_kamstrup_382 Up 2 hours (healthy) 0.0.0.0:1025->1025/tcp, 0.0.0.0:50100->50100/tcp
cowrie Up 2 hours 0.0.0.0:22-23->22-23/tcp
ddospot Up 2 hours 0.0.0.0:19->19/udp, 0.0.0.0:53->53/udp, 0.0.0.0:123->123/udp, 0.0.0.0:1900->1900/udp
dicompot Up 2 hours 0.0.0.0:11112->11112/tcp
dionaea Up 2 hours (healthy) 0.0.0.0:20-21->20-21/tcp, 0.0.0.0:42->42/tcp, 0.0.0.0:81->81/tcp, 0.0.0.0:135->135/tcp, 0.0.0.0:445->445/tcp, 0.0.0.0:1433->1433/tcp, 0.0.0.0:1723->1723/tcp, 0.0.0.0:1883->1883/tcp, 0.0.0.0:3306->3306/tcp, 0.0.0.0:27017->27017/tcp, 0.0.0.0:69->69/udp
elasticpot Up 2 hours 0.0.0.0:9200->9200/tcp
elasticsearch Up 2 hours (healthy) 127.0.0.1:64298->9200/tcp
fatt Up 2 hours
heralding Up 2 hours 0.0.0.0:110->110/tcp, 0.0.0.0:143->143/tcp, 0.0.0.0:465->465/tcp, 0.0.0.0:993->993/tcp, 0.0.0.0:995->995/tcp, 0.0.0.0:1080->1080/tcp, 0.0.0.0:5432->5432/tcp, 0.0.0.0:5900->5900/tcp
honeytrap Up 2 hours
ipphoney Up 2 hours 0.0.0.0:631->631/tcp
kibana Up 2 hours (healthy) 127.0.0.1:64296->5601/tcp
logstash Up 2 hours (healthy)
mailoney Up 2 hours 0.0.0.0:25->25/tcp
map_data Up 2 hours
map_redis Up 2 hours
map_web Up 2 hours 127.0.0.1:64299->64299/tcp
medpot Up 2 hours 0.0.0.0:2575->2575/tcp
nginx Up 2 hours
p0f Up 2 hours
redishoneypot Up 2 hours 0.0.0.0:6379->6379/tcp
sentrypeer Up 2 hours 0.0.0.0:5060->5060/udp
snare Up 2 hours 0.0.0.0:80->80/tcp
spiderfoot Up 2 hours (healthy) 127.0.0.1:64303->8080/tcp
suricata Up 2 hours
tanner Up 2 hours
tanner_api Up 2 hours
tanner_phpox Up 2 hours
tanner_redis Up 2 hours

Elasticsearch docker image:
elasticsearch@3e92191ec47e:/$ curl -s -XGET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "tpotcluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 103,
"active_shards" : 103,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

Kibana docker image log:
kibana@405c7a5ac173:/var/log/kibana$ grep -i elas kibana.log
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T12:30:04.338+00:00","message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"0b950cb646a620e0"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T12:30:08.308+00:00","message":"Error Initializing ML saved objects {"name":"ResponseError","meta":{"body":{"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405},"statusCode":405,"headers":{"x-opaque-id":"unknownId","allow":"POST","x-elastic-product":"Elasticsearch","content-type":"application/json","content-length":"113"},"meta":{"context":null,"request":{"params":{"method":"GET","path":"/_ml/anomaly_detectors","querystring":"","headers":{"user-agent":"Kibana/8.6.2","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1","x-opaque-id":"unknownId","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"options":{"opaqueId":"unknownId","headers":{"x-elastic-product-origin":"kibana","user-agent":"Kibana/8.6.2","x-elastic-client-meta":"es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1"}},"id":7},"name":"elasticsearch-js","connection":{"url":"http://elasticsearch:9200/\",\"id\":\"http://elasticsearch:9200/\",\"headers\":{},\"status\":\"alive\"},\"attempts\":0,\"aborted\":false},\"warnings\":null}}","log":{"level":"ERROR","logger":"plugins.ml"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"0b950cb646a620e0"}}
{"tags":["ML:saved-objects-sync","ML:saved-objects-sync-task","task-run-failed"],"error":{"stack_trace":"ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/datafeeds] and method [GET], allowed: [POST]","status":405}\n at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)\n at Ml.getDatafeeds (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ml.js:699:16)\n at async Promise.all (index 5)\n at checkStatus (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/checks.js:19:108)\n at initSavedObjects (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync.js:311:20)\n at Object.run (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync_task.js:68:19)\n at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:266:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T12:30:11.529+00:00","message":"Task ML:saved-objects-sync "ML:saved-objects-sync-task" failed: ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/datafeeds] and method [GET], allowed: [POST]","status":405}","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"23884546b9795ed8"}}
{"tags":["ML:saved-objects-sync","ML:saved-objects-sync-task","task-run-failed"],"error":{"stack_trace":"ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405}\n at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)\n at Ml.getJobs (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ml.js:808:16)\n at async Promise.all (index 4)\n at checkStatus (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/checks.js:19:108)\n at initSavedObjects (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync.js:311:20)\n at Object.run (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync_task.js:68:19)\n at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:266:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T13:30:11.545+00:00","message":"Task ML:saved-objects-sync "ML:saved-objects-sync-task" failed: ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405}","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"9762a98ccebe2a48"}}
{"tags":["ML:saved-objects-sync","ML:saved-objects-sync-task","task-run-failed"],"error":{"stack_trace":"ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405}\n at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)\n at Ml.getJobs (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ml.js:808:16)\n at async Promise.all (index 4)\n at checkStatus (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/checks.js:19:108)\n at initSavedObjects (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync.js:311:20)\n at Object.run (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync_task.js:68:19)\n at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:266:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T14:30:11.674+00:00","message":"Task ML:saved-objects-sync "ML:saved-objects-sync-task" failed: ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405}","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"2bb61cdd634eb907"}}

[atbohmer]

And it works again without me doing anything, but ok thanks.