I just realized logstash in production needs two service endpoints. One for stuff like postgres (which uses a logstash forwarder AKA filebeat), and another for native logstash sinks (like log4j).
In other words, if you're deploying logstash as an all-in-one logging solution for a microservice app in something like kubernetes, your service should look like this.
logstash 10.108.123.244
not this
logstash 10.108.123.244
Second of all
filebeat is going to run as a separate process in your containers, so you'll want to check its individual logs, to make sure its happy on startup, i.e.
cat `find / -name filebeat`/*
If you don't know where they are.
Broken
2017-08-04T14:15:40Z INFO No non-zero metrics in the last 30s
2017-08-04T14:16:10Z INFO No non-zero metrics in the last 30s
2017-08-04T14:16:28Z ERR Connecting error publishing events (retrying): dial tcp 10.104.131.190:5044: i/o timeout
2017-08-04T14:16:40Z INFO No non-zero metrics in the last 30s
2017-08-04T14:17:10Z INFO No non-zero metrics in the last 30s
2017-08-04T14:17:28Z WARN DNS lookup failure "logstash": lookup logstash on 10.96.0.10:53: no such host
2017-08-04T14:17:28Z ERR Connecting error publishing events (retrying): lookup logstash on 10.96.0.10:53: no such host
2017-08-04T14:17:40Z INFO No non-zero metrics in the last 30s
2017-08-04T14:18:10Z INFO No non-zero metrics in the last 30s
2017-08-04T14:18:29Z INFO Harvester started for file: /var/lib/postgresql/data/pg_log/postgresql_Thu.log
2017-08-04T14:18:39Z INFO Harvester started for file: /var/lib/postgresql/data/pg_log/postgresql_Sat.log
2017-08-04T14:18:39Z INFO Harvester started for file: /var/lib/postgresql/data/pg_log/postgresql_Tue.log
2017-08-04T14:18:39Z INFO Harvester started for file: /var/lib/postgresql/data/pg_log/postgresql_Sun.log
2017-08-04T14:18:39Z INFO Harvester started for file: /var/lib/postgresql/data/pg_log/postgresql_Mon.log
2017-08-04T14:18:39Z INFO Harvester started for file: /var/lib/postgresql/data/pg_log/postgresql_Fri.log
Fixed
2017-08-04T14:18:40Z INFO Non-zero metrics in the last 30s: filebeat.harvester.running=6 registar.states.current=7 registrar.writes=31 filebeat.harvester.open_files=5 libbeat.logstash.published_and_acked_events=57356 libbeat.publisher.published_events=57347 libbeat.logstash.call_count.PublishEvents=31 libbeat.logstash.publish.read_bytes=270 registrar.states.update=57363 filebeat.harvester.started=6 publish.events=57363 libbeat.logstash.publish.write_bytes=2148523
2017-08-04T14:19:10Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=1466320 registrar.writes=24 registrar.states.update=39312 libbeat.logstash.call_count.PublishEvents=24 publish.events=39312 libbeat.logstash.publish.read_bytes=144 libbeat.logstash.published_and_acked_events=39312 libbeat.publisher.published_events=39312
2017-08-04T14:19:40Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_bytes=36 libbeat.logstash.publish.write_bytes=2885 registrar.states.update=20 libbeat.logstash.published_and_acked_events=20 registrar.writes=6 publish.events=20 libbeat.publisher.published_events=20 libbeat.logstash.call_count.PublishEvents=6
2017-08-04T14:20:10Z INFO Non-zero metrics in the last 30s: registrar.states.update=20 libbeat.publisher.published_events=20 libbeat.logstash.publish.write_bytes=2881 libbeat.logstash.published_and_acked_events=20 libbeat.logstash.call_count.PublishEvents=6 libbeat.logstash.publish.read_bytes=36 publish.events=20 registrar.writes=6
2017-08-04T14:20:40Z INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=20 registrar.states.update=20 libbeat.logstash.publish.read_bytes=36 registrar.writes=6 libbeat.logstash.publish.write_bytes=2827 libbeat.logstash.call_count.PublishEvents=6 libbeat.logstash.published_and_acked_events=20 publish.events=20
2017-08-04T14:21:10Z INFO Non-zero metrics in the last 30s: registrar.states.update=20 libbeat.logstash.call_count.PublishEvents=6 libbeat.logstash.publish.read_bytes=36 publish.events=20 libbeat.publisher.published_events=20 registrar.writes=6 libbeat.logstash.published_and_acked_events=20 libbeat.logstash.publish.write_bytes=2809
2017-08-04T14:21:40Z INFO Non-zero metrics in the last 30s: libbeat.logstash.published_and_acked_events=20 libbeat.publisher.published_events=20 publish.events=20 registrar.writes=6 libbeat.logstash.publish.read_bytes=36 libbeat.logstash.publish.write_bytes=2954 libbeat.logstash.call_count.PublishEvents=6 registrar.states.update=20
P.S. my friend (for today) khomol soda showed me this
No comments:
Post a Comment