2

File Beat is unable to send logs from a particular folder, This is the application logs folder.

Things that have been tried :

  • Created a new topic in kafka to retest the settings.
  • Checked for file permission for the folder and the file to send.
  • Updated to the filebeats to 6.7 from 5.5
  • changed from filebeat.prospector to filebeat.inputs

Running Configuration : 

 filebeat.inputs:
    - type: log
      paths:
        - /var/log/containers/*.log
    fields_under_root: true
    output.kafka:
      hosts: ["10.0.0.0:9092"]
      topic: "testtopic"
      codec.json:
        pretty: true

With this i am able to see all the logs in "testtopic"

Not running Configuration :

filebeat.inputs:
    - type: log
      paths:
        - /app/log/server/*.log
    fields_under_root: true
    output.kafka:
      hosts: ["10.0.0.0:9092"]
      topic: "testtopic"
      codec.json:
        pretty: true

Expected Results : Logs from the path /app/log/server/*.log should be sent to Kafka

user3636989
  • 161
  • 1
  • 9

1 Answers1

0

I can connect to kafka this these message:

2023-03-24T10:26:16.198Z    INFO    [publisher_pipeline_output] pipeline/output.go:143  Connecting to kafka(192.168.16.199:10092,192.168.16.199:10093,192.168.16.199:10094)
2023-03-24T10:26:16.198Z    INFO    [publisher] pipeline/retry.go:219   retryer: send unwait signal to consumer
2023-03-24T10:26:16.199Z    INFO    [publisher] pipeline/retry.go:223     done
2023-03-24T10:26:16.199Z    INFO    [publisher_pipeline_output] pipeline/output.go:151  Connection to kafka(192.168.16.199:10092,192.168.16.199:10093,192.168.16.199:10094) established
2023-03-24T10:26:46.181Z    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"id":"/"},"memory":{"id":"/","mem":{"usage":{"bytes":45117440}}}},"cpu":{"system":{"ticks":90,"time":{"ms":102}},"total":{"ticks":430,"time":{"ms":449},"value":430},"user":{"ticks":340,"time":{"ms":347}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":15},"info":{"ephemeral_id":"37db27e4-1920-4071-a677-45d1475d2e06","uptime":{"ms":30095},"version":"7.17.7"},"memstats":{"gc_next":26621000,"memory_alloc":22819736,"memory_sys":49890312,"memory_total":72543720,"rss":110149632},"runtime":{"goroutines":91}},"filebeat":{"events":{"active":4124,"added":4126,"done":2},"harvester":{"open_files":5,"running":5,"started":5}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":2048,"batches":1,"total":2048},"type":"kafka"},"pipeline":{"clients":1,"events":{"active":4117,"filtered":5,"published":4116,"retry":2048,"total":4122},"queue":{"max_events":4096}}},"registrar":{"states":{"current":2,"update":2},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":32},"load":{"1":0.06,"15":0.09,"5":0.13,"norm":{"1":0.0019,"15":0.0028,"5":0.0041}}}}}}
2023-03-24T10:27:16.183Z    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":2764800}}}},"cpu":{"system":{"ticks":110,"time":{"ms":25}},"total":{"ticks":460,"time":{"ms":70},"value":460},"user":{"ticks":350,"time":{"ms":45}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":15},"info":{"ephemeral_id":"37db27e4-1920-4071-a677-45d1475d2e06","uptime":{"ms":60108},"version":"7.17.7"},"memstats":{"gc_next":36801864,"memory_alloc":18895888,"memory_sys":5242880,"memory_total":75547856,"rss":112685056},"runtime":{"goroutines":91}},"filebeat":{"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":2048}},"pipeline":{"clients":1,"events":{"active":4117}}},"registrar":{"states":{"current":2}},"system":{"load":{"1":0.31,"15":0.11,"5":0.18,"norm":{"1":0.0097,"15":0.0034,"5":0.0056}}}}}}

But from kafka cluster, there are nothing!

Quang Vu
  • 21
  • 3