filebeats 收集Nginx日志

1. 介绍

  通过stdin收集日志,通过console输出结果

2. 下载安装

cd /usr/local/src/elk/
sudo wget  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.16-linux-x86_64.tar.gz
tar xf filebeat-5.6.16-linux-x86_64.tar.gz -C /opt/
ln -s /opt/filebeat-5.6.16-linux-x86_64 /opt/filebeat
cd /opt/filebeat
mkdir conf
cd conf
cat>>nginx.yml<<EOF
filebeat.prospectors:
- input_type: stdin
output.console:
    pretty: true
EOF
head -n 2 /var/log/nginx.log |/opt/filebeat/filebeat -e -c /opt/filebeat/conf/nginx.yml 

可以使用迅雷下载,导入到系统中安装。

3. 详细过程

root@master:/home/cmz# cd /usr/local/src/elk/
root@master:/usr/local/src/elk# sudo wget  https://artifacts.elastic.co/downloads/beats/filebeat/fileb
--2019-10-09 16:08:47--  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.16-linux-x
Resolving artifacts.elastic.co (artifacts.elastic.co)... 151.101.110.222, 2a04:4e42:1a::734
Connecting to artifacts.elastic.co (artifacts.elastic.co)|151.101.110.222|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8838693 (8.4M) [application/x-gzip]
Saving to: ‘filebeat-5.6.16-linux-x86_64.tar.gz’

filebeat-5.6.16-linux-x86_64.tar.gz             100%[=================================================

2019-10-09 16:10:18 (97.1 KB/s) - ‘filebeat-5.6.16-linux-x86_64.tar.gz’ saved [8838693/8838693]
root@master:/usr/local/src/elk# tar xf filebeat-5.6.16-linux-x86_64.tar.gz -C /opt/
root@master:/var/log# ln -s /opt/filebeat-5.6.16-linux-x86_64 /opt/filebeat
root@master:/opt/filebeat# mkdir conf
root@master:/opt/filebeat# cd conf/
root@master:/opt/filebeat/conf# egrep -v '#|^$' nginx.yml 
filebeat.prospectors:
- input_type: stdin
output.console:
    pretty: true

root@master:/opt/filebeat/conf# head -n 2 /var/log/nginx.log |/opt/filebeat/filebeat -e -c /opt/filebe
at/conf/nginx.yml 2019/10/09 08:27:41.580046 beat.go:297: INFO Home path: [/opt/filebeat] Config path: [/opt/filebeat] D
ata path: [/opt/filebeat/data] Logs path: [/opt/filebeat/logs]2019/10/09 08:27:41.580065 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.16
2019/10/09 08:27:41.580095 outputs.go:108: INFO Activated console as output plugin.
2019/10/09 08:27:41.580136 metrics.go:23: INFO Metrics logging every 30s
2019/10/09 08:27:41.580149 publish.go:300: INFO Publisher name: master
2019/10/09 08:27:41.580260 async.go:63: INFO Flush Interval set to: 1s
2019/10/09 08:27:41.580268 async.go:64: INFO Max Bulk Size set to: 2048
2019/10/09 08:27:41.580413 beat.go:233: INFO filebeat start running.
2019/10/09 08:27:41.580435 registrar.go:68: INFO No registry file found under: /opt/filebeat/data/regi
stry. Creating a new registry file.2019/10/09 08:27:41.627096 registrar.go:106: INFO Loading registrar data from /opt/filebeat/data/regis
try2019/10/09 08:27:41.627131 registrar.go:123: INFO States Loaded from registrar: 0
2019/10/09 08:27:41.627163 crawler.go:38: INFO Loading Prospectors: 1
2019/10/09 08:27:41.627224 registrar.go:236: INFO Starting Registrar
2019/10/09 08:27:41.627251 sync.go:41: INFO Start sending events to output
2019/10/09 08:27:41.627326 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2019/10/09 08:27:41.627325 prospector.go:124: INFO Starting prospector of type: stdin; id: 14799454217
977376761 2019/10/09 08:27:41.627345 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled pro
spectors: 12019/10/09 08:27:41.627405 log.go:91: INFO Harvester started for file: -
2019/10/09 08:27:41.627440 log.go:114: INFO End of file reached: . Closing because close_eof is enable
d.{
  "@timestamp": "2019-10-09T08:27:41.627Z",
  "beat": {
    "hostname": "master",
    "name": "master",
    "version": "5.6.16"
  },
  "input_type": "stdin",
  "message": "80.82.70.187 - - [09/Oct/2019:08:22:42 +0800] \"GET http://www.baidu.com/cache/global/im
g/gs.gif HTTP/1.1\" 401 188 \"-\" \"Mozilla\"",  "offset": 0,
  "source": "",
  "type": "log"
}
{
  "@timestamp": "2019-10-09T08:27:41.627Z",
  "beat": {
    "hostname": "master",
    "name": "master",
    "version": "5.6.16"
  },
  "input_type": "stdin",
  "message": "80.82.70.187 - - [09/Oct/2019:08:22:42 +0800] \"GET http://www.baidu.com/cache/global/im
g/gs.gif HTTP/1.1\" 401 188 \"-\" \"Mozilla\"",  "offset": 0,
  "source": "",
  "type": "log"
}
2019/10/09 08:28:11.580663 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.cl
osed=1 filebeat.harvester.open_files=-1 filebeat.harvester.started=1 libbeat.publisher.published_events=2 publish.events=3 registrar.states.cleanup=1 registrar.states.update=1 registrar.writes=2