python实时监控logstash日志代码


Posted in Python onApril 27, 2020

实时读取logstash日志,有异常错误keywork即触发报警。

# /usr/bin/env python3
# -*- coding: utf-8 -*-
# __author__ = caozhi
# create_time 2018-11-12,update_time 2018-11-15
# version = 1.0
# 录像高可用报警
# 1 读取日志 使用游标移动
# 2 线上业务日志文件会切割,切割后,读取上一个切割的日志

import os
import sys
import json
import requests
import time
import re

cini = conf.ini'
log_file = logstash.log'

def readconf():
 try:
 with open(cini, 'r+') as f:
  CONF = json.load(f)
 except:
 CONF = {"seek": 0, "inode": 922817, "last_file": logstash.log"}
 writeconf(CONF=CONF)
 print('conf.ini 配置文件缺失,自动创建一个新的配置文件')
 return CONF

def writeconf(CONF):
 with open(cini, 'w+') as e:
 json.dump(CONF, e)

def read_log(log_file, seek):
 try:
 f = open(log_file, 'r')
 except FileNotFoundError:
 f = open(logstash.log', 'r')
 seek = 0
 print('上一个文件读取失败了,请检查切割的日志文件')
 except:
 print('日志文件打开错误,退出程序')
 sys.exit()

f.seek(seek)
line = f.readline()
new_seek = f.tell()
if new_seek == seek:
 print('没有追加日志,退出程序')
 sys.exit()

while line:
 try:
 logstash = json.loads(line)
 except:
 CONF = {"seek": 0, "inode": 922817, "last_file": "/data/logs/lmrs/logstash.log"}
 writeconf(CONF=CONF)
 print('json数据加载错误,重新创建一个新的配置文件')
 sys.exit()

 #if '''re.search(time.strftime("%Y:%H:%M", time.localtime()), logstash.get('log_time')) and '''logstash.get('rtype') == 6 and logstash.get('uri') == '/publish' and logstash.get('event') == 0:
 if logstash.get('rtype') == 6 and logstash.get('uri') == '/publish' and logstash.get('event') == 0:
 value = 1
 stream = logstash.get('name')
 print('{} {}'.format(value, stream))
 record(value=value, stream=stream)
 else:
 value = 0
 stream = 0
 line = f.readline()
seek = f.tell()
f.close
return value, stream, seek

def record(value, stream):
 data = []
 record = {}
 record['metric'] = 'recording_high_availability_monitor'
 record['endpoint'] = os.uname()[1]
 record['timestamp'] = int(time.time())
 record['step'] = 60
 record['value'] = value
 record['counterType'] = 'GAUGE'
 record['Tags'] = '{}={}'.format(int(time.time()), stream)
 data.append(record)

if data:
 print('这是data的json数据')
 print(data)
 falcon_request = requests.post("http://127.0.0.1:1988/v1/push", data=json.dumps(data))
 #falcon_request = requests.post("http://127.0.0.1:1988/v1/push", json=data)
 print('json参数请求返回状态码为:' + str(falcon_request.status_code))
 print('json参数请求返回为:' + str(falcon_request.text))

if __name__ == '__main__':
 print()
 print('***************************************')
 print('本次执行脚本时间:{}'.format(time.strftime("%Y%m%d_%H%M", time.localtime())))
 CONF = readconf()
 print('first_CONF :{}'.format(CONF))
 print('NO1.log_file',log_file)
 last_inode = CONF['inode']
 inode = os.stat(log_file).st_ino
 print('last_inode: {} inode: {}'.format(last_inode, inode))

if inode == last_inode:
 seek = CONF['seek']
 next_file = 0
else:
 log_file = CONF['last_file'] + time.strftime("-%Y%m%d_", time.localtime()) + str(time.strftime("%H%M", time.localtime()))[:-1] + '0'
 next_file = 1
 seek = CONF['seek']

print('NO2.log_file',log_file)
value, stream, seek = read_log(log_file=log_file,seek=seek)

if next_file:
 CONF['seek'] = 0
else:
 CONF['seek'] = seek

CONF['inode'] = os.stat(logstash.log').st_ino
writeconf(CONF=CONF)
print('last_CONF :{}'.format(CONF))

补充知识:logstash 调用exec

我就废话不多说了,还是直接看代码吧!

[elk@Vsftp logstash]$ cat t3.conf 
input {
 stdin {
 } 
} 
filter {
 grok {
 match => [ "message","(?m)\s*%{TIMESTAMP_ISO8601:time}\s*(?<Level>(\S+)).*"]
 }
 date {
 match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]
 }
 mutate {
   add_field =>["type","tailong"]
   add_field =>["messager","%{type}-%{message}"]
   remove_field =>["message"]
  }
} 
output { 
 if ([Level] == "ERROR" or [messager] =~ "Exception" ) and [messager] !~ "温金服务未连接" and [messager] !~ "调用温金代理系统接口错误" and [messager] !~ "BusinessException" {
 exec {
  command => "/bin/smail.pl \"%{messager}\" \"%{type}\" "
 }
 }
 stdout { 
 codec =>rubydebug 
 } 
}
 
Vsftp:/root# cat /bin/smail.pl 
#!/usr/bin/perl 
use Net::SMTP;
use HTTP::Date qw(time2iso str2time time2iso time2isoz); 
use Data::Dumper;
use Getopt::Std;
use vars qw($opt_d );
getopts('d:');
# mail_user should be your_mail@163.com
 $message= "@ARGV";
 $env="$opt_d";
 sub send_mail{
 my $CurrTime = time2iso(time());
 my $to_address = shift;
 my $mail_user = 'zhao.yangjian@163.com';
 my $mail_pwd = 'xx';
 my $mail_server = 'smtp.163.com';
 
 my $from = "From: $mail_user\n";
 my $subject = "Subject: zjcap info\n";
 my $info = "$CurrTime--$message";
 my $message = <<CONTENT; 
 $info
CONTENT
 my $smtp = Net::SMTP->new($mail_server);
 
 $smtp->auth($mail_user, $mail_pwd) || die "Auth Error! $!";
 $smtp->mail($mail_user);
 $smtp->to($to_address);
 
 $smtp->data();  # begin the data
 $smtp->datasend($from); # set user
 $smtp->datasend($subject); # set subject
 $smtp->datasend("\n\n");
 $smtp->datasend("$message\n"); # set content
 $smtp->dataend();
 $smtp->quit();
};
 
send_mail ('zhao.yangjian@163.com'); 
 
2017-01-12 10:19:19,888 jjjjj Exception
{
 "@version" => "1",
 "@timestamp" => "2017-01-12T02:19:19.888Z",
  "host" => "Vsftp",
  "time" => "2017-01-12 10:19:19,888",
  "Level" => "jjjjj",
  "type" => "tailong",
 "messager" => "tailong-2017-01-12 10:19:19,888 jjjjj Exception"
}

以上这篇python实时监控logstash日志代码就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持三水点靠木。

Python 相关文章推荐
介绍Python中的__future__模块
Apr 27 Python
Django发送html邮件的方法
May 26 Python
Python中的id()函数指的什么
Oct 17 Python
Python实现线程状态监测简单示例
Mar 28 Python
使用Python处理BAM的方法
Sep 28 Python
python处理两种分隔符的数据集方法
Dec 12 Python
关于python之字典的嵌套,递归调用方法
Jan 21 Python
利用python将图片版PDF转文字版PDF
May 03 Python
python 利用pyttsx3文字转语音过程详解
Sep 25 Python
在Python中使用filter去除列表中值为假及空字符串的例子
Nov 18 Python
给ubuntu18安装python3.7的详细教程
Jun 08 Python
python3.9和pycharm的安装教程并创建简单项目的步骤
Feb 03 Python
python实现秒杀商品的微信自动提醒功能(代码详解)
Apr 27 #Python
浅析python 定时拆分备份 nginx 日志的方法
Apr 27 #Python
python异步Web框架sanic的实现
Apr 27 #Python
python库skimage给灰度图像染色的方法示例
Apr 27 #Python
python实现密度聚类(模板代码+sklearn代码)
Apr 27 #Python
Django中文件上传和文件访问微项目的方法
Apr 27 #Python
详解Python中namedtuple的使用
Apr 27 #Python
You might like
PHP设计模式 注册表模式(多个类的注册)
2012/02/05 PHP
PHP实现基于3DES算法加密解密字符串示例
2018/08/24 PHP
PHP 模拟登陆功能实例详解
2019/09/10 PHP
javascript下数值型比较难点说明
2010/06/07 Javascript
js判断字符是否是汉字的两种方法小结
2014/01/03 Javascript
JS实现FLASH幻灯片图片切换效果的方法
2015/03/04 Javascript
浅谈Javascript数组索引
2015/07/29 Javascript
jQuery Dialog对话框事件用法实例分析
2016/05/10 Javascript
window.open打开窗口被拦截的快速解决方法
2016/08/04 Javascript
使用bootstrapValidator插件进行动态添加表单元素并校验
2016/09/28 Javascript
Extjs gridpanel 中的checkbox(复选框)根据某行的条件不能选中的解决方法
2017/02/17 Javascript
JavaScript简单拖拽效果(1)
2017/05/17 Javascript
详解angularJs中关于ng-class的三种使用方式说明
2017/06/02 Javascript
JavaScript内存泄漏的处理方式
2017/11/20 Javascript
React Native日期时间选择组件的示例代码
2018/04/27 Javascript
npm 下载指定版本的组件方法
2018/05/17 Javascript
浅探express路由和中间件的实现
2019/09/30 Javascript
JavaScript实现移动小精灵的案例代码
2020/12/12 Javascript
[00:39]DOTA2上海特级锦标赛 Liquid战队宣传片
2016/03/04 DOTA
python回调函数的使用方法
2014/01/23 Python
Python3实现取图片中特定的像素替换指定的颜色示例
2019/01/24 Python
Python使用socket模块实现简单tcp通信
2020/08/18 Python
python matlab库简单用法讲解
2020/12/31 Python
利用CSS3的checked伪类实现OL的隐藏显示的方法
2010/12/18 HTML / CSS
CSS3按钮鼠标悬浮实现光圈效果源码
2016/09/11 HTML / CSS
Alba Moda瑞士网上商店:独家意大利时尚女装销售
2016/11/28 全球购物
英国女性时尚品牌:Apricot
2018/12/04 全球购物
初二物理教学反思
2014/01/29 职场文书
物理系毕业生自荐书范文
2014/02/22 职场文书
任命书怎么写
2014/06/04 职场文书
励志演讲稿3分钟
2014/08/21 职场文书
前台接待岗位职责范本
2015/04/03 职场文书
2016新年晚会开场白
2015/12/03 职场文书
2019开业庆典剪彩仪式主持词!
2019/07/22 职场文书
Spring中的使用@Async异步调用方法
2021/11/01 Java/Android
springboot layui hutool Excel导入的实现
2022/03/31 Java/Android