组件简介

• Grafana

Grafana是一款采用 go 语言开发的开源监控框架，主要用于大规模指标数据的可视化展现,是最流行的针对时序数据做展示的框架工具.

目前最新版本7.1.15,地址:https://dl.grafana.com/oss/release/grafana-7.1.5.linux-amd64.tar.gz

• InfluxDB

InfluxDB是一个使用 Go 语言编写的开源时序数据库,无需外部依赖,InfluxDB旨在作为大量带有时间戳格式数据的后端数据存储,包括DevOps监控,应用程序指标,IoT传感器数据,实时分析等.目前InfluxDB在db-engines时序数据库分类中排名第一.

目前最新版本是1.8.2, 地址:https://dl.influxdata.com/influxdb/releases/influxdb-1.8.2_linux_amd64.tar.gz

• Telegraf

Telegraf 作为收集和报告指标和数据的agent安装在被监控的服务端,将带有时间戳的指标数据写入到InfluxDB.

目前最新版是1.15.2,地址:https://dl.influxdata.com/telegraf/releases/telegraf-1.15.2_linux_amd64.tar.gz

• Redis

目前采用主从复制架构,VIP漂在主节点,主节点发生故障后通过哨兵将VIP切换到从节点,目前使用的Redis版本是5.0.9.

• PostgreSQL

PostgreSQL是开源最流行的关系型数据库,这里之所以介绍PostgreSQL,因为Grafana存储元数据时默认使用的是自带的sqlite3数据库,我们将元数据存储改写入到PostgreSQL数据库中

PostgreSQL数据库安装

PostgreSQL数据库安装请参考之前的博文链接CentOS7下PostgreSQL&PostGIS数据库安装指导手册

创建数据库和用户

postgres=# create database grafana;
CREATE DATABASE
postgres=# create role grafana with password 'mypna123' login;
CREATE ROLE
postgres=# alter database grafana owner to grafana ;
ALTER DATABASE

Grafana安装

解压到/usr/local目录

# groupadd -r dba
# useradd -r -g dba tnuser
# cd /usr/local
# wget https://dl.grafana.com/oss/release/grafana-7.1.5.linux-amd64.tar.gz
# tar -zxf grafana-7.1.5.linux-amd64.tar.gz
# chown -R tnuser.dba grafana-7.1.5
# ln -s grafana-7.1.5 grafana

修改Grafana配置文件,部分参数如下.

[paths]
data = /usr/local/grafana/data  //存放临时文件,sessions等
logs = /usr/local/grafana/data/log  //存放grafana的log
plugins = /usr/local/grafana/data/plugins  //存放grafana的扩展插件
provisioning = /usr/local/grafana/conf/provisioning  //grafana启动和运行时加载的文件
[server]
domain = mypna.com  //填写实际公司内部的域环境信息
# root_url = %(protocol)s://%(domain)s:%(http_port)s/
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana  //如果grafana通过nignx或者apache代理访问,那么这里需要添加额外的URL子目录
serve_from_sub_path = true  //当使用nginx或者Apache代理访问grafana时,这里需要填写true
[database]
type = postgres                    //使用PostgreSQL作为grafana的元数据存储
host = 127.0.0.1:5432
name = grafana
user = grafana
password = mypna123
[auth.anonymous]
enabled = true              //启用匿名访问,否则通过WEB查看grafana监控信息时需要先认证

将Grafana配置成systemctl服务

# cat /usr/lib/systemd/system/grafana.service 
[Unit]
Description=Grafanad instance
Documentation=https://grafana.com/docs/grafana/latest/getting-started
After=syslog.target
After=network.target
Wants=network-online.target

[Service]
User=tnuser
Group=dba
Type=notify
Restart=always

WorkingDirectory=/usr/local/grafana
ExecStart=/usr/local/grafana/bin/grafana-server
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
TimeoutSec=0

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload
# systemctl start grafana
# systemctl status grafana

这个时候再到PostgreSQL grafana数据库中,即可看到初始化的表信息

grafana=# \dt
                  List of relations
 Schema |           Name           | Type  |  Owner  
--------+--------------------------+-------+---------
 public | alert                    | table | grafana
 public | alert_notification       | table | grafana
 public | alert_notification_state | table | grafana
 public | alert_rule_tag           | table | grafana
 public | annotation               | table | grafana
 public | annotation_tag           | table | grafana
 public | api_key                  | table | grafana
 public | cache_data               | table | grafana
 public | dashboard                | table | grafana
 public | dashboard_acl            | table | grafana
 public | dashboard_provisioning   | table | grafana
 public | dashboard_snapshot       | table | grafana
 public | dashboard_tag            | table | grafana
 public | dashboard_version        | table | grafana
 public | data_source              | table | grafana
 public | login_attempt            | table | grafana
 public | migration_log            | table | grafana
 public | org                      | table | grafana
 public | org_user                 | table | grafana
 public | playlist                 | table | grafana
 public | playlist_item            | table | grafana
 public | plugin_setting           | table | grafana
 public | preferences              | table | grafana
 public | quota                    | table | grafana
 public | server_lock              | table | grafana
 public | session                  | table | grafana
 public | star                     | table | grafana
 public | tag                      | table | grafana
 public | team                     | table | grafana
 public | team_member              | table | grafana
 public | temp_user                | table | grafana
 public | test_data                | table | grafana
 public | user                     | table | grafana
 public | user_auth                | table | grafana
 public | user_auth_token          | table | grafana
(35 rows)

登录Grafana的界面,第一次登录时需要设置admin用户的新密码

InfluxDB安装

InfluxDB安装请参考之前的博文链接:InfuxDB简介与安装

创建存放redis指标数据的数据库,并设置retention policy策略为2年,即我们只需要保留两年的监控数据.

> create database redis;
> use redis;
Using database redis
> show retention policies
name    duration shardGroupDuration replicaN default
----    -------- ------------------ -------- -------
autogen 0s       168h0m0s           1        true
> alter retention policy "autogen" on "redis" duration 17520h default;
> show retention policies
name    duration   shardGroupDuration replicaN default
----    --------   ------------------ -------- -------
autogen 17520h0m0s 168h0m0s           1        true

Telegraf安装

在Redis的主备节点分别安装Telegraf agent,这里说明一下,Telegraf采集redis主要通过"info"和"info commandstats"获取指标数据

# groupadd -r dba
# useradd -r -g dba tnuser
# cd /usr/local
# wget https://dl.influxdata.com/telegraf/releases/telegraf-1.15.2_linux_amd64.tar.gz
# tar -zxf telegraf-1.15.2_linux_amd64.tar.gz
# chown -R tnuser.dba telegraf-1.15.2
# ln -s telegraf-1.15.2 telegraf

修改telegraf配置文件,部分参数如下.

[[outputs.influxdb]]      //这里表示采集数据的输出目标
  urls = ["http://10.190.101.4:8086"]   //这里填写InfluxDB所在服务器的IP地址或者域名,这里留意一下是http不是tcp
  database = "redis"    //填写之前在InfluxDB创建的redis数据库
  skip_database_creation = true      //因我们之前创建的redis数据库设置了renention policy,这里跳过自动创建目标数据库
[[inputs.net]]         //采集源,本机的网卡流量信息
[[inputs.netstat]]   //采集源,本机的tcp连接信息
[[inputs.redis]]                    //源,
  servers = ["tcp://localhost:6379"]     //本机默认是localhost

这里需要注意的是,Telegraf针对cpu.disk,memory,swap等默认已开启采集,net和netstat等网络相关的监控功能未开启.

检查修改后的配置文件是否存在语法错误,如果不存在,就会输出具体的指标数据信息,如果存在错误,该test会告知具体错误的地方.

# /usr/local/telegraf/usr/bin/telegraf --test --config /usr/local/telegraf/etc/telegraf/telegraf.conf 

将Telegraf配置成systemctl服务

# cat /usr/lib/systemd/system/telegraf.service
[Unit]
Description=The plugin-driven server agent for reporting metrics into InfluxDB
Documentation=https://docs.influxdata.com/telegraf/v1.15/introduction/getting-started/
After=network.target

[Service]
User=tnuser
Group=dba
Restart=always

ExecStart=/usr/local/telegraf/usr/bin/telegraf -config /usr/local/telegraf/etc/telegraf/telegraf.conf -config-directory /usr/local/telegraf/etc/telegraf/telegraf.d
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -15 $MAINPID

KillMode=control-group

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload
# systemctl start telegraf
# systemctl status telegraf

Telegraf服务启动后,我们在InfluxDB的redis库中可以看到有Telegraf agent新建的measurements.

> show measurements
name: measurements
name
----
cpu
disk
diskio
kernel
mem
net
netstat
processes
redis
redis_cmdstat
redis_keyspace
redis_replication
swap
system

查看Telegraf采集的数据

> select * from "system"
name: system
time                 host              load1 load15 load5 n_cpus n_users uptime uptime_format
----                 ----              ----- ------ ----- ------ ------- ------ -------------
2020-08-31T09:44:50Z ec1s-tempredis-01 0.01  0.05   0.02  2      1       441116 5 days,  2:31
2020-08-31T09:45:00Z ec1s-tempredis-01 0.01  0.05   0.02  2      1       441126 5 days,  2:32
2020-08-31T09:45:10Z ec1s-tempredis-01 0     0.05   0.01  2      1       441136 5 days,  2:32
2020-08-31T09:45:20Z ec1s-tempredis-01 0     0.05   0.01  2      1       441146 5 days,  2:32
2020-08-31T09:45:30Z ec1s-tempredis-01 0     0.05   0.01  2      1       441156 5 days,  2:32
2020-08-31T09:45:40Z ec1s-tempredis-01 0     0.05   0.01  2      1       441166 5 days,  2:32
2020-08-31T09:45:40Z ec1s-tempredis-02 0     0.05   0.01  2      1       440772 5 days,  2:26
2020-08-31T09:45:50Z ec1s-tempredis-01 0     0.05   0.01  2      1       441176 5 days,  2:32
2020-08-31T09:45:50Z ec1s-tempredis-02 0     0.05   0.01  2      1       440782 5 days,  2:26