go get github.com/prometheus/client_golang/prometheus go get github.com/prometheus/client_golang/prometheus/promauto go get github.com/prometheus/client_golang/prometheus/promhttp
sudo vi /etc/grafana/grafana.ini #In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the unified alerts section. Set the enabled property to true.
sudo systemctl restart grafana-server.service
新建报警API
飞书不支持,自己用接口中转一下吧。
报警匹配规则
3. 遇到的问题
3.1 删除old jobs和instance
Expanding on evgenyl’s answer, the exact command would be something like:
1
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="name_of_old_job"}'
Replace name_of_old_job with the name of the job you want deleted.
Reminder that you need to have started prometheus with the –web.enable-admin-api flag
1 2
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="linux-node"}' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="localhost:9100"}'
不过需要注意的是上面的 API 调用并不会立即删除数据,实际数据任然还存在磁盘上,会在后面进行数据清理。