Levon's Blog

微信: L6241425

0%

elasticsearch的磁盘清理

发表于 更新于 分类于 ， 阅读次数： Valine：
本文字数： 4.6k 阅读时长 ≈ 4 分钟

1. 基础命令

1.1 查看健康

1
2
3
4
5
6
7
8
# 查看健康
curl -XGET 'http://localhost:9200/_cat/health?v'
​
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
​
1675044251 02:04:11 elasticsearch yellow          1         1     83  83    0    0       69             0                  -                 54.6%
​
绿色表示一切正常, 黄色表示所有的数据可用但是部分副本还没有分配,红色表示部分数据因为某些原因不可用.

1.2 查看节点和索引

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 查看节点
curl 'localhost:9200/_cat/nodes?v'


ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
127.0.0.1           35          97  62    3.89    3.74     3.74 cdfhilmrstw *      ip-172-31-31-250


# 查看所有索引
curl 'localhost:9200/_cat/indices?v&pretty'


health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat-7.12.0-2022.11.30      Ax1f5QezQHqIuOzNEZGeOg   1   1    1951589            0    751.4mb        751.4mb
green  open   .kibana_task_manager_7.17.7_001 zXfuQUdyRK6GGy5XXSnF3w   1   0         17       381589     41.5mb         41.5mb
yellow open   filebeat-7.12.0-2022.12.01      7S1VPbvTRA6uKaTjtb3e5g   1   1    2270280            0    881.1mb        881.1mb
yellow open   filebeat-7.12.0-2022.12.02      ik_EIJx2SaCHmI4MwxZM7A   1   1    2967501            0      1.1gb          1.1gb

1.3 查看磁盘占用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# 查看占用列表
curl -XGET "http://localhost:9200/_cat/shards?v"  
​
index                                                         shard prirep state          docs   store ip        node
filebeat-7.12.0-2022.12.17                                    0     p      STARTED    16922173   6.9gb 127.0.0.1 ip-172-31-31-250


# 查看占用
curl -XGET 'http://localhost:9200/_cat/allocation?v'
​
83      291.9gb   403.8gb     80.8gb    484.6gb           83 127.0.0.1 127.0.0.1 ip-172-31-31-250
69                                                                               UNASSIGNED


# 查看磁盘
sudo du -hs /var/lib/elasticsearch/

292G    /var/lib/elasticsearch/

2. 索引命令

访问 Elasticsearch 中的数据的 pattern（模式）。该 pattern（模式）可以概括如下 :

1
<REST Verb> /<Index>/<Type>/<ID>

2.1 索引操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# 1. 创建 customer 索引
curl -XPUT 'localhost:9200/customer?pretty&pretty'

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "customer"
}


# 2. 获取所有索引，能够看到新创建的
curl -XGET 'localhost:9200/_cat/indices?v&pretty'
yellow open   customer                        oByO4i3IT96RAHiAyaYc4g   1   1          0            0       226b           226b


# 3. 往 customer 索引增加 external 类型 id 为 1 的数据
curl -XPUT 'localhost:9200/customer/external/1?pretty&pretty' -H 'Content-Type: application/json' -d'{ "name": "John Doe" }'

{
  "_index" : "customer",
  "_type" : "external",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}


# 4. 从索引查询数据
curl -XGET 'localhost:9200/customer/external/1?pretty&pretty'

{
  "_index" : "customer",
  "_type" : "external",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "name" : "John Doe"
  }
}

2.2 删除索引

1
2
3
4
5
6
7
8
9
10
11
12
# 1. 删除一个索引
curl -XDELETE 'localhost:9200/customer?pretty&pretty'
{
  "acknowledged" : true
}

# 2. 获取所有索引
curl -XGET 'localhost:9200/_cat/indices?v&pretty'


# 3. 删除所有数据[谨慎]
curl -XDELETE 'http://localhost:9200/_all'

3. 清理磁盘

3.1 删除索引操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 1. 删除一些索引
curl -XDELETE 'localhost:9200/filebeat-7.12.0-2022.11.30?pretty'
curl -XDELETE 'localhost:9200/filebeat-7.12.0-2022.12.01?pretty'
curl -XDELETE 'localhost:9200/filebeat-7.12.0-2022.12.02?pretty'
curl -XDELETE 'localhost:9200/filebeat-7.12.0-2022.12.03?pretty'

# 2. 获取索引列表
curl -XGET 'localhost:9200/_cat/indices?v&pretty'

# 3. 获取磁盘占用

shards disk.indices disk.used disk.avail disk.total disk.percent host      ip        node
    73      276.9gb   389.3gb     95.3gb    484.6gb           80 127.0.0.1 127.0.0.1 ip-172-31-31-250
    59                                                                               UNASSIGNED

3.2 批量删除操作

1
2
3
4
5
6
7
8
# 1. 获取要删除的索引
curl -XGET 'localhost:9200/_cat/shards' |awk '{print $1}' | grep filebeat| grep 2022 | uniq > /tmp/index_name.tmp

# 2. 循环删除
for i in $(cat /tmp/index_name.tmp);do curl -XDELETE http://localhost:9200/$i;done

# 3. 查询磁盘占用
curl -XGET 'http://localhost:9200/_cat/allocation?v'

3.3 循环删除脚本

vim /home/scripts/del_elasticseatch_index.sh

1
2
3
4
5
6
7
8
9
10
#!/bin/bash
#The index 30 days ago
curl -XGET 'http://localhost:9200/_cat/shards' |awk '{print $1}' |grep `date -d "30 days ago" +%Y.%m.%d` |uniq > /tmp/index_name.tmp

for index_name in `cat /tmp/index_name.tmp`  
do
    curl -XDELETE  http://localhost:9200/$index_name
    echo "${index_name} delete success" >> /home/scripts/del_elasticseatch_index.log
done

1
2
crontab -l
0 3 * * * bash /home/scripts/del_elasticseatch_index.sh

4. 参考文档

可以加首页作者微信，咨询相关问题！