0%

elasticsearch的磁盘清理

1. 基础命令

1.1 查看健康

1
2
3
4
5
6
7
8
# 查看健康
curl -XGET 'http://localhost:9200/_cat/health?v'

epoch     timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent

1675044251 02:04:11 elasticsearch yellow          1         1     83  83    0    0       69             0                  -                 54.6%

绿色表示一切正常, 黄色表示所有的数据可用但是部分副本还没有分配,红色表示部分数据因为某些原因不可用.

1.2 查看节点和索引

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 查看节点
curl 'localhost:9200/_cat/nodes?v'


ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1 35 97 62 3.89 3.74 3.74 cdfhilmrstw * ip-172-31-31-250


# 查看所有索引
curl 'localhost:9200/_cat/indices?v&pretty'


health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open filebeat-7.12.0-2022.11.30 Ax1f5QezQHqIuOzNEZGeOg 1 1 1951589 0 751.4mb 751.4mb
green open .kibana_task_manager_7.17.7_001 zXfuQUdyRK6GGy5XXSnF3w 1 0 17 381589 41.5mb 41.5mb
yellow open filebeat-7.12.0-2022.12.01 7S1VPbvTRA6uKaTjtb3e5g 1 1 2270280 0 881.1mb 881.1mb
yellow open filebeat-7.12.0-2022.12.02 ik_EIJx2SaCHmI4MwxZM7A 1 1 2967501 0 1.1gb 1.1gb

1.3 查看磁盘占用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# 查看占用列表
curl -XGET "http://localhost:9200/_cat/shards?v"  

index                                                         shard prirep state         docs   store ip        node
filebeat-7.12.0-2022.12.17                                    0     p     STARTED    16922173   6.9gb 127.0.0.1 ip-172-31-31-250


# 查看占用
curl -XGET 'http://localhost:9200/_cat/allocation?v'

83      291.9gb   403.8gb     80.8gb    484.6gb           83 127.0.0.1 127.0.0.1 ip-172-31-31-250
69                                                                               UNASSIGNED


# 查看磁盘
sudo du -hs /var/lib/elasticsearch/

292G /var/lib/elasticsearch/

2. 索引命令

访问 Elasticsearch 中的数据的 pattern(模式)。该 pattern(模式)可以概括如下 :

1
<REST Verb> /<Index>/<Type>/<ID>

2.1 索引操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# 1. 创建 customer 索引
curl -XPUT 'localhost:9200/customer?pretty&pretty'

{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "customer"
}


# 2. 获取所有索引,能够看到新创建的
curl -XGET 'localhost:9200/_cat/indices?v&pretty'
yellow open customer oByO4i3IT96RAHiAyaYc4g 1 1 0 0 226b 226b


# 3. 往 customer 索引增加 external 类型 id 为 1 的数据
curl -XPUT 'localhost:9200/customer/external/1?pretty&pretty' -H 'Content-Type: application/json' -d'{ "name": "John Doe" }'

{
"_index" : "customer",
"_type" : "external",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}


# 4. 从索引查询数据
curl -XGET 'localhost:9200/customer/external/1?pretty&pretty'

{
"_index" : "customer",
"_type" : "external",
"_id" : "1",
"_version" : 1,
"_seq_no" : 0,
"_primary_term" : 1,
"found" : true,
"_source" : {
"name" : "John Doe"
}
}

2.2 删除索引

1
2
3
4
5
6
7
8
9
10
11
12
# 1. 删除一个索引
curl -XDELETE 'localhost:9200/customer?pretty&pretty'
{
"acknowledged" : true
}

# 2. 获取所有索引
curl -XGET 'localhost:9200/_cat/indices?v&pretty'


# 3. 删除所有数据[谨慎]
curl -XDELETE 'http://localhost:9200/_all'

3. 清理磁盘

3.1 删除索引操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 1. 删除一些索引
curl -XDELETE 'localhost:9200/filebeat-7.12.0-2022.11.30?pretty'
curl -XDELETE 'localhost:9200/filebeat-7.12.0-2022.12.01?pretty'
curl -XDELETE 'localhost:9200/filebeat-7.12.0-2022.12.02?pretty'
curl -XDELETE 'localhost:9200/filebeat-7.12.0-2022.12.03?pretty'

# 2. 获取索引列表
curl -XGET 'localhost:9200/_cat/indices?v&pretty'

# 3. 获取磁盘占用

shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
73 276.9gb 389.3gb 95.3gb 484.6gb 80 127.0.0.1 127.0.0.1 ip-172-31-31-250
59 UNASSIGNED

3.2 批量删除操作

1
2
3
4
5
6
7
8
# 1. 获取要删除的索引
curl -XGET 'localhost:9200/_cat/shards' |awk '{print $1}' | grep filebeat| grep 2022 | uniq > /tmp/index_name.tmp

# 2. 循环删除
for i in $(cat /tmp/index_name.tmp);do curl -XDELETE http://localhost:9200/$i;done

# 3. 查询磁盘占用
curl -XGET 'http://localhost:9200/_cat/allocation?v'

3.3 循环删除脚本

vim /home/scripts/del_elasticseatch_index.sh

1
2
3
4
5
6
7
8
9
10
#!/bin/bash
#The index 30 days ago
curl -XGET 'http://localhost:9200/_cat/shards' |awk '{print $1}' |grep `date -d "30 days ago" +%Y.%m.%d` |uniq > /tmp/index_name.tmp

for index_name in `cat /tmp/index_name.tmp`
do
curl -XDELETE http://localhost:9200/$index_name
echo "${index_name} delete success" >> /home/scripts/del_elasticseatch_index.log
done

1
2
crontab -l
0 3 * * * bash /home/scripts/del_elasticseatch_index.sh

4. 参考文档

可以加首页作者微信,咨询相关问题!