当前位置: 移动技术网 > 科技>操作系统>Linux > ELK-elasticsearch-6.3.2部署

ELK-elasticsearch-6.3.2部署

2018年09月09日  | 移动技术网科技  | 我要评论

13年高考零分作文,普陀山天气,八字匹配

 

参考博客:linux下elasticsearch.6.2.2集群安装与head、kibana、x-pack..插件的配置安装

参考博客:

参考博客:elk构建mysql慢日志收集平台详解

参考博客:针对logstash吞吐量一次优化

参考博客:elasticstack系列之十八 & elasticsearch5.x xpack 过期新 license 更新

 

1. 主机规划

主机名称

ip信息

内网ip

操作系统

安装软件

备注:运行程序

mini01

10.0.0.11

172.16.1.11

centos 7.4

2g内存

jdk、elasticsearch、kibana

插件head、bigdesk、cerebro

mini02

10.0.0.12

172.16.1.12

centos 7.4

4g内存

jdk、elasticsearch、logstash

 

mini03

10.0.0.13

172.16.1.13

centos 7.4

4g内存

jdk、elasticsearch、logstash

 

mini04

10.0.0.14

172.16.1.14

centos 7.4

jdk、logstash

 

 

添加hosts信息,保证每台都可以相互ping通

1 [root@mini01 ~]# cat /etc/hosts  
2 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
3 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
4 
5 10.0.0.11    mini01
6 10.0.0.12    mini02
7 10.0.0.13    mini03
8 10.0.0.14    mini04
9 10.0.0.15    mini05

 

windows的hosts文件也追加如下信息

1 c:\windows\system32\drivers\etc
2 ########################################## 追加信息如下:
3 10.0.0.11    mini01
4 10.0.0.12    mini02
5 10.0.0.13    mini03
6 10.0.0.14    mini04
7 10.0.0.15    mini05

 

 

2. 添加用户账号

1 # 使用一个专门的用户,避免直接使用root用户
2 # 添加用户、指定家目录并指定用户密码
3 useradd -d /app yun && echo '123456' | /usr/bin/passwd --stdin yun
4 # sudo提权
5 echo "yun  all=(all)       nopasswd: all" >>  /etc/sudoers
6 # 让其它普通用户可以进入该目录查看信息
7 chmod 755 /app/

 

 

3. jdk【java8】

3.1. 软件安装

 1 [yun@mini01 software]# pwd
 2 /app/software
 3 [yun@mini01 software]# tar xf jdk1.8.0_112.tar.gz 
 4 [yun@mini01 software]# ll
 5 total 201392
 6 drwxr-xr-x 8   10  143      4096 dec 20 13:27 jdk1.8.0_112
 7 -rw-r--r-- 1 root root 189815615 mar 12 16:47 jdk1.8.0_112.tar.gz
 8 [yun@mini01 software]# mv jdk1.8.0_112/ /app/
 9 [yun@mini01 software]# cd /app/
10 [yun@mini01 app]# ll
11 total 8
12 drwxr-xr-x  8   10   143 4096 dec 20 13:27 jdk1.8.0_112
13 [yun@mini01 app]# ln -s jdk1.8.0_112/ jdk
14 [yun@mini01 app]# ll
15 total 8
16 lrwxrwxrwx  1 root root    13 may 16 23:19 jdk -> jdk1.8.0_112/
17 drwxr-xr-x  8   10   143 4096 dec 20 13:27 jdk1.8.0_112

 

3.2. 环境变量

 1 [root@mini01 ~]$ pwd
 2 /app
 3 [root@mini01 ~]$ ll -d jdk*  # 选择jdk1.8  
 4 lrwxrwxrwx 1 yun yun   11 mar 15 14:58 jdk -> jdk1.8.0_112
 5 drwxr-xr-x 8 yun yun 4096 dec 20 13:27 jdk1.8.0_112
 6 [root@mini01 profile.d]$ pwd
 7 /etc/profile.d
 8 [root@mini01 profile.d]$ cat jdk.sh # java环境变量   
 9 export java_home=/app/jdk
10 export jre_home=/app/jdk/jre
11 export classpath=.:$java_home/lib/dt.jar:$java_home/lib/tools.jar:$jre_home/lib:$classpath
12 export path=$java_home/bin:$path
13 
14 [root@mini01 profile.d]# source /etc/profile
15 [root@mini01 profile.d]$ java -version  
16 java version "1.8.0_112"
17 java(tm) se runtime environment (build 1.8.0_112-b15)
18 java hotspot(tm) 64-bit server vm (build 25.112-b15, mixed mode)

 

 

4. elasticsearch部署

4.1. 软件部署

1 [yun@mini01 software]$ pwd
2 /app/software
3 [yun@mini01 software]$ tar xf elasticsearch-6.3.2.tar.gz 
4 [yun@mini01 software]$ mv elasticsearch-6.3.2 /app/
5 [yun@mini01 software]$ cd /app/
6 [yun@mini01 ~]$ ln -s elasticsearch-6.3.2/ elasticsearch

 

4.2. 环境变量

       所有安装es的机器都要有该配置

1 [root@mini01 profile.d]# pwd
2 /etc/profile.d
3 [root@mini01 profile.d]# cat es.sh 
4 export es_home="/app/elasticsearch"
5 export path=$es_home/bin:$path
6 
7 [root@mini01 profile.d]# logout
8 [yun@mini01 es-data]$ source /etc/profile  # 重加在环境变量

 

4.3. 配置修改

       由于node.name使用了变量,所以所有集群的该配置都可以一样

 1 [yun@mini01 config]$ pwd
 2 /app/elasticsearch/config
 3 [yun@mini01 config]$ vim elasticsearch.yml
 4 ………………
 5 # ---------------------------------- cluster -----------------------------------
 6 #
 7 # use a descriptive name for your cluster:
 8 #
 9 #cluster.name: my-application
10 # 集群名称
11 cluster.name: zhang-es
12 #
13 # ------------------------------------ node ------------------------------------
14 #
15 # use a descriptive name for the node:
16 #
17 #node.name: node-1
18 # 节点名称
19 node.name: ${hostname}
20 
21 #
22 # add custom attributes to the node:
23 ………………
24 # path to directory where to store the data (separate multiple locations by comma):
25 #
26 #path.data: /path/to/data
27 # 该目录需要创建
28 path.data: /app/es-data
29 
30 #
31 # path to log files:
32 #
33 #path.logs: /path/to/logs
34 # 该目录需要创建
35 path.logs: /app/es-data/logs
36 ………………
37 # lock the memory on startup:
38 #
39 # 锁定内存
40 bootstrap.memory_lock: true
41 #
42 ………………
43 # set the bind address to a specific ip (ipv4 or ipv6):
44 #
45 #network.host: 192.168.0.1
46 # 绑定
47 network.host: 0.0.0.0
48 
49 #
50 # set a custom port for http:
51 #
52 http.port: 9200
53 #
54 ………………
55 # elasticsearch performs poorly when the system is swapping the memory.
56 #
57 #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
58 # 集群节点ip或者主机
59 discovery.zen.ping.unicast.hosts: ["mini01", "mini02", "mini03"]
60 
61 #
62 # prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
63 #
64 # 防止脑裂配置,注意在多master时,这个值应该等于 math.floor(master候选节点数/2)+1
65 # 意思是master候选节点的数目最少达到多少个,才去选举master
66 # 没有这个配置,在多master时容易造成脑裂,出现多个集群
67 # 这里只有一个master就设置成1即可  【默认为1】
68 #discovery.zen.minimum_master_nodes: 
69 ………………
70 #action.destructive_requires_name: true
71 
72 # 下面两行配置为haad和bigdesk插件配置,各服务器一致
73 http.cors.enabled: true
74 http.cors.allow-origin: "*"

 

4.4. 启动es程序

 1 # -d 后台运行   -p 指定pid文件
 2 [yun@mini01 ~]$ elasticsearch -d -p /app/elasticsearch/es.pid  # 添加了环境变量,所以可以在任何地方启动程序 
 3 [yun@mini01 ~]$ cat elasticsearch/es.pid 
 4 2637
 5 [yun@mini01 ~]$ netstat -lntup | grep '9200'
 6 (not all processes could be identified, non-owned process info
 7  will not be shown, you would have to be root to see it all.)
 8 tcp6       0      0 :::9200                 :::*                    listen      2637/java   
 9 [yun@mini01 ~]$ kill -9 2637   # 停止es
10 [yun@mini01 ~]$ ps -ef | grep 'ela'
11 yun        3263   1807  0 20:56 pts/0    00:00:00 grep --color=auto ela

 

4.5. 浏览器访问

1 http://www.lhsxpumps.com/_mini01:9200/    
2 http://mini02:9200/    
3 http://mini03:9200/    

 

4.6. 启动报错与解决

1 # 启动报错如下:
2 [3] bootstrap checks failed
3 [1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
4 [2]: memory locking requested for elasticsearch process but memory is not locked
5 [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

 

4.6.1. 解决1和2

 1 # 将 soft nofile和hard nofile从 65535 改为 131070,如下:
 2 # 添加soft memlock和hard memlock
 3 [yun@mini01 ~]$ tail /etc/security/limits.conf   # 需要退出重新登录才生效
 4 #*               hard    rss             10000
 5 #@student        hard    nproc           20
 6 #@faculty        soft    nproc           20
 7 #@faculty        hard    nproc           50
 8 #ftp             hard    nproc           0
 9 #@student        -       maxlogins       4
10 
11 # end of file
12 * soft nofile 131070
13 * hard nofile 131070
14 * soft memlock unlimited
15 * hard memlock unlimited

 

4.6.2. 解决3

1 [root@mini01 ~]# vim /etc/sysctl.conf  # 追加如下信息
2 ………………
3 
4 vm.max_map_count=655360
5 [root@mini01 ~]# sysctl -p  # 生效

 

 

5. es的xpack 过期新 license 更新

       当我们直接访问es的索引,出现如下信息时,表示license已经过期,需要重新更新

       其中涉及es的账号信息------elastic:是可以 build 这个项目的超级用户,对应默认的密码为:changeme

 

命令行访问

 1 [yun@mini02 ~]$ curl -xget -u elastic:changeme 'http://mini01:9200/_xpack/license'
 2 {
 3   "license" : {
 4     "status" : "expired",
 5     "uid" : "59bc0e32-685b-48a9-bfdb-ddd373f672ab",
 6     "type" : "trial",
 7     "issue_date" : "2018-06-03t08:56:33.376z",
 8     "issue_date_in_millis" : 1528016193376,
 9     "expiry_date" : "2018-07-03t08:56:33.376z",
10     "expiry_date_in_millis" : 1530608193376,
11     "max_nodes" : 1000,
12     "issued_to" : "zhang-es",
13     "issuer" : "elasticsearch",
14     "start_date_in_millis" : -1
15   }
16 }
17 [yun@mini02 ~]$ curl -xget -u elastic:changeme 'http://mini01:9200/_cat/indices'
18 {"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":403}

 

浏览器访问

1 http://www.lhsxpumps.com/_mini01:9200/_cat/indices    

 

获取license 

1 https://register.elastic.co/marvel_register    

       需要你输入你对应的邮箱等基本信息,随便填写,但是邮箱和国家必须是真实的,将对应新下载的 license 上传到对应集群的某一台服务器上即可,我将我下载的一堆很长名字的 license 修改为简单的名称即:

mv xxxxx-license.json license.json

 

更新 license

       我们更新最新的 license 不需要重启相应的 elasticsearch 节点,只需要通过一个命令就可以动态的生效

 1 [yun@mini02 ~]$ curl -xput -u elastic:changeme 'http://mini01:9200/_xpack/license?acknowledge=true' -h "content-type: application/json" -d @license.json 
 2 {"acknowledged":true,"license_status":"valid"}
 3 [yun@mini02 ~]$ curl -xget -u elastic:changeme 'http://mini01:9200/_xpack/license'
 4 {
 5   "license" : {
 6     "status" : "active",
 7     "uid" : "aad141e1-c24b-453c-92d1-0fdf5ac63540",
 8     "type" : "basic",
 9     "issue_date" : "2018-09-07t00:00:00.000z",
10     "issue_date_in_millis" : 1536278400000,
11     "expiry_date" : "2019-09-07t23:59:59.999z",
12     "expiry_date_in_millis" : 1567900799999,
13     "max_nodes" : 100,
14     "issued_to" : "zhang lia (myself)",
15     "issuer" : "web form",
16     "start_date_in_millis" : 1536278400000
17   }
18 }

 

之后就可以正常访问了

1 [yun@mini02 ~]$ curl http://mini01:9200/_cat/indices
2 green open logstash-2018.08.21         mohgsrcbqgyyra5plchepg 5 1     9 0  74.5kb  37.2kb
3 green open nginx-access-log-2018.08.25 tjruocelrpanblj_t943ww 5 1   121 0 652.8kb 321.3kb
4 green open httpd-access-log-2018.08.31 21nenldbtnu49oig9bilnw 5 1   573 0 739.6kb 409.6kb
5 green open index-demo                  ctz0ln39smsqfolam89grg 5 1     2 0  17.9kb   8.9kb
6 green open system-rsyslog-2018.08      zhmpivsqs72dtkqzvhifbq 5 1   154 0 605.9kb 302.9kb
7 ………………

 

 

如对本文有疑问,请在下面进行留言讨论,广大热心网友会与你互动!! 点击进行留言回复

相关文章:

验证码:
移动技术网