扫二维码与项目经理沟通
我们在微信上24小时期待你的声音
解答本文疑问/技术咨询/运营咨询/技术建议/互联网交流
本篇内容介绍了“docker-compose怎么搭建 es/kibana/logstash elk”的有关知识，在实际案例的操作过程中，不少人都会遇到这样的困境，接下来就让小编带领大家学习一下如何处理这些情况吧！希望大家仔细阅读，能够学有所成！
在和硕等地区，都构建了全面的区域性战略布局，加强发展的系统性、市场前瞻性、产品创新能力，以专注、极致的服务理念，为客户提供成都网站制作、做网站 网站设计制作定制网站建设,公司网站建设,企业网站建设,成都品牌网站建设,成都全网营销推广,外贸营销网站建设,和硕网站建设费用合理。
一、准备两台centos7 虚拟机
二、新建 /home/docker-contains/es 再创建node文件夹
分别在node下新建 /data /logs /conf文件
在conf目录下 新建elasticsearch.yml文件如下
cluster.name: elasticsearch-cluster node.name: es01 network.bind_host: 0.0.0.0 network.publish_host: 192.168.65.135 http.port: 9200 transport.tcp.port: 9300 http.cors.enabled: true http.cors.allow-origin: "*" node.master: true node.data: true discovery.zen.ping.unicast.hosts: ["192.168.65.135:9300","192.168.65.136:9300"] discovery.zen.minimum_master_nodes: 1 path.logs: /usr/share/elasticsearch/logs xpack.security.audit.enabled: true ~
回到es目录：
新建docker-compose.yml如下
version: '3' services: es01: image: elasticsearch:6.6.1 container_name: es01 restart: always volumes: - /home/docker_container/es/master/data:/usr/share/elasticsearch/data:rw - /home/docker_container/es/master/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - /home/docker_container/es/master/logs:/user/share/elasticsearch/logs:rw - /home/docker_container/es/master/plugin1:/usr/share/elasticsearch/plugins:rw ports: - "9200:9200" - "9300:9300"
另外一台机器重复上述过程：
新建elasticsearch.yml：
cluster.name: elasticsearch-cluster node.name: es02 network.bind_host: 0.0.0.0 network.publish_host: 192.168.65.136 http.port: 9200 transport.tcp.port: 9300 http.cors.enabled: true http.cors.allow-origin: "*" node.master: true node.data: true discovery.zen.ping.unicast.hosts: ["192.168.65.135:9300","192.168.65.136:9300"] discovery.zen.minimum_master_nodes: 1 path.logs: /usr/share/elasticsearch/logs xpack.security.audit.enabled: true
新建docker-compose.yml:
version: '3' services: es02: image: elasticsearch:6.6.1 container_name: es02 restart: always volumes: - /home/docker-container/es/node1/data:/usr/share/elasticsearch/data:rw - /home/docker-container/es/node1/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - /home/docker-container/es/node1/logs:/user/share/elasticsearch/logs:rw - /home/docker-container/es/node1/plugin1:/usr/share/elasticsearch/plugins:rw ports: - "9200:9200" - "9300:9300"
分别在docker-compose.yml同级目录执行
docker-compose up -d
安装kibana:
docker-compose.yml
version: '3' services: kibana: image: kibana:6.6.1 container_name: kibana volumes: - /home/docker_container/kibana/conf/kibana.yml:/usr/share/kibana/config/kibana.yml restart: always ports: - 5601:5601
kibana.yml:
elasticsearch.hosts: ["http://192.168.65.135:9200"] server.host: "0.0.0.0" xpack.monitoring.ui.container.elasticsearch.enabled: true i18n.locale: zh-CN ~
如果出现 Kibana server is not ready yet
第一点：KB、ES版本不一致（网上大部分都是这么说的）
解决方法：把KB和ES版本调整为统一版本
第二点：kibana.yml中配置有问题（通过查看日志，发现了Error: No Living connections的问题）
解决方法：将配置文件kibana.yml中的elasticsearch.url改为正确的链接，默认为: http://elasticsearch:9200
改为http://自己的IP地址:9200
第三点：浏览器没有缓过来
解决方法：刷新几次浏览器（狂刷 我刷了 6遍才出来）。
logstash:
logstash采用非docker安装，上传logstash-6.4.3.tar.gz至/home/software目录
解压缩gz包，进入到解压后的logstash-6.4.3目录
编辑：vim config/pipelines.yml （如果需要连接多个 只需在后面追加pipeline.id和path.config，切记pipeline.id不能重复）
# Default is path.data/dead_letter_queue # # path.dead_letter_queue: - pipeline.id: table1 path.config: "/home/software/logstash-6.4.3/conf/MySQL_1.conf" - pipeline.id: table2 path.config: "/home/software/logstash-6.4.3/conf/mysql.conf"
mysql.conf:
input {
jdbc {
jdbc_driver_library => "/home/software/logstash-6.4.3/sql/mysql-connector-java-5.1.46.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://ip:3306/test"
jdbc_user => "root"
jdbc_password => ""
schedule => "* * * * *"
statement => "SELECT * FROM user WHERE update_time >= :sql_last_value"
use_column_value => true
tracking_column_type => "timestamp"
tracking_column => "update_time"
last_run_metadata_path => "syncpoint_table"
}
}
output {
elasticsearch {
# ES的IP地址及端口
hosts => ["192.168.65.135:9200","192.168.65.136:9200"]
# 索引名称 可自定义
index => "user"
# 需要关联的数据库中有有一个id字段，对应类型中的id
document_id => "%{id}"
document_type => "user"
}
stdout {
# JSON格式输出
codec => json_lines
}
}mysql_1.conf:
input {
jdbc {
jdbc_driver_library => "/home/software/logstash-6.4.3/sql/mysql-connector-java-5.1.46.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://ip:3306/lvz_goods?autoReconnect=true&useUnicode=true&createDatabaseIfNotExist=true&characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2B8"
jdbc_user => "root"
jdbc_password => ""
schedule => "* * * * *"
statement => "SELECT * FROM lvz_product WHERE update_time >= :sql_last_value"
use_column_value => true
tracking_column_type => "timestamp"
tracking_column => "update_time"
last_run_metadata_path => "syncpoint_table"
}
}
output {
elasticsearch {
# ES的IP地址及端口
hosts => ["192.168.65.135:9200","192.168.65.136:9200"]
# 索引名称 可自定义
index => "goods"
# 需要关联的数据库中有有一个id字段，对应类型中的id
document_id => "%{id}"
document_type => "goods"
}
stdout {
# JSON格式输出
codec => json_lines
}
}注意：需要把mysql连接包放到对应目录；至此三大件安装就完成了
es整合ik和拼音分词器 1下载elasticsearch-analysis-ik-6.6.1.zip
分别上传到两台机器上的node1和 node2目录 ，解压重命名为ik
2下载elasticsearch-analysis-pinyin-6.6.1.zip
分别上传到两台机器上的node1和 node2目录 ，解压重命名为pinyin
编辑docker-compose文件 挂载：
/home/docker-container/es/node1/plugin1:/usr/share/elasticsearch/plugins:rw
重启es，到kibana界面 ：
执行
先删除goods索引
在goods索引自定义ik和拼音分词器，ik_smart_pinyin：
DELETE /goods
PUT /goods
{
"settings": {
"analysis": {
"analyzer": {
"ik_smart_pinyin": {
"type": "custom",
"tokenizer": "ik_smart",
"filter": ["my_pinyin", "word_delimiter"]
},
"ik_max_word_pinyin": {
"type": "custom",
"tokenizer": "ik_max_word",
"filter": ["my_pinyin", "word_delimiter"]
}
},
"filter": {
"my_pinyin": {
"type" : "pinyin",
"keep_separate_first_letter" : true,
"keep_full_pinyin" : true,
"keep_original" : true,
"limit_first_letter_length" : 16,
"lowercase" : true,
"remove_duplicated_term" : true
}
}
}
}
}指定goods索引为ik_smart_pinyin
POST /goods/_mapping/goods
{
"goods": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"attribute_list": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"category_id": {
"type": "long"
},
"created_time": {
"type": "date"
},
"detail": {
"type": "text",
"analyzer":"ik_smart_pinyin",
"search_analyzer":"ik_smart_pinyin"
},
"id": {
"type": "long"
},
"main_image": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"analyzer":"ik_smart_pinyin",
"search_analyzer":"ik_smart_pinyin"
},
"revision": {
"type": "long"
},
"status": {
"type": "long"
},
"sub_images": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"subtitle": {
"type": "text",
"analyzer":"ik_smart",
"search_analyzer":"ik_smart"
},
"updated_time": {
"type": "date"
}
}
}
}安装kafka:
version: '2' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka ports: - "9092" # kafka 把9092端口随机映射到主机的端口 environment: KAFKA_ADVERTISED_HOST_NAME: 192.168.65.135 #本机ip KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_CREATE_TOPICS: test:1:1 volumes: - /var/run/docker.sock:/var/run/docker.sock
启动两个节点 kafka:
docker-compose up -d --scale kafka=2 本机启动一个有两个节点的 Kafka 集群
本地项目集成，kafka和elk;
application.yml
###服务启动端口号 server: port: 8500 ###服务名称(服务注册到eureka名称) eureka: client: service-url: defaultZone: http://localhost:8100/eureka spring: application: name: app-lvz-goods redis: host: 192.168.65.136 port: 6379 password: feilvzhang pool: max-idle: 100 min-idle: 1 max-active: 1000 max-wait: -1 ###数据库相关连接 datasource: username: root password: driver-class-name: com.mysql.jdbc.Driver url: jdbc:mysql://192.168.125.113:3306/lvz_goods?autoReconnect=true&useUnicode=true&createDatabaseIfNotExist=true&characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2B8 data: elasticsearch: ####集群名称 cluster-name: elasticsearch-cluster ####地址 cluster-nodes: 192.168.65.135:9300,192.168.65.136:9300 kafka: bootstrap-servers: 192.168.65.135:32768,192.168.65.135:32769
日志切面：
package com.lvz.shop.elk.aop;
import com.alibaba.fastjson.JSONObject;
import com.lvz.shop.elk.kafka.KafkaSender;
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.annotation.AfterReturning;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
import org.aspectj.lang.annotation.Pointcut;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.web.context.request.RequestContextHolder;
import org.springframework.web.context.request.ServletRequestAttributes;
import javax.servlet.http.HttpServletRequest;
import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
/**
* @description: ELK拦截日志信息
* @author: flz
* @date: 2019/8/9 15:57
*/
@Aspect
@Component
public class AopLogAspect {
@Autowired
private KafkaSender kafkaSender;
// 申明一个切点 里面是 execution表达式
@Pointcut("execution(* com.lvz.shop.*.impl.*.*(..))")
private void serviceAspect() {
}
// 请求method前打印内容
@Before(value = "serviceAspect()")
public void methodBefore(JoinPoint joinPoint) {
ServletRequestAttributes requestAttributes = (ServletRequestAttributes) RequestContextHolder
.getRequestAttributes();
HttpServletRequest request = requestAttributes.getRequest();
// // 打印请求内容
// log.info("===============请求内容===============");
// log.info("请求地址:" + request.getRequestURL().toString());
// log.info("请求方式:" + request.getMethod());
// log.info("请求类方法:" + joinPoint.getSignature());
// log.info("请求类方法参数:" + Arrays.toString(joinPoint.getArgs()));
// log.info("===============请求内容===============");
JSONObject jsonObject = new JSONObject();
SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");// 设置日期格式
//请求时间
jsonObject.put("request_time", df.format(new Date()));
//请求URL
jsonObject.put("request_url", request.getRequestURL().toString());
//请求方法
jsonObject.put("request_method", request.getMethod());
//请求类方法
jsonObject.put("signature", joinPoint.getSignature());
//请求参数
jsonObject.put("request_args", Arrays.toString(joinPoint.getArgs()));
JSONObject requestJsonObject = new JSONObject();
requestJsonObject.put("request", jsonObject);
kafkaSender.send(requestJsonObject);
}
// 在方法执行完结后打印返回内容
@AfterReturning(returning = "o", pointcut = "serviceAspect()")
public void methodAfterReturing(Object o) {
// log.info("--------------返回内容----------------");
// log.info("Response内容:" + gson.toJson(o));
// log.info("--------------返回内容----------------");
JSONObject respJSONObject = new JSONObject();
JSONObject jsonObject = new JSONObject();
SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");// 设置日期格式
jsonObject.put("response_time", df.format(new Date()));
jsonObject.put("response_content", JSONObject.toJSONString(o));
respJSONObject.put("response", jsonObject);
kafkaSender.send(respJSONObject);
}
} kafka消息发送：