springboot集成canal通过rocketmq实现数据同步到elasticsearch
- 2021-11-29 18:07:00
- admin 原创
- 3193
springboot集成canal通过rocketmq实现数据同步到elasticsearch
1、安装rocketmq,windows下安装,以及topic自动创建
下载地址为:http://rocketmq.apache.org/release_notes/release-notes-4.3.0/
cd D:\soft\rocketmq-all-4.5.2-bin-release\bin start mqnamesrv.cmd
cd D:\soft\rocketmq-all-4.5.2-bin-release\bin start mqbroker.cmd -n 127.0.0.1:9876 autoCreateTopicEnable=true
2、安装rocketmq的管理工具rocketmqconsole
cd D:\soft\rocketmq-externals-release-rocketmq-console-1.0.0\rocketmq-console\target java -jar rocketmq-console-ng-1.0.0.jar
下载地址为:https://github.com/apache/rocketmq-externals/tree/release-rocketmq-console-1.0.0
修改配置文件 application.properties
3、canal的搭建
首先下载canaladmin
https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart
修改配置文件,执行sql文件,然后启动bin下面的startup.bat
然后新建一个instance
修改instance里面的配置,修改下面的内容
4、安装canal-deploy
https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz
将conf下面的canal.properties不做修改,然后把canal_local.properties替换为canal.properties一下
启动一下,但是windows下启动会报错,修改startup.bat里面的配置,去掉红色部分的那句话,
此时要看一下rocketmq的版本号,此时springboot里面的rocketmq的版本号要和这个版本号一致,否则不会自动创建topic
5、判断canal是否已经连上rocketmq,查看一下日志,或者看一下rocketmq的rocketmq控制台是否有topic
致此后端搭建已经完成
6、下面是springboot的代码
application.yml
server: port: 8085 rocketmq: name-server: localhost:9876
pom.yml配置
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.6.0</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>com.wanhejia</groupId> <artifactId>rocketmq</artifactId> <version>0.0.1-SNAPSHOT</version> <name>rocketmq</name> <description>Demo project for Spring Boot</description> <properties> <java.version>1.8</java.version> <lombok.version>1.18.12</lombok.version> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <spring-boot.version>2.2.5.RELEASE</spring-boot.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>commons-lang</groupId> <artifactId>commons-lang</artifactId> <version>2.6</version> </dependency> <dependency> <groupId>cn.throwx</groupId> <artifactId>canal-glue-core</artifactId> <version>1.0</version> <scope>s y s t e m</scope> <s y s t e mPath>${pom.basedir}/lib/canal-glue-core.jar</s y s t e mPath> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>${lombok.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>7.7.1</version> </dependency> <!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch --> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>7.7.1</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-elasticsearch</artifactId> </dependency> <dependency> <groupId>org.apache.rocketmq</groupId> <artifactId>rocketmq-spring-boot-starter</artifactId> <version>2.0.4</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.8.1</version> <configuration> <source>1.8</source> <target>1.8</target> <encoding>UTF-8</encoding> </configuration> </plugin> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>
7、项目的代码地址:
https://github.com/liujiawan/rocket-canal
我项目用到的是组件, 非常方便,最终的效果如图所示
联系人: | 郑州-小万 |
---|---|
电话: | 13803993919 |
Email: | 1027060531@qq.com |
QQ: | 1027060531 |
网址: | www.wanhejia.com |