springboot集成canal通过rocketmq实现数据同步到elasticsearch

2021-11-29 18:07:00
admin
原创
210
摘要:springboot集成canal通过rocketmq实现数据同步到elasticsearch

springboot集成canal通过rocketmq实现数据同步到elasticsearch

1、安装rocketmq,windows下安装,以及topic自动创建

下载地址为:http://rocketmq.apache.org/release_notes/release-notes-4.3.0/

cd D:\soft\rocketmq-all-4.5.2-bin-release\bin
start mqnamesrv.cmd

cd D:\soft\rocketmq-all-4.5.2-bin-release\bin
start mqbroker.cmd -n 127.0.0.1:9876 autoCreateTopicEnable=true

2、安装rocketmq的管理工具rocketmqconsole

cd D:\soft\rocketmq-externals-release-rocketmq-console-1.0.0\rocketmq-console\target
java -jar rocketmq-console-ng-1.0.0.jar

下载地址为:https://github.com/apache/rocketmq-externals/tree/release-rocketmq-console-1.0.0

修改配置文件 application.properties

3、canal的搭建

首先下载canaladmin

https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart

修改配置文件,执行sql文件,然后启动bin下面的startup.bat

然后新建一个instance

修改instance里面的配置,修改下面的内容

4、安装canal-deploy

https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz

将conf下面的canal.properties不做修改,然后把canal_local.properties替换为canal.properties一下

启动一下,但是windows下启动会报错,修改startup.bat里面的配置,去掉红色部分的那句话,

此时要看一下rocketmq的版本号,此时springboot里面的rocketmq的版本号要和这个版本号一致,否则不会自动创建topic

5、判断canal是否已经连上rocketmq,查看一下日志,或者看一下rocketmq的rocketmq控制台是否有topic

致此后端搭建已经完成

6、下面是springboot的代码

application.yml

server:
  port: 8085
rocketmq:
  name-server: localhost:9876


pom.yml配置

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.6.0</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.wanhejia</groupId>
    <artifactId>rocketmq</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>rocketmq</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>1.8</java.version>
        <lombok.version>1.18.12</lombok.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <spring-boot.version>2.2.5.RELEASE</spring-boot.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>commons-lang</groupId>
            <artifactId>commons-lang</artifactId>
            <version>2.6</version>
        </dependency>
        <dependency>
            <groupId>cn.throwx</groupId>
            <artifactId>canal-glue-core</artifactId>
            <version>1.0</version>
            <scope>s y s t e m</scope>
            <s y s t e mPath>${pom.basedir}/lib/canal-glue-core.jar</s y s t e mPath>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>${lombok.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.7.1</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch -->
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>7.7.1</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.rocketmq</groupId>
            <artifactId>rocketmq-spring-boot-starter</artifactId>
            <version>2.0.4</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>


7、项目的代码地址:

https://github.com/liujiawan/rocket-canal

我项目用到的是组件, 非常方便,最终的效果如图所示

    发表评论
    评论通过审核之后才会显示。
    文章分类
    联系方式
    联系人: 郑州-小万
    电话: 13803993919
    Email: 1027060531@qq.com
    QQ: 1027060531
    网址: www.wanhejia.com