![]() We run one Debezium connector (in distributed mode on the Kafka connect framework) for each microservice database. This also isolates Debezium from impacting the production OLTP workload that the master CloudSQL instances are handling. Rather than having Debezium connect to dozens of microservice databases directly, we can connect to just a single database. This single Debezium cluster is an operational trick to make it easier for us to operate Debezium. Each CloudSQL instance replicates its data into the Debezium cluster, which consists of two MySQL machines: a primary (active) server and secondary (passive) server. We’ve set up a downstream MySQL cluster specifically for Debezium. These databases run in Google Cloud as CloudSQL MySQL instances with GTIDs enabled. The flow of data starts with each microservice’s MySQL database. This proposed architecture is what we’ve been implementing at WePay, and this post describes how we leverage Debezium and Kafka connect to stream our MySQL databases into Kafka. The write-up triggered a thoughtful post on Debezium’s blog about a proposed equivalent architecture using Kafka connect, Debezium, and Confluent’s schema registry. Their architecture involves a series of homegrown pieces of software to accomplish the task, notably schematizer and MySQL streamer. These included a discussion on how they stream MySQL data into Kafka. Last year, Yelp’s engineering team published an excellent series of posts on their data pipeline. We use Kafka as our data integration layer, so we needed a way to get our database data into it. ![]() ![]() At WePay, we wanted to integrate our microservices and downstream datastores with each other, so every system could get access to the data that it needed. If you’re wondering why you might want to stream database changes into Kafka, I highly suggest reading The Hardest Part About Microservices: Your Data. Notably, using Kafka as a backbone to stream your database data in realtime has become increasingly common. Change data capture has been around for a while, but some recent developments in technology have given it new life.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |