AVRO

How kafka moves data from one system to another. Let’s talk about data transfer in kafka.

Let’s take Weather Data has fields like

• Timestamp
• Country
• Location(Longitude, Latitude)
• Weather

We will store the data into any database.

We might have System A for weather forecasting and System B for hurricane prevention. Data transfer occurred via Kafka.

Let’s says

• System A is producer, which uses PostgreSQL
• System B is Consumer, Which uses MongoDB

How to transfer data from PostgreSQL to MongoDB ??

We can transfer data with the help of kafka, what kind of data format to be put in kafka ??

First format option : CSV (command separated value)

We change PostgreSQL data into CSV file

Another format : JSON

Other format is AVRO

What is AVRO ?

AVRO is like a JSON with schema to enforce validation, so it has JSON benefits. Order is not important and AVRO can also contain data structures.

{
"weather": "LIGHT_RAIN",
"country": "IN",
"Timestamp": "1641010362",
"Geolocation": {
"Lat": -6.3040,
"long": 106.6435
}
}
}

Producer and consumer talk to the same data format. AVRO is written in binary, unlike JSON which is text based.

In JSON, data with many repeating fields can take more bytes, because json writes all repeated fields. In AVRO, such a thing has not happened, So payload size in AVRO is likely smaller than JSON.

AVRO also has schema evolution concept, which means schema can be changed in the future, but producer and consumer does not break.

AVRO drawbacks :

• AVRO programming language support is not as wide as json.
• Official AVRO library are on python, java and c family including C#.
• But other languages, like GO, PHP or others need to define their own library.
• Unlike json, which is easily read since it is text based, avro is binary, which means to actually read and write data, we need to use a library to translate the binary.

Why AVRO ?

• Kafka supports avro better
• Big data ecosystem
• Requests for course update on avro