Kafka Community Spotlight #5
1. Personal
Please tell us about yourself, and where you are from.
I was born in Saint Petersburg and graduated from the university there. During my studies, I started working as a Java developer, and two years after graduation, I moved to Prague and continued my career as a software engineer there.
I started learning programming at school when I was 12, and since then I’ve been constantly programming something.
How do you spend your free time? What are your hobbies?
A usual set of European hobbies: cycling and hiking in the summer, and alpine skiing in the winter. What I enjoy most is hiking in the mountains with my dog.
What does your ideal weekend look like?
It could be a full day cycling trip, a day in the mountains, or a slow morning followed by going to a spa or just reading books.
Last book you read? Or a book you want to recommend to readers?
I recently read Think Distributed Systems by Dominik Tornow. If you want to dive into distributed systems, I highly recommend this book.
Best type of music, best song?
I always like discovering something new. One day it’s rock music, another day it’s rap. My taste is pretty broad, so I don’t really have one specific “best” song.
According to Spotify, the band I listen to the most is Imagine Dragons.
Favorite food? Best cuisine you’d recommend?
I’m not really a foodie. I like different kinds of grilled food because it’s easy to cook, and sushi because it’s easy to order.
Favorite Movie?
Anything by Tarantino. Probably Pulp Fiction — sorry for not being very original 😉
What is the best advice you ever got?
That the path is as important as the result. You should choose a path you genuinely enjoy, no matter what.
Have you studied at a university? If yes, was it related to computers? Did your study help you with your current job?
I got a degree in applied mathematics, and after several years I returned to the university to pursue a PhD in computer science.
It definitely helped me get my first job, but not directly. The diploma itself wasn’t that important, the knowledge was. For me, the academic environment just worked. It was a natural and structured way to gain knowledge. Also, education in Russia is free, which made this path more accessible.
That said, there are many different ways to learn computer science, especially nowadays.
2. Kafka
How did you get into Kafka?
I was working on the integration team at a bank when one day our team leader said to me:
“Hey, there’s a new message broker everyone is talking about. I read about it, but I didn’t understand much. Could you take a look at it?”
That’s how it all began.
What Version of Kafka did you start with?
I don’t remember exactly, probably 0.10.0? It was a long time ago.
When do you think one ought to use Kafka?
The most common answer to this question is: when you need to transfer messages at a high rate. But I don’t fully agree.
I think you should use Kafka when you need a proven, reliable message broker with a strong community behind it.
What do you think is the most common use case(s) for Kafka?
I think the most common use case is the pure publish–subscribe pattern. So simply transferring data between systems.
What do you think are niche(r) use cases for Kafka?
I think one of the more niche use cases is definitely event sourcing. Although I haven’t seen many real-world implementations where Kafka is used directly as long-term storage or as the main source of truth.
Online machine learning also seems very promising. But again, at the moment it’s mostly in the prototype phase rather than production use.
Do you think Kafka has a high entry barrier?
Yes, definitely, but not so much on the development side. With modern frameworks, it’s quite easy to write a service that produces or consumes messages. You just commit, push, deploy, and everything works.
But these frameworks hide many details, and once a problem occurs, you really need to understand what’s going on under the hood. And that’s not easy.
It’s similar when operating Kafka. You install it, and it works… until it doesn’t. The knowledge required to run Kafka reliably in production is significant.
Do you think Kafka is hard to manage?
Yes, as I said, it’s challenging on the operational side. And it’s not just about the brokers, it also involves Kafka Connect, KRaft controllers, and streaming applications.
What’s the most annoying thing in Kafka you can think of?
Annoying? I’m not sure. But I would say one of the toughest mental obstacles when using Kafka is that the brokers don’t validate the data. For architects, that can be quite hard to accept and overcome.
If you had a magic wand and could instantly and frictionlessly contribute/fix one thing to Kafka, what would it be?
Omg — the access control lists. They’re a disaster. I really don’t like working with them. If I could fix one thing, it would definitely be that.
How has Kafka changed over the years from your point of view?
It started as just a broker, but now it’s not only about Kafka itself, it’s about the ecosystem built around it.
At the last Kafka Summit, almost nobody was talking about Kafka. Instead, the focus was on Iceberg, Flink, and AI in different forms.
Why do you think the topic of discussion has changed?
It definitely changed. That’s why it’s no longer called “Kafka Summit” but “Current” 🙂
Kafka as a messaging system has become mature. We understand its strengths and its trade-offs, and it’s widely adopted. When a technology reaches that level of maturity, the interesting discussions naturally move to what can be built on top of it.
So now the focus is less on the broker itself and more on streaming applications, data platforms, Iceberg, Flink, AI — the broader ecosystem around Kafka.
What is the hardest thing you built in or around Kafka?
Some Kafka Streams applications can be truly complex. For example, I worked on an online recommendation system built entirely on Kafka with the help of the Kafka Streams framework.
When should one choose Kafka? When should one not choose it?
Oh, there are plenty of articles about when to use Kafka and when not to.
I think it’s a good choice when you need a reliable messaging solution. Just remember that you need a team that really understands how it works.
It’s also a mistake to try to use it everywhere. People often try to build batch processing on top of Kafka or implement synchronous patterns like request–response. That rarely ends well.
You’ve been writing a fair amount about Flink recently. How come?
Well, we need a solution for data transformation, we need a real-time processing framework. With ksqlDB being decommissioned, that space feels a bit empty, and Flink seems promising to fill it.
But the entry barrier is quite high, and there isn’t that much practical information about Flink available. That’s why I started writing a series of posts about it.
How do you compare Flink to alternatives like Kafka Streams, kSQL (now abandoned), Bento, Connect SMTs, and even regular Consumer/Producer?
Flink is a truly feature-rich framework, but the side effect is that it’s quite complex. SMTs in Kafka Connect are stateless, so once you need stateful transformations, they’re simply not enough. As for Bento, I don’t have much hands-on experience with it.
Kafka Streams is extremely good, but it’s a Java library, so you need to write Java code. For me, that’s fine, I’ve been programming in Java for a long time. But many people prefer to define transformations declaratively. That’s where ksqlDB had its place. Now that it’s gone, we need some kind of replacement, and at the moment Flink seems like the most viable alternative.
Another important difference is that Kafka Streams is a library embedded in your application, while Flink runs as a separate cluster. That’s a significant architectural difference.
There are also multiple “flavors” of Flink now, and it’s not always clear which one to choose. If we’re talking about Apache Flink, I find the overall developer experience a bit disappointing. It still relies on older Java versions and older libraries, and even building a proper Docker image with the right dependencies can take quite some time.
Actually, I expect Flink to become more user-friendly in the near future.
Do you have any contrarian opinions on stream processing?
A lot of people think that stream processing is overrated and not worth the operational complexity. They argue that you should just transfer the data in a simple way and handle everything at the end.
But I think that for real-time transformations, stream processing fits perfectly. Right now it often lacks proper tooling, but I hope that will improve over time.
3. Business/Work
What do you do at your job? What is the day to day of a Kafka Engineer?
Ha-ha, it depends on what happened in production during the night. It can easily start — and end — with fixing critical issues. Debugging and diving into logs is a normal part of the job.
On quieter days, the mornings are usually filled with meetings and the usual agile routines. Then in the afternoon, the technical work begins. Depending on the project, that could mean programming, working on architectural proposals, or setting up operational aspects.
I also teach Kafka on-site. I have a two-day course, and on those days I teach from 9 to 5.
How come you decided to write a book?
I was approached by Manning with the idea of writing the next edition of Kafka in Action. It’s a great book, but I have a very different style of explaining things, so I decided to decline.
But then, after several meetings with the publishers, they asked if I would consider writing a different book, a book for architects. I thought that if I ever wanted to write a book, I should do it now, especially when someone was already interested. So I said yes.
Tell me “why should I read the book”? What’s the ideal audience?
The idea was to explain Kafka concepts without code listings. It’s not tied to any specific programming language, and you don’t even need programming experience to read it.
The book doesn’t provide a collection of ready-made solutions. Instead, it focuses on explaining how Kafka actually works under the hood and why it behaves the way it does.
Once you truly understand the fundamentals, you can design the right solutions yourself, regardless of the tools or frameworks you use.
We have been given a 45% discount code for the book as part of this interview! You can use code
kafkaguy45to get 45% off the book here: Kafka for Architects (valid until March 31, 2026)
What’s the biggest thing you learned while writing Kafka for Architects?
That writing a book is not just about writing the book 😉 A huge part of it is a hidden iceberg. You work with so many people — editors, graphic designers, copywriters, marketers. Writing the text is actually the simplest part.
But seriously, I’m really glad I did it. First, it’s my way of giving back to the community I’ve learned so much from. And the process pushed me to constantly explore new topics and keep my knowledge sharp.
4. General/Parting
What do you think about the Kafka community? (the Apache one, that is)
The Kafka community is very strong. In fact, I think it’s one of the main reasons Kafka became so popular.
Take Apache Pulsar, for example, it has good benchmarks, but the community is smaller. And when you’re running a system in production, the strength of the community really matters.
If you look at some of the newer cloud-based products that implement the Kafka protocol, they’re probably great. But if something goes wrong, you depend entirely on their support. Many engineers (including me) prefer to understand why a problem happened and be able to fix it themselves.
What do you think about queues?
There were no queues in Kafka for such a long time that businesses adapted to working without them. I think they will find their place, but so far I haven’t used them in a real project.
What do you think about schemas?
Having data contracts is a must. Without schemas, large distributed systems quickly become chaotic.
But managing schemas in Kafka can be challenging. Schema evolution, especially when dealing with incompatible changes, requires strong coordination between teams. Some cloud-based Kafka platforms provide additional governance and validation features, but in the open-source ecosystem, much of that responsibility remains with the teams themselves.
And it’s not only about schemas, it’s about governance around Kafka as a whole. Where should topic definitions be stored? Who owns the data? How are changes communicated between teams? These are all very important questions. That’s why I dedicated an entire chapter in my book to how to properly organize a Kafka-based project.
What do you think about Kafka proxies?
You mean a layer between clients and brokers that intercepts traffic? It can be useful for enforcing security and improving observability.
But any additional layer in a distributed system is a potential point of failure and should be introduced carefully.
How do you see the future of Kafka usage and development, 5 years out?
I think the brokers themselves are already quite mature, so I don’t expect any major breakthroughs there.
But streaming technology as a whole will continue to evolve, and I believe we’ll see many interesting products and improvements in the near future.
After all, if several systems need to exchange data, why can’t they just agree on the format and contracts without constant human intervention? 🙂 I think we’ll get closer and closer to that point.
Do you think we’ve innovated in the messaging space in the last 10 years? How have you seen the space change?
I’ve definitely seen significant progress. And it is not only in messaging itself, but in how the entire architecture landscape has changed.
The major shift was moving from monoliths to microservices, and microservices need a communication backbone. That was a big trigger for the evolution of modern messaging systems. We moved from simple message queues to the distributed logs.
At the same time, we moved away from traditional enterprise service buses and started pushing their logic into smaller services. That shift also drove the development of streaming applications.
What other tech besides messaging do you have interest in?
All the developments around AI are very interesting, although the space is quite crowded at the moment.
Now that I’ve finished my book, I’ll have more free time, and I’m planning to focus more on performance engineering and observability.
Anything else you’d like to add?
A lot of people worry about AI and job replacement, but I believe real engineering skills never expire. Tools change, technologies change, programming languages come and go, but understanding how things really work remains valuable.
We’re living in a time of incredible opportunities to push technology forward. So let’s enjoy the party 🙂
Any Social Media channels of yours we should be aware of?
When I have time (which isn’t very often 🙂), I write about big data topics on Medium.