Kafka Community Spotlight #2
1. Personal
Please tell us about yourself, and where you are from.
I’m a builder at heart.
I first discovered programming at around 12 through a video game, Colobot and a few years later with YaBasic. I distinctly remember I could draw shapes so I wanted to write a tic tac toe, but I’m not even sure I knew what a variable was, so it didn’t go well…
It got serious at 15-16 when I met an incredible dude during high school who was knee-deep into programming. He taught me how to use the WAMP stack. I think it was around the PHP5 release. I did a Master’s degree in Computer Science and got my first job in 2010.
I worked for Michelin for 12 years. I almost exclusively did Integration, both batch and real-time.
Datastage, Informatica, Websphere Message Broker, Oracle Fusion Middleware, Talend, and ultimately, Kafka.
Before Kafka, I was a middleware user, building integrations to serve business needs.
With Kafka, I switched sides. As a member of a Platform Team, my “business” was to serve and help other teams make the most out of Kafka.
That’s how I started contributing to AKHQ, and created Michelin’s own namespacing model on top of Kafka (ns4kafka, still being used and actively maintained after 6 years!) and other things that I wish I had if I were a Kafka developer.
I joined Conduktor to bring the same ideas to life, except with a much bigger impact. At Michelin, I helped a few dozens of application teams. At Conduktor I want to think I helped thousands.
Being Product Manager for 3 and half years, I’ve had the opportunity to speak with so many organizations and talk through their problems. It was an incredible experience.
All of that puts me in a very interesting position: I have a clear idea of what we gained with Kafka. But I was already building real-time data pipeline for 8 years, so I also know what we lost.
How do you spend your free time? What are your hobbies?
Board games and building Lego with my family. I enjoy a good restaurant, a good bread and most importantly a good cheese.
Other than that, I’m probably behind a computer. Most likely playing video games. Or building Streemlined.
I lived and breathed motorbikes since I was 14. Like for real, I didn’t have a car license until 31. Sadly, covid and then remote work killed this passion. I still own a beautiful Kawasaki 636 from 2005. I enjoy watching F1, but not Moto GP. No idea why.
I’m generally curious, and I have acquired a set of unrelated skills over the years… solving rubik’s cube, lockpicking, flying a plane, baking sourdough bread.
Any Social Media channels of yours we should be aware of?
I publish once a week on LinkedIn. All my articles are available on Medium as well.
I advocate for common sense and simple / boring tech.
I’m well aware of the paradox: I sell a Kafka product while advocating against complexity. I live happily with that paradox for a very simple reason: I know most teams didn’t choose Kafka. It’s something that they have to live with for reasons beyond their control.
Some of the reasons are actually quite valid! Kafka isn’t bad. Flink isn’t bad. It’s the hammer and nail metaphor from the vendors that’s bad and needs a bit of reality check from the people in the trenches.
2. Kafka
How, when and why did you get into Kafka?
I joined the Michelin Manufacturing domain at a very interesting time.
Michelin had a big program running since 2017 and successfully deployed a few pilots in several plants.
Moving from pilots to full production across dozens of plants proved that extracting data is an organizational challenge before a technical one:
- Data-producing teams aren’t used to sharing their information
- There is no requirement or reward for them to expose data
- Teams lack the proper tools to make data available
- There is no business partnership to help manage:
- Data ownership and governance
- Standardized naming conventions and definitions
To bridge the gap between those organizational roadblocks and the technology, we needed a central nervous system that simplifies data sharing. This is where Kafka shines. Not as a high-throughput distributed middleware but as a means to make application teams aware and responsible of the Data they own, force them to structure it for external consumption, and “step up” their data modeling.
To this day I still deeply believe this is the reason Kafka is a success in organizations that don’t deal with millions of messages per seconds.
I was part of a five-person platform team responsible for deploying Kafka across all Michelin plants. We handled the entire lifecycle. From hardware and networking to automated deployments and application team support.
Our team was small but highly skilled and eager to learn. We taught ourselves the entire stack, from hardware to high-level architecture, by diving into the details and solving problems as we went. In fact, two original members are now at Confluent. We started our journey on Kafka 2.1 using the Confluent Community license.
Do you think Kafka has a high entry barrier?
I think the typical Java developer in a very large company is busy enough focusing on bringing value to his business. If he sticks to kafka-clients, he should read the docs a little bit and he will probably be fine.
The problem starts when he’s being told that in order to filter out and store a few events from an external topic in a database, he has to use Kafka Connect, but also Kafka Streams because his filter can’t be applied through SMT. Because that’s the “standard way”.
Imagine hearing this for the first time:
To store an event in a database, on top of your own business app (probably some code + a database), you need to deploy a Kafka Streams application to filter out a topic, and then configure a Kafka Connector to sink that newly created topic.
You’re literally doubling this team’s required number of artifacts to build.
And version control.
And deploy.
And operate.
And maintain.
What’s the most annoying thing in Kafka you can think of?
I’m sad that Apache Kafka (and Confluent really) just sticks to moving bytes on the core product.
But because Kafka refused to “move up” into the data layer, the community is now forced to build side cars to implement governance features that should have lived on the brokers from day one.
The Schema Registry is a good example for something that should probably be a core component.
I’m not blaming the engineering teams. I suspect they were never given the mandate to build something truly integrated.
Encryption, masking, filtering. A lot of reasonable features that you can’t apply on “just bytes” but that are still heavily requested by organizations. You need proxies (Conduktor, Kroxylicious) to do that.
Those decisions come at the cost of increased complexity for organizations.
I’m also very annoyed by some Kafka influencers.
The ones that use any occasion to cram the tech of their employer down their audience’s throat at the expense of their customer’s best interests. Anatoly Zelenin actually wrote about this recently. He wrote it better than I possibly could:
https://www.linkedin.com/feed/update/urn:li:activity:7417875662813868032/
What’s your favorite feature in Kafka?
Without a doubt Kafka Connect. The people who wrote Kafka Connect have created amazing things.
The Kafka Connect Data API for instance is excellent.
It brings meaning to the abstract bytes of the dumb broker in a consistent way whether your data is plain json, avro, protobuf… all my complaints about Kafka brokers evaporate when I work with Kafka Connect.
The Kafka Connect architecture itself — sure the default deployment model doesn’t work for every organization, but the underlying architecture is generic and extensible. It’s the same code running whether distributed mode, standalone mode, or mirror maker, just with a slightly adapted orchestrator.
I’m so convinced by this architecture that it’s at the core of Streemlined.
Messages flowing through stages in Streemlined are Kafka Connect API objects.
Would you recommend Kafka for side projects?
Unless your side project involves Fraud detection (lol), you should probably just use a SQL database.
If you had a magic wand and could instantly contribute/fix one thing to Kafka, what would it be?
Knowing what we know today about how Kafka was adopted for different use cases than originally intended, and that 80% of Kafka deployments are sub 1Mb/s, I’d swap around the smart clients, dumb brokers philosophy.
Application teams don’t have time to worry about the 100 available configs of their Kafka producers and consumers. They just want .send and .receive along with the guarantee that there are not a thousand ways their code can fail dramatically once in production. The 100 configs should be a Platform Team’s concern.
So yeah, magic wand: dumb clients, smart brokers.
3. Business/Work
Why did you decide to create Streemlined?
If you’ve been following me, you know I’m a fan of keeping things simple. The industry at large wants you to believe that complexity is “inherent” to event-driven systems.
There is this pervasive idea that if you are using Kafka, you’ve entered a realm where “simple” no longer applies, where every minor data transformation requires a new cluster, a new framework, and a month of architectural review.
We’ve been forced to accept that “Enterprise Grade” must mean “Hard to Use.”
With Streemlined, I’m showing a different path. I’m building for the developer who says: “I don’t want to be a ‘Kafka Engineer.’ I want to be a Software Engineer who happens to use Kafka to solve a business problem.”
Streemlined gives you everything you’re forced to rethink about each time you start a new project:
- a development environment
- opinionated error handling
- a test framework
- a decentralized deployment model
- and a monitoring that you can directly connect to your metrics system.
It’s doing the boring stuff so you can focus on your real work, serving your business.
I’ve seen these messes for years, so I know exactly what’s missing. I’m not guessing what to build, I’m building the tool I wish I had back then.
Why did you name it Streemlined?
The domain name was available.
What are your ambitions with it? Paint me the success story looking 3 years out.
This year, I’m optimizing for trust and presence. I have a clear idea of the things I want to build, but I’ve stopped to focus exclusively on meeting customers. Writing content on LinkedIn. Getting into meetings.
I’m counting on the POCs and the first few customers to prioritize my work further.
I’m aiming for 3 customers by the end of the year. I’m confident it’s achievable. I already have a few POCs on the way.
In 3 years, I want Streemlined to be the “obvious” internal standard for large organizations. When a central Platform Team is overwhelmed by hundreds of requests for simple data syncs or basic filtering, I want the answer to be: “Just Use Streemlined.”
It will be the tool that lets the experts focus on the 10% of high-complexity “flagship” projects in Flink or whatever the next shiny thing is, and Streemlined quietly handles the other 90% with total reliability and zero hand-holding.
What’s your opinion on the general VC-backed data infrastructure industry? Are you taking VC money?
I’m conflicted. I’m not a fan of the VC “growth-at-all-costs” machine. It forces companies to do things at the expense of their customers. It makes people lose touch with reality.
But to build something truly great, you need great people. And it is very hard to do without the resources that funding provides. That is my core conflict. I want to keep things simple and avoid the mandatory “explosive growth or die” that VCs demand, but I also want to work with people I couldn’t afford otherwise.
Thankfully I don’t need to worry about that just yet.
What’s in your development toolbox for building a Kafka-adjacent system? Any special IDE, shortcuts, AI agents?
Boring stack. I’m using IntelliJ for the backend, Cursor for the Frontend.
I also have a tiny server where I deploy all the services I need using docker compose and nginx.
Kafka, schema registry, postgres, redis, and several webservices.
In all fairness I should just migrate everything to the Aiven free tier now.
4. General/Parting
What’s your opinion on managed services?
Good. Use them. Managed Postgres. Managed Kafka. Managed Emails.
I don’t believe it’s reasonable to think we can do everything ourselves.
They update the service, they backup the data, they fix things while we sleep.
I care about costs, but I don’t want to manage an email or file server either. So I’m happy to let Google do that for me.
Thoughts on Queues vs Fan-out Log messaging? What’s more popular, what do you prefer?
I really appreciated what Filip Yonov did with his 80% Kafka article where he pointed out how underused Kafka really is. Redpanda also wrote a similar piece I believe.
If we actually looked at the data, I bet we’d find that the “Fan-out” dream is mostly a myth. I’d love to see the stats on:
- Average members per consumer group: My guess is 1.
- Average consumer groups per topic: If you exclude topics without consumer groups, my guess is 1. If you include all topics, it’s probably close to 0.
Most people are just moving data from Point A to Point B. They aren’t doing fanout to ten different microservices; they are just trying to get a record into a database.
And yet I still believe Kafka is the right choice! Because Kafka is not solving tech problems. It’s solving organization problems. If you use a simple queue, you are building a private pipe. If you use Kafka, you are building a public door. Even if only one person is walking through that door today, the fact that the door exists changes how the company thinks about its data. It forces ownership and standards from day one.
The IT world should be forever grateful to Kafka founders and contributors (and Confluent!) for this.
How do you see the future of Kafka usage and development, 5 years out?
Honestly I have no idea.
But I already see the effect of ignoring the long tail of projects. While a company might have one or two ‘flagship’ streaming projects, they usually have hundreds of small ones, like a local plant needing a dashboard or a team just trying to sync data to a database.
These teams don’t have Kafka experts and they have no time to waste, but their work still has to be reliable and easy to manage.
In five years, I hope the ‘expert-only’ era of Kafka is over. If we haven’t made it easy for the ‘regular’ developer to build their own pipelines by then, we’ve failed as an industry.
Anything else you’d like to add?
Thanks for reading!
Check out my product:
https://www.streemlined.io/
Come and say hi on LinkedIn:
https://www.linkedin.com/in/julien-chanaud/