Kafka Importer - with Topic security

How to pass Kafka topic authentication details with KafkaImporter?

Hi @jshah,

I don’t see any option to specify security or encryption. I took at the code, but I am not a Kafka expert, so I will ask internally.

In the meantime, as a workaround, think you could use safka-console-consumer pipelining the messages to Streaminserter.

Something like that
kafka-console-consumer | StreamInserter table database

Every message should be inserted into the target table.

Thanks. I had already tried with the consumer, but that has two concerns

  • How error handling will work? What if consumer reads from Kafka without any issue and KafkaImporter fails?
  • KafkaImporter batch option doesn’t seem to work when used with the console consumer

Hi @jshah,

Your concerns are justified, in fact, I had to ask internally because I’ve not the right skill to give you a satisfying answer. It’s weird the batch option of Kafka importer isn’t working; what’s the issue?
Have you tried with StreamImporter? Same syntax


It looks like batch parameter is ignored when used with Kafka console. One of the other thread from user “Sumit” had a request to use kafka importer at certain time interval rather than batch size. We also need same feature ( if KafkaImporter supports topic security).

No I haven’t tried StreamImporter. I will look into it.

I implemented the time interval but I didn’t do a pr against our repo. I can share the binary, but without topic security yet.

Probably the time interval would be a valuable feature for all the importer’s utilities.

Anyway, when I will get an answer about t security I will report back for sure

Sadly we didn’t support security in the actual version of KafkaImporter, so the workaround I proposed is the only way to load Kafka topic with security enabled

We will take a look to the new libs to investigate how to implement new features.

Tell me if you want to try out the utility with the timer option you requested; I would Tha KS to give me the idea to implement the same on Streamimporter too


Yeah, I would like to try out the timer option. Thanks!