Select Select cluster type to open a list. Once connected, you see information similar to the following text: When working with Kafka, you must know the Apache Zookeeper and Broker hosts. For more general information on planning virtual networks for HDInsight, see Plan a virtual network for Azure HDInsight.
It's often used as a message broker, as it provides functionality similar to a publish-subscribe message queue. From the drop-down list, select the Azure subscription that's used for the cluster. In this Quickstart, you learn how to create an Apache Kafka cluster using the Azure portal.
To set an environment variable with Zookeeper host information, use the command below. For more information on the commands available with the kafka-topics.sh utility, use the following command: Kafka stores records in topics. You have set up your Multi broker Multi-Node Kafka cluster. From the drop-down list, select a region where the cluster is created. Apache Kafka is an open-source, distributed streaming platform. Cluster login username and password: The default login name is admin. Select this check box to use the same password for SSH user as the one you provided for the cluster login user. Conclusion. For the highest availability of your Apache Kafka data, you should rebalance the partition replicas for your topic when: To list topics, use the following command: This command lists the topics available on the Apache Kafka cluster. The virtual machines and managed disks that implement the nodes within an HDInsight cluster are distributed across these fault domains. The command retrieves all Zookeeper hosts, then returns only the first two entries. If you are using an older version of Kafka, replace --bootstrap-server $KAFKABROKERS with --zookeeper $KAFKAZKHOSTS. If you delete the test topic created earlier, then you must recreate it.
To store records into the test topic you created earlier, and then read them using a consumer, use the following steps: To write records to the topic, use the kafka-console-producer.sh utility from the SSH connection: After this command, you arrive at an empty line.
Get the cluster name in lower case from the Azure portal. To verify that the environment variable is set correctly, use the following command: This command returns information similar to the following text: zk0-kafka.eahjefxxp1netdbyklgqj5y1ud.ex.internal.cloudapp.net:2181,zk2-kafka.eahjefxxp1netdbyklgqj5y1ud.ex.internal.cloudapp.net:2181. Replace PASSWORD with the cluster login password, then enter the command: Extract the correctly cased cluster name. This tool must be ran from an SSH connection to the head node of your Apache Kafka cluster. To create an Apache Kafka cluster on HDInsight, use the following steps: From the top menu, select + Create a resource. A resource group is a container of Azure components.