Installing the Latest Version of Kafka on Ubuntu
This guide will walk you through the process of installing the latest version of Apache Kafka on the latest version of Ubuntu.
Prerequisites
- A system running the latest version of Ubuntu
- A user account with sudo privileges
Step 1: Update the System
Before installing Kafka, it’s a good idea to update the system packages. Open a terminal and run the following commands:
sudo apt update
sudo apt upgrade
Step 2: Install Java
Kafka requires Java to run. Install the OpenJDK package by running the following command:
sudo apt install openjdk-11-jdk
Verify the Java installation by running:
java -version
Step 3: Create a separate user for managing kafka
Here’s how you can create a new user (in this case we’ll call it ‘kafka’) and provide it the necessary permissions. You’ll need to perform these operations as a user with sudo permissions.
Create a new user named kafka:
sudo adduser kafka
When prompted, enter and confirm a password for the user. You can also fill in any additional information if you wish. If you don’t want to, simply press ENTER to leave the fields blank.
Add the kafka user to the sudo group:
sudo usermod -aG sudo kafka
Switch to the kafka user:
sudo su - kafka
Running Kafka as a non-root user limits the potential impact of a security vulnerability in Kafka. In a production environment, further hardening might be required according to the specific security guidelines of your organization.
Step 4: Download and Extract Kafka
Visit the Apache Kafka website to find the latest version of Kafka. Download the binary package using wget:
wget https://downloads.apache.org/kafka/{latest_version}/kafka_{latest_version}-bin.tgz
Replace {latest_version} with the actual version number.
Extract the downloaded archive:
tar -xzf kafka_{latest_version}-bin.tgz
Move the extracted directory to /opt:
sudo mv kafka_{latest_version} /opt/kafka
Step 5: Configure Kafka
Create a data directory for Kafka and Zookeeper:
sudo mkdir /opt/kafka/data
sudo mkdir /opt/kafka/data/zookeeper
sudo mkdir /opt/kafka/data/kafka
Change ownership of the directories to the kafka user:
sudo chown -R kafka:kafka /opt/kafka/data
sudo chmod -R 755 /opt/kafka/data
Edit the Zookeeper configuration file:
sudo nano /opt/kafka/config/zookeeper.properties
Add the following line at the end of the file:
dataDir=/opt/kafka/data/zookeeper
Edit the Kafka configuration file:
sudo nano /opt/kafka/config/server.properties
Add the following lines at the end of the file:
log.dirs=/opt/kafka/data/kafka
Step 6: Start Kafka and Zookeeper
Start Zookeeper:
/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
In a new terminal, start Kafka:
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
Kafka and Zookeeper are now running on your Ubuntu system.
Step 7: Configuring System Services for Kafka and Zookeeper using systemctl: A Step-by-Step Guide
To manage Kafka and Zookeeper as system services using systemctl, you need to create service unit configuration files for both Kafka and Zookeeper. Here’s a step-by-step guide on how to achieve this.
Create a Service Unit File for Zookeeper
Open a new file /etc/systemd/system/zookeeper.service with a text editor with root privilege, like this:
sudo nano /etc/systemd/system/zookeeper.service
In the file, add the following contents:
[Unit]
Description=Zookeeper Service
After=network.target
[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
User=kafka
Restart=on-failure
[Install]
WantedBy=multi-user.target
Create a Service Unit File for Kafka
Open a new file /etc/systemd/system/kafka.service with a text editor with root privilege, like this:
sudo nano /etc/systemd/system/kafka.service
In the file, add the following contents:
[Unit]
Description=Kafka Service
After=zookeeper.service
[Service]
Type=simple
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
User=kafka
Restart=on-failure
[Install]
WantedBy=multi-user.target
Reload the Systemd Daemon
Inform systemd about the newly created service files:
sudo systemctl daemon-reload
Enable the Services
To start Kafka and Zookeeper at system boot:
sudo systemctl enable zookeeper
sudo systemctl enable kafka
Start Kafka and Zookeeper Services
You can now start the services using systemctl:
sudo systemctl start zookeeper
sudo systemctl start kafka
** Verify the Services**
Check if the services are running:
sudo systemctl status zookeeper
sudo systemctl status kafka
You should see an output indicating that the services are active (running).
Note that if you want to stop the services, you can do so by running:
sudo systemctl stop kafka
sudo systemctl stop zookeeper
This setup makes Kafka and Zookeeper run as background services managed by systemd. Thus, they will start up automatically with your system, and you can manage them using the systemctl command.
Conclusion
You have successfully installed the latest version of Apache Kafka on your Ubuntu system. You can now start using Kafka for your messaging and streaming applications.