Notes Kafka

Voir aussi :

Notes - queue - messages - middleware - file d'attente de messages

Redpanda (Implémentation en C++)

NATS, RabbitMQ, NSQ

Serveur

Source :

Voir :

https://kafka.apache.org/quickstart

https://www.tutorialkart.com/apache-kafka/install-apache-kafka-on-ubuntu/

https://www.tutorialspoint.com/apache_kafka/apache_kafka_installation_steps.htm

https://www.educba.com/install-kafka/

Install Java

apt-get install default-jdk-headless

Télécharger Kafka https://kafka.apache.org/downloads

Apache Kafka dépend de Zookeeper pour la gestion des clusters. Par conséquent, avant de lancer Kafka, Zookeeper doit être démarré. Il n'est pas nécessaire d'installer explicitement Zookeeper, car il est fourni avec Apache Kafka.

Lancer Zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

Lancer Kafka Server

bin/kafka-server-start.sh config/server.properties

Autres

Kafka/config/server.properties

delete.topic.enable = true
acks=all min.insync.replica=2

Client

Note en vrac

EnableSslCertificateVerification = false security.protocol=SASL_SSL
kafkacat -b localhost:9092 -L kafkacat -b localhost:9092 -L -d broker kafka-topics --list --zookeeper zkinfo kafka-topics --zookeeper zk01.example.com:2181 --list # kafka-consumer-groups --new-consumer --bootstrap-server broker01.example.com:9092 --describe --group flume export KAFKA_HEAP_OPTS=-Xmx1024M kafka-consumer-groups --bootstrap-server host:9092 --list kafka-consumer-groups --bootstrap-server host:9092 --describe --group foo kafka-consumer-groups --bootstrap-server host.server.com:9093 --describe --command-config client.properties

Dockerfile

FROM frolvlad/alpine-python2 COPY root/requirements.txt / COPY root/opt /opt #RUN apk add librdkafka #RUN apk add gcc python2-dev libc-dev librdkafka-dev RUN pip install --upgrade pip RUN pip install --no-cache-dir -r /requirements.txt WORKDIR /opt/app/ #CMD ["gunicorn", "-w 4", "main:app"] CMD ["python", "/opt/app/run.py"]

requirements.txt

#librdkafka==0.11.6 simplejson requests six #avro_json_serializer==0.5 avro_json_serializer #avro==1.8.2 confluent_kafka

RedHat CentOS

# Pour librdkafka ? yum install gcc python-devel # Pour client Oracle yum install libaio # Kafka-cli yum install java-latest-openjdk-headless curl -O http://miroir.univ-lorraine.fr/apache/kafka/2.4.0/kafka_2.13-2.4.0.tgz

Docker

Docker CentOS

Dockerfile

#FROM centos:centos6.9 FROM centos:centos7 COPY root/requirements.txt / COPY root/opt/app /opt/app RUN yum install -y epel-release.noarch RUN yum install -y python-pip # Pour CentOS6 #RUN curl https://bootstrap.pypa.io/2.6/get-pip.py -o get-pip.py #RUN python get-pip.py #RUN pip install --upgrade --ignore-installed pip setuptools #RUN apk add librdkafka #RUN apk add gcc python2-dev libc-dev librdkafka-dev RUN pip install --upgrade pip RUN pip install --no-cache-dir -r /requirements.txt WORKDIR /opt/app #CMD ["gunicorn", "-w 4", "main:app"] CMD ["python", "/opt/app/run.py"]