{{tag>Brouillon}}

= Notes Kafka

Voir aussi : 
* [[Notes queue messages middleware file d attente de message|Notes - queue - messages - middleware - file d'attente de messages]]
* Redpanda (Implémentation en C++)
* NATS, RabbitMQ, NSQ


== Serveur

Source : 

Voir : 
* https://kafka.apache.org/quickstart
* https://www.tutorialkart.com/apache-kafka/install-apache-kafka-on-ubuntu/
* https://www.tutorialspoint.com/apache_kafka/apache_kafka_installation_steps.htm
* https://www.educba.com/install-kafka/

Install Java 
<code bash>
apt-get install default-jdk-headless
</code>

Télécharger Kafka https://kafka.apache.org/downloads 

Apache Kafka dépend de Zookeeper pour la gestion des clusters. Par conséquent, avant de lancer Kafka, Zookeeper doit être démarré. Il n'est pas nécessaire d'installer explicitement Zookeeper, car il est fourni avec Apache Kafka.

Lancer Zookeeper
<code bash>
bin/zookeeper-server-start.sh config/zookeeper.properties
</code>

Lancer Kafka Server
<code bash>
bin/kafka-server-start.sh config/server.properties
</code>


=== Autres

''Kafka/config/server.properties''
<code ini>
delete.topic.enable = true
</code>

<code bash>
acks=all
min.insync.replica=2
</code>



== Client


Note en vrac

<code ini>
EnableSslCertificateVerification = false
security.protocol=SASL_SSL
</code>

<code bash>
kafkacat -b localhost:9092 -L
kafkacat -b localhost:9092 -L -d broker

kafka-topics --list --zookeeper zkinfo
kafka-topics --zookeeper zk01.example.com:2181 --list


# kafka-consumer-groups --new-consumer --bootstrap-server broker01.example.com:9092 --describe --group flume
export KAFKA_HEAP_OPTS=-Xmx1024M
kafka-consumer-groups --bootstrap-server host:9092 --list
kafka-consumer-groups --bootstrap-server host:9092 --describe --group foo

kafka-consumer-groups --bootstrap-server host.server.com:9093 --describe --command-config client.properties
</code>

''Dockerfile''
<code bash>
FROM frolvlad/alpine-python2
COPY root/requirements.txt /
COPY root/opt /opt

#RUN apk add librdkafka
#RUN apk add gcc python2-dev libc-dev librdkafka-dev

RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r /requirements.txt
WORKDIR /opt/app/
#CMD ["gunicorn", "-w 4", "main:app"]
CMD ["python", "/opt/app/run.py"]
</code>

''requirements.txt''
<code python>
#librdkafka==0.11.6
simplejson
requests
six
#avro_json_serializer==0.5
avro_json_serializer
#avro==1.8.2
confluent_kafka
</code>


=== RedHat CentOS

<code bash>
# Pour librdkafka ?
yum install gcc python-devel

# Pour client Oracle
yum install libaio

# Kafka-cli
yum install java-latest-openjdk-headless
curl -O http://miroir.univ-lorraine.fr/apache/kafka/2.4.0/kafka_2.13-2.4.0.tgz
</code>


==== Docker

Docker CentOS

''Dockerfile''
<code bash>
#FROM centos:centos6.9
FROM centos:centos7

COPY root/requirements.txt /
COPY root/opt/app /opt/app

RUN yum install -y epel-release.noarch
RUN yum install -y python-pip

# Pour CentOS6
#RUN curl https://bootstrap.pypa.io/2.6/get-pip.py -o get-pip.py
#RUN python get-pip.py
#RUN pip install --upgrade --ignore-installed pip setuptools

#RUN apk add librdkafka
#RUN apk add gcc python2-dev libc-dev librdkafka-dev

RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r /requirements.txt
WORKDIR /opt/app
#CMD ["gunicorn", "-w 4", "main:app"]
CMD ["python", "/opt/app/run.py"]
</code>

FIXME