• 使用StatefulSet部署有状态应用
    • 部署Zookeeper
    • 部署kafka
    • 参考

    使用StatefulSet部署有状态应用

    StatefulSet 这个对象是专门用来部署用状态应用的，可以为Pod提供稳定的身份标识，包括hostname、启动顺序、DNS名称等。

    下面以在kubernetes1.6版本中部署zookeeper和kafka为例讲解StatefulSet的使用，其中kafka依赖于zookeeper。

    Dockerfile和配置文件见 zookeeper 和 kafka。

    注：所有的镜像基于CentOS系统的JDK制作，为我的私人镜像，外部无法访问，yaml中没有配置持久化存储。

    部署Zookeeper

    Dockerfile中从远程获取zookeeper的安装文件，然后在定义了三个脚本：

    • zkGenConfig.sh：生成zookeeper配置文件
    • zkMetrics.sh：获取zookeeper的metrics
    • zkOk.sh：用来做ReadinessProb

    我们在来看下这三个脚本的执行结果：

    zkGenConfig.sh

    zkMetrics.sh脚本实际上执行的是下面的命令：

    1. $ echo mntr | nc localhost $ZK_CLIENT_PORT >& 1
    2. zk_version    3.4.6-1569965, built on 02/20/2014 09:09 GMT
    3. zk_avg_latency    0
    4. zk_max_latency    5
    5. zk_min_latency    0
    6. zk_packets_received    427879
    7. zk_packets_sent    427890
    8. zk_num_alive_connections    3
    9. zk_outstanding_requests    0
    10. zk_server_state    leader
    11. zk_znode_count    18
    12. zk_watch_count    3
    13. zk_ephemerals_count    4
    14. zk_approximate_data_size    613
    15. zk_open_file_descriptor_count    29
    16. zk_max_file_descriptor_count    1048576
    17. zk_followers    1
    18. zk_synced_followers    1
    19. zk_pending_syncs    0

    zkOk.sh脚本实际上执行的是下面的命令：

    1. $ echo ruok | nc 127.0.0.1 $ZK_CLIENT_PORT
    2. imok

    zookeeper.yaml

    下面是启动三个zookeeper实例的yaml配置文件：

    1. ---
    2. apiVersion: v1
    3. kind: Service
    4. metadata:
    5.   name: zk-svc
    6.   labels:
    7.     app: zk-svc
    8. spec:
    9.   ports:
    10.   - port: 2888
    11.     name: server
    12.   - port: 3888
    13.     name: leader-election
    14.   clusterIP: None
    15.   selector:
    16.     app: zk
    17. ---
    18. apiVersion: v1
    19. kind: ConfigMap
    20. metadata:
    21.   name: zk-cm
    22. data:
    23.   jvm.heap: "1G"
    24.   tick: "2000"
    25.   init: "10"
    26.   sync: "5"
    27.   client.cnxns: "60"
    28.   snap.retain: "3"
    29.   purge.interval: "0"
    30. ---
    31. apiVersion: policy/v1beta1
    32. kind: PodDisruptionBudget
    33. metadata:
    34.   name: zk-pdb
    35. spec:
    36.   selector:
    37.     matchLabels:
    38.       app: zk
    39.   minAvailable: 2
    40. ---
    41. apiVersion: apps/v1beta1
    42. kind: StatefulSet
    43. metadata:
    44.   name: zk
    45. spec:
    46.   serviceName: zk-svc
    47.   replicas: 3
    48.   template:
    49.     metadata:
    50.       labels:
    51.         app: zk
    52.     spec:
    53.       affinity:
    54.         podAntiAffinity:
    55.           requiredDuringSchedulingIgnoredDuringExecution:
    56.             - labelSelector:
    57.                 matchExpressions:
    58.                   - key: "app"
    59.                     operator: In
    60.                     values:
    61.                     - zk
    62.               topologyKey: "kubernetes.io/hostname"
    63.       containers:
    64.       - name: k8szk
    65.         imagePullPolicy: Always
    66.         image: sz-pg-oam-docker-hub-001.tendcloud.com/library/zookeeper:3.4.6
    67.         resources:
    68.           requests:
    69.             memory: "2Gi"
    70.             cpu: "500m"
    71.         ports:
    72.         - containerPort: 2181
    73.           name: client
    74.         - containerPort: 2888
    75.           name: server
    76.         - containerPort: 3888
    77.           name: leader-election
    78.         env:
    79.         - name : ZK_REPLICAS
    80.           value: "3"
    81.         - name : ZK_HEAP_SIZE
    82.           valueFrom:
    83.             configMapKeyRef:
    84.                 name: zk-cm
    85.                 key: jvm.heap
    86.         - name : ZK_TICK_TIME
    87.           valueFrom:
    88.             configMapKeyRef:
    89.                 name: zk-cm
    90.                 key: tick
    91.         - name : ZK_INIT_LIMIT
    92.           valueFrom:
    93.             configMapKeyRef:
    94.                 name: zk-cm
    95.                 key: init
    96.         - name : ZK_SYNC_LIMIT
    97.           valueFrom:
    98.             configMapKeyRef:
    99.                 name: zk-cm
    100.                 key: tick
    101.         - name : ZK_MAX_CLIENT_CNXNS
    102.           valueFrom:
    103.             configMapKeyRef:
    104.                 name: zk-cm
    105.                 key: client.cnxns
    106.         - name: ZK_SNAP_RETAIN_COUNT
    107.           valueFrom:
    108.             configMapKeyRef:
    109.                 name: zk-cm
    110.                 key: snap.retain
    111.         - name: ZK_PURGE_INTERVAL
    112.           valueFrom:
    113.             configMapKeyRef:
    114.                 name: zk-cm
    115.                 key: purge.interval
    116.         - name: ZK_CLIENT_PORT
    117.           value: "2181"
    118.         - name: ZK_SERVER_PORT
    119.           value: "2888"
    120.         - name: ZK_ELECTION_PORT
    121.           value: "3888"
    122.         command:
    123.         - sh
    124.         - -c
    125.         - zkGenConfig.sh && zkServer.sh start-foreground
    126.         readinessProbe:
    127.           exec:
    128.             command:
    129.             - "zkOk.sh"
    130.           initialDelaySeconds: 10
    131.           timeoutSeconds: 5
    132.         livenessProbe:
    133.           exec:
    134.             command:
    135.             - "zkOk.sh"
    136.           initialDelaySeconds: 10
    137.           timeoutSeconds: 5
    138.       securityContext:
    139.         runAsUser: 1000
    140.         fsGroup: 1000

    我们再主要下上面那三个脚本的用途。

    部署kafka

    Kafka的docker镜像制作跟zookeeper类似，都是从远程下载安装包后，解压安装。

    与zookeeper不同的是，只要一个脚本，但是又依赖于我们上一步安装的zookeeper，kafkaGenConfig.sh用来生成kafka的配置文件。

    我们来看下这个脚本。

    1. #!/bin/bash
    2. HOST=`hostname -s`
    3. if [[ $HOST =~ (.*)-([0-9]+)$ ]]; then
    4.         NAME=${BASH_REMATCH[1]}
    5.         ORD=${BASH_REMATCH[2]}
    6. else
    7.         echo "Failed to extract ordinal from hostname $HOST"
    8.         exit 1
    9. fi
    11. MY_ID=$((ORD+1))
    12. sed -i s"/broker.id=0/broker.id=$MY_ID/g" /opt/kafka/config/server.properties
    13. sed -i s'/zookeeper.connect=localhost:2181/zookeeper.connect=zk-0.zk-svc.brand.svc:2181,zk-1.zk-svc.brand.svc:2181,zk-2.zk-svc.brand.svc:2181/g' /opt/kafka/config/server.properties

    该脚本根据statefulset生成的pod的hostname的后半截数字部分作为broker ID，同时再替换zookeeper的地址。

    Kafka.yaml

    下面是创建3个kafka实例的yaml配置。

    1. ---
    2. apiVersion: v1
    3. kind: Service
    4. metadata:
    5.   name: kafka-svc
    6.   labels:
    7.     app: kafka
    8. spec:
    9.   ports:
    10.   - port: 9093
    11.     name: server
    12.   clusterIP: None
    13.   selector:
    14.     app: kafka
    15. ---
    16. apiVersion: policy/v1beta1
    17. kind: PodDisruptionBudget
    18. metadata:
    19.   name: kafka-pdb
    20. spec:
    21.   selector:
    22.     matchLabels:
    23.       app: kafka
    24.   minAvailable: 2
    25. ---
    26. apiVersion: apps/v1beta1
    27. kind: StatefulSet
    28. metadata:
    29.   name: kafka
    30. spec:
    31.   serviceName: kafka-svc
    32.   replicas: 3
    33.   template:
    34.     metadata:
    35.       labels:
    36.         app: kafka
    37.     spec:
    38.       affinity:
    39.         podAntiAffinity:
    40.           requiredDuringSchedulingIgnoredDuringExecution:
    41.             - labelSelector:
    42.                 matchExpressions:
    43.                   - key: "app"
    44.                     operator: In
    45.                     values:
    46.                     - kafka
    47.               topologyKey: "kubernetes.io/hostname"
    48.         podAffinity:
    49.           preferredDuringSchedulingIgnoredDuringExecution:
    50.              - weight: 1
    51.                podAffinityTerm:
    52.                  labelSelector:
    53.                     matchExpressions:
    54.                       - key: "app"
    55.                         operator: In
    56.                         values:
    57.                         - zk
    58.                  topologyKey: "kubernetes.io/hostname"
    59.       terminationGracePeriodSeconds: 300
    60.       containers:
    61.       - name: k8skafka
    62.         imagePullPolicy: Always
    63.         image: sz-pg-oam-docker-hub-001.tendcloud.com/library/kafka:2.10-0.8.2.1
    64.         resources:
    65.           requests:
    66.             memory: "1Gi"
    67.             cpu: 500m
    68.         env:
    69.         - name: KF_REPLICAS
    70.           value: "3"
    71.         ports:
    72.         - containerPort: 9093
    73.           name: server
    74.         command:
    75.         - /bin/bash
    76.         - -c
    77.         - "/opt/kafka/bin/kafkaGenConfig.sh && /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties"
    78.         env:
    79.         - name: KAFKA_HEAP_OPTS
    80.           value : "-Xmx512M -Xms512M"
    81.         - name: KAFKA_OPTS
    82.           value: "-Dlogging.level=DEBUG"
    83.         readinessProbe:
    84.            tcpSocket:
    85.              port: 9092
    86.            initialDelaySeconds: 15
    87.            timeoutSeconds: 1

    参考

    https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

    kubernetes contrib - statefulsets

    http://blog.kubernetes.io/2017/01/running-mongodb-on-kubernetes-with-statefulsets.html