Following this post, you will be able to deploy, configure and use an HashiCorp Vault with Hashicorp Consul, to try it in your Kubernetes Cluster with sample application.

Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.

Consul is a service mesh solution providing a full featured control plane with service discovery, configuration, and segmentation functionality. Each of these features can be used individually as needed, or they can be used together to build a full-service mesh

1- Stack :

• Kubelet : v1.17.2 / v1.18.5
• Kubectl : v1.17.1
• Docker : 19.03.5 / 19.03.8
• Consul : 1.9.3
• Vault : 1.6.2 (Agent 0.8.0)
• Cfssl : 1.2.0
• Kube namespace : vault (if you use a different namespace, it must be changed in service and pod hostnames)
• Architecture : AMD64 / ARM64

2- Consul deployment :

1. First, generate SSL certificates for Consul (can be done in workstation) with Cfssl, after editing configuration files in consul/ca directory

# Generate CA and sign request for Consul
cfssl gencert -initca ca/ca-csr.json | cfssljson -bare ca 
# Generate SSL certificates for Consul
cfssl gencert \ 
-ca=ca.pem \ 
-ca-key=ca-key.pem \ 
-config=ca/ca-config.json \ 
-profile=default \ 
ca/consul-csr.json | cfssljson -bare consul 
# Perpare a GOSSIP key for Consul members communication encryptation
GOSSIP_ENCRYPTION_KEY=$(consul keygen)

2. Create secret with Gossip key and public/private keys

kubectl create secret gesneric consul \
--from-literal="gossip-encryption-key=${GOSSIP_ENCRYPTION_KEY}" \
--from-file=ca.pem \
--from-file=consul.pem \
--from-file=consul-key.pem

3. Deploy 3 Consul members (Statefulset)

kubectl apply -f consul/service.yaml
kubectl apply -f consul/rbac.yaml
kubectl apply -f consul/config.yaml
kubectl apply -f consul/consul.yaml

4. Prepare SSL certificates for Consul client, it will be used by vault consul client (sidecar).

cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca/ca-config.json \
-profile=default \
ca/consul-csr.json | cfssljson -bare client-vault

5. Create secret for Consul client (like members)

kubectl create secret generic client-vault \
--from-literal="gossip-encryption-key=${GOSSIP_ENCRYPTION_KEY}" \
--from-file=ca.pem \
--from-file=client-vault.pem \
--from-file=client-vault-key.pem

3- Vault deployment :

Before deploy Vault, you need to configure a Consul client to give Vault access to Consul members

apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-config
data:
  ...
  concul.config: |
    {
      "verify_incoming": false,
      "verify_outgoing": true,
      "server": false,
      "ca_file": "/etc/tls/ca.pem",
      "cert_file": "/etc/tls/client-vault.pem",
      "datacenter": "vault",
      "key_file": "/etc/tls/client-vault-key.pem",
      "client_addr": "127.0.0.1",
      "ui": false,
      "raft_protocol": 3,
      "retry_join": [ "provider=k8s label_selector=\"app=consul,role=server\" namespace=\"vault\"" ]
    }

The consul client will be deployed as a Sidecar for Vault server, so the "client_addr" must be "127.0.0.1". For certificates parameters, we will use the client-vault secret and same join expression of members configuration.

In another side, we need to configure Vault to request this client by the "listeners" parameter.

apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-config
data:
  vault.config: |
    {
      "ui": true,
      "listener": [{
        "tcp": {
          "address": "0.0.0.0:8200",
          "tls_disable": true
        }
      }],
...

OK, let's deploy

kubectl apply -f vault/service.yaml
kubectl apply -f vault/config.yaml
kubectl apply -f vault/vault.yaml

5- UI:

At this point, we have 3 instances on Consul deployed and 1 instance of Vault connected to Consul members.

We can use a port forwarding to acces UI of Consul and Vault. In our case, we will use an "Ingress" to expose UIs to internet.

kubectl apply -f ingress.yaml

If you use this option with SSL (HTTPS), you need to configure the TLS secret.

6- Vault Injector deployment

• Install vault agent injector (single and simple instance without leader & leader election)

kubectl apply -f vault-injector/serivce.yaml
kubectl apply -f vault-injector/rbac.yaml
kubectl apply -f vault-injector/deployment.yaml
kubectl apply -f vault-injector/webhook.yaml # webhook must be created after deployment

The injector will detect Vault "Annotation" or "Configmap", and will inject an initContainer in the init process of your application Pod to request Vault server for secret. After initialization, an agent will be injected inside the pod to give your application container the requested secret.

For agent injector, we will use our docker image, it's similar of official image with arm64 supporting (At 02/2021 only amd64 arch are distributed by Hashicorp), see docker files.

7- Sample deployment :

We can use UI to configure and use Vault, in this project we use CLI.

1. Start by installing Vault locally (in workspace) for CLI use only

curl https://releases.hashicorp.com/vault/1.6.2/vault_1.6.2_linux_amd64.zip -o vault_1.6.2_linux_amd64.zip
unzip vault_1.6.2_linux_amd64.zip
chmod +x vault
# With ingress, you can use the root url of Vault ui, or use the port forward
export VAULT_ADDR="YOUR_VAULT_HOST"

2. Check the server status and login (using token like UI)

./vault status
>> Key             Value
>> ---             -----
>> Seal Type       shamir
>> Initialized     true
>> Sealed          false
>> Total Shares    1
>> Threshold       1
>> Version         1.6.2
>> Storage Type    consul
>> Cluster Name    vault-cluster-...
>> Cluster ID      ...
>> HA Enabled      true
>> HA Cluster      ..
>> HA Mode         active

./vault login
<< your_token

3. Create key/value for testing

./vault secrets enable kv
./vault kv put kv/myapp/config username="admin" password="adminpassword"

4. Connect Kube to Vault

# Create the service account to access secret
kubectl apply -f myapp/service-account.yaml
# Enable kubernetes support
./vault auth enable kubernetes
# Prepare kube api server data
export SECRET_NAME="$(kubectl get serviceaccount vault-auth  -o go-template='{{ (index .secrets 0).name }}')"
export TR_ACCOUNT_TOKEN="$(kubectl get secret ${SECRET_NAME} -o go-template='{{ .data.token }}' | base64 --decode)"
export K8S_API_SERVER="$(kubectl config view --raw -o go-template="{{ range .clusters }}{{ index .cluster \"server\" }}{{ end }}")"
export K8S_CACERT="$(kubectl config view --raw -o go-template="{{ range .clusters }}{{ index .cluster \"certificate-authority-data\" }}{{ end }}" | base64 --decode)"
# Send kube config to vault
./vault write auth/kubernetes/config kubernetes_host="${K8S_API_SERVER}" kubernetes_ca_cert="${K8S_CACERT}" token_reviewer_jwt="${TR_ACCOUNT_TOKEN}"

5. Create Vault policy and role for "myapp"

Edit policy file myapp/policy.json

path "kv/myapp/*" {
  capabilities = ["read", "list"]
}

Create the application role

./vault policy write myapp-ro myapp/policy.json
./vault write auth/kubernetes/role/myapp-role bound_service_account_names=vault-auth bound_service_account_namespaces=vault policies=default,myapp-ro ttl=15m

6. Deploy "myapp" for testing

Edit annotations for secret output, @see myapp/deployment.yaml

annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-account: "kv/myapp/config"
    vault.hashicorp.com/agent-inject-template-account: |
        {{- with secret "kv/myapp/config" -}}
        dsn://{{ .Data.username }}:{{ .Data.password }}@database:port/mydb?sslmode=disable
        {{- end }}
    vault.hashicorp.com/role: "myapp-role"

Deploy app and log secret

kubectl apply -f myapp/deployment.yaml 
export POD=$(kubectl get pods --selector=app=myapp --output=jsonpath={.items..metadata.name})
kubectl log ${POD} myapp

8- Tips

To activate an HTTP Basic security for Consul UI (it's run without), you can use Nginx ingress annotations, after authentication secret generation.

htpasswd -c auth foo
kubectl create secret generic consul-auth --from-file=auth

Add annotations

nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: consul-auth
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - Consul - MedInvention'

Links

• https://github.com/kelseyhightower/consul-on-kubernetes
• https://github.com/hashicorp/vault-k8s
• https://www.hashicorp.com/blog/whats-next-for-vault-and-kubernetes
• https://github.com/hashicorp/vault-helm
• https://medium.com/hashicorp-engineering/hashicorp-vault-delivering-secrets-with-kubernetes-1b358c03b2a3
• https://github.com/hashicorp/consul
• https://blog.zwindler.fr/2020/08/31/gerez-vos-secrets-kubernetes-dans-vault/

Source @GitHub

After successful Kafka and Zookeeper deployment on Kubernetes following this post, it's time to make it more secure.

1- ZooKeeper DIGEST authentication

Account configuration

(one for Zookeeper nodes, another for Kafka broker) in config.secured.yaml file.

Client section will be used by zkCli.sh helper srcipt to test configuration. In integration environement, Client section must be removed.

apiVersion: v1
kind: Secret
metadata:
  name: zookeeper-jaas
type: Opaque
stringData:
  zookeeper-jaas.conf: |-
    QuorumServer {
          org.apache.zookeeper.server.auth.DigestLoginModule required
          user_zk="passcode";
    };
    QuorumLearner {
          org.apache.zookeeper.server.auth.DigestLoginModule required
          username="zk"
          password="passcode";
    }; 
    Server {
          org.apache.zookeeper.server.auth.DigestLoginModule required
          user_kafka="passcode"
          user_client="passcode";
    };
    Client {
          org.apache.zookeeper.server.auth.DigestLoginModule required
          username="client"
          password="passcode";
    };

Create Secret / ConfigMap and deploy StatefuSet:

kubectl apply -f zookeeper/config.secured.yaml 
kubectl apply -f zookeeper/statefulset.secured.yaml 
# Secret for kafka
kubectl apply -f kafka/config.secured.yaml

2- Kafka SSL encryption

If your want to disable the Host Name verification for some raison, you need to omit the extension parameter -ext and add empty environment variable to Kafka broker KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM="" or use Kafka helper script for every broker.

kafka-configs.sh --bootstrap-server [kafka-0].kafka-broker.kafka.svc.cluster.local:9092 --entity-type brokers --entity-name 0 --alter --add-config "listener.name.internal.ssl.endpoint.identification.algorithm="

Create CA (certificate authority)

openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

Generate server keystore and client keystore

keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA 
keytool -keystore kafka.client.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA

Add generated CA to the trust store

keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert 
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert

Sign the key store (with passcode and ssl.cnf)

You need to update alt_names section of ssl.cnf with list of your brokers hostname.

keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file 
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:passcode -extfile ssl.cnf -extensions req_ext 
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert 
keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed

Sign the client keystore

keytool -keystore kafka.client.keystore.jks -alias localhost -certreq -file cert-file-client 
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file-client -out cert-signed-client -days 365 -CAcreateserial -passin pass:passcode -extfile ssl.cnf -extensions req_ext 
keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert 
keytool -keystore kafka.client.keystore.jks -alias localhost -import -file cert-signed-client

Kafka SSL Kubernetes

Create kubernetes secret from kafka.keystore.jks and kafka.truststore.jks

kubectl create secret generic ssl --from-literal=keystore_password=passcode --from-file=kafka.keystore.jks=ssl/kafka.server.keystore.jks --from-literal=truststore_password=passcode --from-file=kafka.truststore.jks=ssl/kafka.server.truststore.jks

Update Kafka StatefulSet to mount ssl secret and broker, service configuration:

kubectl apply -f kafka/statefulset.ssl.yaml kubectl apply -f kafka/service.ssl.yaml

Testing

Use openssl to debug connectionto valid certificate data:

kubectl exec -ti zk-0 -- openssl s_client -debug -connect kafka-0.kafka-broker.kafka.svc.cluster.local:9093 -tls1

Create client configuration file (client.properties):

security.protocol=SSL
ssl.truststore.location=/opt/kafka/config/kafka.truststore.jks
ssl.truststore.password=passcode
ssl.keystore.location=/opt/kafka/config/client.keystore.jks
ssl.keystore.password=passcode ssl.key.password=passcode
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

Send to test pod:

mkdir -p config 
cp client.properties config/client.properties 
cp ssl/kafka.client.keystore.jks config/client.keystore.jks 
cp ssl/kafka.client.truststore.jks config/kafka.truststore.jks 
kubectl cp config kafka-1:/opt/kafka

Run test:

kubectl exec -ti kafka-1 -- kafka-console-producer.sh --bootstrap-server kafka-0.kafka-broker.kafka.svc.cluster.local:9093 --topic k8s --producer.config /opt/kafka/config/client.properties 
>> Hello with secured connection

kubectl exec -ti kafka-1 -- kafka-console-consumer.sh --bootstrap-server kafka-0.kafka-broker.kafka.svc.cluster.local:9093 --topic k8s -consumer.config /opt/kafka/config/client.properties --from-beginning 
<< Hello with secured connection

Prepare Secret for client:

kubectl create secret generic client-ssl --from-file=ca-certs.pem=ssl/ca-cert --from-file=cert.pem=ssl/cert-signed-client 
kubectl apply -f consumer.secured.yaml kubectl logs consumer-secured

Sources & Links

• Redhat-Kafka
• Confluence-Zookeeper
• Apache-Kafka
• Bitnami-Kafka
• Kafka-SSL
• Vertica-Kafka-SSL
• Kafka-SSL

@GitHub source

Following this post, you will be able to deploy, configure and use an Apache Kafka event streaming platform with Apache Zookeeper , for your integration and development environment easily.

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

1- Stack

• Kubelet : v1.17.2 / v1.18.5
• Kubectl : v1.17.1
• Docker : 19.03.5 / 19.03.8
• Zookeeper : 3.4.10
• Kafka : 2.7.0 (Scala 2.13 / Glib 2.31-r0)
• Kube namespace : kafka (if you use a different namespace, it must be changed in service and pod hostnames)
• Architecture : AMD64 / ARM64
• Python (optional, for client testing) : 3.8

2- Zookeeper deployment

First, deploy a small Zookeeper cluster (2 pods) using a StatefulSet and exposing it with 2 Services, one for client communication and another for Zookeeper cluster communication (leader election).

kubectl apply -f zookeeper/statefulset.yaml kubectl apply -f zookeeper/service.yaml

Next, you can test your deployment :

kubectl exec zk-0 zkCli.sh create /hello world kubectl exec zk-1 zkCli.sh get /hello

For more information, take a tour in the kubernetes blog .

3- Consumer/Producer application case :

You need to deploy a Kafka broker with ZooKeeper as synchronized services :

1. Create 2 Kafka broker with StatefulSet
2. Create first topic (k8s for example), you can use one of available broker hostname or the broker service hostname :

- kafka-0.kafka-broker.kafka.svc.cluster.local

- kafka-1.kafka-broker.kafka.svc.cluster.local

- kafka-broker.kafka.svc.cluster.local

Next, create the first topic and run the first consumer client to check configuration.

kubectl apply -f service.yaml kubectl apply -f statefulset.yaml kubectl exec -ti kafka-0 -- kafka-topics.sh --create --topic=k8s --bootstrap-server kafka-0.kafka-broker.kafka.svc.cluster.local:9092 
kubectl apply -f consumer.yaml kubectl logs consumer

4- Development case (from Workstation with kubectl)

You need to create a custom broker (for host binding) and activate a port forwarding to your workstation, and finally create a development topic :

kubectl apply -f dev-brocker.yaml kubectl port-forward pod/dev-brocker 9092:9092 
kubectl exec -ti dev-brocker -- kafka-topics.sh --create --topic dev-k8s --bootstrap-server 127.0.0.1:9092

• Running python consumer and producer :

pip install kafka-python python ../client/Consumer.py python ../client/Producer.py

• Using Kafka help script client

kubectl exec -ti dev-brocker -- kafka-console-producer.sh --topic=dev-k8s --bootstrap-server 127.0.0.1:9092 
>> Hello World!
>> I'm a Producer
kubectl exec -ti dev-brocker -- kafka-console-consumer.sh --topic=k8s --from-beginning --bootstrap-server 127.0.0.1:9092
<< Hello World!
<< I'm a Producer

5- Secure your Kafka

With a standard Kafka setup, any user or application can write any messages to any topic. It's the same for Zookeeper. So, we need to add a DIGEST authentication layer to Zookeeper (doesn’t support ACL, but we have only Kafka broker as client, DIGEST is sufficient) to authorize only Kafka broker. In Kafka Side we need to add SSL authentication to authorize ony valid client to use services.

Follow the security section documentation

6- Sourcing

• Zookeeper Docker image : we use the kubernetes-zookeeper @kow3ns as base image with 2 modifications:

- Add JVM flags to be injected in Java environment file @see start-zookeeper.sh

echo "JVMFLAGS=\"-Xmx$HEAP -Xms$HEAP $JVMFLAGS\"" >> $JAVA_ENV_FILE

- Extra configuration file path, to be injected in top on configuration file @see start-zookeeper.sh

#Add extra configuration from file (file path in env : EXTRA_CONFIG_FILE) 
if [[ -n "$EXTRA_CONFIG_FILE" ]]; then    
    echo "#Start extra-section" >> $CONFIG_FILE    
    cat $EXTRA_CONFIG_FILE >> $CONFIG_FILE    
    echo "#End of extra-section" >> $CONFIG_FILE
fi

• Kafka Docker image : we use the kafka-docker @wurstmeister as base with 2 modifications :

- For ARM64 arch, switching base image from 'openjdk:8u212-jre-alpine' to 'openjdk:8u201-jre-alpine' to prevent container core dump @see issue.

- For K8S deployment, add a 'KAFKA_LISTENERS_COMMAND' environment parameter to build 'KAFKA_LISTENERS' on fly (to use pod hostname when container started) @see start-kafka.sh

if [[ -n "$KAFKA_LISTENERS_COMMAND" ]]; then
    KAFKA_LISTENERS=$(eval "$KAFKA_LISTENERS_COMMAND")
    export KAFKA_LISTENERS
    unset KAFKA_LISTENERS_COMMAND
fi

7- Tips

• For debugging, you can bypass the Kafka broker for topics management (kafka and ZooKeeper helpers script) :

kubectl exec -ti kafka-0 -- kafka-topics.sh --create --topic k8s --zookeeper zk-cs.kafka.svc.cluster.local:2181 
kubectl exec -ti kafka-0 -- kafka-topics.sh --describe --topic k8s --zookeeper zk-cs.kafka.svc.cluster.local:2181 
kubectl exec zk-1 zkCli.sh ls /brokers/topics

• Building multi-architecture docker image :

docker buildx build --push --platform linux/arm64/v8,linux/amd64 --tag [medinvention]/kubernetes-zookeeper:latest .
docker buildx build --push --platform linux/arm64/v8,linux/amd64 --tag [medinvention]/kafka:latest .

@GitHub source

Let's start by a definition, what's a "Mesh Service" ?

...is a way to control how different parts of an application share data with one another. Unlike other systems for managing this communication, a service mesh is a dedicated infrastructure layer built right into an app. This visible infrastructure layer can document how well (or not) different parts of an app interact, so it becomes easier to optimize communication and avoid downtime as an app grows...

Context

Imagine, you have an application built on many services calls, and you want to know how services communicate inside your cluster, you want to monitor your request/response time and verify your request tracing and routing, you need a mesh service tools.

There are several solution who offering many features, but all this popular tools need a complex configuration to work and redefining of yours deployments and it's not available for ARM architecture.

So i purpose to you "SMS", it's a very simple mesh service for Kubernetes and dedicated to Rest Service (but can work with all HTTP services) and configured to work on ARM (like Raspberry ARMv7) and Amd64 .

Simple Mesh Service - Install

Simple way , with Helm package :

helm repo add medinvention-dev https://mmohamed.github.io/kubernetes-charts

Create your own values file "myvalues.yaml" (optional) :

Caution: Helm package is configured by default to work on ARM architecture, if you want to use it on AMD architecture (AWS, GCP, ...), you must add to all release option "-amd64"

db:
  reuse: false # delete db on install or update
  release: 5.6 # 5.6-amd64

api:
  release: v0.1.0 # v0.1.0-amd64
  ingress:
    host: your ingress host for api...

ui:
  release: v0.1.0 # v0.1.0-amd64
  ingress:
    host: your ingress host for ui...
    
sidecar:
  release: v0.1.0 # v0.1.0-amd64

logger:
  release: v0.1.0 # v0.1.0-amd64

master:
  release: v0.1.0 # v0.1.0-amd64

controller:
  release: v0.1.0 # v0.1.0-amd64

processor:  
  release: v0.1.0 # v0.1.0-amd64

Install (you can create your namespace before, for example kube-sms)

helm install -f myvalues.yaml kube-sms -n kube-sms medinvention-dev/sms

Now, you can view your dashboard on accessing web view (if you have configure an ingress for ui/api) with your credentials defined in your values file (default: admin@sms.dev /admin)

For testing, you can use sample files (from source) to test your install

kubectl apply -f /Sample

You will see your 4 services groups without any connetion

So, to see how it's work, you must execute same request on services, you can use this simple script (inside cluster, from any container)

for i in {1..50}
do
    curl http://product-service.sample-sms.svc.cluster.local:5000/v3
    curl http://product-service.sample-sms.svc.cluster.local:5000/v2
    curl http://product-service.sample-sms.svc.cluster.local:5000/v1
    curl http://product-service.sample-sms.svc.cluster.local:5000
    curl http://product-service.sample-sms.svc.cluster.local:5000/notfound
done

After max 5 minutes (processor run every 5 min), you can see how your services communicate like this

If you have any problem during deployment, you can see logs

kubectl logs -n kube-sms -l "run=controller,app.kubernetes.io/name=SMS,app.kubernetes.io/instance=kube-sms"
kubectl logs -n kube-sms -l "run=master,app.kubernetes.io/name=SMS,app.kubernetes.io/instance=kube-sms"
kubectl logs -n kube-sms -l "run=logger,app.kubernetes.io/name=SMS,app.kubernetes.io/instance=kube-sms"
kubectl logs -n kube-sms -l "run=processor,app.kubernetes.io/name=SMS,app.kubernetes.io/instance=kube-sms"
kubectl logs -n kube-sms -l "run=db,app.kubernetes.io/name=SMS,app.kubernetes.io/instance=kube-sms"
kubectl logs -n kube-sms -l "run=api,app.kubernetes.io/name=SMS,app.kubernetes.io/instance=kube-sms"

Simple Mesh Service - How to

Let's see the V1 of the Review deployment (it's a part of sample) :

...

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-review-v1
  namespace: sample-sms
  annotations:
    medinvention.dev/sms.group: review
    medinvention.dev/sms.port: "5000"
    medinvention.dev/sms.service: review-v1-service
spec:
  selector:
    matchLabels:
      app: review-v1
  replicas: 1
  template:
    metadata:
      labels:
        app: review-v1
    spec:
      containers:
      - name: server
        image: python:3
        command: ['sh', '-c', 'pip install flask && python /var/static/server']
        ports:
          - containerPort: 5000
        volumeMounts:
          - name: review-v1-server
            mountPath: /var/static
      volumes:
        - name: review-v1-server
          configMap:
            name: review-v1-server
...

For our sample we need just 3 annotations :

medinvention.dev/sms.group: review

It's to specify the group of service, for Review services we have 3 version (v1, v2 and v3): we type "review" as group unique name, and they will be used to associate deployment to the services group. Finally it's will be the title in "Review bloc" header in Dashboard interface.

medinvention.dev/sms.port: "5000"

It's a container port who expose your service. In this sample it's 5000 et it will be accepting HTTP request only. This annotation it's not mandatory and by default, 80 will be used.

medinvention.dev/sms.service: review-v1-service

it's the name of service that expose your container port, this annotation is optional for SMS but, you must use service if you want to use same port. For sample "review-v1-service" bind 80 to 5000, so after applying SMS the service will be updated for only target port and you can continue calling service on 80.

if service is defined in another namespace that deployment, you can use this annotation to specify it.

medinvention.dev/sms.servicenamespace: sample-sms

It's done, you can delete sample and start using SMS with your services :)

Simple Mesh Service - Features

Using SMS Dashboard,

In Right pane, you can view some metadata of your services group, like his name, his status and how many services is contained.

You can have statistical data, like request and response Avg time for selected group and success/error request rate.

In bottom, you can see a request status statistic.

In Top bar view, you can filter by date or by namespace. And finally, this view is auto refreshed every 30 second, but you can force a refresh using reload button.

Enjoy

Nota: this is a development release of SMS project available here, it's can have bugs and may doses not work correctly with very big services.

Today, our goal is to deploy a simple application for cluster resources monitoring. Target resources will be CPU charge, Memory consuming, CPU temperature and pods state like in Top view on Linux system.

Project stack

Application

It's a Java Maven multi-module project, source code available here:

• Core module for communication with Cluster API server using Kubernetes Java Client.
• Service module to expose Rest services using Spring Boot and Spring Rest component.
• Webapp module to make a web application based on ReactJs Framework  with Material-UI Framwork.

Metrics

With default install of Kube, we don't have some metrics like CPU and consumed Memory of node, so we will deploy a Metrics Server with our image built for arm-V7 architecture. Metrics server deployment source code available here.

Temperature sensor

To track CPU Temperature of every node, we will deploy a very small pod and use the "/sys/class/thermal" of pod who be inherited from host.

Fan commander

To manage Fan, we will use GPIO utilities of Python RPi library.

Material

Fan

We have a tower for all worker node (north, south, east and west) with a single fan connected to west node (because it's in the top of the tower)

Chipset

To manage fan, we will use a single chipset L293D from Texas Instruments. This chipset allows us to pilot 2 motor with DC power (rotation direction and speed).

it's done with requirements, let's start.

Our Solution

Global view

Our solution is composed by 2 section; logical and material section.

Logical section - Metrics Server

Start by deploy it

kubectl apply -f https://raw.githubusercontent.com/mmohamed/k8s-raspberry/master/kube/metrics.yaml

If success, you must have a new pod deployed

kube-system            metrics-server-5d74                    1/1     Running

Logical section - Application

For building and deployment, we will use Jenkins with a simple pipeline, more information available in this post.

To build application, we start by building the "webapp" module using Node, and the backend application using Maven to get an HTML web application and a Jar file.

For Frontend application, we will use a simple Nginx image and OpenJDK image for the backend.

Application can monitor 5 types of data;

Minimal Cluster Health

It's a minimal cluster health represented by the state of master node and will be appeared as badge of notification: it's can be "OK" or "KO".

Pods

We will use the Kubernetes Client Java to call Cluster API Server by calling "listPodForAllNamespaces" method for pods listing view.

Nodes

Like Pods data, we use "listNode" method to get nodes static information.

CPU & Memory usages

This data is only available with Metrics Server deployed, but standard Kubernetes Client don't have method or data structure to call metrics services.

We have created an extension of standard client to add "V1Beta1NodeMetrics" for metrics data representation. Then we have extended the client "CoreV1Api" to add new method "clusterMetrics" that they call service of Metrics Server on "metrics.k8s.io/v1beta1/nodes" path.

Extension source code are available here.

CPU Temperature

Standard Cluster metrics don't have CPU temperature of nodes as data, so we need to deploy sensors to collect them. We use a DaementSet to deploy a set of Pod to collect and send temperature values to Backend application.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: monitoring-agent
  namespace: monitoring
  labels:
    k8s-app: monitoring-agent
spec:
  selector:
    matchLabels:
      name: monitoring-agent
  template:
    metadata:
      labels:
        name: monitoring-agent
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: monitoring-agent
        image: busybox
        env:
          - name: NODE
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: SERVER
            value: http://monitoring-service.monitoring.svc.cluster.local/k8s/collect/{{token}}/temperature
        command: [ "sh", "-c"]
        args:
        - while true; do
            TEMP=$(cat /sys/class/thermal/thermal_zone0/temp);
            URL="$SERVER?node=$NODE&value=$TEMP";
            wget -qO- $URL;
            sleep 5;
          done;
        imagePullPolicy: IfNotPresent

For this sensor, we use a very small container (busybox) to make a continuous temperature collecting (every 5 seconds) and send it to Backend application using internal DNS service "monitoring-service.monitoring.svc.cluster.local" and a static security token "{{token}}" defined in the Backend app.

We have added a tolerance to deploy a replica of pod into "Master" node despite it's not schedulable.

monitoring-agent-6k7r4     1/1     Running   10.244.4.244   east
monitoring-agent-gzlc8     1/1     Running   10.244.2.181   north
monitoring-agent-hx9fx     1/1     Running   10.244.1.60    south
monitoring-agent-mtkf5     1/1     Running   10.244.3.28    west
monitoring-agent-znczc     1/1     Running   10.244.0.31    master

This data will not be saved anywhere, but only last sent for each node will be stored in a "ConcurrentMap" memory variable, defined in the Backend app and will be sent to Frontend app on Rest service calls.

Outside monitoring

To view and monitor any other system (RPi or not)  from outside of K8S, you can use a small bash script with our application to send data

#!/bin/bash
# nohup sh agent.sh [NODE-NAME] [YOUR-SECURITY-TOKEN] > /tmp/agent.log
if [ -z "$1" ]; then
    echo "Node name required !"
    exit 1
fi

if [ -z "$2" ]; then
    echo "Security Token required !"
    exit 1
fi

attempts=0
server="http[s]://[YOUR-API-BACKEN-URL]/k8s/collect/$1/temperature"

while true; do

    temperature=$(cat /sys/class/thermal/thermal_zone0/temp)

    if [ $? != 0 ] || [ -z "$temperature" ]; then
        echo "Unable to determinate CPU temperature value !"
        exit 1
    fi

    url="$server?node=$2&value=$temperature"

    responseCode=$(curl --silent --output /dev/null --write-out "%{http_code}" $url)

    if [ $? != 0 ] || [ -z "$responseCode" ] || [ $responseCode -ne 200 ]; then
        attempts=$((attempts + 1))
        echo "[ATTEMP-$attempts] Failed sending data to server : $responseCode"
        if [ $attempts = 20 ]; then
            echo "Server response error after 20 attempts !"
            exit 1
        fi;
    else
        attempts=0	
    fi

    sleep 5
done;

Logical section - Fan monitoring

To monitor Fan, Backend app expose 3 services; start, stop and get status of Fan by calling a micro Rest server deployed directly into West node (not with K8S).

We have used Python RPi library with Flask library to communicate with GPIO of West node and to expose some services like on, off and status of Fan by calling GPIO devices.

Starting by Installing libraries

apt-get install rpi.gpio
sudo pip install Flask # use sudo to install Flask bin into PATH

Server code source (replace NODE-FAN-IP by connected node to Fan)

from flask import Flask, jsonify
import RPi.GPIO as GPIO
import os, signal

GPIO.setmode(GPIO.BOARD)

IN1 = 11    # Input Pin 6
IN2 = 13    # Input Pin 7
ENABLE = 15 # Enable Pin 8

GPIO.setup(IN1,GPIO.OUT)
GPIO.setup(IN2,GPIO.OUT)
GPIO.setup(ENABLE,GPIO.OUT)

api = Flask(__name__)
api.config['SERVER_NAME'] = '[NODE-FAN-IP]:5000'

@api.route('/fan/status', methods=['GET'])
def status():
    status = GPIO.input(ENABLE) == GPIO.HIGH
    return jsonify({"status": status, "message": ("FAN ON" if status else "FAN OFF")})

@api.route('/fan/start', methods=['GET'])
def start():
    GPIO.output(IN1,GPIO.HIGH)
    GPIO.output(IN2,GPIO.LOW)
    GPIO.output(ENABLE,GPIO.HIGH)
    return jsonify({"status": True, "message": "FAN started"})

@api.route('/fan/stop', methods=['GET'])
def stop():
    GPIO.output(IN1,GPIO.HIGH)
    GPIO.output(IN2,GPIO.LOW)
    GPIO.output(ENABLE,GPIO.LOW)
    return jsonify({"status": True, "message": "FAN stopped"})

@api.route('/server/shutdown', methods=['GET'])
def shutdown():
    stop()
    os.kill(os.getpid(), signal.SIGINT)
    # not sended
    return jsonify({"status": true, "message": "Server is shutting down..." })

if __name__ == '__main__':
    api.run()
    GPIO.cleanup()

And run the server

nohup sudo python server.py > /tmp/fan-server.log &

Now, we have a Fan server deployed and available only inside cluster (not exposed to outside of LAN). If you want deploying this server using a Pod, it's must privileged Pod (can access to host devices) and you must mount all GPIO devices (available into /dev) to pod with same name and path but it's not guaranteed to work.

Server will be explained into "Material section".

Now, to say to Backend app where it's can pilot fan, we add an environment variable to Backend deployment "FAN_SERVER_URL" and it must be like "http://192.168.1.1:5000". In this way, the switch can make Fan on and off.

The second feature is to make our Backend app run in auto-mode to start Fan when maximal temperature is reached and stop Fan when minimal temperature is reached. To do this, we must specify a maximum value with an environment variable to Backend app "FAN_MAXTEMP", and Fan will be on when any node will have a temperature great than this value and will be off when the maximum temperature of all nodes will be less than 90% of this value.

Material section - L293D

To make fan manageable by RPI, we will use a L293D chipset

We will use a first half part of chipset, so we connect "Input 1" to RPi PIN 11, "Input 2" to RPi PIN 13  and "Enable 1" RPi PIN 15.

PIN 15 is to enable or disable motor management and PIN 11 & 13 is for rotation direction of motor setting. We will power chipset by RPi 5v and in our case we will use same power source to power motor (it's not good idea but it's faster to make it's run). After we will change it by another stable power 5V.

The Fan server play with GPIO output PIN to start and stop Fan, and it's can read state of GPIO to know if the Fan is in run mode or not in all cases.

PIN connection result:

RPi PIN    L293D PIN
11         2
13         7
15         1
4          16
6          4/5
2          8 # should be another PWR source
NAN        3 # Motor +
NAN        6 # Motor -

Nota: you can use another GPIO PIN 11, 13 and 15 if you want but you must adjust server code in PIN declaration section.

To test connection and fan server, we can use a simple Curl command

curl http://[FAN-CONNECTED-NODE-IP]:5000/fan/start
curl http://[FAN-CONNECTED-NODE-IP]:5000/fan/status
curl http://[FAN-CONNECTED-NODE-IP]:5000/fan/stop

When all is done, you can see the fan status in Web view with fan switch status (on/off) and you will see temperature chart evolution. When 60C° value reached, Fan will start, and chart will come down to 54C°.

Start state

And stop state

We can start or stop manually the cluster fan by using the Web view switch.

Realtime monitoring

See how auto-managing fan work.

It's done, enjoy :)

Now we have a ready K8S cluster, we will start to develop, build and deploy some applications.

Let's check cluster health,

NAME     STATUS   ROLES    AGE     VERSION
east     Ready    <none>   3d23h   v1.17.4
master   Ready    master   66d     v1.17.1
north    Ready    <none>   66d     v1.17.1
south    Ready    <none>   66d     v1.17.1
west     Ready    <none>   3d23h   v1.17.4

So, to make an operational CI/CD pipeline, we need a good orchestrator like Jenkins. Go to deploy it.

Orchestrator

We will create a namespace named "jenkins" with a small PVC for master:

apiVersion: v1
kind: Namespace
metadata:
  name: jenkins
  
---  
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins-master-pvc
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

Next, we will create a master deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-master
  namespace: jenkins
  labels:
    app: jenkins-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-master
  template:
    metadata:
      labels:
        app: jenkins-master
    spec:
      containers:
      - name: jenkins-master
        image: medinvention/jenkins-master:arm
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        - containerPort: 50000
        volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins-home
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-master-pvc

The master container exposes 2 ports; 8080 for Web access and 5000 for JNLP communication used by slave (Jenkins executor).

Let's build 2 services to expose ports

apiVersion: v1
kind: Service
metadata:
  name: jenkins-master-service
  namespace: jenkins
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
  selector:
    app: jenkins-master

---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-slave-service
  namespace: jenkins
spec:
  ports:
  - name: jnlp
    protocol: TCP
    port: 50000
    targetPort: 50000
  selector:
    app: jenkins-master 

And finally, the ingress component to access Jenkins GUI from outside of cluster

apiVersion: v1
kind: Secret
metadata:
  name: jenkins-tls
  namespace: jenkins
data:
  tls.crt: {{crt}}
  tls.key: {{key}}
type: kubernetes.io/tls

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: jenkins-master
  namespace: jenkins
  labels:
    app: jenkins-master
spec:
  rules:
    - host: {{host}}
      http:
        paths:
          - backend:
              serviceName: jenkins-master-service
              servicePort: http
            path: /
  tls:
    - hosts:
      - {{host}}
      secretName: jenkins-tls

You must replace {{host}} by your domain, {{crt}} and {{key}} by your SSL Certificate and Private key encoded base 64.

Nice, we have a master of Jenkins deployed

jenkins    jenkins-master-cww    1/1    Running   4d4h    10.244.2.150   north

Executor

To build our application, we will use a Jenkins slave node to preserve master from surcharge.

You must configure a new node with "administration section" to get a secret token needed by slave before deploying it.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins-slave-pvc
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-slave
  namespace: jenkins
  labels:
    app: jenkins-slave
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-slave
  template:
    metadata:
      labels:
        app: jenkins-slave
    spec:
      containers:
      - name: jenkins-slave
        image: medinvention/jenkins-slave:arm
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "250m"
        env:
          - name: "JENKINS_SECRET"
            value: "{{jenkins-secret}}"
          - name: "JENKINS_AGENT_NAME"
            value: "exec-1"
          - name: "JENKINS_DIRECT_CONNECTION"
            value: "jenkins-slave-service.jenkins.svc.cluster.local:50000"
          - name: "JENKINS_INSTANCE_IDENTITY"
            value: "{{jenkins-id}}"
        volumeMounts:
        - mountPath: /var/jenkins
          name: jenkins-home
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-slave-pvc
      nodeSelector:
        name: east

Replace {{jenkins-secret}} by token available after creating node configuration on master, and replace {{jenkins-id}} by ID Token who is sent in HTTP Header response (use a simple curl to extract it , @see).

For our case, we add a "nodeSelector" to force pod assignment to specific node with more capacity.

After a little time, the new node will appear as connected and available.

The application

For this post, I will use a maven project composed by 3 modules, Java Core module, Spring Boot Backend module and an Angular Front module hosted in GitHub. It's a simple monitoring application for your K8S cluster.

CI Pipeline

For our pipelines, we will use a declarative Jenkins pipeline file, we start by Build stage: checkout source and build artifact (Java by Maven and JS by Node)

pipeline {
   agent {
       label  'slave'
   }

   tools {
      maven "AutoMaven"
      nodejs "AutoNode"
   }

   stages {
      stage('Build') {
         steps {
            //checkout
            checkout([$class: 'GitSCM',
                branches: [[name: '*/dev']],
                doGenerateSubmoduleConfigurations: false,
                extensions: [[$class: 'SubmoduleOption',
                              disableSubmodules: false,
                              parentCredentials: false,
                              recursiveSubmodules: true,
                              reference: '',
                              trackingSubmodules: false]], 
                submoduleCfg: [], 
                userRemoteConfigs: [[url: 'https://github.com/mmohamed/k8s-monitoring.git']]])
            // Package
            sh 'mkdir -p $NODEJS_HOME/node'
            sh 'cp -n $NODEJS_HOME/bin/node $NODEJS_HOME/node'
            sh 'cp -rn $NODEJS_HOME/lib/node_modules $NODEJS_HOME/node'
            sh 'ln -sfn $NODEJS_HOME/lib/node_modules/npm/bin/npm-cli.js $NODEJS_HOME/node/npm'
            sh 'ln -sfn $NODEJS_HOME/lib/node_modules/npm/bin/npx-cli.js $NODEJS_HOME/node/npx'
            sh 'export NODE_OPTIONS="--max_old_space_size=256" && export REACT_APP_URL_BASE="https://{{apihostname}}/k8s" && export PATH=$PATH:$NODEJS_HOME/bin && export NODE_PATH=$NODEJS_HOME && mvn install'
            // Copy artifact to Docker build workspace
            sh 'mkdir -p ./service/target/dependency && (cd service/target/dependency; jar -xf ../*.jar) && cd ../..'  
            sh 'mkdir -p ./service/target/_site && cp -r ./webapp/target/classes/static/* service/target/_site'   
         }
      }
      ....
   }
}

I have already configured Node and Maven tools configuration on "Administration section".

In this stage, we start by preparing tools and some environment variables for Node binaries, after we must change {{apihostname}} by our API application hostname for Front component building and call "maven install" goal.

After artifact building, we will extract jar content (it's made container starting faster) and copy ReactJs output files to target directory for copying it to Docker builder node.

Next stage, Prepare Docker workspace to build images: first copy target directory content to Docker workspace (we have defined a node credentials "SSHMaster" in Jenkins master), and we create and copy 2 Docker files, first with OpenJDK for backend java module and second with a simple Nginx for front Web module.

....

stage('Prepare Workspace'){
         steps{
            // Prepare Docker workspace
            withCredentials([sshUserPrivateKey(credentialsId: "SSHMaster", keyFileVariable: 'keyfile')]) {
                sh "ssh -i ${keyfile} [USER]@[NODEIP] 'mkdir -p ~/s2i-k8S/k8s-monitoring-$BUILD_NUMBER'"
                sh "scp -i ${keyfile} -r service/target [USER]@[NODEIP]:~/s2i-k8S/k8s-monitoring-$BUILD_NUMBER"
            }
            // Create Dockerfile for api
            writeFile file: "./Dockerfile.api", text: '''
FROM arm32v7/openjdk:8-jdk 
ARG user=spring
ARG group=spring
ARG uid=1000
ARG gid=1000
RUN groupadd -g ${gid} ${group} && useradd -u ${uid} -g ${gid} -m -s /bin/bash ${user}
ARG DEPENDENCY=target/dependency
COPY --chown=spring:spring ${DEPENDENCY}/BOOT-INF/lib /var/app/lib
COPY --chown=spring:spring ${DEPENDENCY}/META-INF /var/app/META-INF
COPY --chown=spring:spring ${DEPENDENCY}/BOOT-INF/classes /var/app
USER ${user}
ENTRYPOINT ["java","-cp","var/app:var/app/lib/*","dev.medinvention.service.Application"]'''
            // Create Dockerfile for front
            writeFile file: "./Dockerfile.front", text: '''
FROM nginx
EXPOSE 80
COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY target/_site/ /usr/share/nginx/html'''
            // Create config for front
            writeFile file: "./nginx.conf", text: '''
server {
    listen       80;
    server_name  localhost;
    location / {
        root   /usr/share/nginx/html;
        try_files $uri /index.html;
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }    
}'''
            // copy docker and config file
            withCredentials([sshUserPrivateKey(credentialsId: "SSHMaster", keyFileVariable: 'keyfile')]) {
                sh "scp -i ${keyfile} Dockerfile.api [USER]@[NODEIP]:~/s2i-k8S/k8s-monitoring-$BUILD_NUMBER"
                sh "scp -i ${keyfile} Dockerfile.front [USER]@[NODEIP]:~/s2i-k8S/k8s-monitoring-$BUILD_NUMBER"
                sh "scp -i ${keyfile} nginx.conf [USER]@[NODEIP]:~/s2i-k8S/k8s-monitoring-$BUILD_NUMBER"
            }
         }
      }
      
....

Now, we can start building our images with Docker build stage.

....

stage('Docker build'){
         steps{
            withCredentials([sshUserPrivateKey(credentialsId: "SSHMaster", keyFileVariable: 'keyfile')]) {
               sh "ssh -i ${keyfile} [USER]@[NODEIP] 'docker build ~/s2i-k8S/k8s-monitoring-$BUILD_NUMBER -f ~/s2i-k8S/k8s-monitoring-$BUILD_NUMBER/Dockerfile.api -t medinvention/k8s-monitoring-api:arm'"
               sh "ssh -i ${keyfile} [USER]@[NODEIP] 'docker push medinvention/k8s-monitoring-api:arm'"
               sh "ssh -i ${keyfile} [USER]@[NODEIP] 'docker rmi medinvention/k8s-monitoring-api:arm'"
               sh "ssh -i ${keyfile} [USER]@[NODEIP] 'docker build ~/s2i-k8S/k8s-monitoring-$BUILD_NUMBER -f ~/s2i-k8S/k8s-monitoring-$BUILD_NUMBER/Dockerfile.front -t medinvention/k8s-monitoring-front:arm'"
               sh "ssh -i ${keyfile} [USER]@[NODEIP] 'docker push medinvention/k8s-monitoring-front:arm'"
               sh "ssh -i ${keyfile} [USER]@[NODEIP] 'docker rmi medinvention/k8s-monitoring-front:arm'"
            }
         }
      }
      
....

Don't forget to replace [USER] and [NODEIP] by your docker build node (it's can be any of available cluster node)

And finally, start Kubernetes deployment stage.

....

stage('Kubernetes deploy'){
         steps{
            // deploy
            withCredentials([string(credentialsId: 'KubeToken', variable: 'TOKEN'),
                  string(credentialsId: 'TLSKey', variable: 'KEY'),
                  string(credentialsId: 'TLSCrt', variable: 'CRT')
               ]) {
               sh "export TOKEN=$TOKEN && export CRT=$CRT && export KEY=$KEY"
               sh "cd k8s && sh deploy.sh"
            }  
         }
      }
....

For deployment, we can choose Helm packager to make our deployment more industrial or we can make our deployment script like this case

#!/bin/bash

if [ -z "$CRT" ] || [ -z "$KEY" ]; then
    echo "TLS CRT/KEY environment value not found !"
    exit 1
fi

if [ -z "$TOKEN" ]; then
    echo "Kube Token environment value not found !"
    exit 1
fi

echo "Get Kubectl"
curl -s -LO https://storage.googleapis.com/kubernetes-release/release/v1.17.0/bin/linux/arm/kubectl
chmod +x ./kubectl

commitID=$(git log -1 --pretty="%H")

if [ $? != 0 ] || [ -z "$commitID" ]; then
    echo "Unable to determinate CommitID !"
    exit 1
fi

echo "Deploy for CommitID : ${commitID}"

# create new deploy
sed -i "s|{{crt}}|`echo $CRT`|g" api.yaml
sed -i "s|{{key}}|`echo $KEY`|g" api.yaml
sed -i "s|{{host}}|[BACKENDHOSTNAME]|g" api.yaml
sed -i "s|{{commit}}|`echo $commitID`|g" api.yaml

./kubectl --token=$TOKEN apply -f api.yaml
if [ $? != 0 ]; then
    echo "Unable to deploy API !"
    exit 1
fi	

# wait for ready
attempts=0
rolloutStatusCmd="./kubectl --token=$TOKEN rollout status deployment/api -n monitoring"
until $rolloutStatusCmd || [ $attempts -eq 60 ]; do
  $rolloutStatusCmd
  attempts=$((attempts + 1))
  sleep 10
done

# create new deploy
sed -i "s|{{crt}}|`echo $CRT`|g" front.yaml
sed -i "s|{{key}}|`echo $KEY`|g" front.yaml
sed -i "s|{{host}}|[FRONTHOSTNAME]|g" front.yaml
sed -i "s|{{commit}}|`echo $commitID`|g" front.yaml

./kubectl --token=$TOKEN apply -f front.yaml
if [ $? != 0 ]; then
    echo "Unable to deploy Front !"
    exit 1
fi	

# wait for ready
attempts=0
rolloutStatusCmd="./kubectl --token=$TOKEN rollout status deployment/front -n monitoring"
until $rolloutStatusCmd || [ $attempts -eq 60 ]; do
  $rolloutStatusCmd
  attempts=$((attempts + 1))
  sleep 10
done

In this script, we start by installing local "kubectl" client, and check "kube" access token. Next we try to deploy backend first and if we have problem with, we stop deployment process. If success deployment of backend, we proceed with front deployment.

To make Jenkins access to cluster, we need to generate an access token with a ClusterRoleBinding resource.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: jenkins-access
    namespace: jenkins

After running this command to get token, create a secret credential for Jenkins with token content

kubectl -n jenkins describe secret $(kubectl -n jenkins get secret | grep jenkins-access | awk '{print $1}')  

And finally, for backend deployment

---
apiVersion: v1
kind: Namespace
metadata:
   name: monitoring

---   
apiVersion: v1
kind: Secret
metadata:
  name: monitoring-tls
  namespace: monitoring
type: Opaque  
data:
  tls.crt: {{crt}}
  tls.key: {{key}}
type: kubernetes.io/tls

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: monitoring-ingress
  namespace: monitoring
  labels:
    app: api
spec:
  rules:
    - host: {{host}}
      http:
        paths:
          - backend:
              serviceName: monitoring-service
              servicePort: http
            path: /
  tls:
    - hosts:
      - {{host}}
      secretName: monitoring-tls

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: monitoring
  labels:
    app: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
        commit: '{{commit}}'
    spec:
      serviceAccountName: api-access 
      containers:
        - name: api
          image: medinvention/k8s-monitoring-api:arm
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 2
            periodSeconds: 3
            failureThreshold: 1
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 300
            timeoutSeconds: 5
            periodSeconds: 60
            failureThreshold: 1

---
apiVersion: v1
kind: Service
metadata:
  name: monitoring-service
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
  selector:
    app: api

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
  name: api-access
  namespace: monitoring

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: api-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: api-access
    namespace: monitoring

And the front

apiVersion: v1
kind: Secret
metadata:
  name: front-tls
  namespace: monitoring
type: Opaque  
data:
  tls.crt: {{crt}}
  tls.key: {{key}}
type: kubernetes.io/tls

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: front
  namespace: monitoring
  labels:
    app: front
spec:
  rules:
    - host: {{host}}
      http:
        paths:
          - backend:
              serviceName: front-service
              servicePort: http
            path: /
  tls:
    - hosts:
      - {{host}}
      secretName: front-tls

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: front
  namespace: monitoring
  labels:
    app: front
spec:
  replicas: 1
  selector:
    matchLabels:
      app: front
  template:
    metadata:
      labels:
        app: front
        commit: '{{commit}}'
    spec:
      containers:
      - name: front
        image: medinvention/k8s-monitoring-front:arm
        imagePullPolicy: Always
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: front-service
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 80
    targetPort: 80
  selector:
    app: front

Important: if you use same image tag with every build, we must specify "imagePullPolicy: Always" to force kube pulling image with every deployment.

Result

The S2I Job

It's done, you have an operational CI/CD pipeline for your development environment. When you push a new code, your pipeline will be executed automatically to test, build, and deploy your application.

Complete source code available @here.

Enjoy

I have a fresh cluster installed and i will start with a simple deployment of static website.

Docker Image

For our Website, we will use Nginx for arm32 architecture.

FROM nginx

EXPOSE 80

COPY _site/ /usr/share/nginx/html

Just copy static website source code with index file to "_site" directory in Docker context and just build image, tag then push it to "docker.io" registry. It will be available to our cluster.

docker build . -t {{yourrepository}}/home
docker tag {{yourrepository}}/home:latest {{yourrepository}}/home:arm
docker push {{yourrepository}}/home:arm

We must be logged to Docker hub registry before pushing newly image and replacing {{yourrepository}} by our Docker Hub username.

Kubernetes deployment

After prepared image, it's time to deploy it.

Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: front

Create a simple namespace to be a scope for our project.

SSL Secret

apiVersion: v1
kind: Secret
metadata:
  name: home-tls
  namespace: front
type: Opaque  
data:
  tls.crt: {{crt}}
  tls.key: {{key}}
type: kubernetes.io/tls

To have a HTTPS access to our website, we need to get SSL certificate and it's easy and free with letsencrypt.org by just installing CertBot in any node of your cluster and use it to generate certificate for your domain.

You can use "TXT Record" identification method for certificate generation: CertBot give you a string token and you must add it to your DNS configuration of your domain as "TXT Record" to continue generation.

After, you must replace {{crt}} by fullchain and not just cert.pem certificate encoded base64, like {{key}} who is must be replace by the private key encoded also.

# private key
cat privkey.pem | base64
# fullchain certificate
cat fullchain.pem | base64

Deployment & Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: home
  namespace: front
  labels:
    app: home
spec:
  replicas: 1
  selector:
    matchLabels:
      app: home
  template:
    metadata:
      labels:
        app: home
    spec:
      containers:
      - name: home
        image: medinvention/home:arm
        imagePullPolicy: Always
        ports:
        - containerPort: 80

For our deployment, we define one container for built image and export one port 80 for web access. We defined a label selector "app: home" to bind pod to service and create a HTTP end point to access pod service from outside of cluster.

apiVersion: v1
kind: Service
metadata:
  name: home-service
  namespace: front
spec:
  ports:
  - name: http
    port: 80
    targetPort: 80
  selector:
    app: home

With this way, we have bound "home-servcice" to HTTP port of container inside pod deployed. "Kube" use match label configuration to make association between service and pod and define end point.

Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: home
  namespace: front
  labels:
    app: home
spec:
  rules:
    - host: {{host}}
      http:
        paths:
          - backend:
              serviceName: home-service
              servicePort: http
            path: /
  tls:
    - hosts:
      - {{host}}
      secretName: home-tls

To make our service accessible from outside of cluster, we define an "Ingress" object to publish home service using Nginx Load Balancer. Change {{host}} by your host and apply it.

If you check the Load Balancer service, you will find dedicated ports like "80:31782/TCP,443:31179/TCP". You must configure your internet router (or box) to NAT 443 external port to 31179 port of master node.

It's done , enjoy :)

By this memo, i purpose you to discover how deploy a Kubernetes K8S solution on your own cluster composed by some raspberry.

Preparing nodes

We need Raspbian Release with ability to support Docker, so we will use HypriotOS available here.

curl -OJSLs https://github.com/hypriot/image-builder-rpi/releases/download/v1.12.0/hypriotos-rpi-v1.12.0.img.zip
unzip hypriotos-rpi-v1.12.0.img.zip

Now we need utility for Flashing SD Card with forcing some instance parameters, like user, hostname, ... Hypriotos Flash Tool will be most efficience and available here.

curl -LO https://github.com/hypriot/flash/releases/download/2.5.0/flash
chmod +x flash && sudo mv flash /usr/local/bin/flash

Flashing now, i have one master node and four worker nodes

flash --hostname master hypriotos-rpi-v1.12.0.img
flash --hostname north hypriotos-rpi-v1.12.0.img
flash --hostname south hypriotos-rpi-v1.12.0.img
flash --hostname east hypriotos-rpi-v1.12.0.img
flash --hostname west hypriotos-rpi-v1.12.0.img

Configuring

I - Inventary

To configure our cluster, we will use some Ansible playbook, it's more powerful and can be reused with new nodes if we want.

We start by dowloading Ansible playbook available here.

Then, wi start by adjusing host configuration ; setting of master and worker IP, next setting of ssh user and password (for me i use same credentials for all nodes for simplicity)

sed "s/{{masterip}}/[MASTERIP]/" hosts.dist > hosts 
sed -i "s/{{northip}}/[NORTHIP]/" hosts 
sed -i "s/{{southip}}/[SOUTHIP]/"  hosts 
sed -i "s/{{eastip}}/[eastip]/" hosts 
sed -i "s/{{westip}}/[westip]/"  hosts 

sed "s/{{user}}/[USER]/" group_vars/all.yml.dist > group_vars/all.yml
sed -i "s/{{password}}/[PASS]/" group_vars/all.yml

II- Preparing OS

Executing "bootstrap", will configure all raspberry with enabling cgroup for memory , cpu and disabling all swap.

ansible-playbook bootstrap.yml -i hosts --verbose

Next step, installing all common dependencies for Kubernetes on master node before initializing cluster with KubeAdmin.

ansible-playbook master.yml -i hosts --verbose

Finally, install dependencies on worker nodes and join master node.

ansible-playbook node.yml -i hosts --verbose

Set Up Cluster

I- CNI

Kubernetes need a network plugin to manage cluster intra-communication. For our project, i choose to use Flannel with some adjucing (like ARM image arch).

kubectl create -f kube/flannel.yml
kubectl create -f kube/kubedns.yml
# Must be done on all node
sudo sysctl net.bridge.bridge-nf-call-iptables=1

II- Ingress

Ingress manages external access to the services in a cluster, may provide load balancing, SSL termination and name-based virtual hosting. For ower project we use Nginx Ingress.

helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm install nginx-ingress stable/nginx-ingress --set defaultBackend.image.repository=docker.io/medinvention/ingress-default-backend,controller.image.repository=quay.io/kubernetes-ingress-controller/nginx-ingress-controller-arm,defaultBackend.image.tag=latest,controller.image.tag=0.27.1
helm install ingress stable/nginx-ingress --set controller.hostNetwork=true,controller.kind=DaemonSet

For Cluster public IP

# Check public IP if set
kubectl get svc ingress-nginx-ingress-controller -o jsonpath="{.status.loadBalancer.ingress[0].ip}"
# You can set it manual
kubectl patch svc nginx-ingress-controller -p '{"spec": {"type": "LoadBalancer", "externalIPs":["[YOUR-PUBLIC-IP]"]}}'

Trooblshooting

If Pod cannot communicate, or if CoreDNS canno't be ready, run in all node

sudo systemctl stop docker
sudo iptables -t nat -F
sudo iptables -P FORWARD ACCEPT
sudo ip link del docker0
sudo ip link del flannel.1
sudo systemctl start docker

III- Storage

For Storage solution, we can deploy a CEPH server or NFS service and configure cluster to use it. For our project, we will install NFS server on master node

sudo apt-get install nfs-kernel-server nfs-common
sudo systemctl enable nfs-kernel-server
sudo systemctl start nfs-kernel-server

sudo cat >> /etc/exports <<EOF
/data/kubernetes-storage/ north(rw,sync,no_subtree_check,no_root_squash)
/data/kubernetes-storage/ south(rw,sync,no_subtree_check,no_root_squash)
/data/kubernetes-storage/ east(rw,sync,no_subtree_check,no_root_squash)
/data/kubernetes-storage/ west(rw,sync,no_subtree_check,no_root_squash)
EOF

sudo exportfs -a  

On worker node

sudo apt-get install nfs-common

Next, we deploy NFS service

kubectl apply -f storage/nfs-deployment.yml

It's will create new storage class and marked it to default then deploy storage pod.

Now to test configuration

kubectl apply -f storage/nfs-testing.yml

Tricks

I- Cluster backup

To backup our cluster, we need to run

./os/backup.sh # cluset data will be saved in ~/bkp

II- Cluster tear down

To reset cluster, just run on master node

kubeadm reset

It's Ready

After a little time, all nodes must be ready

kubectl get nodes -o wide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
east     Ready    <none>   23h   v1.17.4   192.168.1.30   <none>        Raspbian GNU/Linux 10 (buster)   4.19.75-v7l+     docker://19.3.5
master   Ready    master   63d   v1.17.1   192.168.1.17   <none>        Raspbian GNU/Linux 10 (buster)   4.19.75-v7+      docker://19.3.5
north    Ready    <none>   63d   v1.17.1   192.168.1.54   <none>        Raspbian GNU/Linux 10 (buster)   4.19.75-v7+      docker://19.3.5
south    Ready    <none>   63d   v1.17.1   192.168.1.11   <none>        Raspbian GNU/Linux 10 (buster)   4.19.75-v7+      docker://19.3.5
west     Ready    <none>   23h   v1.17.4   192.168.1.85   <none>        Raspbian GNU/Linux 10 (buster)   4.19.75-v7l+     docker://19.3.5

Enjoy ...

Aloha, I propose you a general presentation of Openshift with a Symfony 4 application as a use case.

First, we start with presentations

I'm an open source container application platform based on Kubernetes container orchestrator for enterprise application development and deployment ¹

What can I say in a few words about Openshift

• It's a PaaS : Plateform-as-a-service
• Three offers availables : Online (used in this Post), Online Dedicated and private (not hosted at RedHat)
• It allows build, deploy and run applications in containers
• His configuration is based on a Docker container engine with Kubernetes Orchestrator
• It offers a microservices oriented architecture.

Among the services of Openshift platform

• Containerization (Source to images ², Docker Repository, Image Stream)
• Route & LoaderBalancer
• Shared Storage
• Resource Management (Quota, Membership, ConfigMap, Secret,...)
• Monitoring (Elasticsearch, Fluentd, Kibana) & Readiness / Liveness

Real life
This is a simple example of a web application deployed in Openshift cluster.

The client (which may be a web browser) requests an URL (an external route to the cluster) of the Web application, the platform Router will dispatch this request to the concerned service (this may be an opening of port 80 of the frontal Pod).

The frontal Pod can then be considered as the FontEnd controller of our application, and which will use the business application also deployed in a Pod in Backend and in communication with the Pod of database (on a TCP service on 3306 for example with MySQL database).

Finally, all Pods use the distributed storage system managed by the platform on virtual file system mounts.

Typically, when you have multiple clusters, you define a front cluster (perhaps behind a proxy) to expose your services to outside of the cluster and another back cluster that will embed a business applications as well as access to the data. Both of these clusters will have different security configurations and requirements, which will allow us to isolate and protect business data.



Use case with a Symfony application
It's a standard Symfony 4 demonstration application with a few modifications to add Memcached support for example and using of MySQL database instead of SQLLite. Source is  available here (branch "openshift").

Let's start !
To begin, we need an OpenShift instance, so there simple registration to demo program (Starter Pack), gives us right to 2GB of memory, 2GB of disk and 4 CPU core free for 60 days.

After login to Web console, we have a view of catalog of images and services supported by the platform:

So we start by creating a project (we're entitled to one at a time). For Openshift, this project is a namespace with own rules of resource management, access and some isolation with other namespaces.
For our Post, we will need an instance of Apache with PHP 7, Opcache and Memcached installed. The PHP image proposed by the platform does not support Memcached, for this reason that we will use a dedicated image available on  DockerHub (Tag "openshift" and source available on GitHub).
However, if we want to use the proposed image, we go through the catalog and we will have this

And if we want to use our image, we go through "Add to Project" then "Import YAML / JSON "

We can copy / paste this content

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  labels:
    app: webapp
  name: webapp
spec:
  output:
    to:
      kind: ImageStreamTag
      name: 'webapp:latest'
  source:
    git:
      ref: openshift
      uri: 'https://github.com/mmohamed/demo.git'
    type: Git
  strategy:
    sourceStrategy:
      from:
        kind: DockerImage
        name: 'medinvention/s2i-ubuntu-php72-apache24:openshift'
    type: Source

Then we have to define an external route for application

We can also go through " Add to Project " then " Deploy image " (Without Git sources) we will have

/!\ Tip: You can use the default PHP image to initiate S2I DeploymentConfig with our GitHub Repository, then modify the configuration to use our specific image without having to define a complete BuildConfig.

So, now we have a functional Build and DeploymentConfig, our first Build and our first Deploy are completed successfully. We have a Pod in Run and Ready to process queries.

It then remains the deployment of a database image (we will use MySQL from catalog), and a MemCached server image. For Memcached, we will use a YAML Template , because it does not exist in catalog, with the import of object definitions in YAML / JSON functionality to import this template and apply it.


Now, we must have a complete environment

We can access to our application with generated public URL

One last tool is missing for continuous deployment; there we can use Jenkins which is proposed by platform in catalog.

We just have to update our Jenkins Job to trigger a build, followed by a deployment when it detects a change in our application sources (hosted at GitHub)

We can see Logs of Job run

Started by an SCM change

No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
....
 > git rev-list --no-walk b9feb776306e16bcba20a6d371e9e83a838d1a2b # timeout=10

Starting "Scale OpenShift Deployment" with deployment config "memcached" from the project "demo-medinvention".
 Scaling to "0" replicas and verifying the replica count is reached ...
Operation will timeout after 180000 milliseconds

Exiting "Scale OpenShift Deployment" successfully, where the deployment "memcached-2" reached "0" replica(s).

Starting the "Trigger OpenShift Build" step with build config "webapp" from the project "demo-medinvention".
  Started build "webapp-9" and waiting for build completion ...
Operation will timeout after 900000 milliseconds
Pulling image "medinvention/s2i-ubuntu-php72-apache24:openshift" ...
Using medinvention/s2i-ubuntu-php72-apache24:openshift as the s2i builder image
---> Installing application source...
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
....
Generating optimized autoload files
ocramius/package-versions:  Generating version class...
ocramius/package-versions: ...done generating version class
Executing script cache:clear [OK]
Executing script assets:install --symlink --relative public [OK]

Executing script security-checker security:check [OK]
---> Clear cache/logs and fixing permissions...

Pushing image docker-registry.default.svc:5000/demo-medinvention/webapp:latest ...
Pushed 0/17 layers, 0% complete
....
Pushed 17/17 layers, 100% complete
Push successful


Exiting "Trigger OpenShift Build" successfully; build "webapp-9" has completed with status:  [Complete].

Starting "Trigger OpenShift Deployment" with deployment config "webapp" from the project "demo-medinvention".
Operation will timeout after 600000 milliseconds

Exiting "Trigger OpenShift Deployment" successfully; deployment "webapp-13" has completed with status:  [Complete].

Starting "Tag OpenShift Image" with the source [image stream:tag] "webapp:latest" from the project "demo-medinvention" and destination stream(s) "webapp" with tag(s) "prod" from the project "demo-medinvention".

Exiting "Tag OpenShift Image" successfully.

Starting "Verify OpenShift Deployment" with deployment config "webapp" from the project "demo-medinvention".
  Waiting on the latest deployment for "webapp" to complete ...
Operation will timeout after 180000 milliseconds


Exiting "Verify OpenShift Deployment" successfully; deployment "webapp-13" has completed with status:  [Complete].

Starting "Scale OpenShift Deployment" with deployment config "memcached" from the project "demo-medinvention".
 Scaling to "1" replicas and verifying the replica count is reached ...
Operation will timeout after 180000 milliseconds


Exiting "Scale OpenShift Deployment" successfully, where the deployment "memcached-2" reached "1" replica(s).
Finished: SUCCESS

Finally,

Now, we have a complete environment with full integration and deployment.

We can pick out two Pipelines :

• First is for our Base Docker image (S2I) ; as I have set up an Auto-Build in DockerHub (1) , it's constantly monitors the repository  of the image in GitHub and as soon as there is a change, a Build is launched in DockerHub and result image is pushed to Registry. In this way, with each new Build of our project in Openshit, the new image will be used (2) .
• The second Pipeline is for our application; it's our Jenkins Job who will be the orchestrator, because as soon as he detects a change in the repository of the sources of our demo application (5) , he will launch a new Build starting with a " Scale down " of the Memcache DeployConfig (because there is a lack of CPU in project resources), followed by a " Build webapp ", then a deployment " Deploy webapp ". Then we Tag the resulting image of the Build (in the Openshift Registry) before checking the deployment. Finally, we finish our Job with a " Scale Up " Memcache DeployConfig .

The functioning of our environment remains simple; when an HTTP request from the client(3) come, it will be dispatched by the Router to webapp service and therefore to application pod on port 8080 (4) . The application will need to communicate with the Memcache server(6) and will have to go through the services, as well as to communicate with Pod of database(7).

Here is an environment that can be used perfectly to develop and experience Openshift freely.
Personally, I fell in love with this solution and this wonderful compilation of tools. I hope you start to love Openshift.

NOTA : This post is the result of my experience with Openshift and my own understanding. If you notice omissions or errors do not hesitate to report it to me with a nice comment.



¹ : RedHat definition - ² :I will do another Post for S2I