Aller au contenu principal
TechLedger Registre technique
Sommaire de la formation

Workload cluster avec kubeadm control-plane et proxmox

coolibra444 vues

3. Workload cluster avec kubeadm control-plane et proxmox

  • Définition des CRDs (Cluster, MachineDeployment, etc.)
  • Création d’un cluster de workload
  • Vérification et gestion des ressources

Définition des CRDs Cluster, MachineDeployment

La commande suivante permet de generer les CRDs pour le workload cluster.

Ajouter les variables suivantes dans le fichier clusterctl-proxmox.yaml ou directement en variables d'env.

## -- Required workload cluster default settings -- ##
PROXMOX_SOURCENODE: "proxmox-node-01"
TEMPLATE_VMID: 1000
ALLOWED_NODES: "[proxmox-node-01]"
VM_SSH_KEYS: "ssh-ed25519 AAAAZDI1NTE5AAAAIGNoeYxPOFEG9sQVNsXLsAdpo coul@home-lab"

## -- networking configuration-- ##
CONTROL_PLANE_ENDPOINT_IP: 10.10.10.41
NODE_IP_RANGES: "[10.10.10.50-10.10.10.56]"
GATEWAY: "10.10.10.1"
FEATURE_GATES: "KubeadmBootstrapFormatIgnition=true"
IP_PREFIX: 24
DNS_SERVERS: "[8.8.8.8,8.8.4.4]"
BRIDGE: "vmbr0"

# -- xl nodes -- ##
BOOT_VOLUME_DEVICE: "scsi0"
BOOT_VOLUME_SIZE: "15"
NUM_SOCKETS: "1"
NUM_CORES: "2"
MEMORY_MIB: "2048"

EXP_CLUSTER_RESOURCE_SET: "true"

 

    • Générer les CRDs.
clusterctl generate cluster tools-cluster-dev \
--kubeconfig ~/.kube/management-kubeconfig --infrastructure proxmox:v0.7.4 --target-namespace my-namespace-dev \
--kubernetes-version v1.31.4 --control-plane-machine-count 1 \
--config clusterctl-proxmox.yaml --worker-machine-count 1 > cluster-proxmox.yaml

 

Création du workload cluster

    • Appliquer sur le  management cluster. On peut personnaliser la config en cas de besoin
kubectl --kubeconfig ~/.kube/management-kubeconfig apply -f  cluster-proxmox.yaml

 

    • Exemple de CRDs personnalisés:
      • workers nodepool,
      • Ajout des node labels et taints
      • kube-proxy en mode ipvs,
      • Installation de kube-vip cloud-provider pour attribuer des ip au service type lb
      • activation des lb svc pour kube-vip
      • Ajout des flannel cni
      • Installation de nginx ingress controller
      • Installation de cert-manager
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: tools-cluster-dev
  namespace: default
  labels:
    cluster: tools-cluster-dev
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 10.73.0.0/16
    services:
      cidrBlocks:
      - 10.72.0.0/16      
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: tools-cluster-dev-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: ProxmoxCluster
    name: tools-cluster-dev

---

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxCluster
metadata:
  name: tools-cluster-dev
  namespace: default
spec:
  allowedNodes:
  - proxmox-node-01
  controlPlaneEndpoint:
    host: 10.10.10.41
    port: 6443
  dnsServers:
  - 8.8.8.8
  - 8.8.4.4
  ipv4Config:
    addresses:
    - 10.10.10.50-10.10.10.56
    gateway: 10.10.10.1
    prefix: 24

---

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: tools-cluster-dev-control-plane
  namespace: default
  labels:
    node-role.kubernetes.io/etcd: "etcd"
spec:
  replicas: 1
  version: v1.32.2
  kubeadmConfigSpec:
    files:
    - content: |
        apiVersion: v1
        kind: Pod
        metadata:
          creationTimestamp: null
          name: kube-vip
          namespace: kube-system
        spec:
          containers:
          - args:
            - manager
            env:
            - name: cp_enable
              value: "true"
            - name: vip_interface
              value: ""
            - name: address
              value: 10.10.10.41
            - name: port
              value: "6443"
            - name: vip_arp
              value: "true"
            - name: vip_leaderelection
              value: "true"
            - name: vip_leaseduration
              value: "15"
            - name: svc_enable
              value: "true" 
            - name: vip_renewdeadline
              value: "10"
            - name: vip_retryperiod
              value: "2"
            image: ghcr.io/kube-vip/kube-vip:v0.8.9
            imagePullPolicy: IfNotPresent
            name: kube-vip
            resources: {}
            securityContext:
              capabilities:
                add:
                - NET_ADMIN
                - NET_RAW
            volumeMounts:
            - mountPath: /etc/kubernetes/admin.conf
              name: kubeconfig
          hostAliases:
          - hostnames:
            - localhost
            - kubernetes
            ip: 127.0.0.1
          hostNetwork: true
          volumes:
          - hostPath:
              path: /etc/kubernetes/admin.conf
              type: FileOrCreate
            name: kubeconfig
        status: {}
      owner: root:root
      path: /etc/kubernetes/manifests/kube-vip.yaml
    - content: |
        #!/bin/bash

        # Copyright 2020 The Kubernetes Authors.
        #
        # Licensed under the Apache License, Version 2.0 (the "License");
        # you may not use this file except in compliance with the License.
        # You may obtain a copy of the License at
        #
        #     http://www.apache.org/licenses/LICENSE-2.0
        #
        # Unless required by applicable law or agreed to in writing, software
        # distributed under the License is distributed on an "AS IS" BASIS,
        # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        # See the License for the specific language governing permissions and
        # limitations under the License.

        set -e

        # Configure the workaround required for kubeadm init with kube-vip:
        # xref: https://github.com/kube-vip/kube-vip/issues/684

        # Nothing to do for kubernetes < v1.29
        KUBEADM_MINOR="$(kubeadm version -o short | cut -d '.' -f 2)"
        if [[ "$KUBEADM_MINOR" -lt "29" ]]; then
          exit 0
        fi

        IS_KUBEADM_INIT="false"

        # cloud-init kubeadm init
        if [[ -f /run/kubeadm/kubeadm.yaml ]]; then
          IS_KUBEADM_INIT="true"
        fi

        # ignition kubeadm init
        if [[ -f /etc/kubeadm.sh ]] && grep -q -e "kubeadm init" /etc/kubeadm.sh; then
          IS_KUBEADM_INIT="true"
        fi

        if [[ "$IS_KUBEADM_INIT" == "true" ]]; then
          sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' \
            /etc/kubernetes/manifests/kube-vip.yaml
        fi
      owner: root:root
      path: /etc/kube-vip-prepare.sh
      permissions: "0700"
    - content: |
        #/bin/sh
        cat >> /run/kubeadm/kubeadm.yaml <<EOF
        ---
        apiVersion: kubeproxy.config.k8s.io/v1alpha1
        kind: KubeProxyConfiguration
        mode: "ipvs"
        ipvs:
          strictARP: true
          scheduler: "wlc"
      owner: root:root
      permissions: "0755"
      path: /tmp/kube-proxy.sh    
    initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          provider-id: proxmox://'{{ ds.meta_data.instance_id }}'       
    joinConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          provider-id: proxmox://'{{ ds.meta_data.instance_id }}'
    preKubeadmCommands:
    - /tmp/kube-proxy.sh
    - /etc/kube-vip-prepare.sh
    users:
    - name: root
      sshAuthorizedKeys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGNoeYxPOFEG9H70hmxRF66WL4MMVQ5XbQVNsXLsAdpo
        coul@home-lab
  machineTemplate:
    metadata:
      labels:
        node-role.kubernetes.io/etcd: "etcd"
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
      kind: ProxmoxMachineTemplate
      name: tools-cluster-dev-control-plane

---

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxMachineTemplate
metadata:
  name: tools-cluster-dev-control-plane
  namespace: default
spec:
  template:
    spec:
      disks:
        bootVolume:
          disk: scsi0
          sizeGb: 15
      format: qcow2
      full: true
      memoryMiB: 2048
      network:
        default:
          bridge: vmbr0
          model: virtio
      numCores: 2
      numSockets: 1
      sourceNode: proxmox-node-01
      templateID: 102

---

############ nodepool wroker ############
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: tools-cluster-dev-workers
  namespace: default
spec:
  clusterName: tools-cluster-dev
  replicas: 1
  selector:
    matchLabels: null
  template:
    metadata:
      labels:
        node-role.kubernetes.io/worker: "worker"
        techledger.node-pool-role: "worker"
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: tools-cluster-dev-worker
      clusterName: tools-cluster-dev
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
        kind: ProxmoxMachineTemplate
        name: tools-cluster-dev-worker
      version: v1.32.2

---

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxMachineTemplate
metadata:
  name: tools-cluster-dev-worker
  namespace: default
spec:
  template:
    spec:
      disks:
        bootVolume:
          disk: scsi0
          sizeGb: 15
      format: qcow2
      full: true
      memoryMiB: 2048
      network:
        default:
          bridge: vmbr0
          model: virtio
      numCores: 2
      numSockets: 1
      sourceNode: proxmox-node-01
      templateID: 102

---

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: tools-cluster-dev-worker
  namespace: default
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            provider-id: proxmox://'{{ ds.meta_data.instance_id }}'
      users:
      - name: root
        sshAuthorizedKeys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGNoeYxPOFEG9H70hmxRF66WL4MMVQ5XbQVNsXLsAdpo
          coul@home-lab

---

############ nodepool monito ############
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: tools-cluster-dev-monito
  namespace: default
spec:
  clusterName: tools-cluster-dev
  replicas: 0
  selector:
    matchLabels: null
  template:
    metadata:
      labels:
        node-role.kubernetes.io/worker: "worker"
        techledger.node-pool-role: "monito"
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: tools-cluster-dev-kct-monito
      clusterName: tools-cluster-dev
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
        kind: ProxmoxMachineTemplate
        name: tools-cluster-dev-pmt-monito
      version: v1.32.2
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxMachineTemplate
metadata:
  name: tools-cluster-dev-pmt-monito
  namespace: default
spec:
  template:
    spec:
      disks:
        bootVolume:
          disk: scsi0
          sizeGb: 15
      format: qcow2
      full: true
      memoryMiB: 2048
      network:
        default:
          bridge: vmbr0
          model: virtio
      numCores: 2
      numSockets: 1
      sourceNode: proxmox-node-01
      templateID: 102

---

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: tools-cluster-dev-kct-monito
  namespace: default
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            provider-id: proxmox://'{{ ds.meta_data.instance_id }}'
          taints:
            - key: "node-role.kubernetes.io/monito"
              value: "true"
              effect: "NoSchedule"        
      users:
      - name: root
        sshAuthorizedKeys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGNoeYxPOFEG9H70hmxRF66WL4MMVQ5XbQVNsXLsAdpo
          coul@home-lab
---

## Addons
# https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm/blob/main/docs/quick-start.md

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: flannel-cni
spec:
  clusterSelector:
    matchLabels:
      cluster: tools-cluster-dev
  repoURL: https://flannel-io.github.io/flannel/
  chartName: flannel
  namespace: flannel-system
  options:
    waitForJobs: true
    wait: true
    timeout: 5m
    install:
      createNamespace: true       
  valuesTemplate: |

    podCidr: "10.73.0.0/16"
---

# apiVersion: addons.cluster.x-k8s.io/v1alpha1
# kind: HelmChartProxy
# metadata:
#   name: cert-manager
# spec:
#   clusterSelector:
#     matchLabels:
#       cluster: tools-cluster-dev
#   repoURL: https://charts.jetstack.io
#   chartName: cert-manager
#   namespace: cert-manager
#   version: 1.17.1
#   options:
#     install:
#       createNamespace: true    
#   valuesTemplate: |

#     installCRDs: true

---
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: ingress-nginx
spec:
  clusterSelector:
    matchLabels:
      cluster: tools-cluster-dev
  repoURL: https://kubernetes.github.io/ingress-nginx
  chartName: ingress-nginx
  namespace: ingress-nginx
  version: 4.12.0
  options:
    install:
      createNamespace: true    
  valuesTemplate: |

    controller:  
      kind: DaemonSet
      hostNetwork: false
      service:
        enabled: true
        external:
          enabled: true
        type: LoadBalancer
        loadBalancerIP: 10.10.10.60
---

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: kube-vip-cloud-provider
spec:
  clusterSelector:
    matchLabels:
      cluster: tools-cluster-dev
  repoURL: https://kube-vip.github.io/helm-charts
  chartName: kube-vip-cloud-provider
  namespace: kube-system
  version: 0.2.5
  options:
    waitForJobs: true
    wait: true
    timeout: 5m         
  valuesTemplate: |

    cm:
      data:
        cidr-global: 10.10.10.60/32

 

 

Vérification et gestion des ressources

    • Une fois le cluster est up, on recupère le kube config

Le kubeconfig du workload cluster est énéré dans un secret dans le management cluster. Le secret est nommé nom-cluster-kubeconfig dans notre cas tools-cluster-dev-kubeconfig

kubectl --kubeconfig ~/.kube/management-kubeconfig \
get secret tools-cluster-dev-kubeconfig \
-o jsonpath='{.data.value}' | base64 --decode > tools-cluster-dev-kubeconfig

 

kubectl --kubeconfig tools-cluster-dev-kubeconfig get no -o wide

 

 

    • Scale up le worker en 2 replicas

Pour scale up/down un nodepool, on passe tout simplement le replicas du crd MachineDeployment  au nombre souhaité. Ensuite on replique avec kubectl apply.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: tools-cluster-dev-workers
  namespace: default
spec:
  clusterName: tools-cluster-dev
  replicas: 2
  selector:
    matchLabels: null
  template:
    metadata:
      labels:
        node-role.kubernetes.io/worker: "worker"
        techledger.node-pool-role: "worker"
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: tools-cluster-dev-worker
      clusterName: tools-cluster-dev
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
        kind: ProxmoxMachineTemplate
        name: tools-cluster-dev-worker
      version: v1.32.2

 

    • Appliquer la nouvelle configuration
kubectl --kubeconfig ~/.kube/management-kubeconfig apply -f  cluster-proxmox.yaml

 

Le deuxième worker a été ajouté

 

 

Le service lb de l'ingress controller a bien pris l'ip 10.10.10.60

 

Pour faire un upgrade, il faut build une nouvelle image avec cpai image builder et la bonne k8s souhaitée. Mettre à jour templateID dans le crd ProxmoxMachineTemplate

et faire un apply.

Commentaires

Connectez-vous pour rejoindre la discussion.

Aucun commentaire pour le moment.