Agent-Based Install Notes

The following are notes on deploying OCP with the NEW (4.12) agent-based installer. Two files are required to build the ISO, “install-config.yaml” and “agent-config.yaml”.

See also

For more detail: Preparing to install with the Agent-based installer

Tip

Check disk performance for etcd with “fio”. It’s critical to have a high performing disk drive for OCP / etcd.

For more info: How to Use ‘fio’ to Check Etcd Disk Performance in OCP

podman run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/openshift-scale/etcd-perf

Important

Install will not complete without NTP, validation to start install will fail. Be sure to have a timesource on your network. For disconnected environments that don’t have an NTP server, you can set up a time source on the local bastion server.

  1. Download the latest openshift-install tool found here:

    OpenShift Installer

    Attention

    The link points to the latest version. Typically you’ll want a specific version. You can find those here by browsing to the version specific directory:

    https://mirror.openshift.com/pub/openshift-v4/amd64/clients/ocp/

  2. Install nmstate

    sudo dnf install /usr/bin/nmstatectl -y

  3. Create a work subdir

    mkdir ./workdir

  4. Create “install-config.yaml” and save in ./workdir

    Tip

    In resource-constrained environments, you can use workload partitioning to isolate OpenShift Container Platform services, cluster management workloads, and infrastructure pods to run on a reserved set of CPUs.

    To enable “workload partitioning” add “cpuPartitioningMode: AllNodes” line right after “baseDomain:” line.

    install-config.yaml
    apiVersion: v1
baseDomain: lab.local
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp1
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.122.0/24
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  baremetal:
    apiVIP: "192.168.122.110"
    ingressVIP: "192.168.122.111"
pullSecret: '{"auths":{"mirror.lab.local:8443":{"auth":"aW5pdDpwYXNzd29yZA=="}}}'
sshKey: |
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQA...
imageDigestSources:
- mirrors:
  - mirror.lab.local:8443/openshift/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - mirror.lab.local:8443/openshift/release-images
  source: quay.io/openshift-release-dev/ocp-release
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  <Use rootCA.pem from your mirror registry here>
  -----END CERTIFICATE-----

    Note

    For SNO set “platform:” to “none: {}”.

    platform:
  none: {}

  5. Create agent-config.yaml and save in ./workdir

    Important

    Repeat “-hostname” block for each host in your config.

    agent-config.yaml - Ethernet Network Example
    apiVersion: v1alpha1
metadata:
  name: ocp1
rendezvousIP: 192.168.122.11
additionalNTPSources:
- 192.168.1.68
hosts:
  - hostname: host11
    role: master
    rootDeviceHints:
      deviceName: "/dev/vda"
    interfaces:
      - name: enp1s0
        macAddress: 52:54:00:f4:16:11
    networkConfig:
      interfaces:
        - name: enp1s0
          type: ethernet
          mtu: 1500
          state: up
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 192.168.122.11
                prefix-length: 24
          ipv6:
            enabled: false
      dns-resolver:
        config:
          search:
            - lab.local
          server:
            - 192.168.1.68
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.122.1
            next-hop-interface: enp1s0
            table-id: 254
    agent-config.yaml - VLAN-TAG Network Example
    apiVersion: v1alpha1
metadata:
  name: ocp1
rendezvousIP: 192.168.122.11
additionalNTPSources:
- 192.168.1.68
hosts:
  - hostname: host11
    role: master
    rootDeviceHints:
      deviceName: "/dev/vda"
    interfaces:
      - name: enp1s0
        macAddress: 52:54:00:f4:16:11
    networkConfig:
      interfaces:
        - name: enp1s0
          type: ethernet
          mtu: 1500
          state: up
        - name: enp1s0.122
          type: vlan
          state: up
          vlan:
            base-iface: enp1s0
            id: 122
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 192.168.122.11
                prefix-length: 24
          ipv6:
            enabled: false
      dns-resolver:
        config:
          search:
            - lab.local
          server:
            - 192.168.1.68
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.122.1
            next-hop-interface: enp1s0.122
            table-id: 254
    agent-config.yaml - Bond with VLAN-TAG Network Example
    apiVersion: v1alpha1
metadata:
  name: ocp1
rendezvousIP: 192.168.122.11
additionalNTPSources:
- 192.168.1.68
hosts:
  - hostname: host11
    role: master
    rootDeviceHints:
      deviceName: "/dev/vda"
    interfaces:
      - name: enp1s0
        macAddress: 52:54:00:f4:16:11
      - name: enp2s0
        macAddress: 52:54:00:f4:17:11
    networkConfig:
      interfaces:
        - name: enp1s0
          type: ethernet
          mtu: 1500
          state: up
        - name: enp2s0
          type: ethernet
          mtu: 1500
          state: up
        - name: bond0
          type: bond
          mtu: 1500
          state: up
          link-aggregation:
            mode: active-backup
            port:
            - enp1s0
            - enp2s0
        - name: bond0.122
          type: vlan
          state: up
          vlan:
            base-iface: bond0
            id: 122
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 192.168.122.11
                prefix-length: 24
          ipv6:
            enabled: false
      dns-resolver:
        config:
          search:
            - lab.local
          server:
            - 192.168.1.68
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.122.1
            next-hop-interface: bond0.122
            table-id: 254

  6. With “openshift-install” downloaded in step 1, run the following command. In my case I’m using a “workdir” dir to supply the required yaml files.

    Tip

    Add the sub directory “openshift” to your workdir for custom manifests. For example adding operators or setting “core” user passwd.

    openshift-install agent create image --dir ./workdir

  7. Boot the VM’s with the ISO created in the previous step. Follow the progress with the following command:

    openshift-install agent wait-for install-complete --dir ./workdir

Note

For my environment I manually set the MAC addresses for the VM’s primary interface using the following patterns.

<host mac='52:54:00:f4:16:11' ip='192.168.122.11'/>
<host mac='52:54:00:f4:16:12' ip='192.168.122.12'/>
<host mac='52:54:00:f4:16:13' ip='192.168.122.13'/>

<host mac='52:54:00:f4:16:21' ip='192.168.122.21'/>
<host mac='52:54:00:f4:16:22' ip='192.168.122.22'/>
<host mac='52:54:00:f4:16:23' ip='192.168.122.23'/>

<host mac='52:54:00:f4:16:31' ip='192.168.122.31'/>
<host mac='52:54:00:f4:16:32' ip='192.168.122.32'/>
<host mac='52:54:00:f4:16:33' ip='192.168.122.33'/>

<host mac='52:54:00:f4:16:41' ip='192.168.122.41'/>
<host mac='52:54:00:f4:16:42' ip='192.168.122.42'/>
<host mac='52:54:00:f4:16:43' ip='192.168.122.43'/>

<host mac='52:54:00:f4:16:51' ip='192.168.122.51'/>
<host mac='52:54:00:f4:16:52' ip='192.168.122.52'/>
<host mac='52:54:00:f4:16:53' ip='192.168.122.53'/>

Add Operators

You can add operators to the install by creating the “openshift” dir in your “workdir”. Doing this saves a step post install.

NMState Example

Create the following yaml files and add to ./workdir/openshift before running openshift-install.

apiVersion: v1
kind: Namespace
metadata:
  labels:
    name: openshift-nmstate
  name: openshift-nmstate
spec:
  finalizers:
  - kubernetes

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: NMState.v1.nmstate.io
  name: openshift-nmstate
  namespace: openshift-nmstate
spec:
  targetNamespaces:
  - openshift-nmstate
  upgradeStrategy: Default

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/kubernetes-nmstate-operator.openshift-nmstate: ""
  name: kubernetes-nmstate-operator
  namespace: openshift-nmstate
spec:
  channel: stable
  installPlanApproval: Automatic
  name: kubernetes-nmstate-operator
  source: cs-redhat-operator-index
  sourceNamespace: openshift-marketplace

Custom Partitioning

Here I have a couple of common examples on how to customize the deployment partitioning; A single block device and four block devices.

Attention

This example is “master” nodes only. If you want to apply to other machine config pools be sure to create the machine config with the appropriate labels.

  1. Based on your environment create one of the following butane file/example.

    • One Device (400G)

      Note

      With a single device the installer will use all the available space across 4 partitions. For this to work “resize” partition 4 and create 3 additional partitions, utilizing the space free’d up from the resized partition.

      Note

      By setting “start_mib: 0” the partition starts where the previous partition ended.

      Note

      By setting “size_mib: 0” all of the avilable space is utilized by this partition.

      Important

      • “wipe_partition_entry: true” - If True, delete existing partition.

      • “wipe_filesystem: true” - If True, ignition will always wipe any preexisting filesystem and create the desired filesystem. The old filesystem will be lost.

      • “with_mount_unit: true” - Create the mount point.

      • “mount_options: [defaults, prjquota]” - The prjquota mount option must be enabled for filesystems used for container storage.

      98-master-partition.bu - One Device (400G)
      variant: openshift
version: 4.14.0
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-master-partition
storage:
  disks:
    - device: /dev/disk/by-path/pci-0000:04:00.0
      partitions:
        - number: 1
          should_exist: true
        - number: 2
          should_exist: true
        - number: 3
          should_exist: true
        - number: 4
          resize: true
          size_mib: 120000
        - label: var-lib-containers
          number: 5
          size_mib: 100000
          start_mib: 0
          wipe_partition_entry: true
        - label: var-lib-etcd
          number: 6
          size_mib: 100000
          start_mib: 0
          wipe_partition_entry: true
        - label: var-lib-prometheus-data
          number: 7
          size_mib: 0
          start_mib: 0
          wipe_partition_entry: true
  filesystems:
    - device: /dev/disk/by-partlabel/var-lib-containers
      format: xfs
      path: /var/lib/containers
      wipe_filesystem: true
      with_mount_unit: true
      mount_options: [defaults, prjquota]
    - device: /dev/disk/by-partlabel/var-lib-etcd
      format: xfs
      path: /var/lib/etcd
      wipe_filesystem: true
      with_mount_unit: true
      mount_options: [defaults, prjquota]
    - device: /dev/disk/by-partlabel/var-lib-prometheus-data
      format: xfs
      path: /var/lib/prometheus/data
      wipe_filesystem: true
      with_mount_unit: true
      mount_options: [defaults, prjquota]

    • Four Device’s (100G each)

      Note

      With four device’s we don’t need to identify the first device. I’m doing this for consistency but am making NO changes.

      Note

      By setting “start_mib: 0” the partition starts where the previous partition ended.

      Note

      By setting “size_mib: 0” all of the avilable space is utilized by this partition.

      Important

      • “wipe_table: true” - Without this the previously installed table is used and partition will not get created.

      • “wipe_partition_entry: true” - If True, delete existing partition.

      • “wipe_filesystem: true” - If True, ignition will always wipe any preexisting filesystem and create the desired filesystem. The old filesystem will be lost.

      • “with_mount_unit: true” - Create the mount point.

      • “mount_options: [defaults, prjquota]” - The prjquota mount option must be enabled for filesystems used for container storage.

      98-master-partition.bu - Four Device’s (100G each)
      variant: openshift
version: 4.14.0
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-master-partition
storage:
  disks:
    - device: /dev/disk/by-path/pci-0000:05:00.0
      partitions:
        - number: 1
          should_exist: true
        - number: 2
          should_exist: true
        - number: 3
          should_exist: true
        - number: 4
          should_exist: true
    - device: /dev/disk/by-path/pci-0000:06:00.0
      wipe_table: true
      partitions:
        - label: var-lib-containers
          number: 1
          size_mib: 0
          start_mib: 0
          wipe_partition_entry: true
    - device: /dev/disk/by-path/pci-0000:07:00.0
      wipe_table: true
      partitions:
        - label: var-lib-etcd
          number: 1
          size_mib: 0
          start_mib: 0
          wipe_partition_entry: true
    - device: /dev/disk/by-path/pci-0000:08:00.0
      wipe_table: true
      partitions:
        - label: var-lib-prometheus-data
          number: 1
          size_mib: 0
          start_mib: 0
          wipe_partition_entry: true
  filesystems:
    - device: /dev/disk/by-partlabel/var-lib-containers
      format: xfs
      path: /var/lib/containers
      wipe_filesystem: true
      with_mount_unit: true
      mount_options: [defaults, prjquota]
    - device: /dev/disk/by-partlabel/var-lib-etcd
      format: xfs
      path: /var/lib/etcd
      wipe_filesystem: true
      with_mount_unit: true
      mount_options: [defaults, prjquota]
    - device: /dev/disk/by-partlabel/var-lib-prometheus-data
      format: xfs
      path: /var/lib/prometheus/data
      wipe_filesystem: true
      with_mount_unit: true
      mount_options: [defaults, prjquota]

  2. Create machine config yaml.

    butane 98-master-partition.bu -o 98-master-partition.yaml

  3. Copy the “yaml” output to your install “working” dir / sub dir “openshift”. By default agent install consumes the machine config in this sub dir.

Set Core User Passwd

For lab purposes it might be beneficial to login as core user with a passwd vs. cert auth. This process will set / override the default random passwd at install time.

Attention

This example is “master” nodes only. If you want to apply to other machine config pools be sure to create the machine config with the appropriate labels.

  1. Use mkpasswd to generate the encrypted passwd. I’m setting the passwd to “core”.

    Note

    If needed:

    sudo dnf install mkpasswd

    mkpasswd core

  2. Create the following butane file, “98-master-core-pass.bu”. I’m setting the passwd to “core” with the “mkpasswd” utility.

    98-master-core-pass.bu
    variant: openshift
version: 4.14.0
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 98-master-core-pass
passwd:
  users:
    - name: core
      password_hash: <mkpasswd_output>

  3. Create machine config yaml.

    Note

    If needed:

    curl -L -k https://mirror.openshift.com/pub/openshift-v4/amd64/clients/butane/latest/butane \
-o ~/.local/bin/butane
chmod +x ~/.local/bin/butane

    butane 98-master-core-pass.bu -o 98-master-core-pass.yaml

  4. Copy the “yaml” output to your install “working” dir / sub dir “openshift”. By default agent install consumes the machine config in this sub dir.

    mkdir -p ./<workdir>/openshift
cp 98-master-core-pass.yam ./<workdir>/openshift

Calico Example

This is a continuation of the previous section. Basically adding a subdir to the working directory and copying the Calico CNI yaml files there, the installer will consume the new informantion.

Attention

In this example I’m not disconnected / using my internal mirror.

  1. Create the <assets_directory> and “openshift” subdir.

    mkdir -p ./workdir/openshift

  2. Create “install-config.yaml” and “agent-config.yaml” files in the <assets_directory>.

    install-config.yaml
    apiVersion: v1
baseDomain: lab.local
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp5
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.122.0/24
  networkType: Calico
  serviceNetwork:
  - 172.30.0.0/16
platform:
  baremetal:
    apiVIP: "192.168.122.150"
    ingressVIP: "192.168.122.151"
pullSecret: 'ADD_YOUR_PULL_SECRET_HERE'
sshKey: |
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQA...
    agent-config.yaml
    apiVersion: v1alpha1
metadata:
  name: ocp5
rendezvousIP: 192.168.122.51
additionalNTPSources:
- 192.168.1.68
hosts:
  - hostname: host51
    role: master
    rootDeviceHints:
      deviceName: "/dev/vda"
    interfaces:
      - name: enp1s0
        macAddress: 52:54:00:f4:16:51
    networkConfig:
      interfaces:
        - name: enp1s0
          type: ethernet
          mtu: 1500
          state: up
        - name: enp1s0.122
          type: vlan
          state: up
          vlan:
            base-iface: enp1s0
            id: 122
          ipv4:
            enabled: true
            dhcp: false
            address:
              - ip: 192.168.122.51
                prefix-length: 24
          ipv6:
            enabled: false
      dns-resolver:
        config:
          search:
            - lab.local
          server:
            - 192.168.1.68
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.122.1
            next-hop-interface: enp1s0.122
            table-id: 254

    Important

    Repeat “-hostname” block for each host in your config.

  3. Download and extract the Calico yaml to workdir/openshift.

    Note

    As of this writing v3.27.0 is the latest.

    wget -qO- https://github.com/projectcalico/calico/releases/download/v3.27.0/ocp.tgz | \
  tar xvz --strip-components=1 -C ./workdir/openshift

  4. Create the ISO

    openshift-install agent create image --dir ./workdir

  5. Monitor the install

    openshift-install agent wait-for install-complete --dir ./workdir

  6. Once the cluster is up and running, check the Calico operator status.

    oc get tigerastatus

IPv6 Only Example

install-config.yaml
apiVersion: v1
baseDomain: lab.local
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp3
networking:
  clusterNetwork:
  - cidr: fd02::/48
    hostPrefix: 64
  machineNetwork:
  - cidr: 2600:1702:4c73:f111::0/64
  networkType: OVNKubernetes
  serviceNetwork:
  - fd03::/112
platform:
  baremetal:
    apiVIPs:
      - 2600:1702:4c73:f111::130
    ingressVIPs:
      - 2600:1702:4c73:f111::131
pullSecret: '{"auths":{"mirror.lab.local:8443":{"auth":"aW5pdDpwYXNzd29yZA=="}}}'
sshKey: |
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQA...
imageDigestSources:
- mirrors:
  - mirror.lab.local:8443/openshift/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - mirror.lab.local:8443/openshift/release-images
  source: quay.io/openshift-release-dev/ocp-release
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  <Use rootCA.pem for mirror registry here>
  -----END CERTIFICATE-----
agent-config.yaml
apiVersion: v1alpha1
metadata:
  name: ocp3
rendezvousIP: 2600:1702:4c73:f111::31
hosts:
  - hostname: host31
    role: master
    rootDeviceHints:
      deviceName: "/dev/vda"
    interfaces:
      - name: enp1s0
        macAddress: 52:54:00:f4:16:31
    networkConfig:
      interfaces:
        - name: enp1s0
          type: ethernet
          mtu: 1500
          state: up
        - name: enp1s0.122
          type: vlan
          state: up
          vlan:
            base-iface: enp1s0
            id: 122
          ipv4:
            enabled: false
            dhcp: false
          ipv6:
            enabled: true
            address:
              - ip: 2600:1702:4c73:f111::31
                prefix-length: 64
      dns-resolver:
        config:
          search:
            - lab.local
          server:
            - 2600:1702:4c73:f110::68
      routes:
        config:
          - destination: '::/0'
            next-hop-address: '2600:1702:4c73:f111::1'
            next-hop-interface: enp1s0.122