User:Hamzy/TripleO mixed undercloud overcloud try9

On a CentOS7 system

[hamzy@oscloud5 ~]$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.3.1611 (Core) 
Release:        7.3.1611
Codename:       Core
[stack@oscloud5 ~]$ uname -a
Linux oscloud5.stglabs.ibm.com 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Unfortunately, it seems that Environment setup for baremetal environment does not explain how to install the undercloud. There are three machines in this scenario:

arch	use	portname1	MAC1	IP1	portname2	MAC2	IP2
x86_64	undercloud	eno2	6c:ae:8b:29:2a:02	9.114.219.30	eno4	6c:ae:8b:29:2a:04	9.114.118.98
x86_64	overcloud control	enp12s0f0	5c:f3:fc:36:dd:68	9.114.219.134	eno3	6c:ae:8b:29:27:fb	9.114.118.??? 13?
ppc64le	overcloud compute	enP3p5s0f2	00:90:fa:74:05:52	9.114.219.49	enP3p5s0f3	00:90:fa:74:05:53	9.114.118.154

So, following Undercloud installation, I perform the following:

[hamzy@oscloud5 ~]$ sudo useradd stack
[hamzy@oscloud5 ~]$ sudo passwd stack
[hamzy@oscloud5 ~]$ echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
[hamzy@oscloud5 ~]$ sudo chmod 0440 /etc/sudoers.d/stack
[hamzy@oscloud5 ~]$ sudo su - stack
[stack@oscloud5 ~]$ sudo hostnamectl set-hostname oscloud5.stglabs.ibm.com
[stack@oscloud5 ~]$ sudo hostnamectl set-hostname --transient oscloud5.stglabs.ibm.com
[stack@oscloud5 ~]$ sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo
[stack@oscloud5 ~]$ sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo
[stack@oscloud5 ~]$ sudo yum install -y python-tripleoclient
[stack@oscloud5 ~]$ cp /usr/share/instack-undercloud/undercloud.conf.sample ~/undercloud.conf
[stack@oscloud5 ~]$ cat << '__EOF__' > instackenv.json
{
    "nodes": [
        {
            "pm_type":"pxe_ipmitool",
            "mac":[
                "5c:f3:fc:36:dd:68"
            ],
            "cpu":"2",
            "memory":"1048576",
            "disk":"1000",
            "arch":"x86_64",
            "pm_user":"user",
            "pm_password":"update",
            "pm_addr":"9.114.219.33"
        },
        {
            "pm_type":"pxe_ipmitool",
            "mac":[
                "00:90:fa:74:05:53"
            ],
            "cpu":"16",
            "memory":"1048576",
            "disk":"1000",
            "arch":"ppc64le",
            "pm_password":"update",
            "pm_addr":"9.114.118.155"
        }
    ]
}
__EOF__

I transfer over the built ppc64le overcloud images:

(OCB=$(dig @192.168.122.1 -4 +short Overcloud.virbr0); UC=9.114.118.98; ssh-keygen -f ~/.ssh/known_hosts -R ${UC}; ssh-keyscan ${UC} >> ~/.ssh/known_hosts; scp -3 hamzy@${OCB}:~/ironic-python-agent.initramfs stack@${UC}:~/ppc64le-ironic-python-agent.initramfs; scp -3 hamzy@${OCB}:~/ironic-python-agent.kernel stack@${UC}:~/ppc64le-ironic-python-agent.kernel; scp -3 hamzy@${OCB}:~/overcloud-full.qcow2 stack@${UC}:~/ppc64le-overcloud-full.qcow2; scp -3 hamzy@${OCB}:~/overcloud-full.initrd stack@${UC}:~/ppc64le-overcloud-full.initrd; scp -3 hamzy@${OCB}:~/overcloud-full.kernel stack@${UC}:~/ppc64le-overcloud-full.kernel)

I then transfer over the built x86_64 overcloud images:

[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/current-tripleo-rdo/ironic-python-agent.tar | tar xvf -
[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/current-tripleo-rdo/overcloud-full.tar | tar xvf -
[stack@oscloud5 ~]$ for FILE in ironic-python-agent.initramfs ironic-python-agent.kernel overcloud-full.initrd overcloud-full.vmlinuz overcloud-full.qcow2; do mv ${FILE} x86_64-${FILE}; done

I then modify undercloud.conf as follows:

[stack@oscloud5 ~]$ cat << __EOF__ | patch -p0
--- undercloud.conf.orig        2017-08-25 12:04:54.935063830 +0000
+++ undercloud.conf 2017-08-25 12:05:17.561063576 +0000
@@ -17,21 +17,25 @@
 # defined by local_interface, with the netmask defined by the prefix
 # portion of the value. (string value)
 #local_ip = 192.168.24.1/24
+local_ip = 9.114.118.98/24
 
 # Network gateway for the Neutron-managed network for Overcloud
 # instances. This should match the local_ip above when using
 # masquerading. (string value)
 #network_gateway = 192.168.24.1
+network_gateway = 9.114.118.98
 
 # Virtual IP or DNS address to use for the public endpoints of
 # Undercloud services. Only used with SSL. (string value)
 # Deprecated group/name - [DEFAULT]/undercloud_public_vip
 #undercloud_public_host = 192.168.24.2
+undercloud_public_host = 9.114.118.98
 
 # Virtual IP or DNS address to use for the admin endpoints of
 # Undercloud services. Only used with SSL. (string value)
 # Deprecated group/name - [DEFAULT]/undercloud_admin_vip
 #undercloud_admin_host = 192.168.24.3
+undercloud_admin_host = 9.114.118.98
 
 # DNS nameserver(s) to use for the undercloud node. (list value)
 #undercloud_nameservers =
@@ -74,6 +78,7 @@
 # Network interface on the Undercloud that will be handling the PXE
 # boots and DHCP for Overcloud instances. (string value)
 #local_interface = eth1
+local_interface = eno4
 
 # MTU to use for the local_interface. (integer value)
 #local_mtu = 1500
@@ -82,18 +87,22 @@
 # instances. This should be the subnet used for PXE booting. (string
 # value)
 #network_cidr = 192.168.24.0/24
+network_cidr = 9.114.118.0/24
 
 # Network that will be masqueraded for external access, if required.
 # This should be the subnet used for PXE booting. (string value)
 #masquerade_network = 192.168.24.0/24
+masquerade_network = 9.114.118.0/24
 
 # Start of DHCP allocation range for PXE and DHCP of Overcloud
 # instances. (string value)
 #dhcp_start = 192.168.24.5
+dhcp_start = 9.114.118.240
 
 # End of DHCP allocation range for PXE and DHCP of Overcloud
 # instances. (string value)
 #dhcp_end = 192.168.24.24
+dhcp_end = 9.114.118.248
 
 # Path to hieradata override file. If set, the file will be copied
 # under /etc/puppet/hieradata and set as the first file in the hiera
@@ -112,12 +121,14 @@
 # doubt, use the default value. (string value)
 # Deprecated group/name - [DEFAULT]/discovery_interface
 #inspection_interface = br-ctlplane
+inspection_interface = br-ctlplane
 
 # Temporary IP range that will be given to nodes during the inspection
 # process.  Should not overlap with the range defined by dhcp_start
 # and dhcp_end, but should be in the same network. (string value)
 # Deprecated group/name - [DEFAULT]/discovery_iprange
 #inspection_iprange = 192.168.24.100,192.168.24.120
+inspection_iprange = 9.114.118.249,9.114.118.250
 
 # Whether to enable extra hardware collection during the inspection
 # process. Requires python-hardware or python-hardware-detect package
__EOF__

And install the undercloud:

[stack@oscloud5 ~]$ time openstack undercloud install 2>&1 | tee output.undercloud.install
...
Undercloud install complete.
...

Ironic needs some different settings to be able to support PXE for ppc64le:

[stack@oscloud5 ~]$ (cd /etc/ironic; cat << '__EOF__' | sudo patch -p0)
--- ironic.conf.orig    2017-11-03 13:54:06.440644679 +0000
+++ ironic.conf 2017-11-03 15:01:47.584431653 +0000
@@ -343,6 +343,7 @@
 # for this option to be unset. (string value)
 # Allowed values: debug, info, warning, error, critical
 #notification_level = <None>
+notification_level = debug
 
 # Directory where the ironic python module is installed.
 # (string value)
@@ -3506,7 +3507,7 @@
 # On ironic-conductor node, template file for PXE
 # configuration. (string value)
 #pxe_config_template = $pybasedir/drivers/modules/pxe_config.template
-pxe_config_template=$pybasedir/drivers/modules/ipxe_config.template
+pxe_config_template=$pybasedir/drivers/modules/pxe_config.template
 
 # On ironic-conductor node, template file for PXE
 # configuration for UEFI boot loader. (string value)
@@ -3517,6 +3518,7 @@
 # configuration per node architecture. For example:
 # aarch64:/opt/share/grubaa64_pxe_config.template (dict value)
 #pxe_config_template_by_arch =
+pxe_config_template_by_arch = ppc64le:$pybasedir/drivers/modules/pxe_config.template
 
 # IP address of ironic-conductor node's TFTP server. (string
 # value)
@@ -3547,7 +3549,7 @@
 
 # Bootfile DHCP parameter. (string value)
 #pxe_bootfile_name = pxelinux.0
-pxe_bootfile_name=undionly.kpxe
+pxe_bootfile_name=pxelinux.0
 
 # Bootfile DHCP parameter for UEFI boot mode. (string value)
 #uefi_pxe_bootfile_name = bootx64.efi
@@ -3556,10 +3558,11 @@
 # Bootfile DHCP parameter per node architecture. For example:
 # aarch64:grubaa64.efi (dict value)
 #pxe_bootfile_name_by_arch =
+pxe_bootfile_name_by_arch = ppc64le:config
 
 # Enable iPXE boot. (boolean value)
 #ipxe_enabled = false
-ipxe_enabled=True
+ipxe_enabled = false
 
 # On ironic-conductor node, the path to the main iPXE script
 # file. (string value)
__EOF__

Ironic also needs patching to the ironic-inspector component.

[stack@oscloud5 ~]$ cat <<"EO_DNSMASQ_CFG" | sudo tee -a /etc/ironic-inspector/dnsmasq.conf
# Client is ppc64le (OPAL) box and doesn't need a chain loader, but does need a custom config
dhcp-match=set:ppc64le,option:client-arch,14
dhcp-boot=tag:ppc64le,
dhcp-option=tag:ppc64le, 209,/tftpboot/ppc64le/deploy.cfg
dhcp-mac=set:hamzy,6c:ae:8b:29:27:fa
dhcp-mac=set:hamzy,6c:ae:8b:29:27:fb
dhcp-mac=set:hamzy,00:90:fa:74:05:52
dhcp-mac=set:hamzy,00:90:fa:74:05:53
dhcp-mac=set:hamzy,00:0a:f7:73:3c:c2
dhcp-mac=set:hamzy,00:0a:f7:73:3c:c3
dhcp-ignore=tag:!hamzy
EO_DNSMASQ_CFG
[stack@oscloud5 ~]$ sudo mkdir /tftpboot/ppc64le; sudo chmod 0755 /tftpboot/ppc64le
[stack@oscloud5 ~]$ cat <<"EO_DEPLOY_CFG" | sudo tee /tftpboot/ppc64le/deploy.cfg
default IPA-Deploy
label IPA-Deploy
    kernel ::/tftpboot/ppc64le/agent.kernel
    initrd ::/tftpboot/ppc64le/agent.ramdisk
    append ipa-inspection-callback-url=http://9.114.118.98:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,numa-topology,logs systemd.journald.forward_to_console=yes ipa-debug=1 ipa-inspection-dhcp-all-interfaces=1 ipa-collect-lldp=1
    sysappend 0x02
EO_DEPLOY_CFG
[stack@oscloud5 ~]$ sudo cp ~stack/ppc64le-ironic-python-agent.kernel /tftpboot/ppc64le/agent.kernel
[stack@oscloud5 ~]$ sudo cp ~stack/ppc64le-ironic-python-agent.initramfs /tftpboot/ppc64le/agent.ramdisk
[stack@oscloud5 ~]$ sudo chmod 0644 /tftpboot/ppc64le/*; sudo chown -R ironic:ironic /tftpboot/ppc64le; sudo chcon -R system_u:object_r:tftpdir_t:s0 /tftpboot/ppc64le/
[stack@oscloud5 ~]$ sudo systemctl restart openstack-ironic-inspector-dnsmasq.service

Find out what version of rpms that you are using. Unfortunately, the patches do not apply cleanly for every release. Look for the short git commit hash.

[stack@oscloud5 ~]$ rpm -qf /usr/lib/python2.7/site-packages/tripleo_common/actions/baremetal.py
openstack-tripleo-common-8.3.1-0.20180103233641.04317ff.el7.centos.noarch
[stack@oscloud5 ~]$ rpm -qf /usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_node.py
python-tripleoclient-8.1.1-0.20171231084754.52b71ea.el7.centos.noarch

The short git commit hash for tripleo-common would be 04317ff and for python-tripleoclient would be 52b71ea. Prepare your system for some patches...

[stack@oscloud5 ~]$ (VER=07; PYTHON_TRIPLEOCLIENT=52b71ea; TRIPLEO_COMMON=04317ff; git config --global user.name "Mark Hamzy"; git config --global user.email hamzy@us.ibm.com; git clone git://git.openstack.org/openstack/tripleo-common; git clone git://git.openstack.org/openstack/python-tripleoclient; cd ~/tripleo-common/; git checkout ${TRIPLEO_COMMON}; cd ~/python-tripleoclient/; git checkout ${PYTHON_TRIPLEOCLIENT})

There is a bug for needing the userid for machines using ipmi that needs to be patched around. There are fixes to tripleo-common for hard coded ironic-python-agent and overcloud names.

[stack@oscloud5 ~]$ (VER=07; TRIPLEO_COMMON=04317ff; cd tripleo-common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/${VER}.${TRIPLEO_COMMON}/0001-fix-usernameless-IPMI.patch | git am)
Applying: fix usernameless IPMI
[stack@oscloud5 ~]$ (VER=07; TRIPLEO_COMMON=04317ff; cd tripleo-common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/${VER}.${TRIPLEO_COMMON}/0001-Support-multiple-architectures.patch | git am)
Applying: Support multiple architectures

There are fixes to python-tripleoclient for hard coded ironic-python-agent and overcloud names.

[stack@oscloud5 ~]$ (VER=07; PYTHON_TRIPLEOCLIENT=52b71ea; cd python-tripleoclient/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/${VER}.${PYTHON_TRIPLEOCLIENT}/0001-Support-multiple-architectures.patch | git am)
Applying: Support multiple architectures

After the patches have been applied, they need to be installed:

[stack@oscloud5 ~]$ (cd python-tripleoclient/; sudo python setup.py install --force) 2>&1 | tee output.install.python-tripleoclient
[stack@oscloud5 ~]$ (cd tripleo-common/; sudo python setup.py install --force) 2>&1 | tee output.install.tripleo-common

Once everything has been patched, update the workbooks and restart all of the necessary services.

[stack@oscloud5 ~]$ (cd /usr/share/openstack-tripleo-common/; openstack workbook update workbooks/validations.yaml; openstack workbook update workbooks/baremetal.yaml)
[stack@oscloud5 ~]$ sudo mistral-db-manage --config-file /etc/mistral/mistral.conf --debug populate
[stack@oscloud5 ~]$ (for SERVICE in openstack-mistral-api.service openstack-mistral-engine.service openstack-mistral-executor.service; do sudo systemctl restart ${SERVICE}; done)
[stack@oscloud5 ~]$ for I in openstack-ironic-conductor.service openstack-ironic-inspector.service openstack-ironic-inspector-dnsmasq.service; do sudo systemctl restart ${I}; done

I then go through the process of uploading images for the overcloud:

[stack@oscloud5 ~]$ source stackrc
(undercloud) [stack@oscloud5 ~]$ AGENT_NAME=x86_64-ironic-python-agent time openstack overcloud image upload --os-image-name x86_64-overcloud-full.qcow2
(undercloud) [stack@oscloud5 ~]$ openstack image list
...
| d8f36173-d143-410d-b5f1-9c25770f5483 | x86_64-bm-deploy-kernel       | active |
| a480cdea-4f5c-4d15-b5de-f8094d7a7e4f | x86_64-bm-deploy-ramdisk      | active |
| 3844db47-6223-4037-8043-8383c0b2e5c6 | x86_64-overcloud-full         | active |
| 8075078a-87a0-4137-8ed7-53c521ea5540 | x86_64-overcloud-full-initrd  | active |
| b15ff007-66ca-42e8-9b85-df5dcf4b32e2 | x86_64-overcloud-full-vmlinuz | active |
...
(undercloud) [stack@oscloud5 ~]$ AGENT_ARCH=ppc64le AGENT_NAME=${AGENT_ARCH}-ironic-python-agent time openstack overcloud image upload --whole-disk --os-image-name ppc64le-overcloud-full.qcow2
(undercloud) [stack@oscloud5 ~]$ openstack image list
...
| ed5aa33a-c6fb-4575-9e71-154186a9d5d6 | ppc64le-bm-deploy-kernel      | active |
| 2ad5c234-e7fc-4572-ae86-41151af31321 | ppc64le-bm-deploy-ramdisk     | active |
| c6473e02-8fbb-4b07-beb8-3407118abe68 | ppc64le-overcloud-full        | active |
...

Now import the baremetal nodes and assign them profiles.

(undercloud) [stack@oscloud5 ~]$ time openstack overcloud node import --provide instackenv.json 2>&1 | tee output.overcloud.node.import
(undercloud) [stack@oscloud5 ~]$ openstack overcloud profiles list
...
| 7f9470ab-b79c-4382-94a7-e4dc2ea7658c |           | available       | None            |                   |
| fdf15be5-2542-42ed-8252-a154c58a12bc |           | available       | None            |                   |
...
(undercloud) [stack@oscloud5 ~]$ (COMPUTE=""; CONTROL=""; while IFS=$' ' read -r -a PROFILES; do NODE=${PROFILES[0]}; ARCH=$(openstack baremetal node show --fields properties --format shell ${NODE} | sed -r -e "s,^.*cpu_arch': u'([^']*).*$,\1,"); echo "ARCH=${ARCH} NODE=${NODE}"; if [ -z "${COMPUTE}" -a "${ARCH}" == "ppc64le" ]; then COMPUTE=${NODE}; echo "Compute:"; ironic node-update ${COMPUTE} replace properties/capabilities=profile:compute,boot_option:local; continue; elif [ -z "${CONTROL}" -a "${ARCH}" == "x86_64" ]; then CONTROL=${PROFILES[0]}; echo "Control:"; ironic node-update ${CONTROL} replace properties/capabilities=profile:control,boot_option:local; continue; else echo "Error: ${NODE}"; openstack baremetal node show --fields properties --format shell ${NODE}; fi; done < <(openstack overcloud profiles list -f value))
(undercloud) [stack@oscloud5 ~]$ openstack overcloud profiles list
...
| 7f9470ab-b79c-4382-94a7-e4dc2ea7658c |           | available       | control         |                   |
| fdf15be5-2542-42ed-8252-a154c58a12bc |           | available       | compute         |                   |
...

Patch the openstack-tripleo-heat-templates locally with our configuration and then do the deploy.

(undercloud) [stack@oscloud5 ~]$ cp -r /usr/share/openstack-tripleo-heat-templates templates
(undercloud) [stack@oscloud5 ~]$ (cd templates/; curl --silent -o - https://hamzy.fedorapeople.org/openstack-tripleo-heat-templates.patch | patch -p1)
(undercloud) [stack@oscloud5 ~]$ cat << '__EOF__' > templates/environments/override-overcloud-image.yaml
parameter_defaults:
  NovaImage: ppc64le-overcloud-full
  ComputeImage: ppc64le-overcloud-full
  controllerImage: x86_64-overcloud-full
  ControllerImage: x86_64-overcloud-full
__EOF__
(undercloud) [stack@oscloud5 ~]$ time openstack overcloud deploy --templates /home/stack/templates -e /home/stack/templates/environments/network-environment.yaml -e /home/stack/templates/environments/network-isolation-custom.yaml -e /home/stack/templates/environments/override-overcloud-image.yaml --control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute 2>&1 | tee output.overcloud.deploy