From Fedora Project Wiki
No edit summary |
No edit summary |
||
| Line 36: | Line 36: | ||
|x86_64 | |x86_64 | ||
|overcloud control | |overcloud control | ||
| | |enp12s0f0 | ||
| | |5c:f3:fc:36:dd:68 | ||
|9.114.219.134 | |9.114.219.134 | ||
|eno3 | |eno3 | ||
Revision as of 20:32, 4 November 2017
On a CentOS7 system
[hamzy@oscloud5 ~]$ lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.3.1611 (Core) Release: 7.3.1611 Codename: Core [stack@oscloud5 ~]$ uname -a Linux oscloud5.stglabs.ibm.com 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Unfortunately, it seems that Environment setup for baremetal environment does not explain how to install the undercloud. There are three machines in this scenario:
| arch | use | portname1 | MAC1 | IP1 | portname2 | MAC2 | IP2 |
| x86_64 | undercloud | eno2 | 6c:ae:8b:29:2a:02 | 9.114.219.30 | eno4 | 6c:ae:8b:29:2a:04 | 9.114.118.98 |
| x86_64 | overcloud control | enp12s0f0 | 5c:f3:fc:36:dd:68 | 9.114.219.134 | eno3 | 6c:ae:8b:29:27:fb | 9.114.118.??? 13? |
| ppc64le | overcloud compute | enP3p5s0f2 | 00:90:fa:74:05:52 | 9.114.219.49 | enP3p5s0f3 | 00:90:fa:74:05:53 | 9.114.118.154 |
So, following Undercloud installation, I perform the following:
[hamzy@oscloud5 ~]$ sudo useradd stack
[hamzy@oscloud5 ~]$ sudo passwd stack
[hamzy@oscloud5 ~]$ echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
[hamzy@oscloud5 ~]$ sudo chmod 0440 /etc/sudoers.d/stack
[hamzy@oscloud5 ~]$ sudo su - stack
[stack@oscloud5 ~]$ sudo hostnamectl set-hostname oscloud5.stglabs.ibm.com
[stack@oscloud5 ~]$ sudo hostnamectl set-hostname --transient oscloud5.stglabs.ibm.com
[stack@oscloud5 ~]$ sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo
[stack@oscloud5 ~]$ sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo
[stack@oscloud5 ~]$ sudo yum install -y python-tripleoclient
[stack@oscloud5 ~]$ cp /usr/share/instack-undercloud/undercloud.conf.sample ~/undercloud.conf
[stack@oscloud5 ~]$ cat << '__EOF__' > instackenv.json
{
"nodes": [
{
"pm_type":"pxe_ipmitool",
"mac":[
"5c:f3:fc:36:dd:68"
],
"cpu":"2",
"memory":"1048576",
"disk":"1000",
"arch":"x86_64",
"pm_user":"user",
"pm_password":"update",
"pm_addr":"9.114.219.33"
},
{
"pm_type":"pxe_ipmitool",
"mac":[
"00:90:fa:74:05:53"
],
"cpu":"16",
"memory":"1048576",
"disk":"1000",
"arch":"ppc64le",
"pm_password":"update",
"pm_addr":"9.114.118.155"
}
]
}
__EOF__
I transfer over the built ppc64le overcloud images:
(OCB=$(dig @192.168.122.1 -4 +short Overcloud.virbr0); UC=9.114.118.98; ssh-keygen -f ~/.ssh/known_hosts -R ${UC}; ssh-keyscan ${UC} >> ~/.ssh/known_hosts; scp -3 hamzy@${OCB}:~/ironic-python-agent.initramfs stack@${UC}:~/ppc64le-ironic-python-agent.initramfs; scp -3 hamzy@${OCB}:~/ironic-python-agent.kernel stack@${UC}:~/ppc64le-ironic-python-agent.kernel; scp -3 hamzy@${OCB}:~/overcloud-full.qcow2 stack@${UC}:~/ppc64le-overcloud-full.qcow2; scp -3 hamzy@${OCB}:~/overcloud-full.initrd stack@${UC}:~/ppc64le-overcloud-full.initrd; scp -3 hamzy@${OCB}:~/overcloud-full.kernel stack@${UC}:~/ppc64le-overcloud-full.kernel)
I then transfer over the built x86_64 overcloud images:
[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/d558d6173970736fde63526da6824d391ada6042_89d45961/ironic-python-agent.tar | tar xvf -
[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/d558d6173970736fde63526da6824d391ada6042_89d45961/overcloud-full.tar | tar xvf -
[stack@oscloud5 ~]$ for FILE in ironic-python-agent.initramfs ironic-python-agent.kernel overcloud-full.initrd overcloud-full.vmlinuz overcloud-full.qcow2; do mv ${FILE} x86_64-${FILE}; done
I then modify undercloud.conf as follows:
[stack@oscloud5 ~]$ cat << __EOF__ | patch -p0 --- undercloud.conf.orig 2017-08-25 12:04:54.935063830 +0000 +++ undercloud.conf 2017-08-25 12:05:17.561063576 +0000 @@ -17,21 +17,25 @@ # defined by local_interface, with the netmask defined by the prefix # portion of the value. (string value) #local_ip = 192.168.24.1/24 +local_ip = 9.114.118.98/24 # Network gateway for the Neutron-managed network for Overcloud # instances. This should match the local_ip above when using # masquerading. (string value) #network_gateway = 192.168.24.1 +network_gateway = 9.114.118.98 # Virtual IP or DNS address to use for the public endpoints of # Undercloud services. Only used with SSL. (string value) # Deprecated group/name - [DEFAULT]/undercloud_public_vip #undercloud_public_host = 192.168.24.2 +undercloud_public_host = 9.114.118.98 # Virtual IP or DNS address to use for the admin endpoints of # Undercloud services. Only used with SSL. (string value) # Deprecated group/name - [DEFAULT]/undercloud_admin_vip #undercloud_admin_host = 192.168.24.3 +undercloud_admin_host = 9.114.118.98 # DNS nameserver(s) to use for the undercloud node. (list value) #undercloud_nameservers = @@ -74,6 +78,7 @@ # Network interface on the Undercloud that will be handling the PXE # boots and DHCP for Overcloud instances. (string value) #local_interface = eth1 +local_interface = eno4 # MTU to use for the local_interface. (integer value) #local_mtu = 1500 @@ -82,18 +87,22 @@ # instances. This should be the subnet used for PXE booting. (string # value) #network_cidr = 192.168.24.0/24 +network_cidr = 9.114.118.0/24 # Network that will be masqueraded for external access, if required. # This should be the subnet used for PXE booting. (string value) #masquerade_network = 192.168.24.0/24 +masquerade_network = 9.114.118.0/24 # Start of DHCP allocation range for PXE and DHCP of Overcloud # instances. (string value) #dhcp_start = 192.168.24.5 +dhcp_start = 9.114.118.240 # End of DHCP allocation range for PXE and DHCP of Overcloud # instances. (string value) #dhcp_end = 192.168.24.24 +dhcp_end = 9.114.118.248 # Path to hieradata override file. If set, the file will be copied # under /etc/puppet/hieradata and set as the first file in the hiera @@ -112,12 +121,14 @@ # doubt, use the default value. (string value) # Deprecated group/name - [DEFAULT]/discovery_interface #inspection_interface = br-ctlplane +inspection_interface = br-ctlplane # Temporary IP range that will be given to nodes during the inspection # process. Should not overlap with the range defined by dhcp_start # and dhcp_end, but should be in the same network. (string value) # Deprecated group/name - [DEFAULT]/discovery_iprange #inspection_iprange = 192.168.24.100,192.168.24.120 +inspection_iprange = 9.114.118.249,9.114.118.250 # Whether to enable extra hardware collection during the inspection # process. Requires python-hardware or python-hardware-detect package __EOF__
And install the undercloud:
[stack@oscloud5 ~]$ time openstack undercloud install 2>&1 | tee output.undercloud.install ... Undercloud install complete. ...
Ironic needs some different settings to be able to support PXE for ppc64le:
[stack@oscloud5 ~]$ (cd /etc/ironic; cat << '__EOF__' | sudo patch -p0) --- ironic.conf.orig 2017-11-03 13:54:06.440644679 +0000 +++ ironic.conf 2017-11-03 15:01:47.584431653 +0000 @@ -343,6 +343,7 @@ # for this option to be unset. (string value) # Allowed values: debug, info, warning, error, critical #notification_level = <None> +notification_level = debug # Directory where the ironic python module is installed. # (string value) @@ -3506,7 +3507,7 @@ # On ironic-conductor node, template file for PXE # configuration. (string value) #pxe_config_template = $pybasedir/drivers/modules/pxe_config.template -pxe_config_template=$pybasedir/drivers/modules/ipxe_config.template +pxe_config_template=$pybasedir/drivers/modules/pxe_config.template # On ironic-conductor node, template file for PXE # configuration for UEFI boot loader. (string value) @@ -3517,6 +3518,7 @@ # configuration per node architecture. For example: # aarch64:/opt/share/grubaa64_pxe_config.template (dict value) #pxe_config_template_by_arch = +pxe_config_template_by_arch = ppc64le:$pybasedir/drivers/modules/pxe_config.template # IP address of ironic-conductor node's TFTP server. (string # value) @@ -3547,7 +3549,7 @@ # Bootfile DHCP parameter. (string value) #pxe_bootfile_name = pxelinux.0 -pxe_bootfile_name=undionly.kpxe +pxe_bootfile_name=pxelinux.0 # Bootfile DHCP parameter for UEFI boot mode. (string value) #uefi_pxe_bootfile_name = bootx64.efi @@ -3556,10 +3558,11 @@ # Bootfile DHCP parameter per node architecture. For example: # aarch64:grubaa64.efi (dict value) #pxe_bootfile_name_by_arch = +pxe_bootfile_name_by_arch = ppc64le:config # Enable iPXE boot. (boolean value) #ipxe_enabled = false -ipxe_enabled=True +ipxe_enabled = false # On ironic-conductor node, the path to the main iPXE script # file. (string value) __EOF__
Ironic needs some patches:
# Don't power off node when already powered off
[stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/; wget --quiet -O - 'https://review.openstack.org/changes/508322/revisions/b5c5ce9c1e6989190cd87a5281934ef6b507fa10/patch?download' | base64 --decode | awk '/^diff --git a\/ironic\/tests\/unit\/conductor\/test_utils.py/{found=1;}
{if (!found) {print;}}' | sudo patch -p1)
# Don't send raw IPMI command
[stack@oscloud5 ~]$ sudo vi /usr/lib/python2.7/site-packages/ironic/drivers/modules/ipmitool.py
...
# timeout_disable = "0x00 0x08 0x03 0x08"
# send_raw(task, timeout_disable)
...
Prepare your system for some patches...
[stack@oscloud5 ~]$ (cd /usr/share/openstack-tripleo-common/; sudo git init .; sudo find . | xargs sudo git add; sudo git commit -m "base") [stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleoclient/; sudo git init .; sudo find . -not -iname \*.pyc -and -not -iname \*.pyo | xargs sudo git add; sudo git commit -m "base") [stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleo_common/; sudo git init .; sudo find . -not -iname \*.pyc -and -not -iname \*.pyo | xargs sudo git add; sudo git commit -m "base")
There is a bug for needing the userid for machines using ipmi that needs to be patched around.
[stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleo_common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/0001-fix-usernameless-IPMI.patch | sudo git am) Applying: fix usernameless IPMI
There is a bug for hardcoded ironic-python-agent and overcloud names:
[stack@oscloud5 ~]$ (cd /usr/share/openstack-tripleo-common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/0001-openstack-tripleo-common.patch | sudo git am) Applying: openstack-tripleo-common [stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleoclient/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/0001-tripleoclient.patch | sudo git am) Applying: tripleoclient [stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleo_common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/0001-tripleo_common.patch | sudo git am) Applying: tripleo_common
[stack@oscloud5 ~]$ (cd /usr/share/openstack-tripleo-common/; openstack workbook update workbooks/validations.yaml; openstack workbook update workbooks/baremetal.yaml)
[stack@oscloud5 ~]$ sudo mistral-db-manage --config-file /etc/mistral/mistral.conf --debug populate
[stack@oscloud5 ~]$ (for SERVICE in openstack-mistral-api.service openstack-mistral-engine.service openstack-mistral-executor.service; do sudo systemctl restart ${SERVICE}; done)
[stack@oscloud5 ~]$ for I in openstack-ironic-conductor.service openstack-ironic-inspector.service openstack-ironic-inspector-dnsmasq.service; do sudo systemctl restart ${I}; done
I then go through the process of uploading images for the overcloud:
[stack@oscloud5 ~]$ source stackrc
(undercloud) [stack@oscloud5 ~]$ AGENT_NAME=x86_64-ironic-python-agent time openstack overcloud image upload --os-image-name x86_64-overcloud-full.qcow2
(undercloud) [stack@oscloud5 ~]$ openstack image list
...
| 35275c4a-0d5e-435d-b2b3-ea79eac5ab43 | bm-deploy-kernel | active |
| d923db60-a3b1-41b0-b9f5-9d233223cfee | bm-deploy-ramdisk | active |
| 4506d0d9-aa0a-48b3-8084-b35f50e33196 | x86_64-overcloud-full | active |
| 3f0be3ae-2624-46fd-9230-f69271b50dd9 | x86_64-overcloud-full-initrd | active |
| ad3e4681-eaab-47bc-8922-5e964973e35d | x86_64-overcloud-full-vmlinuz | active |
...
(undercloud) [stack@oscloud5 ~]$ openstack image delete 35275c4a-0d5e-435d-b2b3-ea79eac5ab43
(undercloud) [stack@oscloud5 ~]$ openstack image delete d923db60-a3b1-41b0-b9f5-9d233223cfee
(undercloud) [stack@oscloud5 ~]$ AGENT_ARCH=x86_64 AGENT_NAME=${AGENT_ARCH}-ironic-python-agent time openstack overcloud image upload --os-image-name x86_64-overcloud-full.qcow2
(undercloud) [stack@oscloud5 ~]$ openstack image list
...
| faa0b855-be45-46c3-b25b-5c79c00c4c10 | x86_64-bm-deploy-kernel | active |
| 2e383335-ffec-4ddc-941d-45cec5b7a212 | x86_64-bm-deploy-ramdisk | active |
| 4506d0d9-aa0a-48b3-8084-b35f50e33196 | x86_64-overcloud-full | active |
| 3f0be3ae-2624-46fd-9230-f69271b50dd9 | x86_64-overcloud-full-initrd | active |
| ad3e4681-eaab-47bc-8922-5e964973e35d | x86_64-overcloud-full-vmlinuz | active |
...
(undercloud) [stack@oscloud5 ~]$ AGENT_ARCH=ppc64le AGENT_NAME=${AGENT_ARCH}-ironic-python-agent time openstack overcloud image upload --whole-disk --os-image-name ppc64le-overcloud-full.qcow2
(undercloud) [stack@oscloud5 ~]$ openstack image list
...
| ed5aa33a-c6fb-4575-9e71-154186a9d5d6 | ppc64le-bm-deploy-kernel | active |
| 2ad5c234-e7fc-4572-ae86-41151af31321 | ppc64le-bm-deploy-ramdisk | active |
| c6473e02-8fbb-4b07-beb8-3407118abe68 | ppc64le-overcloud-full | active |
...
Now import the baremetal nodes and assign them profiles.
(undercloud) [stack@oscloud5 ~]$ time openstack overcloud node import --provide instackenv.json 2>&1 | tee output.overcloud.node.import
(undercloud) [stack@oscloud5 ~]$ openstack overcloud profiles list
...
| 7f9470ab-b79c-4382-94a7-e4dc2ea7658c | | available | None | |
| fdf15be5-2542-42ed-8252-a154c58a12bc | | available | None | |
...
(undercloud) [stack@oscloud5 ~]$ (COMPUTE=""; CONTROL=""; while IFS=$' ' read -r -a PROFILES; do NODE=${PROFILES[0]}; ARCH=$(openstack baremetal node show --fields properties --format shell ${NODE} | sed -r -e "s,^.*cpu_arch': u'([^']*).*$,\1,"); echo "ARCH=${ARCH} NODE=${NODE}"; if [ -z "${COMPUTE}" -a "${ARCH}" == "ppc64le" ]; then COMPUTE=${NODE}; echo "Compute:"; ironic node-update ${COMPUTE} replace properties/capabilities=profile:compute,boot_option:local; continue; elif [ -z "${CONTROL}" -a "${ARCH}" == "x86_64" ]; then CONTROL=${PROFILES[0]}; echo "Control:"; ironic node-update ${CONTROL} replace properties/capabilities=profile:control,boot_option:local; continue; else echo "Error: ${NODE}"; openstack baremetal node show --fields properties --format shell ${NODE}; fi; done < <(openstack overcloud profiles list -f value))
(undercloud) [stack@oscloud5 ~]$ openstack overcloud profiles list
...
| 7f9470ab-b79c-4382-94a7-e4dc2ea7658c | | available | control | |
| fdf15be5-2542-42ed-8252-a154c58a12bc | | available | compute | |
...
Patch the openstack-tripleo-heat-templates locally with our configuration and then do the deploy.
(undercloud) [stack@oscloud5 ~]$ cp -r /usr/share/openstack-tripleo-heat-templates templates (undercloud) [stack@oscloud5 ~]$ (cd templates/; curl --silent -o - https://hamzy.fedorapeople.org/openstack-tripleo-heat-templates.patch | patch -p1) (undercloud) [stack@oscloud5 ~]$ cat << '__EOF__' > templates/environments/override-overcloud-image.yaml parameter_defaults: NovaImage: ppc64le-overcloud-full ComputeImage: ppc64le-overcloud-full controllerImage: x86_64-overcloud-full ControllerImage: x86_64-overcloud-full __EOF__ (undercloud) [stack@oscloud5 ~]$ time openstack overcloud deploy --templates /home/stack/templates -e /home/stack/templates/environments/network-environment.yaml -e /home/stack/templates/environments/network-isolation-custom.yaml -e /home/stack/templates/environments/override-overcloud-image.yaml --control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute 2>&1 | tee output.overcloud.deploy