User:Hamzy/ppc64le Overcloud builder instance

Creating an overcloud image builder instance

Start a QEMU instance

Setup the anaconda kickstart file.

[hamzy@pkvmci853 ~]$ cat << '__EOF__' > libvirt/anaconda-ks-overcloud.cfg
#version=DEVEL
# System authorization information
auth --enableshadow --passalgo=sha512
# Use network installation
url --url="http://mirror.centos.org/altarch/7/os/ppc64le/"
# Use text mode install
text
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=vda
# Keyboard layouts
keyboard --vckeymap=us --xlayouts=''
# System language
lang en_US.UTF-8

# Network information
network  --bootproto=dhcp --device=eth0 --ipv6=auto --activate
network  --hostname=overcloud.virbr0

# Root password
rootpw --iscrypted $6$>?@N4]-Ujl.o~$lUoOP3vNc3Q.D2m4m0Z.wQBC.nMTEvclR.T./xx7slY62l0LOEjxhbiIrqBAl1ADV8tszIaPd86m74A/LyuT20
# System services
services --enabled="chronyd"
# Do not configure the X Window System
skipx
# System timezone
timezone America/Chicago --isUtc
user --groups=wheel --name=hamzy --password=$6$!R}=iNm*FLyE)$0W7wfzyMzHT01HUIV0N4uH0Ixk.6n2AnsPMGr61zxraR0HZq.uS5uvkJDuxzvzyNo07712HNzQ2MAMv5EEnbQ1
EdgLFvZsSMbx1Ee8M1 --iscrypted
# System bootloader configuration
bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda
autopart --type=lvm
# Partition clearing information
clearpart --all --initlabel --drives=vda

%packages
@core
chrony
kexec-tools
screen
tmux
wget
git

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%anaconda
pwpolicy root --minlen=6 --minquality=50 --notstrict --nochanges --notempty
pwpolicy user --minlen=6 --minquality=50 --notstrict --nochanges --notempty
pwpolicy luks --minlen=6 --minquality=50 --notstrict --nochanges --notempty
%end

%post --interpreter=/bin/bash --log=/root/anaconda-post.log
# Post configure tasks
set -x

hostnamectl set-hostname Overcloud.virbr0

SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDB3tqPZ0+AAAAIw6G7XkYyRZecacl96Qocv9+prfN4Hs3bt68yLY7nlfhwYHDc+WFABvwq/jwylsjamznb4XV
s7ZWaiOMidYkAAAA6x+E/Fy4u8WGc/KVdYvhcDg2XvH96ja3nIAbVW9fGZ2rdrRkqqSLNLXC7IWfhsQbG/wF5+ddunBAAAApIuWS+xllgPQvd5O3iKRIqWPH6ZvR6dPx7gc8
a+RkjgqivgAAAARMfEtyQOwKxAHu9XX7w0ICTFOwuswyrRwOlW9piUnieHj6Yc/tpzpwmAAAAFTidQ/vmNdDJC/OKBifRYOeX54+RN7J2bPx66O1EcLdHLlh7c9qj hamzy@
hamzy-tp-w540"

for FILE in /root/.ssh/authorized_keys /home/hamzy/.ssh/authorized_keys
do
        mkdir -p $(dirname ${FILE})
        echo ${SSH_KEY} >> ${FILE}
        chmod 600 ${FILE}
        chmod 700 $(dirname ${FILE})
        if [[ ${FILE} == */root/* ]]
        then
                OWNER="root:root"
        elif [[ ${FILE} == */hamzy/* ]]
        then
                OWNER="hamzy:hamzy"
        fi
        chown ${OWNER} $(dirname ${FILE})
        chown ${OWNER} ${FILE}
done

echo "hamzy ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/hamzy

# Clean for epel-release install
yum clean all
yum update -y
# Install epel, then install pip separately
yum install -y epel-release
yum install -y python-pip
%end
__EOF__

Use virt-install to install CentOS7 inside a VM.

[hamzy@pkvmci853 ~]$ sudo virsh destroy Overcloud; sudo virsh undefine Overcloud
Domain Overcloud destroyed
Domain Overcloud has been undefined
[hamzy@pkvmci853 ~]$ sudo virt-install --virt-type kvm --name Overcloud --memory 32768 --graphics none --disk ~/libvirt/Overcloud.qcow2,format=qcow2 --network=bridge:virbr0 --os-type=linux --os-variant=centos7.0 --location=http://mirror.centos.org/altarch/7/os/ppc64le/ --extra-args="inst.text console=tty0 console=ttyS0,115200 ks=file:/anaconda-ks-overcloud.cfg" --initrd-inject ~/libvirt/anaconda-ks-overcloud.cfg
...
        Installation complete.  Press return to quit
...
Domain creation completed.
Restarting guest.
...

Install dig and copy over ssh keys.

[hamzy@overcloud ~]$ sudo yum install -y bind-utils
[hamzy@overcloud ~]$ (IP="192.168.122.1"; ssh-keygen -f ~/.ssh/known_hosts -R ${IP}; ssh-keyscan ${IP} >> ~/.ssh/known_hosts; scp hamzy@${IP}:~/.ssh/id_rsa* ~/.ssh/)

Optionally, you can update the local hostname name for the VM. Libvirt configures dnsmasq to read the file /var/lib/libvirt/dnsmasq/default.addnhosts for hostnames.

[hamzy@pkvmci853 ~]$ (FILE=/home/hamzy/libvirt/mac-dns.txt; > ${FILE}; for VMNAME in DLRN Overcloud; do MAC1=$(sudo virsh dumpxml ${VMNAME} | awk -F "'" '/mac address/ { print $2; exit }'); MAC2=${MAC1^^}; MAC3=${MAC2//:}; printf "${MAC3}\t${VMNAME}.virbr0\n" >> ${FILE}; done; cat ${FILE})
5254008E0DD2    DLRN.virbr0
525400CBFE22    Overcloud.virbr0
[hamzy@pkvmci853 ~]$ (FILE=/var/lib/libvirt/dnsmasq/default.addnhosts; sudo rm -f ${FILE}; sudo arp-scan --interface=virbr0 --macfile=/home/hamzy/libvirt/mac-dns.txt --localnet | head -n-3 | tail -n+3 | cut -f1,3- | sudo tee -a ${FILE})
192.168.122.104 DLRN.virbr0
192.168.122.147 Overcloud.virbr0

Install diskimage-builder

Diskimage-builder needs a hack patch to disable using python-greenlet from the CentOS repository.

[hamzy@overcloud ~]$ git clone git://git.openstack.org/openstack/diskimage-builder
[hamzy@overcloud ~]$ (cd diskimage-builder; cat << '__EOF__' | git am -)
From b3bfe84ca40e70bce5609e274050e7e23bef00b9 Mon Sep 17 00:00:00 2001
From: Mark Hamzy <hamzy@us.ibm.com>
Date: Tue, 1 Aug 2017 12:44:12 -0500
Subject: [PATCH] fix python-greenlet conflict

---
 diskimage_builder/elements/yum/extra-data.d/99-yum-repo-conf | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/diskimage_builder/elements/yum/extra-data.d/99-yum-repo-conf b/diskimage_builder/elements/yum/extra-data.d/99-yum-repo-conf
index b5b6e98..9fb27d0 100755
--- a/diskimage_builder/elements/yum/extra-data.d/99-yum-repo-conf
+++ b/diskimage_builder/elements/yum/extra-data.d/99-yum-repo-conf
@@ -25,3 +25,5 @@ for file in $DIB_YUM_REPO_CONF; do
     sudo cp -L -f $file $TMP_MOUNT_PATH/etc/yum.repos.d

 done
+
+echo 'exclude=python-greenlet*' | sudo tee -a $TMP_MOUNT_PATH/etc/yum.repos.d/CentOS-Base.repo
--
1.8.3.1

__EOF__
[hamzy@overcloud ~]$ (cd diskimage-builder/; sudo pip install --upgrade --force-reinstall --requirement requirements.txt)
[hamzy@overcloud ~]$ (cd diskimage-builder/; sudo python setup.py install --force)

Configure the repositories that the overcloud building process uses

[hamzy@overcloud ~]$ cat << __EOF__ | sudo tee -a /etc/yum.repos.d/DLRN.repo
[DLRN]
name=DLRN
baseurl=https://trunk.rdoproject.org/centos7/current-passed-ci/
enabled=1
gpgcheck=0
cost=1
__EOF__
[hamzy@overcloud ~]$ cat << __EOF__ | sudo tee -a /etc/yum.repos.d/cloud7-openstack-common-candidate.repo
[cloud7_openstack_common_candidate]
name=cloud7_openstack_common_candidate
baseurl=https://cbs.centos.org/repos/cloud7-openstack-common-candidate/ppc64le/os/
enabled=1
gpgcheck=0
cost=2
__EOF__
[hamzy@overcloud ~]$ cat << __EOF__ | sudo tee -a /etc/yum.repos.d/cloud7-openstack-common-testing.repo
[cloud7_openstack_common_testing]
name=cloud7_openstack_common_testing
baseurl=https://cbs.centos.org/repos/cloud7-openstack-common-testing/ppc64le/os/
enabled=1
gpgcheck=0
cost=3
__EOF__
[hamzy@overcloud ~]$ cat << __EOF__ | sudo tee -a /etc/yum.repos.d/cloud7-openstack-common-release.repo
[cloud7_openstack_common_release]
name=cloud7_openstack_common_release
baseurl=https://cbs.centos.org/repos/cloud7-openstack-common-release/ppc64le/os/
enabled=1
gpgcheck=0
cost=4
__EOF__
[hamzy@overcloud ~]$ cat << __EOF__ | sudo tee -a /etc/yum.repos.d/cloud7-openstack-pike-testing.repo
[cloud7_openstack_pike_testing]
name=cloud7_openstack_pike_testing
baseurl=http://cbs.centos.org/repos/cloud7-openstack-pike-testing/ppc64le/os/
enabled=1
gpgcheck=0
cost=5
__EOF__
[hamzy@overcloud ~]$ sudo yum clean all; sudo yum update -y

erlang-sd_notify is broken

Erlang is broken for ppc64le. You will see the following during the build:

...
Error: Package: erlang-sd_notify-0.1-9.el7.ppc64le (cloud7_openstack_common_candidate)
           Requires: erlang(erl_nif_version) = 2.11
           Installing: erlang-erts-18.3.4.5-4.el7.ppc64le (cloud7_openstack_common_candidate)
               erlang(erl_nif_version) = 2.10
...

So, to fix it, you need to add a missing repository on a web server:

[hamzy@pkvmci853 ~]$ (DIR=/var/www/html/missing-overcloud; cd ${DIR}; sudo cbs download-build --arch=ppc64le --arch=noarch erlang-19.3.6.1-1.el7 --quiet; sudo createrepo .; sudo chcon -R system_u:object_r:httpd_sys_content_t:s0 ${DIR})

And add another repository to the overcloud builder:

[hamzy@overcloud ~]$ cat << __EOF__ | sudo tee -a /etc/yum.repos.d/missing-overcloud.repo
[missing_overcloud]
name=missing_overcloud
baseurl=http://192.168.122.1/missing-overcloud/
enabled=1
gpgcheck=0
cost=6
__EOF__

Install python-tripleoclient

Install python-tripleoclient and patch the overcloud-images.yaml file to use ppc64le instead of amd64.

[hamzy@overcloud ~]$ sudo yum install -y python-tripleoclient

Prepare for local patches

[hamzy@overcloud ~]$ (cd /usr/share/openstack-tripleo-common/; sudo git init .; sudo find . | xargs sudo git add; sudo git commit -m "base")
[hamzy@overcloud ~]$ (cd /usr/lib/python2.7/site-packages/tripleo_common/; sudo git init .; sudo find . -not -iname \*.pyc -and -not -iname \*.pyo | xargs sudo git add; sudo git commit -m "base")

Install patches to allow building of ppc64le images

If https://review.openstack.org/#/c/489476/ has not been accepted, then pull it in.

[hamzy@overcloud ~]$ (cd /usr/share/openstack-tripleo-common; wget --quiet -O - 'https://review.openstack.org/changes/489476/revisions/fba2902dc6aacffd04697dc72dfd5e5b8a427eec/patch?download' | base64 --decode | awk '/^diff --git a\/releasenotes\/notes\/default-arch-selection-d5fd2fcdba725dd4/{found=1;}
{if (!found) {print;}}' | sudo patch -p1)
patching file image-yaml/overcloud-hardened-images-centos7.yaml
patching file image-yaml/overcloud-hardened-images-rhel7.yaml
patching file image-yaml/overcloud-hardened-images.yaml
patching file image-yaml/overcloud-images-centos7.yaml
patching file image-yaml/overcloud-images-rhel7.yaml
patching file image-yaml/overcloud-images.yaml
patching file image-yaml/overcloud-odl-rhel7.yaml
[hamzy@overcloud ~]$ (cd /usr/lib/python2.7/site-packages/tripleo_common/; wget --quiet -O - 'https://review.openstack.org/changes/489476/revisions/fba2902dc6aacffd04697dc72dfd5e5b8a427eec/patch?download' | base64 --decode | awk '/^diff --git a\/tripleo_common\/arch.py/{i++}i' | sudo patch -p2)
patching file arch.py
patching file image/build.py
patching file tests/test_arch.py

Install patch to build a full disk image

[hamzy@overcloud ~]$ (cd /usr/share/openstack-tripleo-common/; cat << '__EOF__' | sudo git am)
From 1e0e7d61247a6ba64786e5be59ec4c7e7a359552 Mon Sep 17 00:00:00 2001
From: root <root@overcloud.virbr0>
Date: Sun, 5 Nov 2017 08:49:34 -0600
Subject: [PATCH] openstack-tripleo-common overcloud-images

---
 image-yaml/overcloud-images.yaml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/image-yaml/overcloud-images.yaml b/image-yaml/overcloud-images.yaml
index c3a0312..52783f9 100644
--- a/image-yaml/overcloud-images.yaml
+++ b/image-yaml/overcloud-images.yaml
@@ -5,6 +5,8 @@ disk_images:
     elements:
       - baremetal
       - dhcp-all-interfaces
+      - vm
+      - devuser
       - overcloud-agent
       - overcloud-full
       - overcloud-controller
-- 
1.8.3.1
__EOF__
Applying: openstack-tripleo-common overcloud-images

Modifying overcloud-full.qcow2 to have a prep partition

[hamzy@overcloud ~]$ cat << '__EOF__' > part.yaml
  - local_loop:
      name: image0

  - partitioning:
      base: image0
      label: mbr
      partitions:
        - name:  boot
          flags: [ boot, primary ]
          size:  4MiB
          type:  "0x41"
        - name:  root
          flags: [ primary ]
          size:  100%
          mkfs:
            label: markfs
            type:  ext4
            mount:
              mount_point: /
              fstab:
                options: "defaults"
                fsck-passno: 1
__EOF__
[hamzy@overcloud ~]$ export DIB_BLOCK_DEVICE_CONFIG=$(cat part.yaml)
[hamzy@overcloud ~]$ sudo /bin/rm -rf overcloud-full*

Run the overcloud building command

[hamzy@overcloud ~]$ (export DIB_DEV_USER_PWDLESS_SUDO=yes; export DIB_DEV_USER_AUTHORIZED_KEYS=/home/ubuntu/hamzy_id_rsa.pub; export DIB_DEV_USER_USERNAME=hamzy; export DIB_DEV_USER_PASSWORD=password; export DIB_YUM_REPO_CONF="/etc/yum.repos.d/DLRN.repo /etc/yum.repos.d/cloud7-openstack-common-candidate.repo /etc/yum.repos.d/cloud7-openstack-common-testing.repo /etc/yum.repos.d/cloud7-openstack-common-release.repo /etc/yum.repos.d/cloud7-openstack-pike-candidate.repo /etc/yum.repos.d/cloud7-openstack-pike-testing.repo /etc/yum.repos.d/missing-overcloud.repo"; export NODE_ARCH=ppc64le; export DIB_CLOUD_IMAGES="http://ftp.unicamp.br/pub/ppc64el/centos/7/images/"; export BASE_IMAGE_FILE="CentOS-7-ppc64le-GenericCloud-1611.qcow2"; time openstack overcloud image build) 2>&1 | tee output.overcloud-image-build)

(Optionally) modify IPA initramfs to have a password for root

[hamzy@overcloud ~]$ cp ironic-python-agent.initramfs ironic-python-agent.initramfs.orig
[hamzy@overcloud ~]$ mv ironic-python-agent.initramfs ironic-python-agent.initramfs.gz
[hamzy@overcloud ~]$ gzip --decompress --to-stdout ironic-python-agent.initramfs.orig > ironic-python-agent.initramfs.cpio
[hamzy@overcloud ~]$ mkdir ironic-python-agent; cd ironic-python-agent; sudo cpio -id --no-preserve-owner --preserve-modification-time < ~/ironic-python-agent.initramfs.cpio)
[hamzy@overcloud ~]$ (PASS=$(openssl passwd -1 -salt xyz password); sudo sed -i -e 's,^root:!!,root:'${PASS}',' ironic-python-agent/etc/shadow)
[hamzy@overcloud ~]$ (cd ironic-python-agent/; sudo find . | sudo cpio -H newc -o > ~/ironic-python-agent.initramfs.cpio)
[hamzy@overcloud ~]$ gzip < ~/ironic-python-agent.initramfs.cpio > ~/ironic-python-agent.initramfs.gz
[hamzy@overcloud ~]$ cp ironic-python-agent.initramfs.gz ironic-python-agent.initramfs