This page is here to show how to setup OpenShift Origin on RHEL6 (or compatible) using the packages published from upstream for v1. These steps are written out to be done by hand. Yes, people can script and/or puppetize these steps. But these are written out so that people can see, and fine tune them.
Note: These steps will get written into scripts here: https://github.com/tdawson/oo-install-scripts
Goal: By the end of this, you should have two machines. A broker machine, and one node machine. You should be able to create applications, that will be put on the node machine. You should be able to check the status of those applications. You should be able to point your web browser to the URL of those applications.
These instructions were created most from the following two places.
- https://www.openshift.com/wiki/build-your-own (Gone)
- https://www.openshift.com/forums/openshift/fedora-18-openshift-origin-setup-steps-and-testing
Initial Setup of Broker and Node Machines
ON BOTH BROKER AND NODE
# Start with a RHEL6 minimal install
yum -y update
# avoid clock skew
yum -y install ntpdate ntp
ntpdate clock.redhat.com
chkconfig ntpd on
service ntpd start
# Add OpenShift Origin Yum Repository
cat <<EOF> /etc/yum.repos.d/openshift-origin.repo
[openshift-origin]
name=openshift-origin
baseurl=https://mirror.openshift.com/pub/openshift-origin/nightly/rhel-6/latest/x86_64/
gpgcheck=0
enabled=1
EOF
# Add OpenShift Origin Dependancy Yum Repository
cat <<EOF> /etc/yum.repos.d/openshift-origin-deps.repo
[openshift-origin-deps]
name=openshift-origin-deps
baseurl=https://mirror.openshift.com/pub/openshift-origin/nightly/rhel-6/dependencies/x86_64/
gpgcheck=0
enabled=1
EOF
# Setup your machine to use ruby193
yum install -y ruby193
cat <<EOF > /etc/profile.d/scl193.sh
# Setup PATH, LD_LIBRARY_PATH and MANPATH for ruby-1.9
ruby19_dir=$(dirname scl enable ruby193 "which ruby"
)
export PATH=$ruby19_dir:$PATH
ruby19_ld_libs=$(scl enable ruby193 "printenv LD_LIBRARY_PATH")
export LD_LIBRARY_PATH=$ruby19_ld_libs:$LD_LIBRARY_PATH
ruby19_manpath=$(scl enable ruby193 "printenv MANPATH")
export MANPATH=$ruby19_manpath:$MANPATH
EOF
cp -f /etc/profile.d/scl193.sh /etc/sysconfig/mcollective
chmod 0644 /etc/profile.d/scl193.sh /etc/sysconfig/mcollective
ON BROKER
export DOMAIN="example.com" export BROKERIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')" export BROKERNAME="broker.example.com" export NODEIP="--- IP Address from Node machine ---" export NODENAME="node.example.com" # Here is the IP Address from Broker machine nm-tool | grep Address | grep -v HW | awk '{print $2}'
ON NODE
export DOMAIN="example.com" export BROKERIP="--- IP Address from Broker machine ---" export BROKERNAME="broker.example.com" export NODEIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')" export NODENAME="node.example.com" # Here is the IP Address from Node machine nm-tool | grep Address | grep -v HW | awk '{print $2}'
Setup and Configure Broker
Broker: Bind DNS
yum -y install bind bind-utils KEYFILE=/var/named/${DOMAIN}.key
setup DNSSEC key pair
cd /var/named/ rm -f K${DOMAIN}* dnssec-keygen -a HMAC-MD5 -b 512 -n USER -r /dev/urandom ${DOMAIN} KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)" cd - rndc-confgen -a -r /dev/urandom echo $KEY
setup permissions for the DNSSEC key pair
restorecon -v /etc/rndc.* /etc/named.* chown -v root:named /etc/rndc.key chmod -v 640 /etc/rndc.key
setup forwarders
echo "forwarders { 8.8.8.8; 8.8.4.4; } ;" >> /var/named/forwarders.conf restorecon -v /var/named/forwarders.conf chmod -v 640 /var/named/forwarders.conf
setup initial DNS database
rm -rvf /var/named/dynamic mkdir -vp /var/named/dynamic cat <<EOF > /var/named/dynamic/${DOMAIN}.db \$ORIGIN . \$TTL 1 ; 1 seconds (for testing only) ${DOMAIN} IN SOA ns1.${DOMAIN}. hostmaster.${DOMAIN}. ( 2011112904 ; serial 60 ; refresh (1 minute) 15 ; retry (15 seconds) 1800 ; expire (30 minutes) 10 ; minimum (10 seconds) ) NS ns1.${DOMAIN}. MX 10 mail.${DOMAIN}. \$ORIGIN ${DOMAIN}. ns1 A 127.0.0.1 EOF
Install the DNSSEC key
cat <<EOF > ${KEYFILE} key ${DOMAIN} { algorithm HMAC-MD5; secret "${KEY}"; }; EOF
Check the key and database
cat /var/named/dynamic/${DOMAIN}.db cat /var/named/${DOMAIN}.key
Set permissions for key and database
chown -Rv named:named /var/named restorecon -rv /var/named
Create the named configuration file
mv /etc/named.conf /etc/named.conf.openshift cat <<EOF > /etc/named.conf // named.conf // // Provided by Red Hat bind package to configure the ISC BIND named(8) DNS // server as a caching only nameserver (as a localhost DNS resolver only). // // See /usr/share/doc/bind*/sample/ for example named configuration files. // options { listen-on port 53 { any; }; directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; allow-query { any; }; recursion yes; /* Path to ISC DLV key */ bindkeys-file "/etc/named.iscdlv.key"; // set forwarding to the next nearest server (from DHCP response) forward only; include "forwarders.conf"; }; logging { channel default_debug { file "data/named.run"; severity dynamic; }; }; // use the default rndc key include "/etc/rndc.key"; controls { inet 127.0.0.1 port 953 allow { 127.0.0.1; } keys { "rndc-key"; }; }; include "/etc/named.rfc1912.zones"; include "${DOMAIN}.key"; zone "${DOMAIN}" IN { type master; file "dynamic/${DOMAIN}.db"; allow-update { key ${DOMAIN} ; } ; }; EOF
Check the named config file
cat /etc/named.conf
setup permissions of named config file
chown -v root:named /etc/named.conf restorecon /etc/named.conf
Edit /etc/resolve.conf file. Add the following at the top
nameserver 127.0.0.1
/etc/dhcp/dhclient-eth0.conf file, or add it if it doesn’t exist, and add the following
prepend domain-name-servers 127.0.0.1;
Setup firewall
lokkit --service=dns
Setup and start service
chkconfig named on service named start
add entries using nsupdate
nsupdate -k ${KEYFILE} > server 127.0.0.1 > update delete broker.example.com A > update add **your broker full name ** 180 A **your broker ip address** (example: update add broker.example.com 180 A 192.168.122.220 ) > send > quit
Test DNS server This is best done before hostname has been set.
ping broker.example.com dig @127.0.0.1 broker.example.com
Broker: DHCP client and hostname
Setup dhcp client
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf echo "supersede host-name \"broker\";" >> /etc/dhcp/dhclient-eth0.conf echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf # Then edit /etc/dhcp/dhclient-eth0.conf and make sure the following lines are set PEERDNS="no" DNS1=**your broker ip address**
Setup hostname
# Edit /etc/sysconfig/network and change the HOSTNAME line to HOSTNAME=broker.example.com
Broker: MongoDB
Install Software
yum -y install mongodb-server
Edit /etc/mongodb.conf and add the following line
smallfiles = true
Setup mongodb service
chkconfig mongod on
Broker: MongoDB: Add broker user
Ensure auth is turned off. Edit /etc/mongodb.conf and ensure auth is commented out
#auth=true
Start the mongodb service
service mongod start
Add OpenShift broker user
/usr/bin/mongo localhost/openshift_broker_dev --eval 'db.addUser("openshift", "<choose a password>")' /usr/bin/mongo localhost/admin --eval 'db.addUser("openshift", "<password chosen above>")'
Stop the mongodb service
service mongod stop
Turn auth back on. Edit /etc/mongodb.conf and ensure auth is not commented out
auth=true
Broker: MongoDB: Firewall and testing
Edit /etc/mongodb.conf and update the bind_ip line to have the broker in it
bind_ip=127.0.0.1,**Your broker ip**
Enable MongoDB access on the firewall.
lokkit --port=27017:tcp
Testing
# Make sure it is off service mongod status # Make sure it starts service mongod start # Make sure our user was added mongo > db.auth('openshift',"<password chosen above>") > show dbs > exit
Broker: Messaging (using QPID)
Activemq on F19 isn't ready for OpenShift production. When it is, we'll use that For now we'll use QPID with mcollective.
Install Software
yum install mcollective-qpid-plugin qpid-cpp-server
Setup Firewall
firewall-cmd --add-port=5672/tcp firewall-cmd --permanent --add-port=5672/tcp firewall-cmd --list-all
Setup and start service
/usr/bin/systemctl enable qpidd.service /usr/bin/systemctl start qpidd.service /usr/bin/systemctl status qpidd.service
Broker: MCollective client ( using QPID)
Install Software
yum -y install mcollective-client
Move original config file out of the way
mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig
Create new client config file. This config file is for using QPID as a messaging platform.
cat <<EOF > /etc/mcollective/client.cfg topicprefix = /topic/ main_collective = mcollective collectives = mcollective libdir = /usr/libexec/mcollective loglevel = debug logfile = /var/log/mcollective-client.log # Plugins securityprovider = psk plugin.psk = unset connector = qpid plugin.qpid.host=${BROKERHOSTNAME} plugin.qpid.secure=false plugin.qpid.timeout=5 # Facts factsource = yaml plugin.yaml = /etc/mcollective/facts.yaml EOF
Broker: broker application
Install software
yum -y install openshift-origin-broker openshift-origin-broker-util rubygem-openshift-origin-auth-remote-user rubygem-openshift-origin-msg-broker-mcollective rubygem-openshift-origin-dns-bind
Modify the broker proxy server name
sed -i -e "s/ServerName .*$/ServerName broker.example.com/" /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf cat /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf
Setup and start service
/usr/bin/systemctl enable httpd.service /usr/bin/systemctl enable ntpd.service /usr/bin/systemctl enable sshd.service
Setup Firewall
firewall-cmd --add-service=ssh firewall-cmd --add-service=http firewall-cmd --add-service=https firewall-cmd --permanent --add-service=ssh firewall-cmd --permanent --add-service=http firewall-cmd --permanent --add-service=https firewall-cmd --list-all
Generate access key
openssl genrsa -out /etc/openshift/server_priv.pem 2048 openssl rsa -in /etc/openshift/server_priv.pem -pubout > /etc/openshift/server_pub.pem ssh-keygen -t rsa -b 2048 -f ~/.ssh/rsync_id_rsa cp -v ~/.ssh/rsync_id_rsa* /etc/openshift/
Setup selinux boolean variables and set file contexts
setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_run_stickshift=on named_write_master_zones=on fixfiles -R rubygem-passenger restore fixfiles -R mod_passenger restore restorecon -rv /var/run restorecon -rv /usr/share/gems/gems/passenger-*
Tweak broker config, if needed
vi /etc/openshift/broker.conf # Might not have to do anything but make sure you have the following lines CLOUD_DOMAIN="example.com" VALID_GEAR_SIZES="small,medium"
Broker: broker plugins and MongoDB user accounts
Create config files from examples
cp /usr/share/gems/gems/openshift-origin-auth-remote-user-*/conf/openshift-origin- auth-remote-user.conf.example /etc/openshift/plugins.d/openshift-origin-auth-remote-user.conf cp /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf.example /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf
Config the DNS plugin
cd /var/named/ KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)" cat $KEYFILE echo $KEY cat <<EOF > /etc/openshift/plugins.d/openshift-origin-dns-bind.conf BIND_SERVER="127.0.0.1" BIND_PORT=53 BIND_KEYNAME="${DOMAIN}" BIND_KEYVALUE="${KEY}" BIND_ZONE="${DOMAIN}" EOF
Configure authentication plugin and add a user
cp -v /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user-basic.conf.sample /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user.conf htpasswd -c -b -s /etc/openshift/htpasswd demo demo # Don't forget your password. <demo password> cat /etc/openshift/htpasswd
Add MongoDB account
grep MONGO /etc/openshift/broker.conf mongo openshift_broker_dev --eval 'db.addUser("openshift", "mooo")' # If you are going to change the username and/or password, change broker.conf
Bundle broker gems
yum -y install rubygem-psych cd /var/www/openshift/broker gem install mongoid bundle --local
Setup and start services
/usr/bin/systemctl enable openshift-broker.service /usr/bin/systemctl start httpd.service /usr/bin/systemctl start openshift-broker.service /usr/bin/systemctl status openshift-broker.service
Test basic broker service
curl -k -u demo:demo https://localhost/broker/rest/api
Setup and Configure Node
Node: Initial setup/configure
ON BROKER
KEYFILE=/var/named/${DOMAIN}.key
Register the node in DNS
oo-register-dns -h node -d ${DOMAIN} -n ${NODEIP} -k ${KEYFILE}
Copy the broker public key to node
scp /etc/openshift/rsync_id_rsa.pub root@${NODENAME}:/root/.ssh/
ON NODE Put the brokers public key in root authorized keys
cat /root/.ssh/rsync_id_rsa.pub >> /root/.ssh/authorized_keys rm -f /root/.ssh/rsync_id_rsa.pub
ON BROKER Test to make sure we can login using our key
ssh -i /root/.ssh/rsync_id_rsa root@${NODENAME} exit
Node: DHCP client and hostname
Configure the dhcp settings
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf echo "supersede host-name \"node\";" >> /etc/dhcp/dhclient-eth0.conf echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf
Set the hostname
echo "node.example.com" > /etc/hostname
Node: MCollective
ON NODE Install Software
yum -y install openshift-origin-msg-node-mcollective
Move original configuration out of the way
mv /etc/mcollective/server.cfg /etc/mcollective/server.cfg.orig
Create new configuration
cat <<EOF > /etc/mcollective/server.cfg topicprefix = /topic/ main_collective = mcollective collectives = mcollective libdir = /usr/libexec/mcollective logfile = /var/log/mcollective.log loglevel = debug daemonize = 1 direct_addressing = n # Plugins securityprovider = psk plugin.psk = unset connector = qpid plugin.qpid.host=${BROKERNAME} plugin.qpid.secure=false plugin.qpid.timeout=5 # Facts factsource = yaml plugin.yaml = /etc/mcollective/facts.yaml EOF
Setup and start services
/bin/systemctl enable mcollective.service /bin/systemctl start mcollective.service
ON BROKER
mco ping # node should show up on mco ping
Node: node application
Install software
yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util yum -y install openshift-origin-cartridge-cron-1.4 openshift-origin-cartridge-diy-0.1
Setup firewall
firewall-cmd --add-service=ssh firewall-cmd --add-service=http firewall-cmd --add-service=https firewall-cmd --permanent --add-service=ssh firewall-cmd --permanent --add-service=http firewall-cmd --permanent --add-service=https firewall-cmd --list-all
Node: PAM namespace module, cgroups, and user quotas
PAM
sed -i -e 's|pam_selinux|pam_openshift|g' /etc/pam.d/sshd for f in "runuser" "runuser-l" "sshd" "su" "system-auth-ac" do t="/etc/pam.d/$f" if ! grep -q "pam_namespace.so" "$t" then echo -e "session\t\trequired\tpam_namespace.so no_unmount_on_close" >> "$t" fi done
CGROUPS
Cgroups Config - Need to still fixup the cgroup configurations
#echo "mount {" >> /etc/cgconfig.conf #echo " cpu = /cgroup/all;" >> /etc/cgconfig.conf #echo " cpuacct = /cgroup/all;" >> /etc/cgconfig.conf #echo " memory = /cgroup/all;" >> /etc/cgconfig.conf #echo " freezer = /cgroup/all;" >> /etc/cgconfig.conf #echo " net_cls = /cgroup/all;" >> /etc/cgconfig.conf #echo "}" >> /etc/cgconfig.conf #restorecon -v /etc/cgconfig.conf #mkdir /cgroup #restorecon -RFvv /cgroup
Cgroups enable and startup services
/bin/systemctl enable cgconfig.service /bin/systemctl enable cgred.service /usr/sbin/chkconfig openshift-cgroups on /bin/systemctl restart cgconfig.service /bin/systemctl restart cgred.service /usr/sbin/service openshift-cgroups restart
DISK QUOTA
# Edit fstab and add usrquota to whichever filesystem # has /var/lib/openshift on it UUID=b9e21eae-4b8c-4936-9f5d-d10631ff535e / ext4 defaults,usrquota 1 1 # reboot or remount mount -o remount / quotacheck -cmug /
Node: SELinux and System Control
Setup SELINUX Booleans
setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_read_user_content=on httpd_enable_homedirs=on httpd_run_stickshift=on allow_polyinstantiation=on
Update selinux file setting
restorecon -rv /var/run restorecon -rv /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid restorecon -rv /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift
SYSTEM CONTROL SETTINGS
echo "# Added for OpenShift" >> /etc/sysctl.d/openshift.conf echo "kernel.sem = 250 32000 32 4096" >> /etc/sysctl.d/openshift.conf echo "net.ipv4.ip_local_port_range = 15000 35530" >> /etc/sysctl.d/openshift.conf echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/openshift.conf sysctl -p /etc/sysctl.d/openshift.conf
Node: SSH, Port Proxy, and Node application
SSH
vi /etc/ssh/sshd_config > AcceptEnv GIT_SSH perl -p -i -e "s/^#MaxSessions .*$/MaxSessions 40/" /etc/ssh/sshd_config perl -p -i -e "s/^#MaxStartups .*$/MaxStartups 40/" /etc/ssh/sshd_config /bin/systemctl restart sshd.service
PORT PROXY
firewall-cmd --add-port=35531-65535/tcp firewall-cmd --permanent --add-port=35531-65535/tcp firewall-cmd --list-all /bin/systemctl enable openshift-port-proxy.service /bin/systemctl restart openshift-port-proxy.service
NODE SETUP
/bin/systemctl enable httpd.service /bin/systemctl enable openshift-gears.service vi /etc/openshift/node.conf > PUBLIC_HOSTNAME="node.example.com" > PUBLIC_IP="192.168.122.161" (Node IP Address) > BROKER_HOST="192.168.122.220" (Broker IP Address) > CLOUD_DOMAIN="example.com" /etc/cron.minutely/openshift-facts
Node: Reboot
We need to reboot to load all the node stuff correctly
reboot
Testing
Test on Broker (after node is back up)
Check Messaging
mco ping
Should look like
node.example.com time=239.51 ms ---- ping statistics ---- 1 replies max: 239.51 min: 239.51 avg: 239.51
Check Broker
curl -k -u demo:demo https://localhost/broker/rest/api
Should look like
{"data":{"API":{"href":"https://localhost/broker/rest/api","method":"GET","optional_params":[],"rel":"API entry point","required_params":[]},"GET_ENVIRONMENT":{"href":"https://localhost/broker/rest/environment","method":"GET","optional_params":[],"rel":"Get environment information","required_params":[]},"GET_USER" ... :id","type":"string","valid_options":[]}]}},"messages":[],"status":"ok","supported_api_versions":[1.0,1.1,1.2,1.3],"type":"links","version":"1.3"}
Check and Setup User
yum -y install rubygem-rhc LIBRA_SERVER=broker.example.com rhc setup
Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot)
OpenShift Client Tools (RHC) Setup Wizard This wizard will help you upload your SSH keys, set your application namespace, and check that other programs like Git are properly installed. The server's certificate is self-signed, which means that a secure connection can't be established to 'broker.example.com'. You may bypass this check, but any data you send to the server could be intercepted by others. Connect without checking the certificate? (yes|no): yes Login to broker.example.com: demo Password: **** OpenShift can create and store a token on disk which allows to you to access the server without using your password. The key is stored in your home directory and should be kept secret. You can delete the key at any time by running 'rhc logout'. Generate a token now? (yes|no) no Saving configuration to /root/.openshift/express.conf ... done No SSH keys were found. We will generate a pair of keys for you. Created: /root/.ssh/id_rsa.pub Your public SSH key must be uploaded to the OpenShift server to access code. Upload now? (yes|no) yes Since you do not have any keys associated with your OpenShift account, your new key will be uploaded as the 'default' key. Uploading key 'default' ... done Checking for git ... found git version 1.8.2.1 Checking common problems .. done Checking your namespace ... none Your namespace is unique to your account and is the suffix of the public URLs we assign to your applications. You may configure your namespace here or leave it blank and use 'rhc create-domain' to create a namespace later. You will not be able to create applications without first creating a namespace. Please enter a namespace (letters and numbers only) |<none>|: demoland
Create an app
rhc domain show -p demo rhc app create test1 diy-0.1 -p demo
Test on Local Machine (after node is back up)
Setup your machine to use broker as a name server (Note: This might mess up normal network operations.)
vi /etc/resolve.conf # At the first line put "nameserver *broker ip address*" nameserver 192.168.122.220
Check and Setup User
yum -y install rubygem-rhc LIBRA_SERVER=broker.example.com rhc setup
Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot)
OpenShift Client Tools (RHC) Setup Wizard This wizard will help you upload your SSH keys, set your application namespace, and check that other programs like Git are properly installed. The server's certificate is self-signed, which means that a secure connection can't be established to 'broker.example.com'. You may bypass this check, but any data you send to the server could be intercepted by others. Connect without checking the certificate? (yes|no): yes Login to broker.example.com: demo Password: **** OpenShift can create and store a token on disk which allows to you to access the server without using your password. The key is stored in your home directory and should be kept secret. You can delete the key at any time by running 'rhc logout'. Generate a token now? (yes|no) no Saving configuration to /root/.openshift/express.conf ... done No SSH keys were found. We will generate a pair of keys for you. Created: /root/.ssh/id_rsa.pub Your public SSH key must be uploaded to the OpenShift server to access code. Upload now? (yes|no) yes Since you do not have any keys associated with your OpenShift account, your new key will be uploaded as the 'default' key. Uploading key 'default' ... done Checking for git ... found git version 1.8.2.1 Checking common problems .. done Checking your namespace ... none Your namespace is unique to your account and is the suffix of the public URLs we assign to your applications. You may configure your namespace here or leave it blank and use 'rhc create-domain' to create a namespace later. You will not be able to create applications without first creating a namespace. Please enter a namespace (letters and numbers only) |<none>|: demoland
Create an app
rhc domain show -p demo rhc app create test2 diy-0.1 -p demo
Check App
You should be able to go to the following URL in your web browser.