Description
Replication testing.
Setup
- You will need at least 2 machines to do replication testing but if you have more that is good too. Start on one server and perform an IPA Installation (see QA:Testcase_freeipa_trust_server_installation). Let's assume that first server was named srv1 (srv1.ipa.example.org).
How to test
First Replica
Creating a replica is a two-step process. A file representing the replica is prepared on an existing IPA server and this file is then used to install on a new replica.
Prepare a Replica
Let's create a replica named srv2
On srv1 run as root:
# ipa-replica-prepare srv2.ipa.example.org
NOTE: if you installed ipa1 with the --setup-dns option you can create DNS records on the fly with the following command:
# ipa-replica-prepare --ip-address=10.1.2.1 srv2.ipa.example.org
Copy the resulting file to srv2:
# scp /var/lib/ipa/replica-info-srv2.ipa.example.org.gpg srv2.ipa.example.org:
Create a couple of entries on the existing IPA server:
# kinit admin # ipa user-add psmith --first=Patty --last=Smith # ipa group-add engineering --desc=Engineers
Installing the Replica
An IPA replica is a full copy of an IPA server. It has the same requirements that any server has, it will just be configured to share data to one or more other IPA servers.
Make sure that iptables is properly configured on all ipa servers or turn it off for testing. ipa-replica-install will still try to validate your installation by executing ipa-replica-conncheck which will test if all required ports are open (from both endpoints)
Start by installing the FreeIPA packages.
On srv2 as root:
# yum install freeipa-server
# ipa-replica-install /root/replica-info-srv2.ipa.example.org.gpg
The installation should exit gracefully with no errors.
Verifying the Installation
We added a couple of entries before we installed the replica, verify that they are available on srv2:
# kinit admin # ipa user-show psmith # ipa group-show engineering # ipa host-show panther # should give a "not found" message
Now we try the reverse and add some entries on srv2 and they should show up on srv1:
# ipa user-add --first=Glen --last=Jones gjones # ipa group-add --desc='Office assistants' assistants
On srv1 run:
# ipa user-show gjones # ipa group-show assistants
Configure the Firewall
New Fedora 19 systems have FirewallD enabled by default, you may want to either open all ports required by FreeIPA server or disable the Firewall for the test and flush iptables rules so that clients do not have issues connecting FreeIPA replica:
# systemctl stop firewalld.service # systemctl disable firewalld.service # iptables -F
CA Replica
By default, a certificate authority is not installed on replicas (unless you specify --setup-ca during ipa-server-install). To add it to srv2, run:
# ipa-ca-install /root/replica-info-srv2.ipa.example.org.gpg
The installation should exit gracefully with no errors.
Second Replica
If you have another machine to test with then continue, otherwise skip to the next set of tests.
Let's create a third server named srv3.
Let's create a replica on srv2 this time. This will result in a replication topology of:
srv1 <-> srv2 <-> srv3
Make sure you have installed a certificate authority on srv2. Only masters with a CA can be used to create replicas.
On srv2 run:
# ipa-replica-prepare srv3.ipa.example.org # scp /var/lib/ipa/replica-info-srv3.ipa.example.org.gpg srv3.ipa.example.org:
On srv3 run:
# yum install freeipa-server
# ipa-replica-install /root/replica-info-srv3.ipa.example.org.gpg
The installation should exit gracefully with no errors.
The same steps in Verifying the Installation should pass.
Entries added on srv3 should appear on both srv2 and srv1.
Management Tools
When there are more than 2 replicas, it is possible to play with the topology.
Testing replication failures
After we created the 2 replicas we are left with a replication topology where srv1 and srv3 both replicate with srv2 but they do not replicate with each other. This means that if srv2 is shut down the other 2 replicas become isolated.
Go on srv2 and shut down ipa:
# ipactl stop
Now go on srv1 and create a new user:
# ipa user-add --first=John --last=Doe jdoe
Switch to srv3 and check that jdoe has not been replicated yet.
# ipa user-show jdoe ipa: ERROR: jdoe: user not found
Go to srv2 and restart ipa:
# ipactl start
Verify that after a ipa is fully started on srv2 that the jdoe user is available:
# ipa user-show jdoe
Do the same on srv3 to verify that replication has been restored and all servers are now synchronized.
Listing servers and replication agreements
You can list all replicating servers in your domain with:
# ipa-replica-manage list srv1.ipa.example.org srv2.ipa.example.org srv3.ipa.example.org
If you want to check which are the replication peers of srv2:
# ipa-replica-manage list srv2.ipa.example.org srv1.ipa.example.org srv3.ipa.example.org
For srv1 and srv3 you should get only srv3 and srv1 respectively.
Use --verbose when asking about a specific server to get more information about its peers.
Changing Topology
To avoid the split brain situation above let's play with topology and add a new link between srv1 and srv3 so that if srv2 is down, the other 2 keeps replicating.
On srv1 issue the following command:
# ipa-replica-manage connect srv3.ipa.example.org
If you have an admin credential cache this command should complete successfully without asking you anything. If you do not have any credentials it will ask you the Directory Manager password.
Once this command completes srv1 and srv3 are linked and replicate to each other.
Try the tests of the previous section, using a new user name and shutting down again srv2. After this new link is created the user should be replicated between srv1 and srv3 even when srv2 is down.
Now let's break a replication agreement so that srv2 and srv3 do not replicate directly anymore. On srv2 do:
# ipa-replica-manage disconnect srv2.ipa.example.org srv3.ipa.example.org
Once this is done the replication topolgy should be: srv2 <=> srv1 <=> srv3 Try the tests of the previous section, but turning off srv1 instead. changes done on srv2 should not be visible in srv3 until you bring up srv1 again.
Forcing a synchronization and re-initializing replicas
Replication can take from a few seconds to a few minutes to complete depending on various factors.
Try forcing an immediate synchronization by issuing a force-sync command on the replica you want to synchronize.
For example to force-sync srv1 from srv2 use this on the srv1 server.
# ipa-replica-manage force-sync --from srv2.ipa.example.org
NOTE: This command accepts on a --from parameter, but not a --to, so it must to be run on the replica that needs to be synchronized.
It may happen that due to compounding issues, replication can't be easily fixed by a simple re-synchronization. In this case a full re-initialization can be performed:
# ipa-replica-manage re-initialize --from srv2.ipa.example.org
This command will wipe clean the directory and re-download all the data from srv2. Any changes to the local tree that were not replicated are lost.
Testing clients
If you have extra machines, install clients on them (see QA:Testcase realmd join sssd or QA:Testcase_realmd_join_cockpit). The client tests should succeed regardless of which replica is used as the client's master.
If you installed FreeIPA with integrated DNS, the server will automatically set LDAP and Kerberos SRV records to the zone ipa.example.com which is then used by clients to autodiscover FreeIPA server it would connect to.
Disposing of a replica
If a replica becomes useless it should be first removed from the topology with the following command (on srv1 or srv2):
# ipa-replica-manage del srv3.ipa.example.org
If it had a CA installed, it's CA replication agreement shall be removed as well:
# ipa-csreplica-manage del srv3.ipa.example.org
If the host is destroyed before this command is run, use the --force switch to remove dangling pointers to the failed replica.
The command above does not stop the replica itself. Once the replica has been removed from the domain, stop it and uninstall it.
On srv3:
# ipa-server-install --uninstall
Expected Results
All the test steps should end with the specified results.