Description
Replication testing.
Setup
- Make sure you have a working FreeIPA server (see QA:Testcase_freeipav2_installation)
- Make sure the CLI works as expected (see QA:Testcase_freeipav2_cli)
- You will need at least 2 machines to do replication testing but if you have more that is good too. Start on one server and perform an IPA Installation. Let's call the first server srv1.
How to test
First Replica
Creating a replica is a two-step process. A file representing the replica is prepared on an existing IPA server and this file is then used to install on a new replica.
Prepare a Replica
Let's create a replica named srv2
On srv1 run as root:
# ipa-replica-prepare srv2.ipa.example.com
NOTE: if you installed ipa1 with the --setup-dns option you can create DNS records on the fly with the following command:
# ipa-replica-prepare --ip-address=10.1.2.1 srv2.ipa.example.com
Copy the resulting file to srv2:
# scp /var/lib/ipa/replica-info-srv2.ipa.example.com.gpg srv2.ipa.example.com:
Create a couple of entries on the existing IPA server:
# kinit admin # ipa user-add --first=Patty --last=Smith psmith # ipa group-add --desc=Engineers engineering
Installing the Replica
An IPA replica is a full copy of an IPA server. It has the same requirements that any server has, it will just be configured to share data to one or more other IPA servers.
NOTE: Make sure iptables is properly configured on all ipa servers or turn it off for testing.
Start by installing the FreeIPA packages.
On srv2 as root:
# yum install freeipa-server
# ipa-replica-install /root/replica-info-srv2.ipa.example.com.gpg
The installation should exit gracefully with no errors.
Verifying the Installation
We added a couple of entries before we installed the replica, verify that they are available on srv2:
# kinit admin # ipa user-show psmith # ipa group-show engineering # ipa host-show panther
Now we try the reverse and add some entries on srv2 and they should show up on srv1:
# ipa user-add --first=Glen --last=Jones gjones # ipa group-add --desc='Office assistants' assistants
On srv1 run:
# ipa user-show gjones # ipa group-show assistants
Second Replica
If you have another replica to test with continue, otherwise skip to the next set of tests.
Let's create a third server named srv3
Let's create a replica on srv2 this time. This will result in a replication topology of:
srv1 <-> srv2 <-> srv3
On srv2 run:
# ipa-replica-prepare srv3.ipa.example.com # scp /var/lib/ipa/replica-info-srv3.ipa.example.com.gpg srv3.ipa.example.com:
On srv3 run:
# yum install freeipa-server
# ipa-replica-install /root/replica-info-srv3.ipa.example.com.gpg
The installation should exit gracefully with no errors.
The same steps in Verifying the Installation should pass.
Entries added on srv3 should appear on both srv2 and srv1.
Management Tools
When there are more than 2 replicas, it is possible to play with the topology.
Testing replication failures
After we created the 2 replicas we are left with a replication topology where srv1 and srv3 both replicate with srv2 bu they do not replicate with each other. This means that if srv2 is shut down the other 2 replicas become isolated.
Go on srv2 and shut down ipa:
# service ipa stop
Now go on srv1 and create a new user:
# ipa user-add --first=John --second=Doe jdoe
Switch to srv3 and check that jdoe has not been replicated yet.
# ipa user-show jdoe ipa: ERROR: jdoe: user not found
Go to srv2 and restart ipa:
# service ipa start
Verify that after a ipa is fully started on srv2 that the jdoe user is available:
# ipa user-show jdoe
Do the same on srv3 to verify that replication has been restored and all servers are now synchronized.
Listing servers and replication agreements
You can list all replicating servers in your domain with:
# ipa-replica-manage list srv1.ipa.example.com srv2.ipa.example.com srv3.ipa.example.com
If you want to check which are the replication peers of srv2:
# ipa-replica-manage list srv2.ipa.example.com srv1.ipa.example.com srv3.ipa.example.com
For srv1 and srv3 you should get only srv3 and srv1 respectively.
Use --verbose when asking about a specific server to get more information about its peers.
Changing Topology
To avoid the split brain situation above let's play with topology and add a new link between srv1 and srv3 so that if srv2 is down, the other 2 keeps replicating.
On srv1 issue the following command:
# ipa-replica-manage connect srv1.ipa.example.com srv3.ipa.example.com
If you have an admin credential cache this command should complete successfully w/o asking you anything. If you do not have any credentials it will ask you the Directory Manager password.
Once this command completes srv1 and srv3 are linked and replicate to each other.
Try the tests of the previous section, using a new user name and shutting down again srv2. After this new link is created the user should be replicated between srv1 and srv3 even when srv2 is down.
Now let's break a replication agreement so that srv2 and srv3 do not replicate directly anymore.
# ipa-replica-manage disconnect srv2.ipa.example.com srv3.ipa.example.com
Once this is done the replication topolgy should be: srv2 <=> srv1 <=> srv3 Try the tests of the previous section, but turning off srv1 instead. changes done on srv2 should not be visible in srv3 until you bring up srv1 again.
Forcing a synchronization and re-initializing replicas
Replication can take from a few seconds to a few minutes to complete depending on various factors.
Try forcing an immediate synchronization by issuing a force-sync command on the replica you want to synchronize.
For example to force-sync srv1 from srv2 use this on the srv1 server.
# ipa-replica-manage force-sync --from srv2.ipa.example,com
NOTE: This commands accepts on a --from parameter, but not a --to, so it must to be run on the replica that needs to be synchronized.
It may happen that due to compounding issues, replication can't be easily fixed by a simple re-synchronization. In this case a full re-initialization can be performed:
# ipa-replica-manage re-initialize --from srv2.ipa.example.com
This command will wipe clean the directory and re-download all the data from srv2. Ant changes to the local tree that were not replicated are lost.
Disposing of a replica
If a replica becomes useless it should be first removed from the topology with the foll0wing command:
# ipa-replica-manage del srv3.ipa.example.com
In case the machine/vm was destroyed before this command is run use the --force switch in order to remove dangling pointers to the failed replica.
The command above does not stop the replica itself. Once the replica has been removed from the domain, stop it and uninstall it.
On srv3:
# ipa-server-install --uninstall
Expected Results
All the test steps should end with the specified results.