QA:Testcase freeipa replication

Description

Replication testing.

Setup

1. You will need at least 2 machines to do replication testing but if you have more that is good too. Start on one server and perform an IPA Installation (see QA:Testcase_freeipav3_installation). Let's assume that first server was named srv1 (srv1.ipa.example.org).

How to test

First Replica

Creating a replica is a two-step process. A file representing the replica is prepared on an existing IPA server and this file is then used to install on a new replica.

Prepare a Replica

Let's create a replica named srv2

On srv1 run as root:

# ipa-replica-prepare srv2.ipa.example.org

NOTE: if you installed ipa1 with the --setup-dns option you can create DNS records on the fly with the following command:

# ipa-replica-prepare --ip-address=10.1.2.1 srv2.ipa.example.org

Copy the resulting file to srv2:

# scp /var/lib/ipa/replica-info-srv2.ipa.example.org.gpg srv2.ipa.example.org:

Create a couple of entries on the existing IPA server:

# kinit admin
# ipa user-add psmith --first=Patty --last=Smith
# ipa group-add engineering --desc=Engineers

Installing the Replica

An IPA replica is a full copy of an IPA server. It has the same requirements that any server has, it will just be configured to share data to one or more other IPA servers.

Firewall
Make sure that iptables is properly configured on all ipa servers or turn it off for testing. ipa-replica-install will still try to validate your installation by executing ipa-replica-conncheck which will test if all required ports are open (from both endpoints)

Start by installing the FreeIPA packages.

On srv2 as root:

# yum install freeipa-server

# ipa-replica-install /root/replica-info-srv2.ipa.example.org.gpg

The installation should exit gracefully with no errors.

Verifying the Installation

We added a couple of entries before we installed the replica, verify that they are available on srv2:

# kinit admin
# ipa user-show psmith
# ipa group-show engineering
# ipa host-show panther  # should give a "not found" message

Now we try the reverse and add some entries on srv2 and they should show up on srv1:

# ipa user-add --first=Glen --last=Jones gjones
# ipa group-add --desc='Office assistants' assistants

On srv1 run:

# ipa user-show gjones
# ipa group-show assistants

Configure the Firewall

New Fedora 19 systems have FirewallD enabled by default, you may want to either open all ports required by FreeIPA server or disable the Firewall for the test and flush iptables rules so that clients do not have issues connecting FreeIPA replica:

# systemctl stop firewalld.service
# systemctl disable firewalld.service
# iptables -F

CA Replica

By default, a certificate authority is not installed on replicas (unless you specify --setup-ca during ipa-server-install). To add it to srv2, run:

# ipa-ca-install /root/replica-info-srv2.ipa.example.org.gpg

The installation should exit gracefully with no errors.

Second Replica

If you have another machine to test with then continue, otherwise skip to the next set of tests.

Let's create a third server named srv3.

Let's create a replica on srv2 this time. This will result in a replication topology of:

srv1 <-> srv2 <-> srv3

Make sure you have installed a certificate authority on srv2. Only masters with a CA can be used to create replicas.

On srv2 run:

# ipa-replica-prepare srv3.ipa.example.org
# scp /var/lib/ipa/replica-info-srv3.ipa.example.org.gpg srv3.ipa.example.org:

On srv3 run:

# yum install freeipa-server

# ipa-replica-install /root/replica-info-srv3.ipa.example.org.gpg

The installation should exit gracefully with no errors.

The same steps in Verifying the Installation should pass.

Entries added on srv3 should appear on both srv2 and srv1.

Management Tools

When there are more than 2 replicas, it is possible to play with the topology.

Testing replication failures

After we created the 2 replicas we are left with a replication topology where srv1 and srv3 both replicate with srv2 but they do not replicate with each other. This means that if srv2 is shut down the other 2 replicas become isolated.

Go on srv2 and shut down ipa:

# ipactl stop

Now go on srv1 and create a new user:

# ipa user-add --first=John --last=Doe jdoe

Switch to srv3 and check that jdoe has not been replicated yet.

# ipa user-show jdoe
ipa: ERROR: jdoe: user not found

Go to srv2 and restart ipa:

# ipactl start

Verify that after a ipa is fully started on srv2 that the jdoe user is available:

# ipa user-show jdoe

Do the same on srv3 to verify that replication has been restored and all servers are now synchronized.

Listing servers and replication agreements

You can list all replicating servers in your domain with:

# ipa-replica-manage list
srv1.ipa.example.org
srv2.ipa.example.org
srv3.ipa.example.org

If you want to check which are the replication peers of srv2:

# ipa-replica-manage list srv2.ipa.example.org
srv1.ipa.example.org
srv3.ipa.example.org

For srv1 and srv3 you should get only srv3 and srv1 respectively.

Use --verbose when asking about a specific server to get more information about its peers.

Changing Topology

To avoid the split brain situation above let's play with topology and add a new link between srv1 and srv3 so that if srv2 is down, the other 2 keeps replicating.

On srv1 issue the following command:

# ipa-replica-manage connect srv3.ipa.example.org

If you have an admin credential cache this command should complete successfully without asking you anything. If you do not have any credentials it will ask you the Directory Manager password.

Once this command completes srv1 and srv3 are linked and replicate to each other.

Try the tests of the previous section, using a new user name and shutting down again srv2. After this new link is created the user should be replicated between srv1 and srv3 even when srv2 is down.

Now let's break a replication agreement so that srv2 and srv3 do not replicate directly anymore. On srv2 do:

# ipa-replica-manage disconnect srv2.ipa.example.org srv3.ipa.example.org

Once this is done the replication topolgy should be: srv2 <=> srv1 <=> srv3 Try the tests of the previous section, but turning off srv1 instead. changes done on srv2 should not be visible in srv3 until you bring up srv1 again.

Forcing a synchronization and re-initializing replicas

Replication can take from a few seconds to a few minutes to complete depending on various factors.

Try forcing an immediate synchronization by issuing a force-sync command on the replica you want to synchronize.

For example to force-sync srv1 from srv2 use this on the srv1 server.

# ipa-replica-manage force-sync --from srv2.ipa.example.org

NOTE: This command accepts on a --from parameter, but not a --to, so it must to be run on the replica that needs to be synchronized.

It may happen that due to compounding issues, replication can't be easily fixed by a simple re-synchronization. In this case a full re-initialization can be performed:

# ipa-replica-manage re-initialize --from srv2.ipa.example.org

Replica re-initialization
This command will wipe clean the directory and re-download all the data from srv2. Any changes to the local tree that were not replicated are lost.

Testing clients

If you have extra machines, install clients on them (see QA:Testcase_freeipav3_installation#Add_a_client). The client tests should succeed regardless of which replica is used as the client's master.

If you installed FreeIPA with integrated DNS, the server will automatically set LDAP and Kerberos SRV records to the zone ipa.example.com which is then used by clients to autodiscover FreeIPA server it would connect to.

Disposing of a replica

If a replica becomes useless it should be first removed from the topology with the following command (on srv1 or srv2):

# ipa-replica-manage del srv3.ipa.example.org

If it had a CA installed, it's CA replication agreement shall be removed as well:

# ipa-csreplica-manage del srv3.ipa.example.org

If the host is destroyed before this command is run, use the --force switch to remove dangling pointers to the failed replica.

The command above does not stop the replica itself. Once the replica has been removed from the domain, stop it and uninstall it.

On srv3:

# ipa-server-install --uninstall

Expected Results

All the test steps should end with the specified results.