Features/FedFS

FedFS

Summary

RFC 5716 introduces the Federated File System (FedFS, for short). FedFS is an extensible standardized mechanism by which system administrators construct a coherent namespace across multiple file servers using file system referrals.

Owner

• Name: Ian Kent

• Email: ikent@redhat.com

Current status

• Targeted release: Fedora 18
• Last updated: Jul 24, 2012
• Percentage of completion: 100%

Detailed Description

RFC 5716 introduces the Federated File System (FedFS, for short). FedFS is an extensible standardized mechanism by which system administrators construct a coherent namespace across multiple file servers using file system referrals.

A file system referral is like a symbolic link to another file system share, but it is not visible to applications. It behaves like an automounted directory where a new file system mount is done when an application first accesses that directory. The arguments of the mount operation are controlled by information returned by the file server.

Today, file system referral mechanisms exist in several network file system protocols. FedFS provides its namespace features by leveraging referral mechanisms already built in to network file system protocols. Thus no change to file system protocols or clients is required.

Currently, the Linux FedFS implementation supports only NFS version 4 referrals. More on NFS version 4 referrals can be found in RFC 3530. FedFS may support other network file system protocols in the future.

Benefit to Fedora

See detailed description. As this is an emerging standard, we want Fedora to be one of the first operating systems to support it.

Chuck Lever also points out that fedfs-utils is part of a larger, ongoing effort to build out storage administration features on Linux file servers. This includes, to name but a few:

• The ability to manage NFS referrals either via FedFS or via a simple command line tool on file servers
• Support for SMB2 in a FedFS framework
• Support for NFSv4.1's fs_locations_info attribute
• Client-side support for NFSv4 migration with transparent state migration
• Support for transparent NFS client access to replicated file sets

Scope

Much of the work for this has already been done. Chuck Lever of Oracle has done an implementation of the required userspace tools for this, and a Fedora package of release 0.8.0 has been built. We'll probably also need to do some documentation and/or articles describing how to set up both the client and server for FedFS.

The upstream project website for this is here:

    http://wiki.linux-nfs.org/wiki/index.php/FedFsUtilsProject

It contains license information, presentation slides, the FedFS LDAP schema, and a project roadmap.

Fedfs is, in many ways, an addition of automount functionality along the line of something that I believe (as the autofs maintainer both upstream and here in Fedora) has been needed for a long time. That is a distributed autofs mount map resource manager and while fedfs isn't quite what I envisioned for autofs the functionality it provides is fairly close. Just how far integration with autofs will go isn't clear yet.

At this stage the fedfs package cotains an autofs program map that is used to provide access to the namespace and that is sufficient to start with. An improvement to autofs that I plan on doing is to add native support to autofs for fedfs in order to provide earier discovery and access to the available fedfs namespaces.

How To Test

Simple example of basic FedFS operation

Assume a test DNS server has been setup for a DNS domain autofs.test (in this case both were setup on zeus.autofs.test) and that the hosts perseus.autofs.test and zeus.autofs.test map to vaild addresses. Also assume that resolve.conf has been amended to use this DNS server.

Also assume we have an LDAP instance that has a naming context of "dc=autofs,dc=test" that is functional on the server and the client can perfom queries.

Further assume that the services used are running. Namely, LDAP, FedFS daemon and NFS services on zeus.autofs.test, and FedFS daemon and autofs services on zeus.autofs.test.

Finally assume that packages fedfs-utils-common, fedfs-utils-nsdbparams, fedfs-utils-server and fedfs-utils-client are installed on perseus.autofs.test. And that fedfs-utils-common, fedfs-utils-server and an nfs-utils built with fedfs-utils-devel installed are installed on zeus.autofs.test.

The DNS and LDAP (OpenLDAP) configuration files can be provided for reference and step by step instructions can be done, if required. But it is assumed that someone wishing to test FedFS will be able to adapt the procedures below to an existing test environment.

What want to setup perseus.autofs.test to provide the domain root, be able to use FedFS to mount the domain root, and to setup and use a FedFS juntion to access an export on zeus.autofs.test upon access to a directory within the domain root. And lastly, setup autofs to use the FedFS program map to mount the root of the domain.

Setup an NSDB (NameSpace DataBase)

1. Set parameters for NSDB connections:
        # nsdbparams(8) is used to set NSDB connection parameters
        nsdbparams update -e "o=fedfs,dc=autofs,dc=test" \
                          -D "cn=Manager,dc=autofs,dc=test" \
                          zeus.autofs.test

2. Add FedFS schema to LDAP server config (which can be found
   in the FedFS common package docunentation directory).

3. Add a FedFS naming context to LDAP server.
        ldapadd -v -x \
                -h zeus.autofs.test \
                -f init.ldif \
                -D "cn=Manager,dc=autofs,dc=test" -W
   where init.ldif contains:
     dn: o=fedfs,dc=autofs,dc=test
     objectclass: organization
     objectClass: top
     o: fedfs

4. Add NCI (NSDB Container information) attributes to the
   naming context LDAP entry:
        nsdb-update-nci -l zeus.autofs.test \
                -D "cn=Manager,dc=autofs,dc=test" \
                -e "o=fedfs,dc=autofs,dc=test"

Add a FedFS junction within a domain root directory

Assuming there is a file system mounted on /vm (or just a directory we can export) on server zues.autofs.test which we want to access under the domain root as <domain root mount point>/vm, we will be exporting /.domainroot-autofs.test as the domain root.

1. Add an entry to /etc/exports on zeus.autofs.test:

        # Add to /etc/exports
        /vm     *(ro)

        # Restart the nfs service or just re-export the table
        exportfs -r

2. Add a junction to the domain root on persues.autofs.test:

        #
        # Tell nfsref the LDAP server (the NSDB) we are using to
        # record file system name (FSN) and file system location
        # (FSL) uuids. This assumes the LDAP connection parameters
        # have been setup as in step 1 of "Setup an NSDB".
        #
        export FEDFS_NSDB_HOST=zeus.themaw.net

        #
        # Add the junction metadata to the directory and update
        # the NSDB with uuid info of the junction.
        #
        mkdir -p /.domainroot-autofs.test/vm
        nfsref --type=nfs-fedfs \
                add /.domainroot-autofs.test/vm \
                zeus.autofs.test /vm

Setup fedfs domain root export (read-only case)

For this we are seeking to mount the domain root exported from host perseus.autofs.test.

1. Add an SRV record for the FedFS file server to DNS:

        _nfs4._domainroot._tcp SRV 0 0 2049 perseus.autofs.test.

2. Restart named to make in available.

        service named restart
or
        systemctl restart named.service

3. Add an entry to /etc/exports on perseus.autofs.test:

        #
        # Created when we added the junction above.
        # mkdir /.domainroot-autofs.test
        #
        /.domainroot-autofs.test        *(ro)

4. Restart NFS:

        service nfs restart
or
        systemctl restart nfs.service

5. Mount using the FedFS mount utility on a local directory:

        mount -v -t fedfs /nfs4/autofs.test /mnt
        mount | grep domainroot
        perseus.autofs.test:/.domainroot-autofs.test/ on /mnt type nfs4 ...

        cd /mnt/vm

        #
        # This check assumes /etc/mtab is symlinked to /proc/mounts
        # as it is in Fedora. Kernel automounted file systems will
        # not be present in the text based /etc/mtab and so will not
        # be seen in it. Look to /proc/mounts instead in this case.
        #
        mount | grep ^zeus.autofs.test.*vm
        zeus.autofs.test:/vm/ on /mnt/vm type nfs4 ...

        # Ha, move out of the directory so it can be umounted
        cd

5. Lastly cleanup:

        #
        # This example includes a specific umount of the junction
        # (/mnt/vm) but such kernel automounted file systems are
        # umounted automatically (when they are not in use) so it
        # may not be present when this step is done.
        #
        umount /mnt/vm
        umount /mnt

Setup autofs to automount the domain root

1. Add a line to /etc/auto.master to automount FedFS root domains:

        #
        # Note that the autofs pseudo option "nobind" probably
        # should be used. In the case here it is required because
        # the FedFS client also hosts the root of the domain and
        # autofs will see the mount is local and perform a bind
        # mount instead of an NFS mount. That, of course, means
        # file system lookups won't be with an NFS file system
        # so NFS referals can't be followed.
        #
        # Also note that the autofs mount point name must be
        # /nfs4 to be able to mount nfs4 root domains.
        #
        echo "/nfs4  /usr/sbin/fedfs-map-nfs4 nobind" >> /etc/auto.master

2. Restart or reload the autofs service:

        service autofs restart
or
        systemctl restart autofs.service

3. Check that we can mount the domain root and the referal:

        # automount the root domain.
        [raven@perseus ~]$ ls /nfs4/autofs.test
        top.txt  vm

        # automount the referal (from a different machine).
        [raven@perseus ~]$ ls /nfs4/autofs.test/vm
        lost+found  test.txt

        # Check they were mounted.
        [raven@perseus ~]$ mount |grep perseus|grep nfs4
        perseus.autofs.test:/.domainroot-autofs.test/ on /nfs4/autofs.test type nfs4 ...
        [raven@perseus ~]$ mount |grep zeus|grep nfs4
        zeus.autofs.test:/vm/ on /nfs4/autofs.test/vm type nfs4 ...

User Experience

The casual Fedora user won't see any changes with this. This should only change the experience for someone who makes the effort to set up and use FedFS.

Dependencies

There are some kernel patches required which are fairly modest in scope and have been included in the upstream kernel since version 3.2.

There are also some changes needed to nfs-utils (specifically mountd) to allow it to process junctions. The change to nfs-utils required by version 0.8.0 has been done although nfs-utils will need to be rebuilt to recognise that the fedfs-utils plugin is now present.

Additionally, LDAP servers need to have the FedFS LDAP schema installed. The schema is not yet included in Openldap so users will need to manually add the schema which is included in the package fedfs-utils-common and placed in docs directory.

/etc/rpc needs to be updated to include the IANA-assigned fedfs RPC program number which was added in glibc-2.15-12, available since Fedora 17.

Contingency Plan

If this package isn't approved for inclusion, we simply won't ship fedfs-utils.

Documentation

The package has a fairly comprehensive set of manpages, but we may also need to create some other "How-To" documentation.

Release Notes

The code provided in this package is a technology preview. The intent is to provide a full and supported Linux FedFS client and server implementation based on this code. Programming and user interfaces may change significantly for the next few releases.

The components in this package are used for managing file system referrals in order to create a global network file system namespace. Installable components include:

• An automounter program map to manage the FedFS domain namespace on FedFS-enabled clients.
• A mount command to mount parts of a FedFS domain namespace.
• A plug-in library that allows programs outside of FedFS to resolve junctions on local file systems.
• An ONC RPC service daemon that runs on file servers enabling the management by remote FedFS ADMIN clients of FedFS junctions.
• A tool called "nfsref" to manage local junctions without requiring fedfsd.
• A set of command-line clients that can access fedfsd instances on remote file servers.
• A set of command-line clients that can manage FedFS entries on an LDAP server acting as a FedFS NSDB.
• A tool to manage NSDB connection parameters on the local host.
• An LDIF format schema to enable an LDAP server to support FedFS objects.

For more information refer to the FedFS project page and the FedFS Documentation page.

Comments and Discussion

• See Talk:Features/FedFS