A shared library exports, among other things, a set of globally defined variables and functions. These functions and variables, along with their types (type of the variable, type of the parameters and return value of the functions) constitute the interface that applications use to get access to the functionalities of the shared library.
In a typical GNU/Linux system, a subset of the properties of this application interface is available directly from the shared library binary. That subset is usually called Application Binary Interface of the shared library, a.k.a ABI.
To make a long story short, it's correct enough to consider that the set of globally defined variables and functions along with their types is part of the ABI of the shared library. If a globally defined and exported variable or function is removed or added to this set or if its type changes, then the ABI of the shared library changes.
Compatibility
When an application links with a given version of a shared library, it uses a subset of the ABI of that library. It thus expects that used ABI subset to have certain properties. There is thus a contract between the application and the library by which the application relies on the library to provide the ABI properties it expects. When that contract is honoured, the library is said to be ABI-compatible with the application.
If the library evolves and stops honouring that contract, then the application won't find the ABI properties it expects from the newer version of the library. The application won't be able to execute as intended. Runtime bugs follow. The newer version of the library is then said to be ABI-incompatible with the application.
For instance, if the application is a C language application that expects the library to export a function named foo which has at least 2 parameters, the first one being an integer and the second one being a character, then, that function foo must always be defined and exported by the library, with at least those two parameters, in that order. Otherwise, the library is 'ABI-incompatible with the application.
Two versions of a shared library are said to be ABI-compatible, if applications that are ABI-compatible with the first version are still ABI-compatible with the second version.