Queued Ingest

Queued Ingest

This feature allows for the separation of bib and authority record
updates and the search (and other) indexing that occurs when a record is
modified in some way.

The Queued Ingest mechanism consists of several parts working together:
 * A set of configuration flags that control when ingest should be
   performed immediately, and when it can be deferred until after the
   transaction commits and control is returned to the user.
 * Refactoring of the in-database ingest triggers to separate deciding
   what should happen to a record given a data modification event, and
   when/how that process should take place.
 * A set of queuing tables used to track which records are to be
   processed and in what ways, when that processing was requested, and
   the ability to group processing requests into named queues that can
   report who made a processing request and for what purpose.
 * A Queued Ingest Coordinator that runs in the background monitoring
   the queuing tables for activity and processes records as they are
   enqueued.  This can run on any server that can connect to the
   database and has the OpenSRF Perl modules installed.
 * A command line tool to be usedby administrators to enqueue records
   for Queued Ingest processing, to create named queues, and to process
   enqueued records either in one queue or all outstanding enqueued
   entries.  This tool can also report on the status of requested Queued
   Ingest processing, whether pending, ongoing, or complete, either for
   all time or since a particular date and time.

The queuing tables added here are not yet included in the IDL for
reporting or Staff interface construction.  They can be added as the
need arises.

To test Queued Ingest:
 * Upgrade the database schema using the appropriate upgrade script.
 * Install OpenILS/src/support-scripts/ingest_ctl in the usual OpenSRF
   bin directory.
 * Start the Queued Ingest Coordinator by using the --coordinator
   parameter to ingest_ctl along with the relevant database parameters.
   Use the --help parameter for more details.
 * Enable the Global Flag called 'ingest.queued.all'
 * Edit, import, and delete some bib and authority records, and use the
   --stats parameter to ingest_ctl to see entries being processed.

Signed-off-by: Mike Rylander <mrylander@gmail.com>