LP#
1979071: Queued Ingest
This feature allows for the separation of bib and authority record
updates and the search (and other) indexing that occurs when a record is
modified in some way.
The Queued Ingest mechanism consists of several parts working together:
* A set of configuration flags that control when ingest should be
performed immediately, and when it can be deferred until after the
transaction commits and control is returned to the user.
* Refactoring of the in-database ingest triggers to separate deciding
what should happen to a record given a data modification event, and
when/how that process should take place.
* A set of queuing tables used to track which records are to be
processed and in what ways, when that processing was requested, and
the ability to group processing requests into named queues that can
report who made a processing request and for what purpose.
* A Queued Ingest Coordinator that runs in the background monitoring
the queuing tables for activity and processes records as they are
enqueued. This can run on any server that can connect to the
database and has the OpenSRF Perl modules installed.
* A command line tool to be usedby administrators to enqueue records
for Queued Ingest processing, to create named queues, and to process
enqueued records either in one queue or all outstanding enqueued
entries. This tool can also report on the status of requested Queued
Ingest processing, whether pending, ongoing, or complete, either for
all time or since a particular date and time.
The queuing tables added here are not yet included in the IDL for
reporting or Staff interface construction. They can be added as the
need arises.
To test Queued Ingest:
* Upgrade the database schema using the appropriate upgrade script.
* Install OpenILS/src/support-scripts/ingest_ctl in the usual OpenSRF
bin directory.
* Start the Queued Ingest Coordinator by using the --coordinator
parameter to ingest_ctl along with the relevant database parameters.
Use the --help parameter for more details.
* Enable the Global Flag called 'ingest.queued.all'
* Edit, import, and delete some bib and authority records, and use the
--stats parameter to ingest_ctl to see entries being processed.
Signed-off-by: Mike Rylander <mrylander@gmail.com>
Signed-off-by: Bill Erickson <berickxx@gmail.com>
Signed-off-by: Galen Charlton <gmc@equinoxOLI.org>