From 4ca8920188e819eadf4614abc3c5f2d978725650 Mon Sep 17 00:00:00 2001 From: Galen Charlton Date: Mon, 1 May 2023 12:20:15 -0400 Subject: [PATCH] LP#1979071: add release notes Signed-off-by: Galen Charlton --- .../Architecture/queued-ingest.adoc | 96 ++++++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100644 docs/RELEASE_NOTES_NEXT/Architecture/queued-ingest.adoc diff --git a/docs/RELEASE_NOTES_NEXT/Architecture/queued-ingest.adoc b/docs/RELEASE_NOTES_NEXT/Architecture/queued-ingest.adoc new file mode 100644 index 0000000000..1b6ba020fb --- /dev/null +++ b/docs/RELEASE_NOTES_NEXT/Architecture/queued-ingest.adoc @@ -0,0 +1,96 @@ +== Queued Ingest == + +This feature allows for the separation of bib and authority record +updates and the search (and other) indexing that occurs when a record is +modified in some way. Prior to this feature, bib and authority records +would be indexed immediately upon an update. + +While individual record ingest has not become a problem with regard to +system performance or interface usability, there exist several batch +operations which aggregate many inserts or updates and whose aggregate +ingest time cost can be significant. These include, but are not +limited to, reingest caused by authority control propagation, reingest +required by the addition or modification of indexing configuration, +cataloging and acquisitions record import and overlay from the staff +interface, and upgrade-time reingest required by structural changes +to the underlying indexing and search system. + +=== New Utility === + +When Queued Ingest is enabled, a new control script, `ingest_ctl`, is +available to perform several functions: + +* Run in the background to process the queues of indexing requests +* Display statistics of queued ingest activity +* Specify that a set of records should be reindexed. + +Here are some examples of how it is used: + +[source,bash] +------------------------------------------------------------------------------------ +# Enqueue records 1-500000 for reingest later, just one worker for the queue +/openils/bin/ingest_ctl --queue-threads 1 + --queue-type biblio + --queue-run-at tomorrow + --queue-owner admin + --queue-name "slowly updating records due to new RDA attributes" + --start-id 1 --end-id 500000 + +# Start the background worker +/openils/bin/ingest_ctl --coordinator --max-child 20 + +# Stop the background worker +/openils/bin/ingest_ctl --coordinator --stop + +# Process whatever you can Right Now +/openils/bin/ingest_ctl --max-child 20 + +# Process a single queue Right Now +/openils/bin/ingest_ctl --queue 1234 --max-child 20 + +# Stats on Queued Ingest processing so far today +/openils/bin/ingest_ctl --stats --since today --totals-only +------------------------------------------------------------------------------------ + +This script also requires the following switches (or environment +variables) in order to connect to the database: + +* `--db_user` (or environment variable `PGUSER`) +* `--db` (or environment variable `PGDATABASE`) +* `--dbpw` (or environment variable `PGPASSWORD`) +* `--db_port` (or environment variable `PGPORT`) + +=== New Settings === + +This feature adds several new global flags: + + +|=== +| Global Flag | Enabled + +| Queued Ingest: Abort transaction on ingest error rather than simply logging an error | no +| Queued Ingest: Queue all bib record updates on authority change propagation, even if bib queuing is not generally enabled | no +| Queued Ingest: Use Queued Ingest for bib record ingest on insert and undelete | no +| Queued Ingest: Use Queued Ingest for authority record ingest on insert and undelete | no +| Queued Ingest: Use Queued Ingest for bib record ingest on update | no +| Queued Ingest: Use Queued Ingest for authority record ingest on update | no +| Queued Ingest: Use Queued Ingest for bib record ingest on delete | no +| Queued Ingest: Use Queued Ingest for authority record ingest on delete | no +| Queued Ingest: Maximum number of database workers allowed for queued ingest processes | yes; default value 20 +| Queued Ingest: Use Queued Ingest for all bib record ingest | no +| Queued Ingest: Use Queued Ingest for all bib and authority record ingest | no +| Queued Ingest: Do NOT use Queued Ingest when creating a new bib, or undeleting a bib, via the MARC editor | yes +| Queued Ingest: Use Queued Ingest for all authority record ingest | no +| Queued Ingest: Do NOT Use Queued Ingest when editing bib records via the MARC Editor | yes +|=== + +This feature does not add any new library settings or permissions. + +=== Upgrade Notes === + +Queued Ingest is not automatically turned on upon upgrade. to enable it, +at minimum the following actions should be taken: + +* Enable at least the "Use Queued Ingest for all bib and authority record + ingest" global flag +* Ensure that `ingest_ctl` is running with the `--coordinator` flag. -- 2.11.0