OS update script staged updates

Support update/upgrade in one stage, followed by dist-upgrade in a
secondary stage with optional reboot in between.  The goal is to reduce
the kernel / grub install errors for prod servers, where grub fails to
install correctly after dist-upgrade, requiring a manual grub reinstall.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1730 Disable hold canceled User Message event def

2.9 Adds a few event defs for generating user-visible hold notification
messages.  One of them was enabled by default.  This commit disables it.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1525 Delete holds history retention user settings

Clear all user settings for managing holds history retention, so all
applicable holds can be aged.

Any new holds history retention starting with this commit will be
starting fresh.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1525 Holds purge settings and cron script

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1725 Enable Renton teacher cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1688 Backstage monthly auths use FTP

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1688 Backstage quarterly file/url name repair

Quarterly files come from the FTP server, monthly files come from the
HTTP server.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1445 2.9 SQL update statement timeouts

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1717 Student card processor ignores duplicate IDs

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1717 Enable Riverview teacher cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage processor -d db-host

Make it easy to override the PGHOST parameter from within the Backstage
processor.  This way we don't have to modify and reset the value from
within CRON when we want to use a DB host that is not the system
default.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

Student importer script sanity checks

1. Skip rows in the CSV file that contain no data for student_id.

2. Avoid trying to create students whose barcode matches an existing
   username in addition to an existing barcode.

3. Avoid inspecting and translating data for non-new accounts when only
   importing new accounts.  This reduces the work load of the script and
   allows it to complete faster.

4. Improve handling/logging for empty files

5. Exit with status 0 for calls to announce with the 'die' flag set to
   true.  In other words, treat them as expected errors that fail
   gracefully without preventing the calling script (sftp-client-agent)
   from exiting on an expected error.

6. Strip non-alpha-numeric characters from student_id values

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1662 Snoqualmie School/Classroom cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

CAT-54 Offline rcpt shows last 4 of barcode

Change printing param for Patron Barcode from the entire barcode to the
last four digits using substring -4.

Add asterisk as filler before last 4 digits of patron barcode on receipt.

Signed-off-by: Kyle Huckins <khuckins@catalystdevworks.com>

JBAS-1696 Patron editor suffix label change

Suffix => Suffix/(Preferred Name)

Signed-off-by: Bill Erickson <berickxx@gmail.com>

CAT-56 Offline Circ Loan Duration Mods

Remove options for 3 and 30 days.

Add options for same day, 42 days, and full year.

Merge in year handling from dev/berick/cat-56-offline-due-date-experiment.
Change ev.target.value handling for the non-option to allow for same-day
handling.

Signed-off-by: Kyle Huckins <khuckins@catalystdevworks.com>
 Changes to be committed:
modified:   Open-ILS/web/opac/locale/en-US/lang.dtd
modified:   Open-ILS/xul/staff_client/chrome/content/circ/offline_checkout.js
modified:   Open-ILS/xul/staff_client/chrome/content/circ/offline_checkout.xul

JBAS-1672 Bellevue teacher cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1699 Linking config remove old maps

Remove the hard-coded bib-to-auth linking maps and the code the
displayed how the linking maps changed when loading them from the DB.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1701 Stale browse entry cleaner stmt timeout

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1699 Linking config loaded from database

Load the bib-to-auth record maps from the
authority.control_set_bib_field data instead of hard-coding the maps in
the script.  This sychronizes the linking configs and the authority data
propagation configs to avoid cases where a mismatch can lead to bib
field deletions.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage processor script

Script to fetch and process monthly and quarterly Backstage data.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Linking script additions and repairs

Teach the bib linker how to find the records to link instead of
requiring external batch files.

Remove aging perl docs.

Update and rename link-all-bibs-daily.sh to leverage the new bib
linker bib-finding parameters.

Remove no longer needed find-bibs-to-link script, which is superseded by
the above.

Update README to reflect changes.

==

Adds a collection of common logging and DB utility to a new DB module
OpenILS::Utils::KCLSScriptUtil.

Updated various Backstage / linking scripts to use this new module.
More migrations to follow.

Auth2Auth gets --auth-mod-since

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1671 Shoreline school/classroom cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1446 KCLS 2.8/2.9 Release notes

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1446 Copy 2.6/2.7 release notes into main branch

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1692 2.7 -> 2.9 SQL Upgrade X-Ports

Port KCLS mods to modified functions back into upgrade script.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1692 2.7 -> 2.9 SQL Upgrade

Signed-off-by: Bill Erickson <berickxx@gmail.com>

Additional Issaquah/Northshore classroom cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage export skips bibs with warnings

So far this only affects bibs with invalid leader lengths.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage quarterly export docs

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage importer db handle resets

Periodically close and reopen the DB connection to avoid using any one
connection for extended periods of time, which can be bad when SQL
queries have memory leaks, as with bib record updates.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage importer process bibs first

When processing a ZIP file, process bib files before authority files.
On large data sets, processing bibs first allows staff to resume normal
bib editing operations earlier.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage importer --marc-file option

Importer now supports processing a single MARC file in addition to
ZIP files as before.  Zip file option is now specified via --zip-file.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Bib batch linker continued

Bug fixes, documentation.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Remove legacy linking/backstage scripts

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Move authority/BS scripts to authority-control

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 bib linking log more info

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Week-spanning bib link batcher script

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Bibs-to-link finder and batch generator

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 B2A linking additional query options

* --all uses direct DB connection
  * Fetching all record ID's via cstore is a bad idea.
* order by ID
* better logging
* --sort-desc option

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 B2A link clean up bogus fast headings; leave OK ones

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage scripts default DB settings

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Auth linker uses 'KCLS' control number identifer

Affects bib-to-auth and auth-to-auth linking.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage update authority 005 date values

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Bib ignores 'fast' headings.

Avoid any attempts at controlling fields whose thesaurus value (ind2=7
and $2) is 'fast'.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 BS search mod bibs once

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage 010z replace records

Replace authority records whose 010a matches the 010z value for inbound
records.

Also avoid looking up deleted records when looking for matches.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage quarterly export docs

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage process bib/auth file names update

Handle inconsistencies in BS results file naming.

* Process files ending in .UTF8 or .MRC
* Treat file with name AUTH as new/modified authority records.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Auth-2-auth linker --file option

Supports a --file option pointing to file of authority record IDs, one
ID per line.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Backstage record export/import scripts

* export-bibs.pl : export bib records to send to Backstage
* process-backstage-files.pl : process bib/auth records returned from BS
* README.adoc : howto docs

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Auth2bib links honors authority 008 #14/#15

Only link 1XX and 7XX bib fields to authority records whose 008 #14
field is 'a'.

Only link 6XX bib fields to authority records whose 008 #15 field is 'a'.

https://www.loc.gov/marc/authority/ad008.html

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 link-new-auths new options, ID printing

--start-auth-id
--max-auth-count

Also, print-x-ids now prints IDs one per line without any logging
header.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1437 Remove deprecated Backstage export functions

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1670 Tukwila teacher cards enabled

Signed-off-by: Bill Erickson <berickxx@gmail.com>

CAT-72 Claims return dialog sanity checks

Avoid the following error when editing hold shelf expire time:

timestamp.js, handle_apply(): TypeError:
    document.getElementById("newRadio1") is null

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1377 New headings report min date

Support applying a minimum report date in the new headings report to
prevent queries that report on (practically) all headings.  This happens
when the start date equals or precedes the deployment date for the
browse headings create_date column, which is set to NOW() by default.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1377 date.getYMD() avoids clobbering date

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1605 Claims returned dialog label adjustments

Change "Use Default Date" to "Use Due Date".

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1459 Authority propagation updates bib editor/date

Cross-porting LP#1588948 - Authority record updates that propagate to
bib records also updates bib editor and edit_date columns.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1638 Patron self-reg cleans spaces

Remove leading / trailing spaces from user-entered values in patron
self-registration app.  In addition to resulting in cleaner data, it
allows the patron editor's duplicate patron search code to work w/ like
values.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1625 Pending patrons create date column

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1635 Physical -> Residence addr in patron summary

Changes the label for the address radio selector toggles in the patron
summary.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

CAT-55 Save Archived Message Columns

Port previous commit's changes over to the
archived message's pane.

Signed-off-by: Kyle Huckins <khuckins@catalystdevworks.com>
 Changes to be committed:
modified:   Open-ILS/web/opac/locale/en-US/lang.dtd
modified:   Open-ILS/xul/staff_client/server/patron/standing_penalties.js
modified:   Open-ILS/xul/staff_client/server/patron/standing_penalties.xul

CAT-55 Save Patron Message Columns

Add Locale line for "Save Columns" in patron messages.
Add Save Column command to standing_penalties xul.
Add function to standing_penalties js to handle saving
columns.

Signed-off-by: Kyle Huckins <khuckins@catalystdevworks.com>
 Changes to be committed:
modified:   Open-ILS/web/opac/locale/en-US/lang.dtd
modified:   Open-ILS/xul/staff_client/server/patron/standing_penalties.js
modified:   Open-ILS/xul/staff_client/server/patron/standing_penalties.xul

KMAIN-1828 Full Patron Messages

Add new action on messages: "View Penalty/Message."

Add new xul window to display full note of message in
larger font than displayed on main messages page.

Add localization for new content.

Signed-off-by: Kyle Huckins <khuckins@catalystdevworks.com>
modified:   Open-ILS/web/opac/locale/en-US/lang.dtd
modified:   Open-ILS/xul/staff_client/chrome/content/main/constants.js
modified:   Open-ILS/xul/staff_client/server/patron/standing_penalties.js
modified:   Open-ILS/xul/staff_client/server/patron/standing_penalties.xul
new file:   Open-ILS/xul/staff_client/server/patron/view_standing_penalty.js
new file:   Open-ILS/xul/staff_client/server/patron/view_standing_penalty.xul

CAT-76 Remove author subfield 'e' from TPAC display/links

Avoid displaying 1XX or 7XX subfield 'e' values for authors and prevent
said values from populating auto-generated author search links.

Commit msg modified by Bill for clarity.

Signed-off-by: Billy Horn <bhorn@catalystdevworks.com>

CAT-50 Tighten stop sign page display

* Resize stop sign image to 80x85 from the original 80x113.
* Decrease margin between stop_sign.png and text.
* Remove blank space from bottom of stop_sign.png

Signed-off-by: Kyle Huckins <khuckins@catalystdevworks.com>

CAT-72 Claims returned default date option

Gives staff an option to use a default date for claims returned.

Minor cleanup and doc/comment additions by Bill.

Signed-off-by: Billy Horn <bhorn@catalystdevworks.com>
Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1604 Vandelay queue delete speedup indexes

Add indexes to make deleting vandelay queue data faster, specifically
when vandelay records link to import items and acq lineitems.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

CAT-77 Renewal override cancel error message repair.

Add check to make sure payload circ has a value.

-- Minor cleanup by Bill.

Signed-off-by: Billy Horn <bhorn@catalystdevworks.com>
Signed-off-by: Bill Erickson <berickxx@gmail.com>

CAT-79 Hide place-hold-for-me staff action

Add visibility: hidden to span containing the target radio

Signed-off-by: Billy Horn <bhorn@catalystdevworks.com>

JBAS-1443 Added content uses EAN / Misc.

Consider values from MARC 024 $a with indicator 1 values of 3 (EAN) or 8
(Misc/Unknown) as valid added content lookup keys.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1415 Apply net_access_level in self-reg

During self-registration, if a patron is less than 17 years old, apply
the "Under 17 Plus" net_access_level to the staged user.  If the patron
is 17 or older, apply "17 and Up Only".

It shouldn't happen in practice since we require a DoB, but if no DoB is
present, no net_access_level value is applied.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1415 Staged users get net_access_level

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1150 Record bucket 'deleted' column

Includes new 'deleted' column on 'mvr' class.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-905 Reduce copy status checkout alerts

Avoid copy status checkout alerts for these:

4    -- Missing
104  -- Evaluations
107  -- Display
112  -- Cleared
114  -- Workroom
115  -- For Opening
103  -- Story Time

Signed-off-by: Bill Erickson <berickxx@gmail.com>

CAT-69 Volume merge bug

Create function in update_items to handle
find_or_create_volume api call.

Create helper function to match acns and get
the amount of copies in supplied volume in PO.

Create helper function to get number copies from
current volume.

Clean up gather_copies().

Remove instances of direct attempts at modifying
IDL properties, replace with the proper method
calls.

Add helper function to set the acn for the items in the
current PO.

Add FM_ACN_RETRIEVE.authoritative call to find_or_create
helper function.

Set temp_acn's label class, prefix, and suffix
to their object's id within g.find_or_create_volume().

Remove unnecessary loop from g.render_batch_button()

Fix incorrect iteration style to
array-friendly one.

Signed-off-by: Kyle Huckins <khuckins@catalystdevworks.com>
 Changes to be committed:
modified:   Open-ILS/xul/staff_client/server/cat/update_items.js

JBAS-1377 New headings report UI / report view.

Adds a new reporter.cataloged_browse_entry DB view for finding browse
headings based on when their bibs were cataloged.

Adds a UI for searching and displayin such headings.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1377 Date handling utility JS functions

Signed-off-by: Bill Erickson <berickxx@gmail.com>
Conflicts:
Open-ILS/web/js/dojo/openils/Util.js

JBAS-1640 Riverview school/classroom cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1598 Fed. Way school cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1643 Teacher districts must be enabled

Before the SFTP client will process a teacher account file, the district
code must be added to a new TEACHER_DISTRICTS configuration list at the
top of the file.

Also, the SFTP client not skips any files not conforming to the correct
csv file name convention.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1660 School teacher cards use upper-case ID's

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1643 Teacher cards pad ID to 4 chars min.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1598 Student importer exits on CSV parse error

An unterminated quote, for example, now causes the student generator to
stop processing entirely.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

SQL demo server insert protection: payflow org settings

Signed-off-by: Bill Erickson <berickxx@gmail.com>

OS updates apply autoremove first

Experiment running 'autoremove' before 'dist-upgrade' to avoid occaisonal
problems with botched grub installs leading to un-bootable servers.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1504 Mercer Island classroom card file rename

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1504 Mercer Island student/classroom cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1586 Teacher Ecard uses profile 903

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1586 Teacher accounts avoid activity extensions

Avoid applying activity-based expire_date extensions on teacher ecard
accounts, just like student and classroom cards.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1586 Student account gen scripts support --teacher

* Using new Teacher Ecard profile
* 10-year expire
* Default to 1900-01-01 DoB.
* Slightly modified alert message
* Insert 't' between district code and ID in the barcode.

sftp-client-agent.sh treats all CSV files with 'teacher' in the filename
as teacher files.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1586 Teach Ecard new profile

* Entry for permission.grp_tree
* Hold policy configuration to prevent holds for new profile
* Entry in patron editor UI for sorting the new profile in the group selector.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

Copy auditor purge runs 30 mins

Avoid triggering Zabbix alerts about backed replication (and, indeed,
limit clogged replication) by running the copy audit purge for 30 mins
total.  This results in ~3.5M rows deleted per night while still within
the cleanup phase.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1590 Northshore school cards

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1544 Payflow xact balance comparison repair

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1544 Paypal skip silent post URL option

Org unit setting to avoid sending the Paypal Silent Post URL during
secure token generation.  This is useful if you want to use a specific
server for handling silent posts that may not match the hostname of the
catalog paged accessed by the browser.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1588 CC payment balances match active data

Ensure that the total amount to be tracked as payments in EG after a
successfull PayPal payment match the PP payment amount.  Exit early and
return a non-success to PP when the totals don't match.

This imbalance can occur when a transaction is paid by some other client
(secondary PP instance, staff client) after the PP transaction has been
initiated with the original transaction balance owed amount.

IOW, don't accept payments for transactions that were paid off through
some other mechanism while the PP transaction was in process.

Signed-off-by: Bill Erickson <berickxx@gmail.com>

JBAS-1587 CC always avoids negative transactions

At the middle layer, prevent the credit card transaction retrieval code
from ever accessing negative transactions.  This fixes a bug where
patrons could accidentally attempt to pay against negative transactions
by navigating directly to /eg/opac/biblio/main_payment_form with no
transactions provided in the URL, defaulting to all open (including
negative) transactions.

Signed-off-by: Bill Erickson <berickxx@gmail.com>