working/random.git
9 years agoLog last processed record collab/berick/pingest
Bill Erickson [Wed, 13 Apr 2016 14:29:04 +0000 (10:29 -0400)]
Log last processed record

With each batch completed log message, log the ID of the last record
processed.  Useful for multiple runs of the script.

Signed-off-by: Bill Erickson <berickxx@gmail.com>
9 years agoLog more info on batch completion
Bill Erickson [Tue, 12 Apr 2016 15:49:04 +0000 (11:49 -0400)]
Log more info on batch completion

Log the last record processed for non-browse reingest.  Make logging
more consistent across browse and non-browse reingest stages.

Signed-off-by: Bill Erickson <berickxx@gmail.com>
9 years agoLogging consolidation and syslog support
Bill Erickson [Tue, 12 Apr 2016 14:43:17 +0000 (10:43 -0400)]
Logging consolidation and syslog support

Log all messages to syslog.  See $syslog_* variables to modify behavior.
Add support for --log-stdout option, which causes all log messages to be
cloned to STDOUT.

Signed-off-by: Bill Erickson <berickxx@gmail.com>
9 years agoMin/Max ID clarification; --sort-id-desc
Bill Erickson [Tue, 12 Apr 2016 14:18:36 +0000 (10:18 -0400)]
Min/Max ID clarification; --sort-id-desc

Replace start-id/end-id arguments with min-id/max-id to avoid any
confusion when using descending sort options.

Add support for --sort-id-desc to process records in reverse order of
record ID.  This works better across multiple batches when you don't
know what both the min/max ID values are goign to be ahead of time.

Signed-off-by: Bill Erickson <berickxx@gmail.com>
9 years agopingest --newest-first option
Bill Erickson [Wed, 2 Mar 2016 15:57:56 +0000 (10:57 -0500)]
pingest --newest-first option

When set, --newest-first causes the script to processes records in
descending order of edit_date, so that more recently edited/created
records are ingested first.

Signed-off-by: Bill Erickson <berickxx@gmail.com>
9 years agopingest supports max/min ID, duration, more ops
Bill Erickson [Tue, 16 Feb 2016 17:59:55 +0000 (12:59 -0500)]
pingest supports max/min ID, duration, more ops

From the new help text:

    --batch-size
        Number of records to process per batch

    --max-child
        Max number of worker processes

    --skip-browse
    --skip-attrs
    --skip-search
    --skip-facets
        Skip the selected reingest component

    --start-id
        Start processing at this record ID.

    --end-id
        Stop processing when this record ID is reached

    --max-duration
        Stop processing after this many total seconds have passed.

    --help
        Show this help text.

Signed-off-by: Bill Erickson <berickxx@gmail.com>
9 years agoCloning pingest from git.mvlcstaff.org/jason/evergreen_utilities.git
Bill Erickson [Tue, 16 Feb 2016 18:09:18 +0000 (13:09 -0500)]
Cloning pingest from git.mvlcstaff.org/jason/evergreen_utilities.git

Signed-off-by: Bill Erickson <berickxx@gmail.com>
13 years agoAdd quick README to the master branch master
Thomas Berezansky [Thu, 22 Mar 2012 17:06:59 +0000 (13:06 -0400)]
Add quick README to the master branch

Otherwise the default clone branch will be empty.

Signed-off-by: Thomas Berezansky <tsbere@mvlc.org>