From 1f1ebf26dfa0345e64966de1e82ccc286c7b7478 Mon Sep 17 00:00:00 2001 From: Remington Steed Date: Wed, 15 Jan 2020 08:59:43 -0500 Subject: [PATCH] Docs: Fix sections for Antora: "Developer Resources" and "Getting Data from Evergreen" This commit does the known tasks to prepare each main docs section for Antora: - Add chapter labels to nav file items - To match existing docs "chapter" level headings, combine adoc files as needed so that each "chapter" from the old docs structure has a single file linked from the nav - Upgrade all heading levels in each adoc file so that the top-most heading is a "Level 1" heading (example: "= Chapter Name =") - Add ":toc:" below top heading of each file NOTE: Previous commit had a duplicate title, but should have said "for section 'Using the Public Access Catalog'". Signed-off-by: Remington Steed --- docs-antora/modules/api/nav.adoc | 6 +-- docs-antora/modules/development/nav.adoc | 9 ++-- .../modules/development/pages/data_opensearch.adoc | 5 +- .../modules/development/pages/data_supercat.adoc | 19 +++---- .../modules/development/pages/data_unapi.adoc | 5 +- .../modules/development/pages/intro_opensrf.adoc | 61 +++++++++++----------- .../modules/development/pages/introduction.adoc | 3 +- docs-antora/modules/development/pages/pgtap.adoc | 7 +-- .../modules/development/pages/support_scripts.adoc | 29 +++++----- .../pages/updating_translations_launchpad.adoc | 7 +-- 10 files changed, 80 insertions(+), 71 deletions(-) diff --git a/docs-antora/modules/api/nav.adoc b/docs-antora/modules/api/nav.adoc index 0d713e0af1..2ce92424f3 100644 --- a/docs-antora/modules/api/nav.adoc +++ b/docs-antora/modules/api/nav.adoc @@ -1,5 +1,5 @@ * xref:api:introduction.adoc[Getting Data from Evergreen] -** xref:development:data_supercat.adoc[] -** xref:development:data_unapi.adoc[] -** xref:development:data_opensearch.adoc[] +** xref:development:data_supercat.adoc[Using Supercat] +** xref:development:data_unapi.adoc[Using UnAPI] +** xref:development:data_opensearch.adoc[Using Opensearch as a developer] diff --git a/docs-antora/modules/development/nav.adoc b/docs-antora/modules/development/nav.adoc index dd0c451b42..4c9464e4ee 100644 --- a/docs-antora/modules/development/nav.adoc +++ b/docs-antora/modules/development/nav.adoc @@ -1,6 +1,7 @@ * xref:development:introduction.adoc[Developer Resources] -** xref:development:support_scripts.adoc[] -** xref:development:pgtap.adoc[] -** xref:development:intro_opensrf.adoc[] -** xref:development:updating_translations_launchpad.adoc[] +** xref:development:support_scripts.adoc[Support Scripts] +** xref:development:pgtap.adoc[Developing with pgTAP tests] +** xref:development:intro_opensrf.adoc[Easing gently into OpenSRF] +** xref:development:updating_translations_launchpad.adoc[Updating translations +using Launchpad] diff --git a/docs-antora/modules/development/pages/data_opensearch.adoc b/docs-antora/modules/development/pages/data_opensearch.adoc index 218410a380..9e2a1514d7 100644 --- a/docs-antora/modules/development/pages/data_opensearch.adoc +++ b/docs-antora/modules/development/pages/data_opensearch.adoc @@ -1,6 +1,7 @@ -== Using Opensearch as a developer == += Using Opensearch as a developer = +:toc: -=== Introduction === +== Introduction == Evergreen responds to OpenSearch requests. This can be a good way to get search results delivered in a format that you prefer. diff --git a/docs-antora/modules/development/pages/data_supercat.adoc b/docs-antora/modules/development/pages/data_supercat.adoc index 0cea5f52bd..ff4489c9b4 100644 --- a/docs-antora/modules/development/pages/data_supercat.adoc +++ b/docs-antora/modules/development/pages/data_supercat.adoc @@ -1,6 +1,7 @@ -== Using Supercat == += Using Supercat = +:toc: -=== Introduction === +== Introduction == You can use SuperCat to get data about ISBNs, metarecords, bibliographic records, and authority records. @@ -8,7 +9,7 @@ records, and authority records. Throughout this section, replace `` with the domain or subdomain of your Evergreen installation to try these examples on your own system. -=== ISBNs === +== ISBNs == Given one ISBN, Evergreen can return a list of related records and ISBNs, including alternate editions and translations. To use the Supercat @@ -56,9 +57,9 @@ the following list of catalog record IDs and ISBNs: ---------------------------------------------------------------------------- -=== Records === +== Records == -==== Record formats ==== +=== Record formats === First, determine which format you'd like to receive data in. To see the available formats for bibliographic records, visit @@ -110,7 +111,7 @@ data for a given bibliographic record. ============================================================================ -==== Retrieve records ==== +=== Retrieve records === You can retrieve records using URLs in the following format: ---- @@ -197,7 +198,7 @@ returns the following record. ---------------------------------------------------------------------------- -==== Recent records ==== +=== Recent records === SuperCat can return feeds of recently edited or created authority and bibliographic records: @@ -213,7 +214,7 @@ Note the following features: Example: http://gapines.org/opac/extras/feed/freshmeat/atom/biblio/import/10/2008-01-01 -===== Filtering by Org Unit ===== +==== Filtering by Org Unit ==== You can generate a similar list, with the added ability to limit by Org Unit, using the item-age browse axis. @@ -235,7 +236,7 @@ Modifying the 'atom' portion of the URL to 'html-full' will produce an HTML page Example: http://gapines.org/opac/extras/browse/html-full/item-age/ARL-BOG/1/10 -===== Additional Filters ===== +==== Additional Filters ==== If you'd like to limit to a particular status, you can append `?status=0` where `0` is the ID number of the status you'd like to limit to. If a diff --git a/docs-antora/modules/development/pages/data_unapi.adoc b/docs-antora/modules/development/pages/data_unapi.adoc index cd9a27146a..5d6fcb18b4 100644 --- a/docs-antora/modules/development/pages/data_unapi.adoc +++ b/docs-antora/modules/development/pages/data_unapi.adoc @@ -1,6 +1,7 @@ -== Using UnAPI == += Using UnAPI = +:toc: -=== URL format === +== URL format == Evergreen's unAPI support includes access to many record types. For example, the following URL would fetch diff --git a/docs-antora/modules/development/pages/intro_opensrf.adoc b/docs-antora/modules/development/pages/intro_opensrf.adoc index e1be38f374..d512978569 100644 --- a/docs-antora/modules/development/pages/intro_opensrf.adoc +++ b/docs-antora/modules/development/pages/intro_opensrf.adoc @@ -1,6 +1,7 @@ -== Easing gently into OpenSRF == += Easing gently into OpenSRF = +:toc: -=== Abstract === +== Abstract == The Evergreen open-source library system serves library consortia composed of hundreds of branches with millions of patrons - for example, http://www.georgialibraries.org/statelibrarian/bythenumbers.pdf[the Georgia @@ -11,7 +12,7 @@ article introduces OpenSRF, demonstrates how to build OpenSRF services through simple code examples, and explains the technical foundations on which OpenSRF is built. -=== Introducing OpenSRF === +== Introducing OpenSRF == OpenSRF is a message routing network that offers scalability and failover support for individual services and entire servers with minimal development and deployment overhead. You can use OpenSRF to build loosely-coupled applications @@ -48,7 +49,7 @@ of your servers; or if the problem is that your service is resource-hungry, you could add an inexpensive server to your cluster and dedicate it to running that resource-hungry service. -==== Programming language support ==== +=== Programming language support === If you need to develop an entirely new OpenSRF service, you can choose from a number of different languages in which to implement that service. OpenSRF @@ -79,7 +80,7 @@ The recommended option for would-be developers of another language binding is to use the Python implementation as the cleanest basis for a port to another language. -==== OpenSRF communication flows over XMPP ==== +=== OpenSRF communication flows over XMPP === The XMPP messaging service underpins OpenSRF, requiring an XMPP server such as http://www.ejabberd.im/[ejabberd]. When you start OpenSRF, the first XMPP @@ -98,7 +99,7 @@ ID `6285` that has connected to the `private.localhost` domain using the `opensrf@private.localhost/opensrf.simple-text_drone_at_localhost_6285`. [#OpenSRFOverHTTP] -==== OpenSRF communication flows over HTTP ==== +=== OpenSRF communication flows over HTTP === Any OpenSRF service registered with the public router is accessible via the OpenSRF HTTP Translator. The OpenSRF HTTP Translator implements the http://www.open-ils.org/dokuwiki/doku.php?id=opensrf_over_http[OpenSRF-over-HTTP @@ -218,7 +219,7 @@ Evergreen `open-ils.pcrud` public service offers the same functionality as additional authentication and authorization layer in `open-ils.pcrud` prevents unchecked access to Evergreen's data. -==== Stateless and stateful connections ==== +=== Stateless and stateful connections === OpenSRF supports both _stateless_ and _stateful_ connections. When an OpenSRF client issues a `REQUEST` message in a _stateless_ connection, the router @@ -242,7 +243,7 @@ set of database INSERT, UPDATE, and DELETE statements within a transaction. .CONNECT, REQUEST, and DISCONNECT flow in a stateful connection image:media/CONNECT.png[CONNECT, REQUEST, and DISCONNECT flow in a stateful connection] -=== Enough jibber-jabber: writing an OpenSRF service === +== Enough jibber-jabber: writing an OpenSRF service == Imagine an application architecture in which 10 lines of Perl or Python, using the data types native to each language, are enough to implement a method that can then be deployed and invoked seamlessly across hundreds of servers. You @@ -303,7 +304,7 @@ inheritance is available in this package (named by the special Perl symbol procedure is how we introduce a method to the rest of the OpenSRF world. [#serviceRegistration] -==== Registering a service with the OpenSRF configuration files ==== +=== Registering a service with the OpenSRF configuration files === Two files control most of the configuration for OpenSRF: @@ -419,12 +420,12 @@ offers their services via the public router as well as the private router. Once you have defined the new service, you must restart the OpenSRF Router to retrieve the new configuration and start or restart the service itself. -==== Calling an OpenSRF method ==== +=== Calling an OpenSRF method === OpenSRF clients in any supported language can invoke OpenSRF services in any supported language. So let's see a few examples of how we can call our fancy new `opensrf.simple-text.reverse()` method: -===== Calling OpenSRF methods from the srfsh client ===== +==== Calling OpenSRF methods from the srfsh client ==== `srfsh` is a command-line tool installed with OpenSRF that you can use to call OpenSRF methods. To call an OpenSRF method, issue the `request` command and pass the OpenSRF service and method name as the first two arguments; then pass a list @@ -448,7 +449,7 @@ Request Time in seconds: 0.016718 -------------------------------------------------------------------------------- [#opensrfIntrospection] -===== Getting documentation for OpenSRF methods from the srfsh client ===== +==== Getting documentation for OpenSRF methods from the srfsh client ==== The `srfsh` client also gives you command-line access to retrieving metadata about OpenSRF services and methods. For a given OpenSRF method, for example, @@ -527,7 +528,7 @@ implementation. <9> `argc` is an integer describing the minimum number of arguments that must be passed to this method. -===== Calling OpenSRF methods from Perl applications ===== +==== Calling OpenSRF methods from Perl applications ==== To call an OpenSRF method from Perl, you must connect to the OpenSRF service, issue the request to the method, and then retrieve the results. @@ -618,7 +619,7 @@ contains the `length` and `word_count` keys we defined in the method. <10> The `OpenSRF::AppSession->disconnect()` method closes the XMPP client connection and cleans up resources associated with the session. -==== Accepting and returning more interesting data types ==== +=== Accepting and returning more interesting data types === Of course, the example of accepting a single string and returning a single string is not very interesting. In real life, our applications tend to pass @@ -653,7 +654,7 @@ returned to the caller. As a caller of a given method, you must rely on the documentation used to register to determine the data structures - if the developer has added the appropriate documentation. -==== Accepting and returning Evergreen objects ==== +=== Accepting and returning Evergreen objects === OpenSRF is agnostic about objects; its role is to pass JSON back and forth between OpenSRF clients and services, and it allows the specific clients and @@ -810,7 +811,7 @@ the given instance has been deleted, updated, or created respectively. Evergreen can then act in batch mode over the collection to perform the requested actions on any of the instances that have been flagged for action. -==== Returning streaming results ==== +=== Returning streaming results === In the previous implementation of the `opensrf.simple-text.split` method, we returned a reference to the complete array of results. For small values being @@ -867,7 +868,7 @@ connection object on each element of the array. <2> Registering the method as a streaming method instructs OpenSRF to also register an atomic variant (`opensrf.simple-text.split.atomic`). -==== Error! Warning! Info! Debug! ==== +=== Error! Warning! Info! Debug! === As hard as it may be to believe, it is true: applications sometimes do not behave in the expected manner, particularly when they are still under development. The server language bindings for OpenSRF include integrated @@ -934,7 +935,7 @@ To see everything that is happening in OpenSRF, try leaving your logging level set to INTERNAL for a few minutes - just ensure that you have a lot of free disk space available if you have a moderately busy system! -==== Caching results: one secret of scalability ==== +=== Caching results: one secret of scalability === If you have ever used an application that depends on a remote Web service outside of your control-say, if you need to retrieve results from a microblogging service-you know the pain of latency and dependability (or the @@ -986,7 +987,7 @@ entry, with the cache key, value to be stored ("here"), and the timeout value in seconds to ensure that we do not return stale data on subsequent calls -==== Initializing the service and its children: child labour ==== +=== Initializing the service and its children: child labour === When an OpenSRF service is started, it looks for a procedure called `initialize()` to set up any global variables shared by all of the children of the service. The `initialize()` procedure is typically used to retrieve @@ -1003,7 +1004,7 @@ procedure (if any) to clean up any resources associated with the child process. Similarly, when the OpenSRF service is stopped, it calls the `DESTROY()` procedure to clean up any remaining resources. -==== Retrieving configuration settings ==== +=== Retrieving configuration settings === The settings for OpenSRF services are maintained in the `opensrf.xml` XML configuration file. The structure of the XML document consists of a root element `` containing two child elements: @@ -1045,12 +1046,12 @@ libraries locally cache the configuration file to avoid network roundtrips for every request and enable the developer to request specific values without having to manually construct XPath expressions. -=== Getting under the covers with OpenSRF === +== Getting under the covers with OpenSRF == Now that you have seen that it truly is easy to create an OpenSRF service, we can take a look at what is going on under the covers to make all of this work for you. -==== Get on the messaging bus - safely ==== +=== Get on the messaging bus - safely === One of the core innovations of OpenSRF was to use the Extensible Messaging and Presence Protocol (XMPP, more colloquially known as Jabber) as the messaging bus that ties OpenSRF services together across servers. XMPP is an "XML @@ -1078,7 +1079,7 @@ The opensrf and router user names, passwords, and domain names, along with the list of services that should be public, are contained in the `opensrf_core.xml` configuration file. -==== Message body format ==== +=== Message body format === OpenSRF was an early adopter of JavaScript Object Notation (JSON). While XMPP is an XML protocol, the Evergreen developers recognized that the compactness of the JSON format offered a significant reduction in bandwidth for the volume of @@ -1145,7 +1146,7 @@ The content of the `` element of the OpenSRF request and result should look familiar; they match the structure of the <> that we previously dissected. -==== Registering OpenSRF methods in depth ==== +=== Registering OpenSRF methods in depth === Let's explore the call to `__PACKAGE__->register_method()`; most of the elements of the hash are optional, and for the sake of brevity we omitted them in the previous example. As we have seen in the results of the <>, a @@ -1168,7 +1169,7 @@ that you should pass to `__PACKAGE__->register_method()`: *** `desc`: a description of the return value *** `type`: the data type of the return value: for example, string, integer, boolean, number, array, or hash -=== Evergreen-specific OpenSRF services === +== Evergreen-specific OpenSRF services == Evergreen is currently the primary showcase for the use of OpenSRF as an application architecture. Evergreen 2.6.0 includes the following @@ -1232,7 +1233,7 @@ flexibility as OpenSRF services, but have the disadvantage of not being integrated into the same configuration and control infrastructure as the OpenSRF services. -=== Evergreen after one year: reflections on OpenSRF === +== Evergreen after one year: reflections on OpenSRF == http://projectconifer.ca[Project Conifer] has been live on Evergreen for just over a year now, and as one of the primary technologists I have had to work @@ -1240,7 +1241,7 @@ closely with the OpenSRF infrastructure during that time. As such, I am in a position to identify some of the strengths and weaknesses of OpenSRF based on our experiences. -==== Strengths of OpenSRF ==== +=== Strengths of OpenSRF === As a service infrastructure, OpenSRF has been remarkably reliable. We initially deployed Evergreen on an unreleased version of both OpenSRF and Evergreen due @@ -1284,7 +1285,7 @@ to a single server where we can inspect the complete set of messages for the entire cluster in context, rather than trying to piece them together across servers. -==== Weaknesses ==== +=== Weaknesses === The primary weakness of OpenSRF is the lack of either formal or informal documentation for OpenSRF. There are many frequently asked questions on the @@ -1329,7 +1330,7 @@ framework is in place, more developers might be willing to develop and contribute patches as they could sanity check their own code without an intense effort before exposing it to their peers. -=== Summary === +== Summary == In this article, I attempted to provide both a high-level and detailed overview of how OpenSRF works, how to build and deploy new OpenSRF services, how to make requests to OpenSRF method from OpenSRF clients or over HTTP, and why you @@ -1341,7 +1342,7 @@ Conifer after a year in production, with some thoughts about areas where the right application of skills could make a significant difference to the Evergreen and OpenSRF projects. -=== Appendix: Python client === +== Appendix: Python client == Following is a Python client that makes the same OpenSRF calls as the Perl client: diff --git a/docs-antora/modules/development/pages/introduction.adoc b/docs-antora/modules/development/pages/introduction.adoc index 81ea47416f..8fd3a0a5de 100644 --- a/docs-antora/modules/development/pages/introduction.adoc +++ b/docs-antora/modules/development/pages/introduction.adoc @@ -1,4 +1,5 @@ -== Introduction == += Introduction = +:toc: Developers can use this part to learn more about the programming languages, communication protocols and standards used in Evergreen. diff --git a/docs-antora/modules/development/pages/pgtap.adoc b/docs-antora/modules/development/pages/pgtap.adoc index b86fd2f2ad..0b8a15677c 100644 --- a/docs-antora/modules/development/pages/pgtap.adoc +++ b/docs-antora/modules/development/pages/pgtap.adoc @@ -1,6 +1,7 @@ -== Developing with pgTAP tests == += Developing with pgTAP tests = +:toc: -=== Setting up pgTAP on your development server === +== Setting up pgTAP on your development server == Currently, Evergreen pgTAP tests expect a version of pgTAP (0.93) that is not yet available in the packages for most Linux distributions. @@ -19,7 +20,7 @@ Therefore, you will have to install pgTAP from source as follows: CREATE EXTENSION pgtap; ------------------------------------------------------------------------------ -=== Running pgTAP tests === +== Running pgTAP tests == The pgTAP tests can be found in subdirectories of `Open-ILS/src/sql/Pg/` as follows: diff --git a/docs-antora/modules/development/pages/support_scripts.adoc b/docs-antora/modules/development/pages/support_scripts.adoc index 9167894140..04e993cb36 100644 --- a/docs-antora/modules/development/pages/support_scripts.adoc +++ b/docs-antora/modules/development/pages/support_scripts.adoc @@ -1,4 +1,5 @@ -== Support Scripts == += Support Scripts = +:toc: Various scripts are included with Evergreen in the `/openils/bin/` directory (and in the source code in `Open-ILS/src/support-scripts` and @@ -59,7 +60,7 @@ to more thorough documentation, if available. [#authority_control_fields] -=== authority_control_fields: Connecting Bibliographic and Authority records === +== authority_control_fields: Connecting Bibliographic and Authority records == indexterm:[authority control] @@ -97,7 +98,7 @@ Here is how the matching works: [#marc_export] -=== marc_export: Exporting Bibliographic Records into MARC files === +== marc_export: Exporting Bibliographic Records into MARC files == indexterm:[marc_export] indexterm:[MARC records,exporting,using the command line] @@ -144,12 +145,12 @@ cat /home/opensrf/records.txt | ./marc_export --store -i -c /openils/conf/opensr `marc_export` does not output progress as it executes. ==================== -==== Options ==== +=== Options === The *marc_export* support script includes several options. You can find a complete list by running `./marc_export -h`. A few key options are also listed below: -===== --descendants and --library ===== +==== --descendants and --library ==== The `marc_export` script has two related options, `--descendants` and `--library`. Both options take one argument of an organizational unit @@ -170,7 +171,7 @@ All of the specified org. units and their descendants will be included in the output. You can also combine `--library` and `--descendants` options when necessary. -===== --items ===== +==== --items ==== The `--items` option will add an 852 field for every relevant item to the MARC record. This 852 field includes the following information: @@ -193,18 +194,18 @@ record. This 852 field includes the following information: |=================================== -===== --since ===== +==== --since ==== You can use the `--since` option to export records modified after a certain date and time. -===== --store ===== +==== --store ==== By default, marc_export will use the reporter storage service, which should work in most cases. But if you have a separate reporter database and you know you want to talk directly to your main production database, then you can set the `--store` option to `cstore` or `storage`. -===== --uris ===== +==== --uris ==== The `--uris` option (short form: `-u`) allows you to export records with located URIs (i.e. electronic resources). When used by itself, it will export only records that have located URIs. When used in conjunction with `--items`, @@ -217,7 +218,7 @@ holdings both physical and electronic. [#pingest_pl] -=== Parallel Ingest with pingest.pl === +== Parallel Ingest with pingest.pl == indexterm:[pgingest.pl] indexterm:[MARC records,importing,using the command line] @@ -234,7 +235,7 @@ NOTE: The browse ingest is presently done in a single process over all of the input records as it cannot run in parallel with itself. It does, however, run in parallel with the other ingests. -==== Command Line Options ==== +=== Command Line Options === pingest.pl accepts the following command line options: @@ -308,7 +309,7 @@ SELECT reporter.refresh_materialized_simple_record(); [#importing_authority_records_from_command_line] -=== Importing Authority Records from Command Line === +== Importing Authority Records from Command Line == indexterm:[marc2are.pl] indexterm:[pg_loader.pl] @@ -358,7 +359,7 @@ that command looks like: psql -U evergreen -h localhost -d evergreen -f pg_loader-output.sql ---- -=== Juvenile-to-adult batch script === +== Juvenile-to-adult batch script == The batch `juv_to_adult.srfsh` script is responsible for toggling a patron from juvenile to adult. It should be set up as a cron job. @@ -368,7 +369,7 @@ library setting named "Juvenile Age Threshold" (`global.juvenile_age_threshold`) When no library setting value is present at a given patron's home library, the value passed in to the script will be used as a default. -=== MARC Stream Importer === +== MARC Stream Importer == indexterm:[MARC records,importing,using the command line] diff --git a/docs-antora/modules/development/pages/updating_translations_launchpad.adoc b/docs-antora/modules/development/pages/updating_translations_launchpad.adoc index c3dca9badd..9b177395f9 100644 --- a/docs-antora/modules/development/pages/updating_translations_launchpad.adoc +++ b/docs-antora/modules/development/pages/updating_translations_launchpad.adoc @@ -1,10 +1,11 @@ -== Updating translations using Launchpad == += Updating translations using Launchpad = +:toc: This document describes how to update the translations in an Evergreen branch by pulling them from Launchpad, as well as update the files to be translated in Launchpad by updating the POT files in the Evergreen master branch. -=== Prerequisites === +== Prerequisites == You must install all of the Python prerequisites required for building translations, per http://evergreen-ils.org/dokuwiki/doku.php?id=evergreen-admin:customizations:i18n @@ -16,7 +17,7 @@ http://evergreen-ils.org/dokuwiki/doku.php?id=evergreen-admin:customizations:i18 * http://pypi.python.org/pypi/simplejson/[simplejson] * http://lxml.de/[lxml] -=== Updating the translations === +== Updating the translations == . Check out the latest translations from Launchpad by branching the Bazaar repository: -- 2.11.0