DBCacheContrib < TWiki

TWiki Web>DBCacheContrib (01 Nov 2019, TWikiAdminUser) (raw view)
---+!! <nop>%TOPIC%
<!--
   Contributions to this contrib are appreciated. Please update the contrib page at
   http://twiki.org/cgi-bin/view/Plugins/DBCacheContrib or provide feedback at
   http://twiki.org/cgi-bin/view/Plugins/DBCacheContribDev.
   If you are a TWiki contributor please update the contrib in the SVN repository.
-->
<sticky>
<div style="float:right; background-color:#EBEEF0; margin:0 0 20px 20px; padding: 0 10px 0 10px;">
%TOC{ title="Page contents" depth="2" }%
</div>
</sticky>
%SHORTDESCRIPTION%

---++ Summary of Contents

This module supports structured queries over a database built on the fly from the forms in TWiki topics. It does not support any tags, as it is provided as a service for other plugins that want to treat a TWiki web as a simple database; for example, the TWiki:Plugins/FormQueryPlugin, which supports the display of query results.

The plugin encapsulates code that was formerly in the "engine room" of the <nop>FormQueryPlugin. It has been abstracted out in the belief that it will be useful to anyone who wants to do simple search operations from a plugin.

---+++ Features
   * Perform complex queries on the TWiki database
   * Cache TWiki topics for rapid queries

---+++ How the database gets built

You can think of the database as an array of all the topics in a web. Each array entry is a map (or hash in perl terms) that maps a set of field names to values.

Each topic in the web automatically gets a number of standard fields, generated by reading the metadata from the topic (see %SYSTEMWEB%.TWikiMetaData)
   * =name=        - name of the topic
   * =parent=      - name of parent topic
   * =_up=         - _reference_ to the Map of the parent topic, if it exists
   * =attachments= - array of Maps, each of which contains:
      *  =_up= - _reference_ to the Map for the topic
      *  =name= - attachment name
      * =attr= - e.g hidden
      * =comment= - attachment comment
      * =path= - client path used to upload attachment
      * =size= - size in Kb
      * =user= - who uploaded the attachment
      * =version= - e.g. 1.3
   * =info= - Map containing:
      * =_up= - reference to the Map for the topic
      * =author= - most recent author
      * =date= - date of last change
      * =format= - topic format version
      * =version= - topic version number
   * =moved= - Map containing:
      * =_up= - reference to the Map for the topic
      * =by= - who moved it
      * =date= - when they moved it
      * =from= - where they moved it from
      * =to= - where they moved it to
   * =form= - form type
   *  _form name_ - e.g. if a "My<nop>Form" is attached, this will be
          =My<nop>Form=. This is a reference to a Map containing a key for each
          field in the form. Each key maps to the value in the form data for
          that key. The Map will also have an =_up= reference to the Map for
          the topic.
   * =text= - raw text of the topic)

The sub-Maps created for =info=, _form name_, =moved=, and each row in =attachments= also have a _reference_ back to the topic Map, called  =_up=.

Other fields may be added by subclasses. Refer to the documentation for the plugin that is using the DBCache for more details.

---+++ The cache

To achieve best perfomance the plugin caches the database read from the TWiki topics. The cache is stored in the work area for the DBCacheContrib (see ={RCS}{WorkAreaDir}= in =configure=). If any topic changes in the web, this cache is automatically rebuilt. The cache file can be deleted at any point with no ill effects.

---+++ Extending or customising

Extension or customisation is welcome, as long as all extensions are described and code provided back to the author.

The module is shipped with a perl build file, which should be used for installation and testing. Testing is done using CPAN:Test::Unit, and is invoked using the 'test' build target. Writing tests is a useful way of feeding back bugs as well. I can't encourage you enough to maintain and extend the tests!

---++ Detailed Documentation

Clients use the DBCache by defining a subclass of the =TWiki::Contrib::DBCacheContrib= class. The following POD documentation describes the methods of this class and the various other classes provided by the plugin..


---++ class DBCacheContrib

General purpose cache that treats TWiki topics as hashes. Useful for
rapid read and search of the database. Only works on one web.

Typical usage:
<verbatim>
  use TWiki::Contrib::DBCacheContrib;

  $db = new TWiki::Contrib::DBCacheContrib( $web ); # always done
  $db->load(); # may be always done, or only on demand when a tag is parsed that needs it

  # the DB is a hash of topics keyed on their name
  foreach my $topic ($db->getKeys()) {
     my $attachments = $db->get($topic)->get("attachments");
     # attachments is an array
     foreach my $val ($attachments->getValues()) {
       my $aname = $attachments->get("name");
       my $acomment = $attachments->get("comment");
       my $adate = $attachments->get("date");
       ...
     }
  }
</verbatim>
As topics are loaded, the readTopicLine method gives subclasses an opportunity to apply special processing to indivual lines, for example to extract special syntax such as %ACTION lines, or embedded tables in the text. See FormQueryPlugin for an example of this.


---+++ =new($web, $cacheName)=
   * =$web= name of web to create the object for.
   * =$cacheName= name of cache file (default "_DBCache")

Construct a new DBCache object.


---+++ readTopicLine($topic, $meta, $line, $fh) --> text
   * $topic - name of the topic being read
   * $meta - reference to the hash object for this topic
   * line - the line being read
   * $fh - the file handle of the file
   * __return__ text to insert in place of _line_ in the text field of the topic
Called when reading a topic that is being cached, this method is invoked on each line
in the topic. It is designed to be overridden by subclasses; the default implementation
does nothing. The sort of expected activities will be (for example) reading tables and
adding them to the hash for the topic.


---+++ onReload($topics)
   * =$topics= - perl array of topic names that have just been loaded (or reloaded)
Designed to be overridden by subclasses. Called when one or more topics had to be
read from disc rather than from the cache. Passed a list of topic names that have been read.


---+++ load( [updateCache]  ) -> ($readFromCache, $readFromFile, $removed)

Load the web into the database.
Returns a list containing 3 numbers that give the number of topics
read from the cache, the number read from file, and the number of previously
cached topics that have been removed.

if  $TWiki::cfg{DBCache}{AlwaysUpdateCache}  is set to FALSE (defaults to TRUE for compatibility)
then avoid calling _updateCache unless requested. DBCachePlugin now only asked for it from
the afterSaveHandler and from the new REST updateCache handler


---++ package TWiki::Contrib::DBCacheContrib::Search

Search operators work on the fields of a TWiki::Contrib::DBCacheContrib::Map.
%STARTSECTION{"searchoperators"}%
Fields are given by name, and values by strings or numbers. Strings should always be surrounded by 'single-quotes'. Strings which are regular expressions (RHS of =, != =~ operators) use 'perl' regular expression syntax (google for =perlre= for help). Numbers can be signed integers or decimals. Single quotes in values may be escaped using backslash (\).

The following operators are available:
| *Operator* | *Result* | *Meaning* |
| <code>=</code> | Boolean | LHS exactly matches the regular expression on the RHS. The expression must match the whole string. |
| <code>!=</code> | Boolean | Inverse of = |
| <code>=~</code> | Boolean | LHS contains RHS i.e. the RHS is found somewhere in the field value. |
| <code>&lt;</code> | Boolean | Numeric < |
| <code>&gt;</code> | Boolean | Numeric > |
| <code>&gt;=</code> | Boolean | Numeric >= |
| <code>&lt;=</code> | Boolean | Numeric <= |
| =lc= | String | Unary lower case |
| =uc= | String | Unary UPPER CASE |
| =EARLIER_THAN= | BOOLEAN | Date is earlier than the given date |
| =LATER_THAN= | Boolean | LHS is later than the given date (string containing a date e.g. '1 Apr 2003') |
| =WITHIN_DAYS= | Boolean | Date (which must be in the future) is within n _working_ days of todays date |
| <code>!</code> | Boolean | Unary NOT |
| =AND= | Boolean | AND |
| =OR= | Boolean | OR |
| <code>()</code> | any | Bracketed subexpression |

Dates for =EARLIER_THAN=, =LATER_THAN= and =WITHIN_DAYS= must be dates in the format expected by =Time::ParseDate= (like the ActionTrackerPlugin). =WITHIN_DAYS= works out the number of _working_ days (i.e. excluding Saturday and Sunday). Apologies in advance if your weekend is offset &plusmn; a day! Integers will automatically be converted to dates, by assuming they represent a number of seconds since midnight GMT on 1st January 1970.

%ENDSECTION{"searchoperators"}%

---+++ Example
Get a list of attachments that have a date earlier than 1st January 2000
<verbatim>
  $db = new TWiki::Contrib::DBCacheContrib::DBCache( $web ); # always done
  $db->load();
  my $search = new TWiki::Contrib::DBCacheContrib::Search("date EARLIER_THAN '1st January 2000'");

  foreach my $topic ($db->getKeys()) {
     my $attachments = $topic->get("attachments");
     foreach my $val ($attachments->getValues()) {
       if ($search->matches($val)) {
          print $val->get("name") . "\n";
       }
     }
  }
</verbatim>

A search object implements the "matches" method as its general
contract with the rest of the world.


---+++ =new($string)=
   * =$string= - string containing an expression to parse
Construct a new search node by parsing the passed expression.


---+++ =toString()= -> string
Generates a string representation of the object.


--+++ =addOperator(%oper)
Add an operator to the parser

=%oper= is a hash, containing the following fields:
   * =name= - operator string
   * =prec= - operator precedence, positive non-zero integer.
     Larger number => higher precedence.
   * =arity= - set to 1 if this operator is unary, 2 for binary. Arity 0
     is legal, should you ever need it.
   * =exec= - the handler to implement the new operator


---++ package TWiki::Contrib::DBCacheContrib::FileTime

Object that handles a file/time tuple for use in Storable and
=TWiki::Contrib::DBCacheContrib::Archive=.


---+++ =new($file)=
   * =$file= - filename
Construct from a file name


---+++ =uptodate()= -> boolean
Check the file time against what is seen on disc. Return 1 if consistent, 0 if inconsistent.


---+++ =toString()= -> string
Generates a string representation of the object.


---+++ =write()=
TWiki::Contrib::DBCacheContrib::Archive hook


---+++ =read()=
TWiki::Contrib::DBCacheContrib::Archive hook


---++ package TWiki::Contrib::DBCacheContrib::Array

Generic array object. This is required because perl arrays are not objects,
and cannot be subclassed e.g. for serialisation. To avoid lots of horrid
code to handle special cases of the different perl data structures, we use
this array object instead.


---+++ =new()=
Create a new, empty array object


---+++ =add($object)=
   * =$object= any perl data type
Add an element to the end of the array


---+++ =find($object)= -> integer
   * $object datum of the same type as the content of the array
Uses "==" to find the given element in the array and return it's index


---+++ =remove($index)=
   * =$index= - integer index
Remove an entry at an index from the array.


---+++ =get($key, $root)= -> datum
   * =$k= - key
   * $root - what # refers to
*Subfield syntax*
   * =get("9", $r)= where $n is a number will get the 9th entry in the array
   * =get("[9]", $r)= will also get the 9th entry
   * =get(".9", $r)= will also get the 9th entry
   * =get(".X", $r)= will return the sum of the subfield =X= of each entry
   * =get("[?<i>search</i>]", $r)= will perform the given search over the entries in the array. Always returns an array result, even when there is only one result. For example: <code>[?name='Sam']</code> will return an array of all the entries that have their subfield =name= set to =Sam=.
   * =#= means "reset to root". So =get("#[3]", $r)= will return the 4th entry of $r (assuming $r is an array!).
   * =get("[*X]", $r)= will get a new array made from subfield X of each entry in this array.

Where the result of a subfield expansion is another object (a Map or an Array) then further subfield expansions can be used. For example,
<verbatim>
get("parent.UserTable[?SubTopic='ThisTopic'].UserName", $web);
</verbatim>

See also =TWiki::Contrib::DBCacheContrib::Map= for syntax that applies to maps.


---+++ =size()= -> integer
Get the size of the array


---+++ =sum($field)= -> number
   * =$field= - name of a field in the class of objects stored by this array
Returns the sum of values of the given field in the objects stored in this array.


---+++ =search($search)= -> search result
   * =$search= - TWiki::Contrib::DBCacheContrib::Search object to use in the search
Search the array for matches with the given object.
values. Return a =TWiki::Contrib::DBCacheContrib::Array= of matching entries.


---+++ =getValues()= -> perl array

Get a "perl" array of the values in the array, suitable for use with =foreach=


---+++ =toString($limit, $level, $strung)= -> string
   * =$limit= - recursion limit for expansion of elements
   * =$level= - currentl recursion level
Generates an HTML string representation of the object.


---+++ write($archive)
   * =$archive= - the TWiki::Contrib::DBCacheContrib::Archive being written to
Writes this object to the archive. Archives are used only if Storable is not available. This
method must be overridden by subclasses is serialisation of their data fields is required.


---+++ read($archive)
   * =$archive= - the TWiki::Contrib::DBCacheContrib::Archive being read from
Reads this object from the archive. Archives are used only if Storable is not available. This
method must be overridden by subclasses is serialisation of their data fields is required.


---++ package TWiki::Contrib::DBCacheContrib::Map
Generic map object for mapping names to things. A name is defined as
  =name = \w+ | \w+ "." name=
The . indicates a field reference in a sub-map.
Objects in the map are either strings, or other objects that must
support toString.


---+++ =new($string)=
   * $string - optional attribute string in standard TWiki syntax
Create a new, empty array object. Optionally parse a standard attribute
string containing name=value pairs. The
value may be a word or a quoted string (no escapes!)


---+++ =fastget($k)= -> datum
   * =$k= - key
Get the value for a key, but without any subfield field expansion


---+++ =get($k, $root)= -> datum
   * =$k= - key
   * =$root= what # refers to
Get the value corresponding to key =$k=; return undef if not set.

*Subfield syntax*
   * =get("X",$r)= will get the subfield named =X=.
   * =get("X.Y",$r)= will get the subfield =Y= of the subfield named =X=.
   * =get("[X]",$r) = will get the subfield named =X= (so X[Y] and X.Y are synonymous)..
   * =#= means "reset to root". So =get("#.Y", $r) will return the subfield =Y= of $r (assuming $r is a map!), as will =get("#[Y]"=.

Where the result of a subfield expansion is another object (a Map or an Array) then further subfield expansions can be used. For example,
<verbatim>
get("UserTable[0].Surname", $web);
</verbatim>

See also =TWiki::Contrib::DBCacheContrib::Array= for syntax that applies to arrays.


---+++ =set($k, $v)=
   * =$k= - key
   * =$v= - value
Set the given key, value pair in the map.


---+++ =size()= -> integer
Get the size of the map


---+++ =remove($index)= -> old value
   * =$index= - integer index
Remove an entry at an index from the array. Return the old value.


---+++ =getKeys()= -> perl array

Get a "perl" array of the keys in the map, suitable for use with =foreach=


---+++ =getValues()= -> perl array

Get a "perl" array of the values in the Map, suitable for use with =foreach=


---+++ =search($search)= -> search result
   * =$search= - TWiki::Contrib::DBCacheContrib::Search object to use in the search
Search the map for keys that match with the given object.
values. Return a =TWiki::Contrib::DBCacheContrib::Array= of matching keys.


---+++ =toString($limit, $level, $strung)= -> string
   * =$limit= - recursion limit for expansion of elements
   * =$level= - currentl recursion level
Generates an HTML string representation of the object.


---+++ write($archive)
   * =$archive= - the TWiki::Contrib::DBCacheContrib::Archive being written to
Writes this object to the archive. Archives are used only if Storable is not available. This
method must be overridden by subclasses is serialisation of their data fields is required.


---+++ read($archive)
   * =$archive= - the TWiki::Contrib::DBCacheContrib::Archive being read from
Reads this object from the archive. Archives are used only if Storable is not available. This
method must be overridden by subclasses is serialisation of their data fields is required.



<!--
   * Set STUB = %$STUB%
   * Set SHORTDESCRIPTION = Reusable code that treats TWiki forms as if they were table rows in a database
-->

---++ Installation Instructions

---+++ {DBCache}{AlwaysUpdateCache}

If  $TWiki::cfg{DBCache}{AlwaysUpdateCache}  is set to FALSE (defaults to TRUE for compatibility) then <nop>DBCacheContrib will avoid calling =_updateCache= unless requested. =_updateCache= accesses every topic file in the web, so its an unnecessary performance hit.  <nop>DBCacePlugin now only requests =_updateCache= from the afterSaveHandler and from the new REST updateCache handler.

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server where TWiki is running.

Like many other TWiki extensions, this module is shipped with a fully
automatic installer script written using the Build<nop>Contrib.
   * If you have TWiki 4.2 or later, you can install from the =configure= interface (Go to Plugins->Find More Extensions)
      * See the [[http://twiki.org/cgi-bin/view/Plugins/BuildContribInstallationSupplement][installation supplement]] on TWiki.org for more information.
   * If you have any problems, then you can still install manually from the command-line:
      1 Download one of the =.zip= or =.tgz= archives
      1 Unpack the archive in the root directory of your TWiki installation.
      1 Run the installer script ( =perl &lt;module&gt;_installer= )
      1 Run =configure= and enable the module, if it is a plugin.
      1 Repeat for any missing dependencies.
   * If you are *still* having problems, then instead of running the installer script:
      1 Make sure that the file permissions allow the webserver user to access all files.
      1 Check in any installed files that have existing =,v= files in your existing install (take care *not* to lock the files when you check in)
      1 Manually edit !LocalSite.cfg to set any configuration variables.

%IF{"defined 'SYSTEMWEB'" else="<div class='twikiAlert'>%X% WARNING: SYSTEMWEB is not defined in this TWiki. Please add these definitions to your %MAINWEB%.TWikiPreferences, if they are not already there:<br><pre>   * <nop>Set SYSTEMWEB = %<nop>TWIKIWEB%<br>   * <nop>Set USERSWEB = %<nop>MAINWEB%</pre></div>"}%


---++ Contrib Info

|  Author: | TWiki:Main.CrawfordCurrie |
|  Copyright: | This code is based on an original development of Motorola Inc. and is protected by the following copyrights: <br />&copy; 2002-2003 Motorola Inc. All Rights Reserved. <br />Portions copyright &copy; 2004. Crawford Currie http://www.c-dot.co.uk <br /> &copy; 2004-2011, TWiki:TWiki.TWikiContributor |
|  License: | As required for the publication of all extensions to TWiki, this software is published under the terms of the [[http://www.gnu.org/copyleft/gpl.html][GNU General Public License]]. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. |
|  Version: | 21578 (2011-07-07) |
|  Change History: | <!-- versions below in reverse order -->&nbsp; |
|  2011-07-07: | TWikibug:Item6711: Small doc fix in Config.spec |
|  2010-04-29: | TWikibug:Item6433: Doc improvements |
|  16347 | remove META data from text hash; include META data in all hash. TWiki:Main/MichaelDaum |
|  16346 | caching all topic elements to an =all= field to allow th search in all of the text and the formfields like the normal grep-based SEARCH does. TWiki:Main/MichaelDaum |
|  15868 | fixed WITHIN_DAYS and EARLIER_THAN. TWiki:Main/MichaelDaum |
|  15583 | made query parser pluggable so that other plugins can implement their own predicates. TWiki:Main/MichaelDaum |
|  15019 | added {DBCache}{AlwaysUpdateCache} to remove the updateCache from every operation. TWiki:Main/SvenDowideit |
|  13562 | Bugs:Item3985 - fixed failures with hierarchical webs |
|  13527 | Moved the cache into the extensions work areas, instead of the web directory |
|  12943 | Bugs:Item3659: added automatic conversion of integers to dates |
|  12923 | added REF operator;  added link to web object to hashes;  fixed parent relation to end in WebHome; added "web" property to topic hashes; caching META:PREFERENCES now |
|  11537 | Added lc and uc operators for case-insensitive searches |
|  9303 | TWikibug:Item1844 - don't die on broken symlinks |
|  8682 | TWikibug:Item1580 - one-char fix that makes the difference |
|  8110 | TWikibug:Item663 - formatting and text fixes |
|  7552 | TWikibug:Item997 - test update |
|  7274 | TWikibug:Item719 - onReload() is not a static method. |
|  7262 | TWikibug:Item719 - !MichaelDaum's patch (almost) to correct parameters to onReload |
|  7260 | TWikibug:Item727 - made it clean the form name using normaliseWebTopicName |
|  6353 | TWikibug:Item380 - do as the man says; make all $/ local |
|  5720 | Updated tests |
|  5719 | Fix for correct handling of parent relations |
|  5229 | Small improvement to the way it handles errors from Storable and Archive |
|  5223 | Documentation fixes, adding gifs. |
|  5048 | Cairo readiness |
|  5036 | Split from !SharedCode |
|  5031 | Moving to new name |
|  5030 | About to rename |
|  5019 | Improved topic data model, cleaned up tests |
|  5008 | Added extended access syntax, [?], [*] etc. |
|  5006 | Doc fixes |
|  5005 | Poddified documentation |
|  5003 | Initial version |
|  8 Jul 2004 | Initial version, split out from <nop>FormQueryPlugin |
|  Dependencies: | <table border="1"><tr><th>Name</th><th>Version</th><th>Description</th></tr><tr><td align="left">Time::ParseDate</td><td align="left">&gt;=2003.0211</td><td align="left">Required. Available from [[http://cpan.uwinnipeg.ca/dist/Time-modules][CPAN]].</td></tr><tr><td align="left">Storable</td><td align="left">&gt;=2.07</td><td align="left">Recommended; accelerates cache handling. Available from [[http://cpan.uwinnipeg.ca/dist/Storable][CPAN]]</td></tr></table> |
|  Perl Version: | 5.0 |
|  Plugin Home: | http://TWiki.org/cgi-bin/view/Plugins/%TOPIC% |
|  Feedback: | http://TWiki.org/cgi-bin/view/Plugins/%TOPIC%Dev |
|  Appraisal: | http://TWiki.org/cgi-bin/view/Plugins/%TOPIC%Appraisal |

__Related Topics:__ DBCachePlugin, %SYSTEMWEB%.DeveloperDocumentationCategory, %SYSTEMWEB%.AdminDocumentationCategory, %SYSTEMWEB%.TWikiPreferences
Topic revision: r19 - 01 Nov 2019 - TWikiAdminUser
Account
- Log In
- Register User
Edit
Attach
Copyright © 1999-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
Note: Please contribute updates to this topic on TWiki.org at TWiki:TWiki.DBCacheContrib.