window.manualConvertUnicodeToUTF8 = function (s)\n{\n var re=/[^\su0000-\su007F]/g ;\n return( s.replace( re, function( $0 ) { return( "&#" +\n$0.charCodeAt(0).toString() + ";" ) ; }));\n\n}

[[Introduction]]

\nfunction onClickToolbarCloseOthers(e) {\nif (!e) var e = window.event;\noldVal = config.options.chkAnimate;\nconfig.options.chkAnimate = false;\ncloseAllTiddlers();\nif(this.parentNode.id) {\nvar title = this.parentNode.id.substr(7);\ndisplayTiddler(e.target,title,0);\n}\nconfig.options.chkAnimate = oldVal;\n}\n\nfunction addCloseOthersButton(ignored, args) {\nvar theToolbar = document.getElementById("toolbar" + args[0]);\nif(theToolbar) {\nif(!args[1]) {\ncreateTiddlyButton(theToolbar, "close others", "close others", onClickToolbarCloseOthers);\ninsertSpacer(theToolbar);\n}\n}\n\nreturn args;\n}\n\nAspects.addAfter(this, "createTiddlerToolbar", addCloseOthersButton);

//Thanks to Roman Porotnikov\n//http:www.jroller.com/page/deep/20030701\n\n//NB this systemConfig needs to be evaluated before other ones \n//that use the Aspects so the name starts with "."\n//since they're loaded alphabetically\n//should really put it into the main source code, but\n//wanted to keep everything upgrade-proof\n\nAspects = new Object();\n\nAspects.addBefore = function(obj, fname, before) {\nvar oldFunc = obj[fname];\nobj[fname] = function() {\nreturn oldFunc.apply(this, before(arguments, oldFunc, this));\n};\n};\n\nAspects.addAfter = function(obj, fname, after) {\nvar oldFunc = obj[fname];\nobj[fname] = function() {\nreturn after(oldFunc.apply(this, arguments), arguments, oldFunc, this);\n};\n};\n\nAspects.addAround = function(obj, fname, around) {\nvar oldFunc = obj[fname];\nobj[fname] = function() {\nreturn around(arguments, oldFunc, this);\n};\n};

///////////////////////////////////////////////////////////////\n// Requires http://www.digitaldimsum.co.uk/#_.FunctionDecorator\n///////////////////////////////////////////////////////////////\nconfig.options.txtBackupDir = "backup";\nconfig.shadowTiddlers.AdvancedOptions += "\sn\sn Backup Subdirectory: <<option txtBackupDir>>\sn ";\n\nfunction alterBackupDir(args) {\n var path = args[0];\n var re = new RegExp("\s.[0-9]+\s.html");\n if (re.test(path)) {\n var backSlash = true;\n var dirPathPos = path.lastIndexOf("\s\s");\n if (dirPathPos == -1) {\n dirPathPos = path.lastIndexOf("/");\n backSlash = false;\n }\n var backupPath = path.substr(0,dirPathPos) + (backSlash ? "\s\s" : "/");\n backupPath += config.options.txtBackupDir + path.substr(dirPathPos, path.length - dirPathPos);\n \n args[0] = backupPath;\n }\n return args;\n}\n\nAspects.addBefore(this,"saveFile",alterBackupDir);\n

See [[TiddlyWiki.com|http://www.tiddlywiki.com]] for descriptions, examples, and tutorials on TiddlyWiki.\n\nIf you want to create your own TiddlyWiki page, you can start by saving a copy of [[this file|http://avweb.etxb.ericsson.se/~etxuwig/empty_tiddlywiki.html]].\n

RDBMS

A relational database layer on top of Mnesia

'Rdbms' is mainly a mnesia activity module for Mnesia that adds support for constraint checking.\n\nMnesia is a robust distributed DBMS, but it performs very little validation on the actual data. Basically, the only things that it does validate (apart from ensuring ACID properties, which is no small thing) is that the record tag and number of attributes are correct. There is no support for verifying the type of data, or indeed to check relations between different tables.\n\n'Rdbms' does this, in a way that is almost totally transparent to the Mnesia user. It supports definition and validation of type, bounds, access rights, and relational constraints.

This document describes the 'rdbms' add-on to Mnesia. I've decided to arrange it as a TiddlyWiki in order to attempt a [[User Guide]], [[Reference Manual]], [[design log]], and [[tutorials]] in one single document. We'll se how it goes.\n\nA good starting point is [[What is rdbms?]]\n

in ''rdbms_index''. For example, read-only file system tables would be difficult to index. If one were to allow "partially complete" indexes, one could allow ''update_on_read'' indexes. Basically, one would [[rebuild_index]] once, and then refresh a portion of the index every time an object is read. This will not guarantee that the index is up to date, but should keep it mostly accurate, as long as updates are reasonably rare.\n

In [[module rdbms_index_load]], there is code for creating an index table on the fly. We should perhaps have a function, ''rebuild_indexes'', that does the same thing on demand for existing indexes.

This module is a callback for the mnesia_schema:do_restore/1 function, which is used by [[module rdbms_index]] in order to (re-)build an index at runtime. The reason for doing it this way, is that normal writes are not allowed within a schema transaction (which, for various reasons, is probably a good thing.)

[[Data model]]\n\nThe following API functions exist in rdbms:\n- [[module rdbms]]\n- [[module rdbms_index]]\n- [[module rdbms_props]]\n- [[module rdbms_mailmerge]]\n- [[module rdbms_wsearch]]

''Rdbms'' metadata can be specified while creating tables, using the following functions in [[module rdbms]]:\n<<<\nrdbms:create_table(Name, Options) % where Options = [{K,V}], rdbms metadata given as {rdbms, Properties}\nrdbms:do_create_table(Name, Options) \n<<<\n\nand for existing tables using the following functions in [[module rdbms_props]]:\n\n<<<\nset_property(Tab, Key, Value)\ndo_set_property(Tab, Key, Value)\nset_global_type(Class, Name, Value)\ndo_set_global_type(Class, Name, Value)\n<<<\n\nThe following types of table metadata can be specified:\n- [[references]]\n- [[acl]]\n- [[verify]]\n- [[{typedef, Name}]]\n- [[{attr, Attr, Prop}]]\n

This defines an attribute property for the given table (or a global attribute property).\n\nFor a table property, ''Attr'' must refer to an existing attribute in the table, or to a "compound_attribute" (see below).\n\nThe following properties can be specified for table attributes:\n\n- ''type'' - see [[attribute types]]\n- ''required'' - ''true | false''\n- ''key_type'' - ''primary | secondary | none''\n- ''default'' - ''{'value', term()} | {'auto', {M, F}}'')

- [[rebuild_indexes]], [[update_on_read]]\n- [[optimize type checks]]\n- [[rdbms_rofs]]

There is some code in [[module rdbms_props]] that normalizes type definitions. Part of the purpose is to expand all typedefs, but another reason is to try to simplify the type expressions. Obviously, e.g. {list, {'or', [..., any]}} can be simplified to {list, any}, which means that the list elements do not 't have to be inspected. This code lacks sophistication (to say the least), and could be made much better.\n\nBTW, should {list, any} be read as "a non-empty list", or "any list, including nil"? I think I prefer the former, since we have no other good expression for non-empty list.

<<<\n''type'' = builtinType | complexType | altType\n''logicalType'' = {logicalOp, [type]} | {'not', type} | 'true' | 'false'\n''logicalOp'' = 'and' | 'or' | 'andalso' | 'orelse'\n''comType'' = {compOp, value}\n''compOp'' = '==' | '=/=' | '>' | '<' | '=<' | '>='\n''builtinType'' = 'atom' | 'string' | 'binary' | 'number' | 'integer' | 'float' | 'oid' | 'any' |\n 'list' | 'nil' | 'tuple' | 'function' | 'pid' | 'port' | 'reference'\n''complexType'' = tupleType | listType | enumType\n''tupleType'' = {'tuple', [type]}\n''listType'' = {'list', type}\n''enumType'' = {'enum', [value]}\n<<<\n\nSpecifically, ''{'and', []}'' and ''{'or',[]}'' are the same as ''no_type'', and \n- '''nil''' means the empty list\n- ''{'list', 'false'}'' is the same as '''nil'''\n- ''{'list', 'any'} and ''{'list', 'undefined'}'' both mean a non-empty list

The following types are supported by ''rdbms'':\n\n- ''undefined'' - specifically matches the value 'undefined'\n- ''no_type'' - is an internal representation for "no type has been defined". It cannot be set.\n- ''any'' - means specifically that any value is allowed.\n- ''oid'' - built-in type for use as globally unique identifier. See [[oid]]\n- Simple builtin types (''atom, string, binary, number, integer, float, oid, any, pid, port, reference, list, nil, tuple, function, boolean'')\n- The distinct types 'true' and 'false'\n- [[global type]] (''{type, typeRef}'', see TypeDefs)\n- [[complex type]] (tuples or lists)\n

This user guide covers the following topics\n- [[activating rdbms]]\n- [[specifying types]]\n- [[referential integrity]]\n- [[parameterized indexes]]\n- [[fragmented tables]]

[[Introduction]]\n[[What is rdbms?]]\n[[Reference Manual]]\n[[User Guide]]\n[[Ongoing work]]\nTiddlyWiki\n[[TiddlyWiki Tutorial|http://www.blogjones.com/TiddlyWikiTutorial.html]]\n\n\n<<newTiddler>>\n<<newJournal "DD MMM YYYY">>

The 'rdbms' library supports parameterized indexes. These indexes are regular mnesia tables, and index values are determined by a user-supplied callback function, which operates either on a single attribute, or on the whole object. This is the main difference between 'rdbms' indexes and mnesia indexes, for which the index value is always just the value of the identified attribute.\n\nThe 'rdbms' indexes can also be ''fragmented'', since they are regular mnesia tables. This should lead to better scalability in cluster databases.\n\nThe following types of index are supported:\n* ''bag'' - unordered indexes of the sort mnesia supports today (except for the differences mentioned above.)\n* ''ordered'' - ordered_set indexes where the key is {{{{~IndexValue, ~ObjectKey}}}}\n* ''set'' - like ''bag'', but with the restriction that the index value must be present in only one object.\n* ''weighted'' - also ordered_sets. The index function is expected to return {{{[{~IndexValue, Weight}]}}}, where {{{Weight}}} can be any term. The key in the index table becomes {{{{~IndexValue, Weight, ~ObjectKey}}}}

The rdbms module implements the actual activity module for mnesia. To enable rdbms, add the environment variable {{{{access_module, rdbms}}}} to Mnesia. It is also possible to select (or override) rdbms for each transaction through the function {{{mnesia:activity(Type, Fun, Args, Module)}}}.\n\nRdbms uses the naming convention that {{{do_xxxxx(...)}}} signifies a function that must be run within a schema transaction. Normally, there is one function that starts a schema transaction implicitly (e.g. {{{add_properties/1}}}, and a corresponding function that assumes an existing scheme transaction (e.g. {{{do_add_properties/1}}}). The idea is to be able to perform complex data initialization (possibly creating the entire database) within one transaction. Note, however, that Mnesia does not allow insertion of normal data within a schema transaction.\n\nFunctions:\n* [[rdbms:activity/1]]\n* [[rdbms:drop_references/1]], [[rdbms:do_drop_references/1]]\n* [[rdbms:load_schema/1]], [[rdbms:load_schema/2]]\n* [[rdbms:create_table/2]], [[rdbms:do_create_table/2]]\n* [[rdbms:delete_table/1]], [[rdbms:do_delete_table/1]]\n* [[rdbms:make_simple_form/1]]\n* [[rdbms:null_value/0]]

{{{rdbms:null_value() -> undefined}}}\n\nErlang doesn't have a proper {{{null}}} value, but the convention is to use the atom {{{undefined}}}.

{{{rdbms:activity(Fun)}}}\n\nEquvalent to {{{mnesia:activity(transaction, Fun, [], rdbms)}}}.\n\nStarts a mnesia transaction, using the 'rdbms' activity module. If mnesia has been started with the environment varliable {{{{access_module, rdbms}}}}, the function {{{mnesia:activity(transaction, Fun)}}} will also have the same effect. If another activity module is the default, {{{rdbms:activity(Fun)}}} will override the setting, and use 'rdbms' instead.

{{{rdbms:create_table(Tab, Options) -> {atomic, ok} | {aborted, Reason}}}}\n\nWorks like {{{mnesia:create_table(Tab, Options)}}}, except that it accepts the additional option {{{{rdbms, ~RdbmsOptions}}}}.\n

The rdbms_props module contains functions for manipulating and accessing meta-data.\n\n* Table Properties\n** [[references]]\n** [[indexes]]\n** [[acl]]\n** [[read_filter]]\n** [[write_filter]]\n*Attribute Properties\n** [[{attr,Attr,type}]]\n** [[{attr,Attr,default}]]\n

''rdbms'' is activated whenever the {{{mnesia:activity()}}} function is called with ''rdbms'' as callback module. For example:\n\n<<<\n{{{mnesia:activity(transaction, F, [], rdbms)}}}\n<<<\n\nwhere F is the function to execute within the transaction.\n\n<<<\nNote that the old {{{mnesia:transaction(F)}}} function _never_ uses any callback module other than the default (mnesia).\n<<<\n\nAnother way to activate ''rdbms'' is to specify the mnesia environment variable {{{{access_module,rdbms}}}}. This can be done in three different ways.\n\n- from the command line: {{{erl ... -mnesia access_module rdbms}}}\n- from the sys.config file: {{{{mnesia, [{access_module, rdbms}]}}}}\n- by starting mnesia with {{{mnesia:start([{access_module,rdbms}])}}}\n\nWhen ''rdbms'' has been activated as the default access module, it can still be overridden using the mnesia:activity/4 function.

One of the main contributions of 'rdbms' is ''parameterized indexes''.\n\nMnesia allows defining an index on any non-key attribute. The following restrictions apply:\n* Only one index per attribute\n* The index is unsorted with 'bag' semantics\n* The index value is always the exact value of the given attribute\n* For fragmented tables, each fragment gets its own index, and an index lookup on the table is broadcast to all nodes that have fragments.\n\nThe 'rdbms' indexes have the following properties:\n* Several (named) indexes can exist on each attribute, or on the whole record\n* The index values are determined by a user-supplied callback function\n* Indexes can be sorted or unsorted, bags or sets. A special type is /weighted index/\n* An index can be specified as 'unique', meaning that each index value can only point to at most one object\n* Indexes are regular mnesia tables, and can have their own replication and fragmentation properties\n\nTo illustrate the use of parameterized indexes, observe the SNMP attribute for mnesia tables. It is implemented as a separate "indexing" feature inside mnesia, specific to SNMP, and not using the built-in indexing functionality. The SNMP oid "index" must be composed from the primary key, something that essentially forces the designers to create tables customized for SNMP. Had the 'rdbms' indexing support existed, the SNMP index could have either been composed from any combination of existing attribute values (or data derived from existing values) - or additional non-key attributes could have been added to define the SNMP index. This would have been much less intrusive on the application design.\n\nAnother interesting use of parameterized indexes is to extract the words, or word stems, from attributes containing text. This is not possible with the standard mnesia indexes, but with 'rdbms' indexes, it's rather straightforward.\n\n(Since indexes are regular mnesia tables, it ought to be possible to create indexes for them as well. The author has been unable so far to think of a practical use for this.)\n\nAdding an index is either done through the [[rdbms:create_table/2]] function, or by calling [[rdbms_index:do_add_indexes/2]]. Each index is specified with a tuple:\n\n<<<\n{{~AttrPos, ~IndexName}, Module, Function, ~XtraArg, Options}\n<<<\n\nIf the index is to act on the whole object, AttrPos = 1. Otherwise, AttrPos should be an integer or an atom (attribute name) indicating the position in the object.\n\nThe indexing function is called as {{{Module:Function(Data, XtraArg) -> [IxValue]}}}.\nIt should return a list of index values. Data is either the whole object, or the value of an attribute. XtraArg is statically defined when the index is created.\n\nThe following utility functions have been predefined in [[module rdbms.erl]]:\n* {{{ix(Value,[]) -> [Value].}}}\n* {{{ix_list(Value, []) -> Value}}}, "making each element in a list into an index value\n* {{{ix_vals(Obj, ~PosList) -> [ [element(P,Obj) || P <- ~PosList] ]}}}\n