Wed Nov 10 18:09:46 PST 2004
- Previous message: [Slony1-commit] By cbbrowne: New Directory
- Next message: [Slony1-commit] By cbbrowne: Minor changes to comments
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Log Message:
-----------
Add in material from Slony-I wiki that will become an "Administration
Guide."
Added Files:
-----------
slony1-engine/doc/adminguide:
README (r1.1)
SlonyAddThings.txt (r1.1)
SlonyAdministrationScripts.txt (r1.1)
SlonyConcepts.txt (r1.1)
SlonyDDLChanges.txt (r1.1)
SlonyDefineCluster.txt (r1.1)
SlonyDefineSet.txt (r1.1)
SlonyDropThings.txt (r1.1)
SlonyFAQ.txt (r1.1)
SlonyFAQ01.txt (r1.1)
SlonyFAQ02.txt (r1.1)
SlonyFAQ03.txt (r1.1)
SlonyFAQ04.txt (r1.1)
SlonyFAQ05.txt (r1.1)
SlonyFAQ06.txt (r1.1)
SlonyFAQ07.txt (r1.1)
SlonyFAQ08.txt (r1.1)
SlonyFAQ09.txt (r1.1)
SlonyFAQ10.txt (r1.1)
SlonyFAQ11.txt (r1.1)
SlonyFAQ12.txt (r1.1)
SlonyFAQ13.txt (r1.1)
SlonyFAQ14.txt (r1.1)
SlonyFAQ15.txt (r1.1)
SlonyFAQ16.txt (r1.1)
SlonyFAQ17.txt (r1.1)
SlonyFAQ18.txt (r1.1)
SlonyHandlingFailover.txt (r1.1)
SlonyHelp.txt (r1.1)
SlonyHowtoFirstTry.txt (r1.1)
SlonyIAdministration.txt (r1.1)
SlonyInstallation.txt (r1.1)
SlonyIntroduction.txt (r1.1)
SlonyListenPaths.txt (r1.1)
SlonyListenerCosts.txt (r1.1)
SlonyMaintenance.txt (r1.1)
SlonyPrerequisites.txt (r1.1)
SlonyReshapingCluster.txt (r1.1)
SlonySlonConfiguration.txt (r1.1)
SlonySlonik.txt (r1.1)
SlonyStartSlons.txt (r1.1)
-------------- next part --------------
--- /dev/null
+++ doc/adminguide/SlonyFAQ18.txt
@@ -0,0 +1,42 @@
+%META:TOPICINFO{author="guest" date="1099973679" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ Replication Fails - Unique Constraint Violation
+
+Replication has been running for a while, successfully, when a node encounters a "glitch," and replication logs are filled with repetitions of the following:
+
+<verbatim>
+DEBUG2 remoteWorkerThread_1: syncing set 2 with 5 table(s) from provider 1
+DEBUG2 remoteWorkerThread_1: syncing set 1 with 41 table(s) from provider 1
+DEBUG2 remoteWorkerThread_1: syncing set 5 with 1 table(s) from provider 1
+DEBUG2 remoteWorkerThread_1: syncing set 3 with 1 table(s) from provider 1
+DEBUG2 remoteHelperThread_1_1: 0.135 seconds delay for first row
+DEBUG2 remoteHelperThread_1_1: 0.343 seconds until close cursor
+ERROR remoteWorkerThread_1: "insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '34', '35090538', 'D', '_rserv_ts=''9275244''');
+delete from only public.epp_domain_host where _rserv_ts='9275244';insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '34', '35090539', 'D', '_rserv_ts=''9275245''');
+delete from only public.epp_domain_host where _rserv_ts='9275245';insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '26', '35090540', 'D', '_rserv_ts=''24240590''');
+delete from only public.epp_domain_contact where _rserv_ts='24240590';insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '26', '35090541', 'D', '_rserv_ts=''24240591''');
+delete from only public.epp_domain_contact where _rserv_ts='24240591';insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '26', '35090542', 'D', '_rserv_ts=''24240589''');
+delete from only public.epp_domain_contact where _rserv_ts='24240589';insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '11', '35090543', 'D', '_rserv_ts=''36968002''');
+delete from only public.epp_domain_status where _rserv_ts='36968002';insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '11', '35090544', 'D', '_rserv_ts=''36968003''');
+delete from only public.epp_domain_status where _rserv_ts='36968003';insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '24', '35090549', 'I', '(contact_id,status,reason,_rserv_ts) values (''6972897'',''64'','''',''31044208'')');
+insert into public.contact_status (contact_id,status,reason,_rserv_ts) values ('6972897','64','','31044208');insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '24', '35090550', 'D', '_rserv_ts=''18139332''');
+delete from only public.contact_status where _rserv_ts='18139332';insert into "_oxrsapp".sl_log_1 (log_origin, log_xid, log_tableid, log_actionseq, log_cmdtype, log_cmddata) values ('1', '919151224', '24', '35090551', 'D', '_rserv_ts=''18139333''');
+delete from only public.contact_status where _rserv_ts='18139333';" ERROR: duplicate key violates unique constraint "contact_status_pkey"
+ - qualification was:
+ERROR remoteWorkerThread_1: SYNC aborted
+</verbatim>
+
+The transaction rolls back, and Slony-I tries again, and again, and again. The problem is with one of the _last_ SQL statements, the one with log_cmdtype = 'I'. That isn't quite obvious; what takes place is that Slony-I groups 10 update queries together to diminish the number of network round trips.
+
+A _certain_ cause for this has not yet been arrived at. The factors that *appear* to go together to contribute to this scenario are as follows:
+
+ * The "glitch" seems to coincide with some sort of outage; it has been observed both in cases where databases were suffering from periodic "SIG 11" problems, where backends were falling over, as well as when temporary network failure seemed likely.
+
+ * The scenario seems to involve a delete transaction having been missed by Slony-I.
+
+By the time we notice that there is a problem, the missed delete transaction has been cleaned out of sl_log_1, so there is no recovery possible.
+
+What is necessary, at this point, is to drop the replication set (or even the node), and restart replication from scratch on that node.
+
+In Slony-I 1.0.5, the handling of purges of sl_log_1 are rather more conservative, refusing to purge entries that haven't been successfully synced for at least 10 minutes on all nodes. It is not certain that that will prevent the "glitch" from taking place, but it seems likely that it will leave enough sl_log_1 data to be able to do something about recovering from the condition or at least diagnosing it more exactly. And perhaps the problem is that sl_log_1 was being purged too aggressively, and this will resolve the issue completely.
+
--- /dev/null
+++ doc/adminguide/SlonyConcepts.txt
@@ -0,0 +1,51 @@
+%META:TOPICINFO{author="guest" date="1098328042" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Slony-I Concepts
+
+In order to set up a set of Slony-I replicas, it is necessary to understand the following major abstractions that it uses:
+
+ * Cluster
+ * Node
+ * Replication Set
+ * Provider and Subscriber
+
+---+++ Cluster
+
+In Slony-I terms, a Cluster is a named set of PostgreSQL database instances; replication takes place between those databases.
+
+The cluster name is specified in each and every Slonik script via the directive:
+<verbatim>
+cluster name = 'something';
+</verbatim>
+
+If the Cluster name is 'something', then Slony-I will create, in each database instance in the cluster, the namespace/schema '_something'.
+
+---+++ Node
+
+A Slony-I Node is a named PostgreSQL database that will be participating in replication.
+
+It is defined, near the beginning of each Slonik script, using the directive:
+<verbatim>
+ NODE 1 ADMIN CONNINFO = 'dbname=testdb host=server1 user=slony';
+</verbatim>
+
+The CONNINFO information indicates a string argument that will ultimately be passed to the PQconnectdb() libpq function.
+
+Thus, a Slony-I cluster consists of:
+ * A cluster name
+ * A set of Slony-I nodes, each of which has a namespace based on that cluster name
+
+---+++ Replication Set
+
+A replication set is defined as a set of tables and sequences that are to be replicated between nodes in a Slony-I cluster.
+
+You may have several sets, and the "flow" of replication does not need to be identical between those sets.
+
+---+++ Provider and Subscriber
+
+Each replication set has some "master" node, which winds up being the _only_ place where user applications are permitted to modify data in the tables that are being replicated. That "master" may be considered the master "provider node;" it is the main place from which data is provided.
+
+Other nodes in the cluster will subscribe to the replication set, indicating that they want to receive the data.
+
+The "master" node will never be considered a "subscriber." But Slony-I supports the notion of cascaded subscriptions, that is, a node that is subscribed to the "master" may also behave as a "provider" to other nodes in the cluster.
+
--- /dev/null
+++ doc/adminguide/SlonyDDLChanges.txt
@@ -0,0 +1,19 @@
+%META:TOPICINFO{author="guest" date="1097252100" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---+++ Database Schema Changes (DDL)
+
+When changes are made to the database schema, e.g. adding fields to a table, it is necessary for this to be handled rather carefully, otherwise different nodes may get rather deranged because they disagree on how particular tables are built.
+
+If you pass the changes through Slony-I via the EXECUTE SCRIPT (slonik) / ddlscript(set,script,node) (stored function), this allows you to be certain that the changes take effect at the same point in the transaction streams on all of the nodes.
+
+It's worth making a couple of comments on "special things" about EXECUTE SCRIPT:
+
+ * The script must not contain transaction BEGIN or END statements, as the script is already executed inside a transaction. In PostgreSQL version 8, the introduction of nested transactions may change this requirement somewhat, but you must still remain aware that the actions in the script are wrapped inside a transaction.
+
+ * If there is _anything_ broken about the script, or about how it executes on a particular node, this will cause the slon daemon for that node to panic and crash. If you restart the node, it will, more likely than not, try to *repeat* the DDL script, which will, almost certainly, fail the second time just as it did the first time. I have found this scenario to lead to a need to go to the "master" node to delete the event to stop it from continuing to fail.
+
+ * For slon to, at that point, "panic" is probably the _correct_ answer, as it allows the DBA to head over to the database node that is broken, and manually fix things before cleaning out the defective event and restarting slon. You can be certain that the updates made _after_ the DDL change on the provider node are queued up, waiting to head to the subscriber. You don't run the risk of there being updates made that depended on the DDL changes in order to be correct.
+
+Unfortunately, this nonetheless implies that the use of the DDL facility is somewhat fragile and dangerous. Making DDL changes should not be done in a sloppy or cavalier manner. If your applications do not have fairly stable SQL schemas, then using Slony-I for replication is likely to be fraught with trouble and frustration.
+
+There is an article on how to manage Slony schema changes here: [[http://www.varlena.com/varlena/GeneralBits/88.php][http://www.varlena.com/varlena/GeneralBits/88.php]]
--- /dev/null
+++ doc/adminguide/SlonyInstallation.txt
@@ -0,0 +1,65 @@
+%META:TOPICINFO{author="guest" date="1099002364" format="1.0" version="1.3"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Slony-I Installation
+
+You should have obtained the Slony-I source from the previous step. Unpack it.
+
+<verbatim>
+gunzip slony.tar.gz
+tar xf slony.tar
+</verbatim>
+
+This will create a directory Slony-I under the current directory with the Slony-I sources. Head into that that directory for the rest of the installation procedure.
+
+---+++ Short Version
+
+<verbatim>
+./configure --with-pgsourcetree=/whereever/the/source/is
+gmake all
+gmake install
+<verbatim>
+
+---+++ Configuration
+
+The first step of the installation procedure is to configure the source tree
+for your system. This is done by running the configure script. Configure
+needs to know where your PostgreSQL source tree is, this is done with the
+--with-pgsourcetree= option.
+
+---+++ Example
+
+<verbatim>
+./configure --with-pgsourcetree=/usr/local/src/postgresql-7.4.3
+</verbatim>
+
+This script will run a number of tests to guess values for various dependent
+variables and try to detect some quirks of your system. Slony-I is known to
+need a modified version of libpq on specific platforms such as Solaris2.X on
+SPARC this patch can be found at [[http://developer.postgresql.org/~wieck/slony1/download/threadsafe-libpq-742.diff.gz][http://developer.postgresql.org/~wieck/slony1/download/threadsafe-libpq-742.diff.gz]].
+
+---+++ Build
+
+To start the build process, type
+
+<verbatim>
+gmake all
+</verbatim>
+
+Be sure to use GNU make; on BSD systems, it is called gmake; on Linux, GNU make is typically the native "make", so the name of the command you type in may vary somewhat. The build may take anywhere from a few seconds to 2 minutes depending on how fast your hardware is at compiling things. The last line displayed should be
+<verbatim>
+All of Slony-I is successfully made. Ready to install.
+</verbatim>
+
+---+++ Installing Slony-I
+
+To install Slony-I, enter
+
+<verbatim>
+gmake install
+</verbatim>
+
+This will install files into postgresql install directory as specified by the
+--prefix option used in the PostgreSQL configuration. Make sure you have
+appropriate permissions to write into that area. Normally you need to do this either
+as root or as the postgres user.`
+
--- /dev/null
+++ doc/adminguide/SlonyFAQ.txt
@@ -0,0 +1,26 @@
+%META:TOPICINFO{author="guest" date="1099972260" format="1.0" version="1.3"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Frequently Asked Questions
+
+Not all of these are "frequently asked;" some represent _trouble found that seemed worth documenting_.
+
+Note that numerous of these items are version-specific, describing how to deal with error conditions that are dealt with automatically by later releases. In particular, version 1.0.5 resolved numerous issues where changing replication configuration didn't quite work entirely right. Many of these items should therefore ultimately get reorganized to indicate version specificity. There are quite enough issues with 1.0.2 that people should probably prefer to use 1.0.5 unless there's very special reason to stay with 1.0.2 for the time being.
+
+ * SlonyFAQ01 I looked for the _clustername namespace, and it wasn't there.
+ * SlonyFAQ02 Some events moving around, but no replication
+ * SlonyFAQ03 Cluster name with "-" in it
+ * SlonyFAQ04 slon does not restart after crash
+ * SlonyFAQ05 ps finds passwords on command line
+ * SlonyFAQ06 Slonik fails - cannot load PostgreSQL library - PGRES_FATAL_ERROR load '$libdir/xxid';
+ * SlonyFAQ07 Table indexes with FQ namespace names
+ * SlonyFAQ08 Subscription fails "transaction still in progress"
+ * SlonyFAQ09 ERROR: duplicate key violates unique constraint "sl_table-pkey"
+ * SlonyFAQ10 I need to drop a table from a replication set
+ * SlonyFAQ11 I need to drop a sequence from a replication set
+ * SlonyFAQ12 Slony-I: cannot add table to currently subscribed set 1
+ * SlonyFAQ13 Some nodes start consistently falling behind
+ * SlonyFAQ14 I started doing a backup using pg_dump, and suddenly Slony stops
+ * SlonyFAQ15 The slons spent the weekend out of commission [for some reason], and it's taking a long time to get a sync through.
+ * SlonyFAQ16 I pointed a subscribing node to a different parent and it stopped replicating
+ * SlonyFAQ17 After dropping a node, sl_log_1 isn't getting purged out anymore
+ * SlonyFAQ18 Replication Fails - Unique Constraint Violation
--- /dev/null
+++ doc/adminguide/SlonyFAQ01.txt
@@ -0,0 +1,12 @@
+%META:TOPICINFO{author="guest" date="1098326312" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ I looked for the _clustername namespace, and it wasn't there.
+
+If the DSNs are wrong, then slon instances can't connect to the nodes.
+
+This will generally lead to nodes remaining entirely untouched.
+
+Recheck the connection configuration. By the way, since slon links to
+libpq, you could have password information stored in $HOME/.pgpass,
+partially filling in right/wrong authentication information there.
+
--- /dev/null
+++ doc/adminguide/SlonyFAQ09.txt
@@ -0,0 +1,16 @@
+%META:TOPICINFO{author="guest" date="1098326891" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ ERROR: duplicate key violates unique constraint "sl_table-pkey"
+
+I tried setting up a second replication set, and got the following error:
+
+<verbatim>
+<stdin>:9: Could not create subscription set 2 for oxrslive!
+<stdin>:11: PGRES_FATAL_ERROR select "_oxrslive".setAddTable(2, 1, 'public.replic_test', 'replic_test__Slony-I_oxrslive_rowID_key', 'Table public.replic_test without primary key'); - ERROR: duplicate key violates unique constraint "sl_table-pkey"
+CONTEXT: PL/pgSQL function "setaddtable_int" line 71 at SQL statement
+</verbatim>
+
+The table IDs used in SET ADD TABLE are required to be unique ACROSS
+ALL SETS. Thus, you can't restart numbering at 1 for a second set; if
+you are numbering them consecutively, a subsequent set has to start
+with IDs after where the previous set(s) left off.
--- /dev/null
+++ doc/adminguide/SlonyIAdministration.txt
@@ -0,0 +1,32 @@
+%META:TOPICINFO{author="guest" date="1098325777" format="1.0" version="1.6"}%
+%META:TOPICPARENT{name="WebHome"}%
+---++ Slony-I Replication
+
+Slony-I is a replication system for use with [[http://www.postgresql.org PostgreSQL]].
+
+ * SlonyIntroduction - Introduction to Slony - how it works (triggers) what it can do; what it cannot do
+ * SlonyPrerequisites - Software and network stuff you need as prerequisites
+ * SlonyInstallation - Compiling it, installing and making sure all components are there
+ * SlonySlonik - an introduction to what Slonik is about
+ * SlonyConcepts - Clusters, nodes, sets
+ * SlonyDefineCluster - defining the network of nodes (includes some "best practices" on numbering them)
+ * SlonyDefineSet - defining sets of tables/sequences to be replicated (make sure YOU define a primary key!)
+ * SlonyAdministrationScripts - the "altperl" tools
+ * SlonyStartSlons - watchdog script; where should slon run?
+ * SlonySlonConfiguration - what are the options, how should they be chosen
+ * SlonySubscribeNodes
+ * SlonyMonitoring - what to expect in the logs, scripts to monitor things
+ * SlonyMaintenance - making sure things are vacuumed; making sure slons run; reading logs
+ * SlonyReshapingCluster - moving nodes, moving master between nodes
+ * SlonyHandlingFailover - Switchover and Failover
+ * SlonyListenPaths - care and feeding of sl_listen
+ * SlonyAddThings - adding tables, sequences, nodes
+ * SlonyDropThings - dropping tables, sequences, nodes
+ * SlonyDDLChanges - changing database schema
+ * SlonyHowtoFirstTry - Replicating Your First Database
+ * SlonyHelp - where to go for more help
+ * SlonyFAQ - Frequently Asked Questions
+
+Note to would-be authors: If you wish to add additional sections, please feel free. It would be preferable if new material maintained the prefix "Slony" as my backup scripts look specifically for that file prefix.
+
+This set of documentation will, once it is more complete, be transformed into DocBook and added into the Slony-I documentation tree. As a result, please contribute only if you are willing for it to be added there under the traditional BSD-style license used by PostgreSQL...
--- /dev/null
+++ doc/adminguide/SlonyMaintenance.txt
@@ -0,0 +1,27 @@
+%META:TOPICINFO{author="guest" date="1099541824" format="1.0" version="1.4"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Slony-I Maintenance
+
+Slony-I actually does most of its necessary maintenance itself, in a "cleanup" thread:
+
+ * Deletes old data from various tables in the Slony-I cluster's namespace, notably entries in sl_log_1, sl_log_2 (not yet used), and sl_seqlog.
+
+ * Vacuum certain tables used by Slony-I. As of 1.0.5, this includes pg_listener; in earlier versions, you must vacuum that table heavily, otherwise you'll find replication slowing down because Slony-I raises plenty of events, which leads to that table having plenty of dead tuples.
+
+ In some versions (1.1, for sure; possibly 1.0.5) there is the option of not bothering to vacuum any of these tables if you are using something like pg_autovacuum to handle vacuuming of these tables. Unfortunately, it has been quite possible for pg_autovacuum to not vacuum quite frequently enough, so you probably want to use the internal vacuums. Vacuuming pg_listener "too often" isn't nearly as hazardous as not vacuuming it frequently enough.
+
+ Unfortunately, if you have long-running transactions, vacuums cannot clear out dead tuples that are newer than the eldest transaction that is still running. This will most notably lead to pg_listener growing large and will slow replication.
+
+---++ Watchdogs: Keeping Slons Running
+
+There are a couple of "watchdog" scripts available that monitor things, and restart the slon processes should they happen to die for some reason, such as a network "glitch" that causes loss of connectivity.
+
+You might want to run them...
+
+---++ Log Files
+
+Slon daemons generate some more-or-less verbose log files, depending on what debugging level is turned on. You might assortedly wish to:
+
+ * Use a log rotator like Apache rotatelogs to have a sequence of log files so that no one of them gets too big;
+
+ * Purge out old log files, periodically.
--- /dev/null
+++ doc/adminguide/SlonyFAQ12.txt
@@ -0,0 +1,15 @@
+%META:TOPICINFO{author="guest" date="1098327131" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ Slony-I: cannot add table to currently subscribed set 1
+
+ I tried to add a table to a set, and got the following message:
+
+<verbatim>
+ Slony-I: cannot add table to currently subscribed set 1
+</verbatim>
+
+You cannot add tables to sets that already have subscribers.
+
+The workaround to this is to create ANOTHER set, add the new tables to
+that new set, subscribe the same nodes subscribing to "set 1" to the
+new set, and then merge the sets together.
--- /dev/null
+++ doc/adminguide/SlonyFAQ02.txt
@@ -0,0 +1,20 @@
+%META:TOPICINFO{author="guest" date="1099541541" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ Some events moving around, but no replication
+
+Slony logs might look like the following:
+
+<verbatim>
+DEBUG1 remoteListenThread_1: connected to 'host=host004 dbname=pgbenchrep user=postgres port=5432'
+ERROR remoteListenThread_1: "select ev_origin, ev_seqno, ev_timestamp, ev_minxid, ev_maxxid, ev_xip, ev_type, ev_data1, ev_data2, ev_data3, ev_data4, ev_data5, ev_data6, ev_data7, ev_data8 from "_pgbenchtest".sl_event e where (e.ev_origin = '1' and e.ev_seqno > '1') order by e.ev_origin, e.ev_seqno" - could not receive data from server: Operation now in progress
+</verbatim>
+
+On AIX and Solaris (and possibly elsewhere), both Slony-I _and PostgreSQL_ must be compiled with the --enable-thread-safety option. The above results when PostgreSQL isn't so compiled.
+
+What breaks here is that the libc (threadsafe) and libpq (non-threadsafe) use different memory locations for errno, thereby leading to the request failing.
+
+Problems like this crop up with disadmirable regularity on AIX and Solaris; it may take something of an "object code audit" to make sure that _ALL_ of the necessary components have been compiled and linked with --enable-thread-safety.
+
+For instance, I ran into the problem one that LD_LIBRARY_PATH had been set, on Solaris, to point to libraries from an old PostgreSQL compile.
+That meant that even though the database had been compiled with --enable-thread-safety, and slon had been compiled against that, slon was being dynamically linked to the "bad old thread-unsafe version," so slon didn't work. It wasn't clear that this was the case until I ran "ldd" against slon.
+
--- /dev/null
+++ doc/adminguide/SlonyFAQ10.txt
@@ -0,0 +1,28 @@
+%META:TOPICINFO{author="guest" date="1099542035" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ I need to drop a table from a replication set
+
+This can be accomplished several ways, not all equally desirable ;-).
+
+ * You could drop the whole replication set, and recreate it with just the tables that you need. Alas, that means recopying a whole lot of data, and kills the usability of the cluster on the rest of the set while that's happening.
+
+ * If you are running 1.0.5 or later, there is the command SET DROP TABLE, which will "do the trick."
+
+ * If you are still using 1.0.1 or 1.0.2, the _essential_ functionality of SET DROP TABLE involves the functionality in droptable_int(). You can fiddle this by hand by finding the table ID for the table you want to get rid of, which you can find in sl_table, and then run the following three queries, on each host:
+
+<verbatim>
+ select _slonyschema.alterTableRestore(40);
+ select _slonyschema.tableDropKey(40);
+ delete from _slonyschema.sl_table where tab_id = 40;
+</verbatim>
+
+The schema will obviously depend on how you defined the Slony-I
+cluster. The table ID, in this case, 40, will need to change to the
+ID of the table you want to have go away.
+
+You'll have to run these three queries on all of the nodes, preferably
+firstly on the "master" node, so that the dropping of this propagates
+properly. Implementing this via a SLONIK statement with a new Slony
+event would do that. Submitting the three queries using EXECUTE
+SCRIPT could do that. Also possible would be to connect to each
+database and submit the queries by hand.
--- /dev/null
+++ doc/adminguide/SlonyListenerCosts.txt
@@ -0,0 +1,14 @@
+%META:TOPICINFO{author="guest" date="1099541111" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyIntroduction"}%
+---++ Slony-I Communications Costs
+
+The cost of communications grows in a quadratic fashion in several directions as the number of replication nodes in a cluster increases. Note the following relationships:
+
+ * It is necessary to have a sl_path entry allowing connection from each node to every other node. Most will normally not need to be used for a given replication configuration, but this means that there needs to be n(n-1) paths. It is probable that there will be considerable repetition of entries, since the path to "node n" is likely to be the same from everywherein the network.
+
+ * It is similarly necessary to have a sl_listen entry indicating how data flows from every node to every other node. This again requires configuring n(n-1) "listener paths."
+
+ * Each SYNC applied needs to be reported back to all of the other nodes participating in the set so that the nodes all know that it is safe to purge sl_log_1 and sl_log_2 data, as any "forwarding" node may take over as "master" at any time. One might expect SYNC messages to need to travel through n/2 nodes to get propagated to their destinations; this means that each SYNC is expected to get transmitted n(n/2) times. Again, this points to a quadratic growth in communications costs as the number of nodes increases.
+
+This points to it being a bad idea to have the large communications network resulting from the number of nodes being large. Up to a half dozen nodes seems pretty reasonable; every time the number of nodes doubles, this can be expected to quadruple communications overheads.
+
--- /dev/null
+++ doc/adminguide/SlonyListenPaths.txt
@@ -0,0 +1,110 @@
+%META:TOPICINFO{author="guest" date="1099870294" format="1.0" version="1.7"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Slony Listen Paths
+
+If you have more than two or three nodes, and any degree of usage of cascaded subscribers (_e.g._ - subscribers that are subscribing through a subscriber node), you will have to be fairly careful about the configuration of "listen paths" via the Slonik STORE LISTEN and DROP LISTEN statements that control the contents of the table sl_listen.
+
+The "listener" entries in this table control where each node expects to listen in order to get events propagated from other nodes. You might think that nodes only need to listen to the "parent" from whom they are getting updates, but in reality, they need to be able to receive messages from _all_ nodes in order to be able to conclude that SYNCs have been received everywhere, and that, therefore, entries in sl_log_1 and sl_log_2 have been applied everywhere, and can therefore be purged.
+
+---+++ How Listening Can Break
+
+On one occasion, I had a need to drop a subscriber node (#2) and recreate it. That node was the data provider for another subscriber (#3) that was, in effect, a "cascaded slave." Dropping the subscriber node initially didn't work, as slonik informed me that there was a dependant node. I repointed the dependant node to the "master" node for the subscription set, which, for a while, replicated without difficulties.
+
+I then dropped the subscription on "node 2," and started resubscribing it. That raised the Slony-I SET_SUBSCRIPTION event, which started copying tables. At that point in time, events stopped propagating to "node 3," and while it was in perfectly OK shape, no events were making it to it.
+
+The problem was that node #3 was expecting to receive events from node #2, which was busy processing the SET_SUBSCRIPTION event, and was not passing anything else on.
+
+We dropped the listener rules that caused node #3 to listen to node 2, replacing them with rules where it expected its events to come from node #1 (the "master" provider node for the replication set). At that moment, "as if by magic," node #3 started replicating again, as it discovered a place to get SYNC events.
+
+---+++ How The Listen Configuration Should Look
+
+The simple cases tend to be simple to cope with. We'll look at a fairly complex set of nodes.
+
+Consider a set of nodes, 1 thru 6, where 1 is the "master," where 2-4 subscribe directly to the master, and where 5 subscribes to 2, and 6 subscribes to 5.
+
+Here is a "listener network" that indicates where each node should listen for messages coming from each other node:
+
+<verbatim>
+ 1| 2| 3| 4| 5| 6|
+--------------------------------------------
+ 1 0 2 3 4 2 2
+ 2 1 0 1 1 5 5
+ 3 1 1 0 1 1 1
+ 4 1 1 1 0 1 1
+ 5 2 2 2 2 0 6
+ 6 5 5 5 5 5 0
+</verbatim>
+
+Row 2 indicates all of the listen rules for node 2; it gets events for nodes 1, 3, and 4 throw node 1, and gets events for nodes 5 and 6 from node 5.
+
+The row of 5's at the bottom, for node 6, indicate that node 6 listens to node 5 to get events from nodes 1-5.
+
+The set of slonik SET LISTEN statements to express this "listener network" are as follows:
+
+<verbatim>
+ store listen (origin = 1, receiver = 2, provider = 1);
+ store listen (origin = 1, receiver = 3, provider = 1);
+ store listen (origin = 1, receiver = 4, provider = 1);
+ store listen (origin = 1, receiver = 5, provider = 2);
+ store listen (origin = 1, receiver = 6, provider = 5);
+ store listen (origin = 2, receiver = 1, provider = 2);
+ store listen (origin = 2, receiver = 3, provider = 1);
+ store listen (origin = 2, receiver = 4, provider = 1);
+ store listen (origin = 2, receiver = 5, provider = 2);
+ store listen (origin = 2, receiver = 6, provider = 5);
+ store listen (origin = 3, receiver = 1, provider = 3);
+ store listen (origin = 3, receiver = 2, provider = 1);
+ store listen (origin = 3, receiver = 4, provider = 1);
+ store listen (origin = 3, receiver = 5, provider = 2);
+ store listen (origin = 3, receiver = 6, provider = 5);
+ store listen (origin = 4, receiver = 1, provider = 4);
+ store listen (origin = 4, receiver = 2, provider = 1);
+ store listen (origin = 4, receiver = 3, provider = 1);
+ store listen (origin = 4, receiver = 5, provider = 2);
+ store listen (origin = 4, receiver = 6, provider = 5);
+ store listen (origin = 5, receiver = 1, provider = 2);
+ store listen (origin = 5, receiver = 2, provider = 5);
+ store listen (origin = 5, receiver = 3, provider = 1);
+ store listen (origin = 5, receiver = 4, provider = 1);
+ store listen (origin = 5, receiver = 6, provider = 5);
+ store listen (origin = 6, receiver = 1, provider = 2);
+ store listen (origin = 6, receiver = 2, provider = 5);
+ store listen (origin = 6, receiver = 3, provider = 1);
+ store listen (origin = 6, receiver = 4, provider = 1);
+ store listen (origin = 6, receiver = 5, provider = 6);
+</verbatim>
+
+How we read these listen statements is thus...
+
+When on the "receiver" node, look to the "provider" node to provide events coming from the "origin" node.
+
+The tool "init_cluster.pl" in the "altperl" scripts produces optimized listener networks in both the tabular form shown above as well as in the form of Slonik statements.
+
+There are three "thorns" in this set of roses:
+
+ * If you change the shape of the node set, so that the nodes subscribe differently to things, you need to drop sl_listen entries and create new ones to indicate the new preferred paths between nodes. There is no automated way at this point to do this "reshaping."
+
+ * If you _don't_ change the sl_listen entries, events will likely continue to propagate so long as all of the nodes continue to run well. The problem will only be noticed when a node is taken down, "orphaning" any nodes that are listening through it.
+
+ * You might have multiple replication sets that have _different_ shapes for their respective trees of subscribers. There won't be a single "best" listener configuration in that case.
+
+ * In order for there to be an sl_listen path, there _must_ be a series of sl_path entries connecting the origin to the receiver. This means that if the contents of sl_path do not express a "connected" network of nodes, then some nodes will not be reachable. This would typically happen, in practice, when you have two sets of nodes, one in one subnet, and another in another subnet, where there are only a couple of "firewall" nodes that can talk between the subnets. Cut out those nodes and the subnets stop communicating.
+
+---+++ Open Question
+
+I am not certain what happens if you have multiple listen path entries for one path, that is, if you set up entries allowing a node to listen to multiple receivers to get events from a particular origin. Further commentary on that would be appreciated!
+
+---+++ Generating listener entries via heuristics
+
+It ought to be possible to generate sl_listen entries dynamically, based on the following heuristics. Hopefully this will take place in version 1.1, eliminating the need to configure this by hand.
+
+Configuration will (tentatively) be controlled based on two data sources:
+
+ * sl_subscribe entries are the first, most vital control as to what listens to what; we know there must be a "listen" entry for a subscriber node to listen to its provider for events from the provider, and there should be direct "listening" taking place between subscriber and provider.
+
+ * sl_path entries are the second indicator; if sl_subscribe has not already indicated "how to listen," then a node may listen directly to the event's origin if there is a suitable sl_path entry
+
+ * If there is no guidance thus far based on the above data sources, then nodes can listen indirectly if there is an sl_path entry that points to a suitable sl_listen entry...
+
+A stored procedure would run on each node, rewriting sl_listen each time sl_subscribe or sl_path are modified.
+
--- /dev/null
+++ doc/adminguide/SlonyPrerequisites.txt
@@ -0,0 +1,54 @@
+%META:TOPICINFO{author="guest" date="1098242633" format="1.0" version="1.5"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Requirements:
+
+Any platform that can run PostgreSQL should be able to run Slony-I.
+
+The platforms that have received specific testing at the time of this release are
+FreeBSD-4X-i368, FreeBSD-5X-i386, FreeBSD-5X-alpha, osX-10.3, Linux-2.4X-i386
+Linux-2.6X-i386 Linux-2.6X-amd64, Solaris-2.8-SPARC, Solaris-2.9-SPARC, AIX 5.1 and
+OpenBSD-3.5-sparc64.
+
+There have been reports of success at running Slony-I hosts that are running PostgreSQL on Microsoft Windows(tm). At this time, the "binary" applications (e.g. - slonik, slon) do not run on Windows(tm), but a slon running on one of the Unix-like systems has no reason to have difficulty connect to a PostgreSQL instance running on Windows(tm).
+
+It ought to be possible to port slon and slonik to run on Windows; the conspicuous challenge is of having a POSIX-like pthreads implementation for slon, as it uses that to have multiple threads of execution. There are reports of there being a pthreads library for Windows(tm), so nothing should prevent some interested party from volunteering to do the port.
+
+---+++ Software needed
+
+ * GNU make. Other make programs will not work. GNU make is often installed under the name gmake; this document will therefore always refer to it by that name. (On Linux-based systems GNU make is typically the default make, and is called "make") To test to see if your make is GNU make enter "make version." Version 3.76 or later will suffice; previous versions may not.
+
+ * You need an ISO/ANSI C compiler. Recent versions of GCC work.
+
+ * You also need a recent version of PostgreSQL *source*. Slony-I depends on namespace support so you must have version 7.3 or newer to be able to build and use Slony-I. Rod Taylor has "hacked up" a version of Slony-I that works with version 7.2; if you desperately need that, look for him on the PostgreSQL Hackers mailing list. It is not anticipated that 7.2 will be supported by any official Slony-I release.
+
+ * GNU packages may be included in the standard packaging for your operating system, or you may need to look for source code at your local GNU mirror (see [[http://www.gnu.org/order/ftp.html][http://www.gnu.org/order/ftp.html]] for a list) or at [[ftp://ftp.gnu.org/gnu][ftp://ftp.gnu.org/gnu]].)
+
+ * If you need to obtain PostgreSQL source, you can download it from your favorite PostgreSQL mirror (see [[http://www.postgresql.org/mirrors-www.html][http://www.postgresql.org/mirrors-www.html]] for a list), or via [[http://bt.postgresql.org/][BitTorrent]].
+
+Also check to make sure you have sufficient disk space. You will need
+approximately 5MB for the source tree during build and installation.
+
+---+++ Getting Slony-I Source
+
+You can get the Slony-I source from [[http://developer.postgresql.org/~wieck/slony1/download/][http://developer.postgresql.org/~wieck/slony1/download/]]
+
+---+++ Time Synchronization
+
+All the servers used within the replication cluster need to have their Real Time Clocks in sync. This is to ensure that slon doesn't error with messages indicating that slave is already ahead of the master during replication. We recommend you have ntpd running on all nodes, with subscriber nodes using the "master" provider node as their time server.
+
+It is possible for Slony-I to function even in the face of there being some time discrepancies, but having systems "in sync" is usually pretty important for distributed applications.
+
+---+++ Network Connectivity
+
+It is necessary that the hosts that are to replicate between one another have _bidirectional_ network communications to the PostgreSQL instances. That is, if node B is replicating data from node A, it is necessary that there be a path from A to B and from B to A. It is recommended that all nodes in a Slony-I cluster allow this sort of bidirection communications from any node in the cluster to any other node in the cluster.
+
+Note that the network addresses need to be consistent across all of the nodes. Thus, if there is any need to use a "public" address for a node, to allow remote/VPN access, that "public" address needs to be able to be used consistently throughout the Slony-I cluster, as the address is propagated throughout the cluster in table sl_path.
+
+A possible workaround for this, in environments where firewall rules are particularly difficult to implement, may be to establish SSHTunnels that are created on each host that allow remote access through IP address 127.0.0.1, with a different port for each destination.
+
+Note that slonik and the slon instances need no special connections to communicate with one another; they just need to be able to get access to the PostgreSQL databases.
+
+An implication of the communications model is that the extended network in which a Slony-I cluster operates must be able to be treated as being secure. If there is a remote location where you cannot trust the Slony-I node to be considered "secured," this represents a vulnerability that affects _all_ the nodes throughout the cluster. In effect, the security policies throughout the cluster can only be considered as stringent as those applied at the _weakest_ link. Running a full-blown Slony-I node at a branch location that can't be kept secure compromises security for *every node* in the cluster.
+
+In the future plans is a feature whereby updates for a particular replication set would be serialized via a scheme called "log shipping." The data stored in sl_log_1 and sl_log_2 would be written out to log files on disk. These files could be transmitted in any manner desired, whether via scp, FTP, burning them onto DVD-ROMs and mailing them, or even by recording them on a USB "flash device" and attaching them to birds, allowing a sort of "avian transmission protocol." This will allow one way communications so that "subscribers" that use log shipping would have no need for access to other Slony-I nodes.
+
--- /dev/null
+++ doc/adminguide/SlonyAddThings.txt
@@ -0,0 +1,11 @@
+%META:TOPICINFO{author="guest" date="1099541886" format="1.0" version="1.4"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Adding Things to Replication
+
+You may discover that you have missed replicating things that you wish you were replicating.
+
+This can be fairly easily remedied.
+
+You cannot directly use SET ADD TABLE or SET ADD SEQUENCE in order to add tables and sequences to a replication set that is presently replicating; you must instead create a new replication set. Once it is identically subscribed (e.g. - the set of subscribers is _identical_ to that for the set it is to merge with), the sets may be merged together using MERGE SET.
+
+Up to and including 1.0.2, there is a potential problem where if MERGE_SET is issued when other subscription-related events are pending, it is possible for things to get pretty confused on the nodes where other things were pending. This problem was resolved in 1.0.5.
--- /dev/null
+++ doc/adminguide/SlonyFAQ07.txt
@@ -0,0 +1,12 @@
+%META:TOPICINFO{author="guest" date="1098326722" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ Table indexes with FQ namespace names
+
+<verbatim>
+set add table (set id = 1, origin = 1, id = 27, full qualified name = 'nspace.some_table', key = 'key_on_whatever',
+ comment = 'Table some_table in namespace nspace with a candidate primary key');
+</verbatim>
+
+If you have
+ key = 'nspace.key_on_whatever'
+the request will FAIL.
--- /dev/null
+++ doc/adminguide/SlonyIntroduction.txt
@@ -0,0 +1,49 @@
+%META:TOPICINFO{author="guest" date="1099539982" format="1.0" version="1.6"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ An Introduction to Slony-I
+
+---+++ Why yet another replication system?
+
+Slony-I was born from an idea to create a replication system that was not tied
+to a specific version of PostgreSQL, which is allowed to be started and stopped on
+an existing database with out the need for a dump/reload cycle.
+
+---+++ What Slony-I is
+
+Slony-I is a "master to multiple slaves" replication system supporting cascading and slave promotion. The big picture for the development of Slony-I is as a master-slave system that includes all features and capabilities needed to replicate large databases to a reasonably limited number of slave systems. "Reasonable," in this context, is probably no more than a few dozen servers. If the number of servers grows beyond that, the cost of communications becomes prohibitively high. SlonyListenerCosts
+
+Slony-I is a system intended for data centers and backup sites, where the normal mode of operation is that all nodes are available all the time, and where all nodes can be secured. If you have nodes that are likely to regularly drop onto and off of the network, or have nodes that cannot be kept secure, Slony-I may not be the ideal replication solution for you.
+
+There are plans for a "file-based log shipping" extension where updates would be serialized into files. Given that, log files could be distributed by any means desired without any need of feedback between the provider node and those nodes subscribing via "log shipping."
+
+---+++ What Slony-I is not
+
+Slony-I is not a network management system.
+
+Slony-I does not have any functionality within it to detect a node failure, or automatically promote a node to a master or other data origin.
+
+Slony-I is not multi-master; it's not a connection broker, and it doesn't make you coffee and toast in the morning.
+
+(That being said, the plan is for a subsequent system, Slony-II, to provide "multimaster" capabilities, and be "bootstrapped" using Slony-I. But that is a separate project, and expectations for Slony-I should not be based on hopes for future projects.)
+
+---+++ Why doesn't Slony-I do automatic fail-over/promotion?
+
+This is the job of network monitoring software, not Slony. Every site's configuration and fail-over path is different. For example, keep-alive
+monitoring with redundant NIC's and intelligent HA switches that guarantee race-condition-free takeover of a network address and disconnecting the
+"failed" node vary in every network setup, vendor choice, hardware/software combination. This is clearly the realm of network management software and not
+Slony-I.
+
+Let Slony-I do what it does best: provide database replication.
+
+---+++ Current Limitations
+
+Slony-I does not automatically propagate schema changes, nor does it have any ability to replicate large objects. There is a single common reason for these limitations, namely that Slony-I operates using triggers, and neither schema changes nor large object operations can raise triggers suitable to tell Slony-I when those kinds of changes take place.
+
+There is a capability for Slony-I to propagate DDL changes if you submit them as scripts via the slonik EXECUTE SCRIPT operation. That is not "automatic;" you have to construct an SQL DDL script and submit it.
+
+If you have those sorts of requirements, it may be worth exploring the use of PostgreSQL 8.0 PITR (Point In Time Recovery), where WAL logs are replicated to remote nodes. Unfortunately, that has two attendant limitations:
+
+ * PITR replicates _all_ changes in _all_ databases; you cannot exclude data that isn't relevant;
+ * A PITR replica remains dormant until you apply logs and start up the database. You cannot use the database and apply updates simultaneously. It is like having a "standby server" which cannot be used without it ceasing to be "standby."
+
+There are a number of distinct models for database replication; it is impossible for one replication system to be all things to all people.
--- /dev/null
+++ doc/adminguide/SlonyDefineCluster.txt
@@ -0,0 +1,13 @@
+%META:TOPICINFO{author="guest" date="1097952385" format="1.0" version="1.4"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Defining Slony-I Clusters
+
+A Slony-I cluster is the basic grouping of database instances in which replication takes place. It consists of a set of PostgreSQL database instances in which is defined a namespace specific to that cluster.
+
+Each database instance in which replication is to take place is identified by a node number.
+
+For a simple install, it may be reasonable for the "master" to be node #1, and for the "slave" to be node #2.
+
+Some planning should be done, in more complex cases, to ensure that the numbering system is kept sane, lest the administrators be driven insane. The node numbers should be chosen to somehow correspond to the shape of the environment, as opposed to (say) the order in which nodes were initialized.
+
+It may be that in version 1.1, nodes will also have a "name" attribute, so that they may be given more mnemonic names. In that case, the node numbers can be cryptic; it will be the node name that is used to organize the cluster.
--- /dev/null
+++ doc/adminguide/SlonyDefineSet.txt
@@ -0,0 +1,37 @@
+%META:TOPICINFO{author="guest" date="1097702299" format="1.0" version="1.3"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Defining Slony-I Replication Sets
+
+Defining the nodes indicated the shape of the cluster of database servers; it is now time to determine what data is to be copied between them. The groups of data that are copied are defined as "sets."
+
+A replication set consists of the following:
+
+ * Keys on tables that are to be replicated that have no suitable primary key
+
+ * Tables that are to be replicated
+
+ * Sequences that are to be replicated
+
+---+++ Primary Keys
+
+Slony-I *needs* to have a primary key on each table that is replicated. PK values are used as the primary identifier for each tuple that is modified in the source system. There are three ways that you can get Slony-I to use a primary key:
+
+ * If the table has a formally identified primary key, SET ADD TABLE can be used without any need to reference the primary key. Slony-I will pick up that there is a primary key, and use it.
+
+ * If the table hasn't got a primary key, but has some *candidate* primary key, that is, some index on a combination of fields that is UNIQUE and NOT NULL, then you can specify the key, as in
+
+<verbatim>
+ SET ADD TABLE (set id = 1, origin = 1, id = 42, full qualified name = 'public.this_table', key = 'this_by_that', comment='this_table has this_by_that as a candidate primary key');
+</verbatim>
+
+ Notice that while you need to specify the namespace for the table, you must /not/ specify the namespace for the key, as it infers the namespace from the table.
+
+ * If the table hasn't even got a candidate primary key, you can ask Slony-I to provide one. This is done by first using TABLE ADD KEY to add a column populated using a Slony-I sequence, and then having the SET ADD TABLE include the directive key=serial, to indicate that Slony-I's own column should be used.
+
+It is not terribly important whether you pick a "true" primary key or a mere "candidate primary key;" it is, however, recommended that you have one of those instead of having Slony-I populate the PK column for you. If you don't have a suitable primary key, that means that the table hasn't got any mechanism from your application's standpoint of keeping values unique. Slony-I may therefore introduce a new failure mode for your application, and this implies that you had a way to enter confusing data into the database.
+
+---+++ Grouping tables into sets
+
+It will be vital to group tables together into a single set if those tables are related via foreign key constraints. If tables that are thus related are _not_ replicated together, you'll find yourself in trouble if you switch the "master provider" from one node to another, and discover that the new "master" can't be updated properly because it is missing the contents of dependent tables.
+
+If a database schema has been designed cleanly, it is likely that replication sets will be virtually synonymous with namespaces. All of the tables and sequences in a particular namespace will be sufficiently related that you will want to replicate them all. Conversely, tables found in different schemas will likely NOT be related, and therefore should be replicated in separate sets.
--- /dev/null
+++ doc/adminguide/SlonyFAQ08.txt
@@ -0,0 +1,57 @@
+%META:TOPICINFO{author="guest" date="1098326832" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ Subscription fails "transaction still in progress"
+
+I'm trying to get a slave subscribed, and get the following
+messages in the logs:
+
+<verbatim>
+DEBUG1 copy_set 1
+DEBUG1 remoteWorkerThread_1: connected to provider DB
+WARN remoteWorkerThread_1: transactions earlier than XID 127314958 are still in progress
+WARN remoteWorkerThread_1: data copy for set 1 failed - sleep 60 seconds
+</verbatim>
+
+Oops. What I forgot to mention, as well, was that I was trying to add
+TWO subscribers, concurrently.
+
+That doesn't work out: Slony-I won't work on the COPY commands
+concurrently. See src/slon/remote_worker.c, function copy_set()
+
+This has the (perhaps unfortunate) implication that you cannot
+populate two slaves concurrently. You have to subscribe one to the
+set, and only once it has completed setting up the subscription
+(copying table contents and such) can the second subscriber start
+setting up the subscription.
+
+It could also be possible for there to be an old outstanding
+transaction blocking Slony-I from processing the sync. You might want
+to take a look at pg_locks to see what's up:
+
+<verbatim>
+sampledb=# select * from pg_locks where transaction is not null order by transaction;
+ relation | database | transaction | pid | mode | granted
+----------+----------+-------------+---------+---------------+---------
+ | | 127314921 | 2605100 | ExclusiveLock | t
+ | | 127326504 | 5660904 | ExclusiveLock | t
+(2 rows)
+</verbatim>
+
+See? 127314921 is indeed older than 127314958, and it's still running.
+
+<verbatim>
+$ ps -aef | egrep '[2]605100'
+postgres 2605100 205018 0 18:53:43 pts/3 3:13 postgres: postgres sampledb localhost COPY
+</verbatim>
+
+This happens to be a COPY transaction involved in setting up the
+subscription for one of the nodes. All is well; the system is busy
+setting up the first subscriber; it won't start on the second one
+until the first one has completed subscribing.
+
+By the way, if there is more than one database on the PostgreSQL
+cluster, and activity is taking place on the OTHER database, that will
+lead to there being "transactions earlier than XID whatever" being
+found to be still in progress. The fact that it's a separate database
+on the cluster is irrelevant; Slony-I will wait until those old
+transactions terminate.
--- /dev/null
+++ doc/adminguide/SlonyHowtoFirstTry.txt
@@ -0,0 +1,282 @@
+%META:TOPICINFO{author="guest" date="1098196188" format="1.0" version="1.3"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Replicating Your First Database
+
+In this example, we will be replicating a brand new pgbench database. The
+mechanics of replicating an existing database are covered here, however we
+recommend that you learn how Slony-I functions by using a fresh new
+non-production database.
+
+The Slony-I replication engine is trigger-based, allowing us to replicate
+databases (or portions thereof) running under the same postmaster.
+
+This example will show how to replicate the pgbench database running on
+localhost (master) to the pgbench slave database also running on localhost
+(slave). We make a couple of assumptions about your PostgreSQL configuration:
+
+ 1 You have tcpip_socket=true in your postgresql.conf and
+ 1 You have enabled access in your cluster(s) via pg_hba.conf
+
+The REPLICATIONUSER needs to be a PostgreSQL superuser. This is typically
+postgres or pgsql.
+
+You should also set the following shell variables:
+
+<verbatim>
+CLUSTERNAME=slony_example
+MASTERDBNAME=pgbench
+SLAVEDBNAME=pgbenchslave
+MASTERHOST=localhost
+SLAVEHOST=localhost
+REPLICATIONUSER=pgsql
+PGBENCHUSER=pgbench
+</verbatim>
+Here are a couple of examples for setting variables in common shells:
+
+ * bash, sh, ksh
+<verbatim>
+ export CLUSTERNAME=slony_example
+</verbatim>
+ * (t)csh:
+<verbatim>
+ setenv CLUSTERNAME slony_example
+</verbatim>
+
+-----
+*Gotcha:* If you're changing these variables to use different hosts for MASTERHOST and SLAVEHOST, be sure _not_ to use localhost for either of them. This will result in an error similar to the following:
+
+<verbatim>
+ERROR remoteListenThread_1: db_getLocalNodeId() returned 2 - wrong database?
+</verbatim>
+
+-- Main.SteveSimms - 16 Oct 2004
+-----
+
+---+++ Creating the pgbenchuser
+
+<verbatim>
+createuser -A -D $PGBENCHUSER
+</verbatim>
+
+---+++ Preparing the databases
+
+<verbatim>
+createdb -O $PGBENCHUSER -h $MASTERHOST $MASTERDBNAME
+createdb -O $PGBENCHUSER -h $SLAVEHOST $SLAVEDBNAME
+
+pgbench -i -s 1 -U $PGBENCHUSER -h $MASTERHOST $MASTERDBNAME
+</verbatim>
+
+Because Slony-I depends on the databases having the pl/pgSQL procedural
+language installed, we better install it now. It is possible that you have
+installed pl/pgSQL into the template1 database in which case you can skip this
+step because it's already installed into the $MASTERDBNAME.
+
+<verbatim>
+createlang plpgsql -h $MASTERHOST $MASTERDBNAME
+</verbatim>
+
+Slony-I does not yet automatically copy table definitions from a master when a
+slave subscribes to it, so we need to import this data. We do this with
+pg_dump.
+
+<verbatim>
+pg_dump -s -U $REPLICATIONUSER -h $MASTERHOST $MASTERDBNAME | psql -U $REPLICATIONUSER -h $SLAVEHOST $SLAVEDBNAME
+</verbatim>
+
+To illustrate how Slony-I allows for on the fly replication subscription, lets
+start up pgbench. If you run the pgbench application in the foreground of a
+separate terminal window, you can stop and restart it with different
+parameters at any time. You'll need to re-export the variables again so they
+are available in this session as well.
+
+The typical command to run pgbench would look like:
+
+<verbatim>
+pgbench -s 1 -c 5 -t 1000 -U $PGBENCHUSER -h $MASTERHOST $MASTERDBNAME
+</verbatim>
+
+This will run pgbench with 5 concurrent clients each processing 1000
+transactions against the pgbench database running on localhost as the pgbench
+user.
+
+---+++ Configuring the Database for Replication.
+
+Creating the configuration tables, stored procedures, triggers and
+configuration is all done through the slonik tool. It is a specialized
+scripting aid that mostly calls stored procedures in the master/salve (node)
+databases. The script to create the initial configuration for the simple
+master-slave setup of our pgbench database looks like this:
+
+<verbatim>
+#!/bin/sh
+
+slonik <<_EOF_
+ #--
+ # define the namespace the replication system uses in our example it is
+ # slony_example
+ #--
+ cluster name = $CLUSTERNAME;
+
+ #--
+ # admin conninfo's are used by slonik to connect to the nodes one for each
+ # node on each side of the cluster, the syntax is that of PQconnectdb in
+ # the C-API
+ # --
+ node 1 admin conninfo = 'dbname=$MASTERDBNAME host=$MASTERHOST user=$REPLICATIONUSER';
+ node 2 admin conninfo = 'dbname=$SLAVEDBNAME host=$SLAVEHOST user=$REPLICATIONUSER';
+
+ #--
+ # init the first node. Its id MUST be 1. This creates the schema
+ # _$CLUSTERNAME containing all replication system specific database
+ # objects.
+
+ #--
+ init cluster ( id=1, comment = 'Master Node');
+
+ #--
+ # Because the history table does not have a primary key or other unique
+ # constraint that could be used to identify a row, we need to add one.
+ # The following command adds a bigint column named
+ # _Slony-I_$CLUSTERNAME_rowID to the table. It will have a default value
+ # of nextval('_$CLUSTERNAME.s1_rowid_seq'), and have UNIQUE and NOT NULL
+ # constraints applied. All existing rows will be initialized with a
+ # number
+ #--
+ table add key (node id = 1, fully qualified name = 'public.history');
+
+ #--
+ # Slony-I organizes tables into sets. The smallest unit a node can
+ # subscribe is a set. The following commands create one set containing
+ # all 4 pgbench tables. The master or origin of the set is node 1.
+ #--
+ create set (id=1, origin=1, comment='All pgbench tables');
+ set add table (set id=1, origin=1, id=1, fully qualified name = 'public.accounts', comment='accounts table');
+ set add table (set id=1, origin=1, id=2, fully qualified name = 'public.branches', comment='branches table');
+ set add table (set id=1, origin=1, id=3, fully qualified name = 'public.tellers', comment='tellers table');
+ set add table (set id=1, origin=1, id=4, fully qualified name = 'public.history', comment='history table', key = serial);
+
+ #--
+ # Create the second node (the slave) tell the 2 nodes how to connect to
+ # each other and how they should listen for events.
+ #--
+
+ store node (id=2, comment = 'Slave node');
+ store path (server = 1, client = 2, conninfo='dbname=$MASTERDBNAME host=$MASTERHOST user=$REPLICATIONUSER');
+ store path (server = 2, client = 1, conninfo='dbname=$SLAVEDBNAME host=$SLAVEHOST user=$REPLICATIONUSER');
+ store listen (origin=1, provider = 1, receiver =2);
+ store listen (origin=2, provider = 2, receiver =1);
+_EOF_
+</verbatim>
+
+
+Is the pgbench still running? If not start it again.
+
+At this point we have 2 databases that are fully prepared. One is the master
+database in which bgbench is busy accessing and changing rows. It's now time
+to start the replication daemons.
+
+On $MASTERHOST the command to start the replication engine is
+
+<verbatim>
+slon $CLUSTERNAME "dbname=$MASTERDBNAME user=$REPLICATIONUSER host=$MASTERHOST"
+</verbatim>
+
+Likewise we start the replication system on node 2 (the slave)
+
+<verbatim>
+slon $CLUSTERNAME "dbname=$SLAVEDBNAME user=$REPLICATIONUSER host=$SLAVEHOST"
+</verbatim>
+
+Even though we have the slon running on both the master and slave and they are
+both spitting out diagnostics and other messages, we aren't replicating any
+data yet. The notices you are seeing is the synchronization of cluster
+configurations between the 2 slon processes.
+
+To start replicating the 4 pgbench tables (set 1) from the master (node id 1)
+the the slave (node id 2), execute the following script.
+
+<verbatim>
+#!/bin/sh
+slonik <<_EOF_
+ # ----
+ # This defines which namespace the replication system uses
+ # ----
+ cluster name = $CLUSTERNAME;
+
+ # ----
+ # Admin conninfo's are used by the slonik program to connect
+ # to the node databases. So these are the PQconnectdb arguments
+ # that connect from the administrators workstation (where
+ # slonik is executed).
+ # ----
+ node 1 admin conninfo = 'dbname=$MASTERDBNAME host=$MASTERHOST user=$REPLICATIONUSER';
+ node 2 admin conninfo = 'dbname=$SLAVEDBNAME host=$SLAVEHOST user=$REPLICATIONUSER';
+
+ # ----
+ # Node 2 subscribes set 1
+ # ----
+ subscribe set ( id = 1, provider = 1, receiver = 2, forward = no);
+_EOF_
+</verbatim>
+
+
+Any second here, the replication daemon on $SLAVEHOST will start to copy the
+current content of all 4 replicated tables. While doing so, of course, the
+pgbench application will continue to modify the database. When the copy
+process is finished, the replication daemon on $SLAVEHOST will start to catch
+up by applying the accumulated replication log. It will do this in little
+steps, 10 seconds worth of application work at a time. Depending on the
+performance of the two systems involved, the sizing of the two databases, the
+actual transaction load and how well the two databases are tuned and
+maintained, this catchup process can be a matter of minutes, hours, or
+eons.
+
+You have now successfully set up your first basic master/slave replication
+system, and the 2 databases once the slave has caught up contain identical
+data. That's the theory. In practice, it's good to check that the datasets
+are in fact the same.
+
+The following script will create ordered dumps of the 2 databases and compare
+them. Make sure that pgbench has completed it's testing, and that your slon
+sessions have caught up.
+
+<verbatim>
+#!/bin/sh
+echo -n "**** comparing sample1 ... "
+psql -U $REPLICATIONUSER -h $MASTERHOST $MASTERDBNAME >dump.tmp.1.$$ <<_EOF_
+ select 'accounts:'::text, aid, bid, abalance, filler
+ from accounts order by aid;
+ select 'branches:'::text, bid, bbalance, filler
+ from branches order by bid;
+ select 'tellers:'::text, tid, bid, tbalance, filler
+ from tellers order by tid;
+ select 'history:'::text, tid, bid, aid, delta, mtime, filler,
+ "_Slony-I_${CLUSTERNAME}_rowID"
+ from history order by "_Slony-I_${CLUSTERNAME}_rowID";
+_EOF_
+psql -U $REPLICATIONUSER -h $SLAVEHOST $SLAVEDBNAME >dump.tmp.2.$$ <<_EOF_
+ select 'accounts:'::text, aid, bid, abalance, filler
+ from accounts order by aid;
+ select 'branches:'::text, bid, bbalance, filler
+ from branches order by bid;
+ select 'tellers:'::text, tid, bid, tbalance, filler
+ from tellers order by tid;
+ select 'history:'::text, tid, bid, aid, delta, mtime, filler,
+ "_Slony-I_${CLUSTERNAME}_rowID"
+ from history order by "_Slony-I_${CLUSTERNAME}_rowID";
+_EOF_
+
+if diff dump.tmp.1.$$ dump.tmp.2.$$ >$CLUSTERNAME.diff ; then
+ echo "success - databases are equal."
+ rm dump.tmp.?.$$
+ rm $CLUSTERNAME.diff
+else
+ echo "FAILED - see $CLUSTERNAME.diff for database differences"
+fi
+</verbatim>
+
+Note that there is somewhat more sophisticated documentation of the process in the Slony-I source code tree in a file called slony-I-basic-mstr-slv.txt.
+
+If this script returns "FAILED" please contact the developers at
+[[http://slony.org/]]
--- /dev/null
+++ doc/adminguide/SlonyFAQ16.txt
@@ -0,0 +1,50 @@
+%META:TOPICINFO{author="guest" date="1098327540" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ I pointed a subscribing node to a different parent and it stopped replicating
+
+We noticed this happening when we wanted to re-initialize a node,
+where we had configuration thus:
+
+ * Node 1 - master
+ * Node 2 - child of node 1 - the node we're reinitializing
+ * Node 3 - child of node 3 - node that should keep replicating
+
+The subscription for node 3 was changed to have node 1 as provider,
+and we did DROP SET/SUBSCRIBE SET for node 2 to get it repopulating.
+
+Unfortunately, replication suddenly stopped to node 3.
+
+The problem was that there was not a suitable set of "listener paths"
+in sl_listen to allow the events from node 1 to propagate to node 3.
+The events were going through node 2, and blocking behind the
+SUBSCRIBE SET event that node 2 was working on.
+
+The following slonik script dropped out the listen paths where node 3
+had to go through node 2, and added in direct listens between nodes 1
+and 3.
+
+<verbatim>
+cluster name = oxrslive;
+ node 1 admin conninfo='host=32.85.68.220 dbname=oxrslive user=postgres port=5432';
+ node 2 admin conninfo='host=32.85.68.216 dbname=oxrslive user=postgres port=5432';
+ node 3 admin conninfo='host=32.85.68.244 dbname=oxrslive user=postgres port=5432';
+ node 4 admin conninfo='host=10.28.103.132 dbname=oxrslive user=postgres port=5432';
+try {
+ store listen (origin = 1, receiver = 3, provider = 1);
+ store listen (origin = 3, receiver = 1, provider = 3);
+ drop listen (origin = 1, receiver = 3, provider = 2);
+ drop listen (origin = 3, receiver = 1, provider = 2);
+}
+</verbatim>
+Immediately after this script was run, SYNC events started propagating
+again to node 3.
+
+This points out two principles:
+
+ * If you have multiple nodes, and cascaded subscribers, you need to be quite careful in populating the STORE LISTEN entries, and in
+ modifying them if the structure of the replication "tree" changes.
+
+ * Version 1.1 probably ought to provide better tools to help manage this.
+
+The issues of "listener paths" are discussed further at SlonyListenPaths
+
--- /dev/null
+++ doc/adminguide/SlonyDropThings.txt
@@ -0,0 +1,72 @@
+%META:TOPICINFO{author="guest" date="1099633479" format="1.0" version="1.3"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Dropping things from Slony Replication
+
+There are several things you might want to do involving dropping things from Slony-I replication.
+
+---+++ Dropping A Whole Node
+
+If you wish to drop an entire node from replication, the Slonik command DROP NODE should do the trick.
+
+This will lead to Slony-I dropping the triggers (generally that deny the ability to update data), restoring the "native" triggers, dropping the schema used by Slony-I, and the slon process for that node terminating itself.
+
+As a result, the database should be available for whatever use your application makes of the database.
+
+This is a pretty major operation, with considerable potential to cause substantial destruction; make sure you drop the right node!
+
+The operation will fail if there are any nodes subscribing to the node that you attempt to drop, so there is a bit of failsafe.
+
+SlonyFAQ17 documents some extra maintenance that may need to be done on sl_confirm if you are running versions prior to 1.0.5.
+
+---+++ Dropping An Entire Set
+
+If you wish to stop replicating a particular replication set, the Slonik command DROP SET is what you need to use.
+
+Much as with DROP NODE, this leads to Slony-I dropping the Slony-I triggers on the tables and restoring "native" triggers. One difference is that this takes place on *all* nodes in the cluster, rather than on just one node. Another difference is that this does not clear out the Slony-I cluster's namespace, as there might be other sets being serviced.
+
+This operation is quite a bit more dangerous than DROP NODE, as there _isn't_ the same sort of "failsafe." If you tell DROP SET to drop the _wrong_ set, there isn't anything to prevent "unfortunate results."
+
+---+++ Unsubscribing One Node From One Set
+
+The UNSUBSCRIBE SET operation is a little less invasive than either DROP SET or DROP NODE; it involves dropping Slony-I triggers and restoring "native" triggers on one node, for one replication set.
+
+Much like with DROP NODE, this operation will fail if there is a node subscribing to the set on this node.
+
+---+++ Warning!!!
+
+For all of the above operations, "turning replication back on" will require that the node copy in a *full* fresh set of the data on a provider. The fact that the data was recently being replicated isn't good enough; Slony-I will expect to refresh the data from scratch.
+
+---+++ Dropping A Table From A Set
+
+In Slony 1.0.5 and above, there is a Slonik command SET DROP TABLE that allows dropping a single table from replication without forcing the user to drop the entire replication set.
+
+If you are running an earlier version, there is a "hack" to do this:
+
+You can fiddle this by hand by finding the table ID for the table you want to get rid of, which you can find in sl_table, and then run the following three queries, on each host:
+
+<verbatim>
+ select _slonyschema.alterTableRestore(40);
+ select _slonyschema.tableDropKey(40);
+ delete from _slonyschema.sl_table where tab_id = 40;
+</verbatim>
+
+The schema will obviously depend on how you defined the Slony-I cluster. The table ID, in this case, 40, will need to change to the ID of the table you want to have go away.
+
+You'll have to run these three queries on all of the nodes, preferably firstly on the "master" node, so that the dropping of this propagates
+properly. Implementing this via a Slonik statement with a new Slony event would do that. Submitting the three queries using EXECUTE
+SCRIPT could do that. Also possible would be to connect to each database and submit the queries by hand.
+
+---+++ Dropping A Sequence From A Set
+
+Just as with SET DROP TABLE, version 1.0.5 introduces the operation SET DROP SEQUENCE.
+
+If you are running an earlier version, here are instructions as to how to drop sequences:
+
+The data that needs to be deleted to stop Slony from continuing to replicate the two sequences identified with Sequence IDs 93 and 59 are thus:
+
+<verbatim>
+delete from _oxrsorg.sl_seqlog where seql_seqid in (93, 59);
+delete from _oxrsorg.sl_sequence where seq_id in (93,59);
+</verbatim>
+
+Those two queries could be submitted to all of the nodes via ddlscript() / EXECUTE SCRIPT, thus eliminating the sequence everywhere "at once." Or they may be applied by hand to each of the nodes.
--- /dev/null
+++ doc/adminguide/SlonyFAQ13.txt
@@ -0,0 +1,23 @@
+%META:TOPICINFO{author="guest" date="1099542268" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ Some nodes start consistently falling behind
+
+I have been running Slony-I on a node for a while, and am seeing
+system performance suffering.
+
+I'm seeing long running queries of the form:
+<verbatim>
+ fetch 100 from LOG;
+</verbatim>
+
+This is characteristic of pg_listener (which is the table containing
+NOTIFY data) having plenty of dead tuples in it. That makes NOTIFY
+events take a long time, and causes the affected node to gradually
+fall further and further behind.
+
+You quite likely need to do a VACUUM FULL on pg_listener, to vigorously clean it out, and need to vacuum pg_listener really frequently. Once every five minutes would likely be AOK.
+
+Slon daemons already vacuum a bunch of tables, and cleanup_thread.c contains a list of tables that are frequently vacuumed automatically. In Slony-I 1.0.2, pg_listener is not included. In 1.0.5 and later, it is regularly vacuumed, so this should cease to be a direct issue.
+
+There is, however, still a scenario where this will still "bite." Vacuums cannot delete tuples that were made "obsolete" at any time after the start time of the eldest transaction that is still open. Long running transactions will cause trouble, and should be avoided, even on "slave" nodes.
+
--- /dev/null
+++ doc/adminguide/SlonyFAQ17.txt
@@ -0,0 +1,20 @@
+%META:TOPICINFO{author="guest" date="1099632840" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ sl_log_1 grows after dropping a node
+
+After dropping a node, sl_log_1 isn't getting purged out anymore.
+
+This is a common scenario in versions before 1.0.5, as the "clean up" that takes place when purging the node does not include purging out old entries from the Slony-I table, sl_confirm, for the recently departed node.
+
+The node is no longer around to update confirmations of what syncs have been applied on it, and therefore the cleanup thread that purges log entries thinks that it can't safely delete entries newer than the final sl_confirm entry, which rather curtails the ability to purge out old logs.
+
+In version 1.0.5, the "drop node" function purges out entries in sl_confirm for the departing node. In earlier versions, this needs to be done manually. Supposing the node number is 3, then the query would be:
+
+<verbatim>
+delete from _namespace.sl_confirm where con_origin = 3 or con_received = 3;
+</verbatim>
+
+General "being careful" dictates starting with a BEGIN, looking at the contents of sl_confirm before, ensuring that only the expected records are purged, and then, only after that, confirming the change with a COMMIT. If you delete confirm entries for the wrong node, that could ruin your whole day.
+
+You'll need to run this on each node that remains...
+
--- /dev/null
+++ doc/adminguide/SlonyFAQ11.txt
@@ -0,0 +1,37 @@
+%META:TOPICINFO{author="guest" date="1099542075" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ I need to drop a sequence from a replication set
+
+If you are running 1.0.5 or later, there is a SET DROP SEQUENCE
+command in Slonik to allow you to do this, parallelling SET DROP
+TABLE.
+
+If you are running 1.0.2 or earlier, the process is a bit more manual.
+
+Supposing I want to get rid of the two sequences listed below,
+whois_cachemgmt_seq and epp_whoi_cach_seq_, we start by needing the
+seq_id values.
+
+<verbatim>
+oxrsorg=# select * from _oxrsorg.sl_sequence where seq_id in (93,59);
+ seq_id | seq_reloid | seq_set | seq_comment
+--------+------------+---------+-------------------------------------
+ 93 | 107451516 | 1 | Sequence public.whois_cachemgmt_seq
+ 59 | 107451860 | 1 | Sequence public.epp_whoi_cach_seq_
+(2 rows)
+</verbatim>
+
+The data that needs to be deleted to stop Slony from continuing to
+replicate these are thus:
+
+<verbatim>
+delete from _oxrsorg.sl_seqlog where seql_seqid in (93, 59);
+delete from _oxrsorg.sl_sequence where seq_id in (93,59);
+</verbatim>
+
+Those two queries could be submitted to all of the nodes via
+ddlscript() / EXECUTE SCRIPT, thus eliminating the sequence everywhere
+"at once." Or they may be applied by hand to each of the nodes.
+
+Similarly to SET DROP TABLE, this should be in place for Slony-I version
+1.0.5 as SET DROP SEQUENCE.
--- /dev/null
+++ doc/adminguide/SlonySlonik.txt
@@ -0,0 +1,17 @@
+%META:TOPICINFO{author="guest" date="1097691381" format="1.0" version="1.4"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+*Introduction*
+
+Slonik is a command line utility designed specifically to setup and modify configurations of the Slony-I replication system.
+
+*General outline*
+
+The slonik command line utility is supposed to be used embedded into shell scripts and reads commands from files or stdin.
+
+It reads a set of Slonik statements, which are written in a scripting language with syntax similar to that of SQL, and performs the set of configuration changes on the slony nodes specified in the script.
+
+Nearly all of the real configuration work is actually done by calling stored procedures after loading the Slony-I support base into a database. Slonik was created because these stored procedures have special requirements as to on which particular node in the replication system they are called. The absence of named parameters for stored procedures makes it rather hard to do this from the psql prompt, and psql lacks the ability to maintain multiple connections with open transactions to multiple databases.
+
+The format of the Slonik "language" is very similar to that of SQL, and the parser is based on a similar set of formatting rules for such things as numbers and strings. Note that slonik is declarative, using literal values throughout, and does _not_ have the notion of variables. It is anticipated that Slonik scripts will typically be *generated* by scripts, such as Bash or Perl, and these sorts of scripting languages already have perfectly good ways of managing variables.
+
+A detailed list of Slonik commands can be found here: [[http://gborg.postgresql.org/project/slony1/genpage.php?slonik_commands][http://gborg.postgresql.org/project/slony1/genpage.php?slonik_commands]]
--- /dev/null
+++ doc/adminguide/SlonyFAQ03.txt
@@ -0,0 +1,14 @@
+%META:TOPICINFO{author="guest" date="1098326455" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ Cluster name with "-" in it
+
+I tried creating a CLUSTER NAME with a "-" in it. That didn't work.
+
+Slony-I uses the same rules for unquoted identifiers as the PostgreSQL
+main parser, so no, you probably shouldn't put a "-" in your
+identifier name.
+
+You may be able to defeat this by putting "quotes" around identifier
+names, but it's liable to bite you some, so this is something that is
+probably not worth working around.
+
--- /dev/null
+++ doc/adminguide/SlonyHandlingFailover.txt
@@ -0,0 +1,129 @@
+%META:TOPICINFO{author="guest" date="1097850667" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Doing switchover and failover with Slony-I
+
+---+++ Foreword
+
+ Slony-I is an asynchronous replication system. Because of that, it
+ is almost certain that at the moment the current origin of a set
+ fails, the last transactions committed have not propagated to the
+ subscribers yet. They always fail under heavy load, and you know
+ it. Thus the goal is to prevent the main server from failing.
+ The best way to do that is frequent maintenance.
+
+ Opening the case of a running server is not exactly what we
+ all consider professional system maintenance. And interestingly,
+ those users who use replication for backup and failover
+ purposes are usually the ones that have a very low tolerance for
+ words like "downtime". To meet these requirements, Slony-I has
+ not only failover capabilities, but controlled master role transfer
+ features too.
+
+ It is assumed in this document that the reader is familiar with
+ the slonik utility and knows at least how to set up a simple
+ 2 node replication system with Slony-I.
+
+---+++ Switchover
+
+ We assume a current "origin" as node1 (AKA master) with one
+ "subscriber" as node2 (AKA slave). A web application on a third
+ server is accessing the database on node1. Both databases are
+ up and running and replication is more or less in sync.
+
+ * Step 1)
+
+ * At the time of this writing switchover to another server requires the application to reconnect to the database. So in order to avoid any complications, we simply shut down the web server. Users who use pg_pool for the applications database connections can shutdown the pool only.
+
+ * Step 2)
+
+ * A small slonik script executes the following commands:
+<verbatim>
+ lock set (id = 1, origin = 1);
+ wait for event (origin = 1, confirmed = 2);
+ move set (id = 1, old origin = 1, new origin = 2);
+ wait for event (origin = 1, confirmed = 2);
+</verbatim>
+
+ After these commands, the origin (master role) of data set 1
+ is now on node2. It is not simply transferred. It is done
+ in a fashion so that node1 is now a fully synchronized subscriber
+ actively replicating the set. So the two nodes completely switched
+ roles.
+
+ * Step 3)
+
+ * After reconfiguring the web application (or pgpool) to connect to the database on node2 instead, the web server is restarted and resumes normal operation.
+
+ Done in one shell script, that does the shutdown, slonik, move
+ config files and startup all together, this entire procedure
+ takes less than 10 seconds.
+
+ It is now possible to simply shutdown node1 and do whatever is
+ required. When node1 is restarted later, it will start replicating
+ again and eventually catch up after a while. At this point the
+ whole procedure is executed with exchanged node IDs and the
+ original configuration is restored.
+
+
+---+++ Failover
+
+ Because of the possibility of missing not-yet-replicated
+ transactions that are committed, failover is the worst thing
+ that can happen in a master-slave replication scenario. If there
+ is any possibility to bring back the failed server even if only
+ for a few minutes, we strongly recommended that you follow the
+ switchover procedure above.
+
+ Slony does not provide any automatic detection for failed systems.
+ Abandoning committed transactions is a business decision that
+ cannot be made by a database. If someone wants to put the
+ commands below into a script executed automatically from the
+ network monitoring system, well ... its your data.
+
+ * Step 1)
+
+ * The slonik command
+<verbatim>
+ failover (id = 1, backup node = 2);
+</verbatim>
+
+ causes node2 to assume the ownership (origin) of all sets that
+ have node1 as their current origin. In the case there would be
+ more nodes, All direct subscribers of node1 are instructed that
+ this is happening. Slonik would also query all direct subscribers
+ to figure out which node has the highest replication status
+ (latest committed transaction) for each set, and the configuration
+ would be changed in a way that node2 first applies those last
+ minute changes before actually allowing write access to the
+ tables.
+
+ In addition, all nodes that subscribed directly from node1 will
+ now use node2 as data provider for the set. This means that
+ after the failover command succeeded, no node in the entire
+ replication setup will receive anything from node1 any more.
+
+ * Step 2)
+
+ Reconfigure and restart the application (or pgpool) to cause it
+ to reconnect to node2.
+
+ * Step 3)
+
+ After the failover is complete and node2 accepts write operations
+ against the tables, remove all remnants of node1's configuration
+ information with the slonik command
+
+<verbatim>
+ drop node (id = 1, event node = 2);
+</verbatim>
+
+---+++ After failover, getting back node1
+
+ After the above failover, the data stored on node1 must be
+ considered out of sync with the rest of the nodes. Therefore, the
+ only way to get node1 back and transfer the master role to it is
+ to rebuild it from scratch as a slave, let it catch up and then
+ follow the switchover procedure.
+
+Based on [[http://gborg.postgresql.org/project/slony1/genpage.php?howto_failover]]
+
--- /dev/null
+++ doc/adminguide/SlonySlonConfiguration.txt
@@ -0,0 +1,61 @@
+%META:TOPICINFO{author="guest" date="1097952716" format="1.0" version="1.6"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+Slon parameters:
+
+usage: slon [options] clustername conninfo
+
+<verbatim>
+Options:
+-d <debuglevel> verbosity of logging (1..8)
+-s <milliseconds> SYNC check interval (default 10000)
+-t <milliseconds> SYNC interval timeout (default 60000)
+-g <num> maximum SYNC group size (default 6)
+-c <num> how often to vacuum in cleanup cycles
+-p <filename> slon pid file
+-f <filename> slon configuration file
+</verbatim>
+
+*-d*
+<verbatim>
+The eight levels of logging are:
+- Error
+- Warn
+- Config
+- Info
+- Debug1
+- Debug2
+- Debug3
+- Debug4
+</verbatim>
+
+*-s*
+
+A SYNC event will be sent at least this often, regardless of whether update activity is detected.
+
+Short sync times keep the master on a "short leash," updating the slaves more frequently. If you have replicated sequences that are frequently updated _without_ there being tables that are affected, this keeps there from being times when only sequences are updated, and therefore _no_ syncs take place.
+
+Longer sync times allow there to be fewer events, which allows somewhat better efficiency.
+
+*-t*
+
+The time before the SYNC check interval times out.
+
+*-g*
+
+Number of SYNC events to try to cram together. The default is 6, which is probably suitable for small systems that can devote only very limited bits of memory to slon. If you have plenty of memory, it would be reasonable to increase this, as it will increase the amount of work done in each transaction, and will allow a subscriber that is behind by a lot to catch up more quickly.
+
+Slon processes usually stay pretty small; even with large value for this option, slon would be expected to only grow to a few MB in size.
+
+*-c*
+
+How often to vacuum (_e.g._ - how many cleanup cycles to run before vacuuming).
+
+Set this to zero to disable slon-initiated vacuuming. If you are using something like pg_autovacuum to initiate vacuums, you may not need for slon to initiate vacuums itself. If you are not, there are some tables Slony-I uses that collect a LOT of dead tuples that should be vacuumed frequently.
+
+*-p*
+
+The location of the PID file for the slon process.
+
+*-f*
+
+The location of the slon configuration file.
--- /dev/null
+++ doc/adminguide/SlonyStartSlons.txt
@@ -0,0 +1,19 @@
+%META:TOPICINFO{author="guest" date="1097257140" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Slon Daemons
+
+The programs that actually perform Slony-I replication are the "slon" daemons.
+
+You need to run one "slon" instance for each node in a Slony-I cluster. It is not essential that these daemons run on any particular host, but there are some principles worth considering:
+
+ * Each slon needs to be able to communicate quickly with the database whose "node controller" it is. Therefore, if a Slony-I cluster runs across some form of Wide Area Network, the slon processes should run on or nearby the databases each is controlling. If you break this rule, there is no immediate disaster, but the added latency introduced to monitoring events on the slon's "own node" will cause it to replicate in a somewhat less timely manner.
+
+ * The fastest results would be achieved by having each slon run on the database server that it is servicing. If it runs somewhere within a fast local network, performance will not be noticeably degraded.
+
+ * It is an attractive idea to run many of the slon processes for a cluster on one machine, as this makes it easy to monitor them both in terms of log files and process tables from one location. This eliminates the need to login to several hosts in order to look at log files or to restart slon instances.
+
+There are two "watchdog" scripts currently available:
+
+ * tools/altperl/slon_watchdog.pl - an "early" version that basically wraps a loop around the invocation of slon, restarting any time it may fail
+ * tools/altperl/slon_watchdog2.pl - a somewhat more intelligent version that periodically polls the database, checking to see if a SYNC has taken place recently. We have had VPN connections that occasionally fall over without signalling the application, so that the slon stops working, but doesn't actually die; this polling accounts for that...
+
--- /dev/null
+++ doc/adminguide/SlonyFAQ06.txt
@@ -0,0 +1,21 @@
+%META:TOPICINFO{author="guest" date="1099541323" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ Slonik fails - cannot load PostgreSQL library - PGRES_FATAL_ERROR load '$libdir/xxid';
+
+ When I run the sample setup script I get an error message similar
+to:
+
+<verbatim>
+<stdin>:64: PGRES_FATAL_ERROR load '$libdir/xxid'; - ERROR: LOAD:
+could not open file '$libdir/xxid': No such file or directory
+</verbatim>
+
+Evidently, you haven't got the xxid.so library in the $libdir directory that the PostgreSQL instance is using. Note that the Slony-I components need to be installed in the PostgreSQL software installation for _each and every one_ of the nodes, not just on the "master node."
+
+This may also point to there being some other mismatch between the PostgreSQL binary instance and the Slony-I instance. If you compiled
+Slony-I yourself, on a machine that may have multiple PostgreSQL builds "lying around," it's possible that the slon or slonik binaries
+are asking to load something that isn't actually in the library directory for the PostgreSQL database cluster that it's hitting.
+
+Long and short: This points to a need to "audit" what installations of PostgreSQL and Slony you have in place on the machine(s).
+Unfortunately, just about any mismatch will cause things not to link up quite right. See also SlonyFAQ02 concerning threading issues on Solaris
+...
--- /dev/null
+++ doc/adminguide/SlonyHelp.txt
@@ -0,0 +1,14 @@
+%META:TOPICINFO{author="guest" date="1099494380" format="1.0" version="1.5"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+If you are having problems with Slony-I, you have several options for help:
+
+ * [[http://slony.info/][http://slony.info/]] - the official "home" of Slony
+ * Documentation on the Slony-I Site- Check the documentation on the Slony website: [[http://gborg.postgresql.org/project/slony1/genpage.php?howto_idx][http://gborg.postgresql.org/project/slony1/genpage.php?howto_idx]]
+ * Other Documentation - There are several articles here ([[http://www.varlena.com/varlena/GeneralBits/Tidbits/index.php#Replication][http://www.varlena.com/varlena/GeneralBits/Tidbits/index.php#Replication]] that may be helpful.
+ * IRC - There are usually some people on #slony on irc.freenode.net who may be able to answer some of your questions. There is also a bot named "rtfm_please" that you may want to chat with.
+ * Mailing lists - The answer to your problem may exist in the Slony1-general mailing list archives, or you may choose to ask your question on the Slony1-general mailing list. The mailing list archives, and instructions for joining the list may be found here: [[http://gborg.postgresql.org/mailman/listinfo/slony1][http://gborg.postgresql.org/mailman/listinfo/slony1]]
+ * If your Russian is much better than your English, then [[http://kirov.lug.ru/wiki/Slony][KirovOpenSourceCommunity: Slony]] may be the place to go
+
+---+++ Other Information Sources
+
+ * [[http://comstar.dotgeek.org/postgres/slony-config/][slony-config]] - A Perl tool for configuring Slony nodes using config files in an XML-based format that the tool transforms into a Slonik script
--- /dev/null
+++ doc/adminguide/SlonyReshapingCluster.txt
@@ -0,0 +1,15 @@
+%META:TOPICINFO{author="guest" date="1099633739" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Reshaping A Cluster
+
+If you rearrange the nodes so that they serve different purposes, this will likely lead to the subscribers changing a bit.
+
+This will require doing several things:
+
+ * If you want a node that is a subscriber to become the "master" provider for a particular replication set, you will have to issue the slonik MOVE SET operation to change that "master" provider node.
+
+ * You may subsequently, or instead, wish to modify the subscriptions of other nodes. You might want to modify a node to get its data from a different provider, or to change it to turn forwarding on or off. This can be accomplished by issuing the slonik SUBSCRIBE SET operation with the new subscription information for the node; Slony-I will change the configuration.
+
+ * If the directions of data flows have changed, it is doubtless appropriate to issue a set of DROP LISTEN operations to drop out obsolete paths between nodes and SET LISTEN to add the new ones. At present, this is not changed automatically; at some point, MOVE SET and SUBSCRIBE SET might change the paths as a side-effect. See SlonyListenPaths for more information about this. In version 1.1 and later, it is likely that the generation of sl_listen entries will be entirely automated, where they will be regenerated when changes are made to sl_path or sl_subscribe, thereby making it unnecessary to even think about SET LISTEN.
+
+The "altperl" toolset includes a "init_cluster.pl" script that is quite up to the task of creating the new SET LISTEN commands; it isn't smart enough to know what listener paths should be dropped.
--- /dev/null
+++ doc/adminguide/SlonyAdministrationScripts.txt
@@ -0,0 +1,22 @@
+%META:TOPICINFO{author="guest" date="1098117183" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyIAdministration"}%
+---++ Slony Administration Scripts
+
+In the "altperl" directory in the CVS tree, there is a sizable set of Perl scripts that may be used to administer a set of Slony-I instances, which support having arbitrary numbers of nodes.
+
+Most of them generate Slonik scripts that are then to be passed on to the slonik utility to be submitted to all of the Slony-I nodes in a particular cluster. At one time, this embedded running slonik on the slonik scripts. Unfortunately, this turned out to be a pretty large calibre "foot gun," as minor typos on the command line led, on a couple of occasions, to pretty calamitous actions, so the behaviour has been changed so that the scripts simply submit output to standard output. An administrator should review the slonik script before submitting it to Slonik.
+
+---+++ Node/Cluster Configuration - cluster.nodes
+
+The UNIX environment variable SLONYNODES is used to determine what Perl configuration file will be used to control the shape of the nodes in a Slony-I cluster.
+
+What variables are set up...
+
+---+++ Set configuration - cluster.set1, cluster.set2
+
+The UNIX environment variable SLONYSET is used to determine what Perl configuration file will be used to determine what objects will be contained in a particular replication set.
+
+Unlike SLONYNODES, which is essential for ALL of the slonik-generating scripts, this only needs to be set when running create_set.pl, to control what tables will be in a particular replication set.
+
+What variables are set up...
+
--- /dev/null
+++ doc/adminguide/SlonyFAQ04.txt
@@ -0,0 +1,31 @@
+%META:TOPICINFO{author="guest" date="1099542005" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ slon does not restart after crash
+
+After an immediate stop of postgresql (simulation of system crash)
+in pg_catalog.pg_listener a tuple with
+relname='_${cluster_name}_Restart' exists. slon doesn't start cause it
+thinks another process is serving the cluster on this node. What can
+I do? The tuples can't be dropped from this relation.
+
+The logs claim that "Another slon daemon is serving this node already"
+
+It's handy to keep a slonik script like the following one around to
+run in such cases:
+
+<verbatim>
+twcsds004[/opt/twcsds004/OXRS/slony-scripts]$ cat restart_org.slonik
+cluster name = oxrsorg ;
+node 1 admin conninfo = 'host=32.85.68.220 dbname=oxrsorg user=postgres port=5532';
+node 2 admin conninfo = 'host=32.85.68.216 dbname=oxrsorg user=postgres port=5532';
+node 3 admin conninfo = 'host=32.85.68.244 dbname=oxrsorg user=postgres port=5532';
+node 4 admin conninfo = 'host=10.28.103.132 dbname=oxrsorg user=postgres port=5532';
+restart node 1;
+restart node 2;
+restart node 3;
+restart node 4;
+</verbatim>
+
+'restart node n' cleans up dead notifications so that you can restart the node.
+
+As of version 1.0.5, the startup process of slon looks for this condition, and automatically cleans it up.
--- /dev/null
+++ doc/adminguide/SlonyFAQ14.txt
@@ -0,0 +1,37 @@
+%META:TOPICINFO{author="guest" date="1099541409" format="1.0" version="1.2"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ I started doing a backup using pg_dump, and suddenly Slony stops
+
+Ouch. What happens here is a conflict between:
+
+ * pg_dump, which has taken out an AccessShareLock on all of the tables in the database, including the Slony-I ones, and
+
+ * A Slony-I sync event, which wants to grab a AccessExclusiveLock on the table sl_event.
+
+The initial query that will be blocked is thus:
+
+<verbatim>
+ select "_slonyschema".createEvent('_slonyschema, 'SYNC', NULL);
+</verbatim>
+
+(You can see this in pg_stat_activity, if you have query display turned on in postgresql.conf)
+
+The actual query combination that is causing the lock is from the function Slony_I_ClusterStatus(), found in slony1_funcs.c, and is localized in the code that does:
+
+<verbatim>
+ LOCK TABLE %s.sl_event;
+ INSERT INTO %s.sl_event (...stuff...)
+ SELECT currval('%s.sl_event_seq');
+</verbatim>
+
+The LOCK statement will sit there and wait until pg_dump (or whatever else has pretty much any kind of access lock on sl_event) completes.
+
+Every subsequent query submitted that touches sl_event will block behind the createEvent call.
+
+There are a number of possible answers to this:
+
+ * Have pg_dump specify the schema dumped using --schema=whatever, and don't try dumping the cluster's schema.
+
+ * It would be nice to add an "--exclude-schema" option to pg_dump to exclude the Slony cluster schema. Maybe in 8.0 or 8.1...
+
+Note that 1.0.5 uses a more precise lock that is less exclusive that relieves this problem.
--- /dev/null
+++ doc/adminguide/SlonyFAQ05.txt
@@ -0,0 +1,9 @@
+%META:TOPICINFO{author="guest" date="1098326597" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ ps finds passwords on command line
+
+ If I run a "ps" command, I, and everyone else, can see passwords
+on the command line.
+
+Take the passwords out of the Slony configuration, and put them into
+$(HOME)/.pgpass.
--- /dev/null
+++ doc/adminguide/README
@@ -0,0 +1,6 @@
+This will become an "administrative guide" to the use of Slony-I.
+
+At present, it consists of copies of the Wiki at
+http://cbbrowne.dyndns.info:8741/cgi-bin/twiki/view/Sandbox/SlonyIAdministration
+
+That will eventually be transformed into DocBook.
--- /dev/null
+++ doc/adminguide/SlonyFAQ15.txt
@@ -0,0 +1,20 @@
+%META:TOPICINFO{author="guest" date="1098327360" format="1.0" version="1.1"}%
+%META:TOPICPARENT{name="SlonyFAQ"}%
+---++ The slons spent the weekend out of commission [for some reason], and it's taking a long time to get a sync through.
+
+You might want to take a look at the sl_log_1/sl_log_2 tables, and do
+a summary to see if there are any really enormous Slony-I transactions
+in there. Up until at least 1.0.2, there needs to be a slon connected
+to the master in order for SYNC events to be generated.
+
+If none are being generated, then all of the updates until the next
+one is generated will collect into one rather enormous Slony-I
+transaction.
+
+Conclusion: Even if there is not going to be a subscriber around, you
+_really_ want to have a slon running to service the "master" node.
+
+Some future version (probably 1.1) may provide a way for SYNC counts
+to be updated on the master by the stored function that is invoked by
+the table triggers.
+
- Previous message: [Slony1-commit] By cbbrowne: New Directory
- Next message: [Slony1-commit] By cbbrowne: Minor changes to comments
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the Slony1-commit mailing list