Log Message:
-----------
Added a section to "Best Practices" with tips on handling Very Large Sets

Changed config discussion to reflect recent parameter change - maximum
SYNC grouping is now 10,000, rather than just 100.

Modified Files:
--------------
    slony1-engine/doc/adminguide:
        bestpractices.sgml (r1.6 -> r1.7)
        bookindex.sgml (r1.8 -> r1.9)
        slonconf.sgml (r1.6 -> r1.7)

-------------- next part --------------
Index: slonconf.sgml
===================================================================
RCS file: /usr/local/cvsroot/slony1/slony1-engine/doc/adminguide/slonconf.sgml,v
retrieving revision 1.6
retrieving revision 1.7
diff -Ldoc/adminguide/slonconf.sgml -Ldoc/adminguide/slonconf.sgml -u -w -r1.6 -r1.7
--- doc/adminguide/slonconf.sgml
+++ doc/adminguide/slonconf.sgml
@@ -248,7 +248,7 @@
           only one <command>SYNC</command> available, even
           <option>-g60</option> will apply just that one. As soon as a
           subscriber catches up, it will apply every single
-          <command>SYNC</command> by itself.  Range: [0,100], default:
+          <command>SYNC</command> by itself.  Range: [0,10000], default:
           6
         </para>
       </listitem>
Index: bookindex.sgml
===================================================================
RCS file: /usr/local/cvsroot/slony1/slony1-engine/doc/adminguide/bookindex.sgml,v
retrieving revision 1.8
retrieving revision 1.9
diff -Ldoc/adminguide/bookindex.sgml -Ldoc/adminguide/bookindex.sgml -u -w -r1.8 -r1.9
--- doc/adminguide/bookindex.sgml
+++ doc/adminguide/bookindex.sgml
@@ -25,13 +25,13 @@
   <primaryie> DDL changes 
   </primaryie>
   <secondaryie> changing the database schema ,
-    <ulink url="ddlchanges.html" role="AEN1665">Database Schema Changes (DDL)</ulink>
+    <ulink url="ddlchanges.html" role="AEN1702">Database Schema Changes (DDL)</ulink>
   </secondaryie>
 </indexentry>
 
 <indexentry>
   <primaryie> Frequently Asked Questions about Slony-I,
-    <ulink url="faq.html" role="AEN2427"></ulink>
+    <ulink url="faq.html" role="AEN2464"></ulink>
   </primaryie>
 </indexentry>
 
@@ -47,7 +47,7 @@
 
 <indexentry>
   <primaryie>cluster_name configuration parameter,
-    <ulink url="slon-config-connection.html" role="AEN3542">Connection settings</ulink>
+    <ulink url="slon-config-connection.html" role="AEN3610">Connection settings</ulink>
   </primaryie>
 </indexentry>
 
@@ -55,19 +55,19 @@
   <primaryie>configuration
   </primaryie>
   <secondaryie>of the slon daemon,
-    <ulink url="runtime-config.html" role="AEN3440"></ulink>
+    <ulink url="runtime-config.html" role="AEN3508"></ulink>
   </secondaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>conn_info configuration parameter,
-    <ulink url="slon-config-connection.html" role="AEN3552">Connection settings</ulink>
+    <ulink url="slon-config-connection.html" role="AEN3620">Connection settings</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>Controlled switchover,
-    <ulink url="failover.html#AEN1005" role="AEN1007"> Controlled Switchover</ulink>
+    <ulink url="failover.html#AEN1042" role="AEN1044"> Controlled Switchover</ulink>
   </primaryie>
 </indexentry>
 
@@ -76,7 +76,13 @@
 
 <indexentry>
   <primaryie>desired_sync_time configuration parameter,
-    <ulink url="slon-config-interval.html" role="AEN3622">Event Tuning</ulink>
+    <ulink url="slon-config-interval.html" role="AEN3690">Event Tuning</ulink>
+  </primaryie>
+</indexentry>
+
+<indexentry>
+  <primaryie>drop_indices_for_copy configuration parameter,
+    <ulink url="slon-config-interval.html" role="AEN3702">Event Tuning</ulink>
   </primaryie>
 </indexentry>
 
@@ -85,7 +91,7 @@
 
 <indexentry>
   <primaryie>failover upon system failure,
-    <ulink url="failover.html#AEN1032" role="AEN1034"> Failover</ulink>
+    <ulink url="failover.html#AEN1069" role="AEN1071"> Failover</ulink>
   </primaryie>
 </indexentry>
 
@@ -94,37 +100,37 @@
 
 <indexentry>
   <primaryie>listen paths,
-    <ulink url="listenpaths.html" role="AEN1093">Slony-I listen paths</ulink>
+    <ulink url="listenpaths.html" role="AEN1130">Slony-I listen paths</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>locking issues,
-    <ulink url="locking.html" role="AEN1268">Locking Issues</ulink>
+    <ulink url="locking.html" role="AEN1305">Locking Issues</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>log_level configuration parameter,
-    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3491">Logging</ulink>
+    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3559">Logging</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>log_pid configuration parameter,
-    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3500">Logging</ulink>
+    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3568">Logging</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>log_timestamp configuration parameter,
-    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3509">Logging</ulink>
+    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3577">Logging</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>log_timestamp_format configuration parameter,
-    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3518">Logging</ulink>
+    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3586">Logging</ulink>
   </primaryie>
 </indexentry>
 
@@ -151,7 +157,7 @@
 
 <indexentry>
   <primaryie>pid_file configuration parameter,
-    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3530">Logging</ulink>
+    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3598">Logging</ulink>
   </primaryie>
 </indexentry>
 
@@ -193,13 +199,13 @@
 
 <indexentry>
   <primaryie>Slony-I for PostgreSQL version upgrades,
-    <ulink url="versionupgrade.html" role="AEN2097">Using Slony-I for PostgreSQL Upgrades</ulink>
+    <ulink url="versionupgrade.html" role="AEN2134">Using Slony-I for PostgreSQL Upgrades</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>sql_on_connection configuration parameter,
-    <ulink url="slon-config-connection.html" role="AEN3562">Connection settings</ulink>
+    <ulink url="slon-config-connection.html" role="AEN3630">Connection settings</ulink>
   </primaryie>
 </indexentry>
 
@@ -217,37 +223,37 @@
 
 <indexentry>
   <primaryie>sync_group_maxsize configuration parameter,
-    <ulink url="slon-config-interval.html" role="AEN3598">Event Tuning</ulink>
+    <ulink url="slon-config-interval.html" role="AEN3666">Event Tuning</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>sync_interval configuration parameter,
-    <ulink url="slon-config-interval.html" role="AEN3575">Event Tuning</ulink>
+    <ulink url="slon-config-interval.html" role="AEN3643">Event Tuning</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>sync_interval_timeout configuration parameter,
-    <ulink url="slon-config-interval.html" role="AEN3584">Event Tuning</ulink>
+    <ulink url="slon-config-interval.html" role="AEN3652">Event Tuning</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>syslog configuration parameter,
-    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3463">Logging</ulink>
+    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3531">Logging</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>syslog_facility configuration parameter,
-    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3472">Logging</ulink>
+    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3540">Logging</ulink>
   </primaryie>
 </indexentry>
 
 <indexentry>
   <primaryie>syslog_ident configuration parameter,
-    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3482">Logging</ulink>
+    <ulink url="runtime-config.html#SLON-CONFIG-LOGGING" role="AEN3550">Logging</ulink>
   </primaryie>
 </indexentry>
 
@@ -256,7 +262,7 @@
 
 <indexentry>
   <primaryie>vac_frequency configuration parameter,
-    <ulink url="slon-config-interval.html" role="AEN3612">Event Tuning</ulink>
+    <ulink url="slon-config-interval.html" role="AEN3680">Event Tuning</ulink>
   </primaryie>
 </indexentry>
 
Index: bestpractices.sgml
===================================================================
RCS file: /usr/local/cvsroot/slony1/slony1-engine/doc/adminguide/bestpractices.sgml,v
retrieving revision 1.6
retrieving revision 1.7
diff -Ldoc/adminguide/bestpractices.sgml -Ldoc/adminguide/bestpractices.sgml -u -w -r1.6 -r1.7
--- doc/adminguide/bestpractices.sgml
+++ doc/adminguide/bestpractices.sgml
@@ -284,6 +284,83 @@
 <xref linkend="slonik"> scripts.</para>
 </listitem>
 
+<listitem><para> Handling Very Large Replication Sets </para>
+
+<para> Some users have set up replication on replication sets that are
+tens to hundreds of gigabytes in size, which puts some added
+<quote>strain</quote> on the system, in particular where it may take
+several days for the <command>COPY_SET</command> event to complete.
+Here are some principles that have been observed for dealing with
+these sorts of situtations.</para>
+
+<itemizedlist>
+
+<listitem><para> Drop all indices other than the primary key index
+while the <command>COPY_SET</command> event is run. </para>
+
+<para> When data is copied into a table that has indices on it,
+&postgres; builds the indices incrementally, on the fly.  This is much
+slower than simply copying the data into the table, and then
+recreating each index <quote>ex nihilo</quote>, as the latter can take
+substantial advantage of sort memory. </para>
+
+<para> In a future release, it is hopeful that indices will be dropped
+and recreated automatically, which would eliminate this.</para>
+</listitem>
+
+<listitem><para> If there are large numbers of updates taking place as
+the large set is being copied, this can lead to the subscriber being
+behind by some enormous number of <command>SYNC</command> events.</para>
+
+<para> If a <command> SYNC </command> is generated about once per
+second, that leads to the subscriber <quote>falling behind</quote> by
+around 90,000 <command>SYNC</command>s per day, possibly for several
+days.  </para>
+
+<para> There will correspondingly be an <emphasis>enormous</emphasis>
+growth of <xref linkend="table.sl-log-1"> and <xref
+linkend="table.sl-seqlog">.  Unfortunately, once the
+<command>COPY_SET</command> completes, users have found that the
+queries against these tables wind up reverting to <command>Seq
+Scans</command> so that even though a particular
+<command>SYNC</command> processing event is only processing a small
+number of those 90,000 <command>SYNC</command> events, it still reads
+through the entire table.  In such a case, you may never see
+replication catch up.
+</para> </listitem>
+
+<listitem><para> Several things can be done that will help, involving
+careful selection of <xref linkend="slon"> parameters:</para>
+
+<itemizedlist>
+
+<listitem><para> On the subscriber's <xref linkend="slon">, increase
+the number of <command>SYNC</command> events processed together, with
+the <xref linkend= "slon-config-sync-group-maxsize"> parameter to some
+value that allows it to process a significant portion of the
+outstanding <command>SYNC</command> events. </para> </listitem>
+
+<listitem><para> On the subscriber's <xref linkend="slon">, set the
+<xref linkend="slon-config-desired-sync-time"> to 0, as the adaptive
+<command>SYNC</command> grouping system will start with small
+groupings that will, under these circumstances, perform
+poorly. </para> </listitem>
+
+<listitem><para> Increase the <xref
+linkend="slon-config-sync-interval"> on the origin's <xref
+linkend="slon"> so that <command>SYNC</command> events are generated
+less frequently.  If a <command>SYNC</command> is only generated once
+per minute instead of once per second, that will cut down the number
+of events by a factor of 60. </para> </listitem>
+</itemizedlist>
+
+<listitem><para> It is likely to be worthwhile to use <xref
+linkend="slon-config-vac-frequency"> to deactivate <xref
+linkend="slon">-initiated vacuuming in favor of running your own
+vacuum scripts, as there will be a buildup of unpurgeable data while
+the data is copied and the subscriber starts to catch up. </para>
+</listitem>
+</listitem>
 </itemizedlist>
 
 </sect1>