Interesting concept, and that would be accurate.  The tables
do have large rows.  For example, the body of the Email could
be a multi-meg attachment.  Basically we are storing Email, and
we allow up to 10M attachments.

I should have mentioned that we are using PostgreSQL 8.0.3.

Just to be clear, I should change SLON_DATA_FETCH_SIZE = 1
and leave SLON_COMMANDS_PER_LINE = 10 because I'm using >= 7.4
correct?

Second question is, if I'm going to go ahead and hack on the
code some, I figure I will go to 1.1.2.  Anything that I
should worry about for the upgrade?  Will I need to re-replicate
everything?  I suppose I should just go read the upgrade
notes, but I'm looking for those "oh yeah, by the way ...."
items.

Thanks.

--
David A. Niblett               | email: niblettda at gru.net
Network Administrator          | Phone: (352) 334-3400
Gainesville Regional Utilities | Web: http://www.gru.net/



-----Original Message-----
From: Christopher Browne [mailto:cbbrowne at ca.afilias.info] 
Sent: Monday, October 24, 2005 2:27 PM
To: Niblett, David A
Cc: 'slony1-general at gborg.postgresql.org'
Subject: Re: [Slony1-general] Excessive Swap Memory Usage on Slave


Niblett, David A wrote:

>I'm hoping that it's just a stupid user configuration error that I've 
>made. I can't seem to find anything in the archives that really fits 
>the bill here.
>  
>
Question:  Do some of the tables have Extremely Large rows?  e.g. - rows
that are megabytes in size?

The typical place where things "break down" in the way you describe is if
you are using Slony-I to replicate sets of data that occasionally include
rows that are really large.

The main work "cursor" on the subscriber draws in 100 records at a time from
the provider to apply them on the subscriber.

Supposing a bunch of those records were 50MB in size, apiece, this would
lead to memory consumption for that slon growing to accomodate those records
in memory.  And that memory, once claimed, doesn't readily return to the OS
until the slon terminates.

I have some ideas as to how version 1.2 might address this; in the interim,
you might try modifying how much data is fetched in slon.h.

Cut down SLON_DATA_FETCH_SIZE from 100 (used for PG 7.3) to 10, or from 10
(for >= PG 7.4) to 1 and the "high water mark" for memory consumption should
fall considerably.

#ifdef    SLON_CHECK_CMDTUPLES
#define SLON_COMMANDS_PER_LINE        1
#define SLON_DATA_FETCH_SIZE        100
#define SLON_WORKLINES_PER_HELPER    (SLON_DATA_FETCH_SIZE * 4)
#else
#define SLON_COMMANDS_PER_LINE        10
#define SLON_DATA_FETCH_SIZE        10
#define SLON_WORKLINES_PER_HELPER    (SLON_DATA_FETCH_SIZE * 50)
#endif