On 7/11/2008 11:00 AM, chris wrote:
> Cyril SCETBON <scetbon at echo.fr> writes:
>> I've got a lot of updates (~1000 w/s) on the master and on the nearest
>> slave (like others) I've got a big replication lag. It seems that
>> fetching lines from the cursor is not taking much time, but processing
>> events is not well managed.
>>
>> Here you can find an extract of the log that shows that the proposed
>> size for grouping is often just 3 and not greater :
>>
>> http://pastebin.com/f2a2974a
>>
>> The slon parameters used are : -g 1000 -o 0
>>
>> Any idea to speed up the processing ?
> 
> There's something a bit confusing in the logs; it's not affecting how
> the grouping is actually working.  The "just 3" looks to be when it's
> evaluating sync grouping for events coming from *other* nodes than the
> provider, which is pretty much irrelevant since those syncs don't lead
> to any actual work.

Correct. The 3 or 4 group size is for non origins. The current group 
size for node 1, which seems to be the only origin in the system, is 
actually 255.

> 
> Looks to me like the processing of data from node #1 is working out
> about as can be expected on a node that is processing a lot of data.
> There may be relevant/material improvements to handling of this in
> 2.0; I don't think there's anything to be done in terms of
> configuration.

I seem to remember that Slony had some problems with large numbers of 
sets. Although it seems that in this particular case it only accounts 
for 0.3 out of 135 seconds to completely process 255 sync events.

Anyhow, I counted 31 sets with a total of 513 tables which have a pretty 
strange pattern. Up to set 29, all the odd set id's have 17 tables while 
the even set id's have 18 tables. Is this one of those misdesigned 
applications that creates a separate set of tables per user or the like?


Jan

-- 
Anyone who trades liberty for security deserves neither
liberty nor security. -- Benjamin Franklin