http://www.slony.info/bugzilla/show_bug.cgi?id=235

--- Comment #3 from Daniel Kahn Gillmor <dkg at fifthhorseman.net> 2011-08-24 01:05:30 PDT ---
I think the logic behind the "ramping up" is that if the admin says "group at
most 3000 SYNCs at a time" (sync_group_maxsize) and "i want each run to take
about 60 seconds" (desired_sync_time), and it turns out that we can process
about 10 SYNCs per second, then if we just jump straight to 3000 SYNCs, the run
would take 5 times the recommended maximum duration.

So the current logic does a small number of SYNCs and then tries to predict
(based on some implicit assumption that the number of SYNCs varies linearly
with the duration of the SYNC group) what the ideal number of SYNCs would be to
hit the desired_sync_time.

Aside from overflow issues, the big problem with this approach is that the
number of SYNCs processed per group does *not* necessarily vary linearly with
the time it takes for the group to be processed.  Or, if it does vary linearly,
it is with a constant offset that can grow substantially if there are a large
number of elements in sl_log_1 or sl_log_2.

In my case, with > 8 GiB in sl_log_1, a SYNC group of size 1 takes several
hundred seconds, and a sync group of size 3000 *also* takes several hundred
seconds.  (maybe due to the full tablescan of sl_log_1?) 

As a result, the estimate of how many SYNCs to process in a group stays low,
which makes slony take even longer to finish processing its data :(

-- 
Configure bugmail: http://www.slony.info/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.
You are the assignee for the bug.