Problem:  I can't setup replication, copy of a large table (45 GB)
does not complete.


Data:  Master has a 58 GB database (according to psql "\l") and over
45 GB is in a single table (including TOAST and indexes).

Slony logs show that the COPY of other tables start and complete, but
this 45 GB table, the COPY starts, and is still running after 11
hours.

(When the database was half the size it is now, it'd take about an
hour or 90 minutes to copy to our DR using Slony.)

tcpdump shows Slony is still trying to copy this table.   (I see what
looks like the contents of this table going from Master to DR)

Other indicators of trouble:


/usr/bin/test_slony_state-dbi.pl tells me:

Summary of sl_confirm aging
   Origin   Receiver   Min SYNC   Max SYNC  Age of latest SYNC  Age of
eldest SYNC
=================================================================================
        1          2        156        156      11:43:00      11:43:00    1
        2          1       4141       4205      00:00:00      00:10:00    0


Node: 1 sl_seqlog tuples = 225368 > 200000
================================================
Number of tuples in Slony-I table sl_seqlog is 225368 which
exceeds 200000.

You may wish to investigate whether or not a node is down, or perhaps
if sl_confirm entries have not been propagating properly.


Node: 2 sl_seqlog tuples = 225368 > 200000
================================================
Number of tuples in Slony-I table sl_seqlog is 225368 which
exceeds 200000.

You may wish to investigate whether or not a node is down, or perhaps
if sl_confirm entries have not been propagating properly.

--- end of output from test_slony_state-dbi.pl ---

Bucardo's check_postgres.pl reports:

	POSTGRES_SEQUENCE CRITICAL: [master database] sl_log_status=100% (calls left=0)


The master's Slony log looks like this:
2011-02-13 10:35:53 PSTDEBUG2 localListenThread: Received event 1,11085 SYNC
2011-02-13 10:35:55 PSTDEBUG2 syncThread: new sl_action_seq 64567 - SYNC 11086
2011-02-13 10:35:55 PSTDEBUG2 localListenThread: Received event 1,11086 SYNC
2011-02-13 10:35:57 PSTDEBUG2 syncThread: new sl_action_seq 64578 - SYNC 11087
2011-02-13 10:35:57 PSTDEBUG2 localListenThread: Received event 1,11087 SYNC
2011-02-13 10:35:58 PSTDEBUG2 remoteListenThread_2: LISTEN
2011-02-13 10:35:59 PSTDEBUG2 syncThread: new sl_action_seq 64634 - SYNC 11088
2011-02-13 10:35:59 PSTDEBUG2 localListenThread: Received event 1,11088 SYNC
2011-02-13 10:35:59 PSTDEBUG2 remoteListenThread_2: queue event 2,4320 SYNC
2011-02-13 10:35:59 PSTDEBUG2 remoteListenThread_2: UNLISTEN
2011-02-13 10:35:59 PSTDEBUG2 remoteWorkerThread_2: Received event 2,4320 SYNC
2011-02-13 10:35:59 PSTDEBUG2 calc sync size - last time: 1 last
length: 9995 ideal: 6 proposed size: 3
2011-02-13 10:35:59 PSTDEBUG2 remoteWorkerThread_2: SYNC 4320 processing
2011-02-13 10:35:59 PSTDEBUG2 remoteWorkerThread_2: no sets need
syncing for this event
2011-02-13 10:36:01 PSTDEBUG2 syncThread: new sl_action_seq 64638 - SYNC 11089
2011-02-13 10:36:01 PSTDEBUG2 localListenThread: Received event 1,11089



The Slave's Slony log shows:

2011-02-13 10:37:02 PSTDEBUG2 remoteListenThread_1: queue event 1,11116 SYNC
2011-02-13 10:37:04 PSTDEBUG2 remoteListenThread_1: queue event 1,11117 SYNC
2011-02-13 10:37:06 PSTDEBUG2 remoteListenThread_1: queue event 1,11118 SYNC
2011-02-13 10:37:08 PSTDEBUG2 remoteListenThread_1: queue event 1,11119 SYNC
2011-02-13 10:37:10 PSTDEBUG2 syncThread: new sl_action_seq 1 - SYNC 4327
2011-02-13 10:37:12 PSTDEBUG2 localListenThread: Received event 2,4327 SYNC
2011-02-13 10:37:16 PSTDEBUG2 remoteListenThread_1: queue event 1,11120 SYNC
2011-02-13 10:37:18 PSTDEBUG2 remoteListenThread_1: queue event 1,11121 SYNC
2011-02-13 10:37:20 PSTDEBUG2 syncThread: new sl_action_seq 1 - SYNC 4328
2011-02-13 10:37:24 PSTDEBUG2 localListenThread: Received event 2,4328 SYNC


The Slony server is not breaking a sweat (according to vmstat) - CPU
is 95% idle.

Any suggestions?

Best,
-at