On Tue, 2006-03-21 at 08:35 -0800, Ujwal S. Setlur wrote:
> I tried tuning a bunch fof things, but the subscriber
> never did catch up, so I restarted replication from
> scratch. Make me a little nervous...

Chris,

I don't recall whether Slony would scale up the group size automatically
to a large transaction boundary if a normal group was smaller than the
largest transaction size in the time period it is covering.

Ujwal,

If not, bumping up the group size to a setting of several thousand, -g
10000 is standard operation here, it can catch it up in a hurry.

The catch is that each batch may take a couple of hours to process,
although you will be dealing with significantly more data than that.
Once it catches up group sizes transferred will automatically step back
down to a more reasonable size (copies everything available at the time
-- if it is only a few SYNCs then that is what it deals with).


I found that by the time the initial replication stage was complete
(data copy) the sl_log_1 structure can grow to be several hundred
million tuples in length.

Slony is not yet ready for handling a large (> 2TB) database with 2 or
more full copy slaves. The initial data copy simply becomes far too
painful.

400GB at 2B transactions/year already requires nearly double hardware
(CPU & RAM) to get through the initial data copy as would otherwise be
required for standard DB operation. Lack of VACUUM during this time is a
killer. Emptying sl_log_1 with everything being in a single transaction
at the beginning is also painful.

> --- "Jim C. Nasby" <jnasby at pervasive.com> wrote:
> 
> > Check the archives. There's a couple paramaters you
> > can tune to help the
> > situation (IIRC -g is one of them).
> > 
> > On Sat, Mar 18, 2006 at 08:02:25AM -0800, Ujwal S.
> > Setlur wrote:
> > > Hi,
> > > 
> > > The slon process on my subscriber had terminated
> > > complaining about a server process having crashed.
> > We
> > > noticed this after only about 10 days. I guess I
> > need
> > > to do something about that.
> > > 
> > > I then restarted slon on both origin and
> > subscriber,
> > > but the subscriber is not catching up. I am
> > monitoring
> > > the sl_lag_time, and every now and then, it goes
> > down,
> > > but not by much. Mostly it is slowly going up.
> > > 
> > > The origin is getting fed with quite a bit of
> > data, so
> > > maybe slon is finding it hard to catch up?
> > > 
> > > I am using slon 1.1.5 on postgresql 8.0.3. I am
> > also
> > > using a "g" value of 100.
> > > 
> > > Any ideas?
> > > 
> > > Thanks,
> > > 
> > > Ujwal
> > > 
> > > __________________________________________________
> > > Do You Yahoo!?
> > > Tired of spam?  Yahoo! Mail has the best spam
> > protection around 
> > > http://mail.yahoo.com 
> > > _______________________________________________
> > > Slony1-general mailing list
> > > Slony1-general at gborg.postgresql.org
> > >
> >
> http://gborg.postgresql.org/mailman/listinfo/slony1-general
> > > 
> > 
> > -- 
> > Jim C. Nasby, Sr. Engineering Consultant     
> > jnasby at pervasive.com
> > Pervasive Software      http://pervasive.com   
> > work: 512-231-6117
> > vcard: http://jim.nasby.net/pervasive.vcf      
> > cell: 512-569-9461
> > 
> > 
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
> _______________________________________________
> Slony1-general mailing list
> Slony1-general at gborg.postgresql.org
> http://gborg.postgresql.org/mailman/listinfo/slony1-general
> 
--