Vivek Khera wrote:

>
> On Feb 18, 2005, at 10:58 AM, Christopher Browne wrote:
>
>> I would instead go to the node with empty sl_listen and do:
>>
>> -# select _somecluster.storelisten(2,3,2), 
>> _somecluster.storelisten(3,2,3);
>>
>> That will add both entries on that node, and propagate the event to 
>> add them everywhere else.
>>
>
> Cool.  that added the listens... but now my slon process on node2 
> won't start.  it is failing with this query:
>
> ERROR  remoteWorkerThread_3: "begin transaction; set transaction 
> isolation level
>  serializable; lock table "_mailermailer".sl_config_lock; select 
> "_mailermailer"
> .storeListen_int(2, 1, 3); notify "_mailermailer_Event"; notify 
> "_mailermailer_C
> onfirm"; insert into "_mailermailer".sl_event     (ev_origin, 
> ev_seqno, ev_times
> tamp,      ev_minxid, ev_maxxid, ev_xip, ev_type , ev_data1, ev_data2, 
> ev_data3
>    ) values ('3', '2215', '2005-02-18 10:30:42.529048', '3286814', 
> '3286815', ''
> , 'STORE_LISTEN', '2', '1', '3'); insert into "_mailermailer".sl_confirm
> (con_origin, con_received, con_seqno, con_timestamp)    values (3, 2, 
> '2215', CU
> RRENT_TIMESTAMP); commit transaction;" PGRES_FATAL_ERROR ERROR:  
> insert or updat
> e on table "sl_listen" violates foreign key constraint 
> "sl_listen-sl_path-ref"
> DETAIL:  Key (li_provider,li_receiver)=(1,3) is not present in table 
> "sl_path".
> DEBUG1 syncThread: thread done
>
>
> ie, it is attempting to insert a listen path for a node that is 
> non-existent now.

Ouch.

There's evidently a STORE LISTEN event queued that can't succeed because 
node 1 is gone.

Methinks you'll need to rummage thru sl_event, probably on BOTH nodes, 
looking for events that reference node 1, and get rid of those events.  
That's a real ugly thing to do, but I don't see much alternative :-(.

Note that the problem on node 2 is almost certainly NOT with events 
stored on node 2; it is with events that node 2 is pulling from node 3.  
You have to fix node 3's data for node 2 to come back online...