Karl Denninger wrote:
> But on the other clients slony instantly pitches a fit and claims that
> the ID of the master is incorrect (which is, by the way, correct - the
> connection now points to node 4 instead of node 2.)  I thought I could
> get around this with a pair of "store path" statements to change the
> node Ids for each client doing this:
>
>   
That's because they're still connecting with the previous conninfo but
you figured that out later.
> include <slonik.preamble>;
> store path (server=1, client=4, conninfo='dbname=ticker
> host=genesis.denninger.net user=slony port=55432 password=xxxxx');
> store path (server= 4, client=1, conninfo='dbname=ticker
> host=colo1.denninger.net user=slony port=5432 password=xxxxx');
>
> That logs a bunch of messages for provider configuration updates in the
> syslog as expected when it is executed.  The slonik.preamble contains:
>
> CLUSTER NAME = tickerforum;
> node 4 admin conninfo='dbname=ticker host=colo1.denninger.net user=slony
> password=xxxx port=5432';
> node 1 admin conninfo='dbname=ticker host=localhost user=slony port=5432
> password=xxxx';
>
> (The divergence between the path stored into the dbms and the admin info
> is due to the fact that the OTHER nodes "See" this machine - node 1 -
> through a firewall that redirects ports, but to connect locally on this
> machine I can't "bounce" off from the inside.  This is "good" and has
> worked to set it up and replicate....)
>
> No dice; I get lots of these:
>
> Aug 22 23:40:08 dbms slon[25559]: [1830-1] CONFIG version for
> "dbname=ticker host=colo1.denninger.net user=slony port=5432
> password=xxxx" is 80400
> Aug 22 23:40:08 dbms slon[25559]: [1831-1] ERROR  remoteListenThread_2:
> db_getLocalNodeId() returned 4 - wrong database?
> Aug 22 23:40:19 dbms slon[25559]: [1832-1] CONFIG version for
> "dbname=ticker host=colo1.denninger.net user=slony port=5432
> password=xxxx" is 80400
> Aug 22 23:40:19 dbms slon[25559]: [1833-1] ERROR  remoteListenThread_2:
> db_getLocalNodeId() returned 4 - wrong database?
>
> Over and over again....
>
> It looks like the path statement was ignored and attempting to update it
> does nothing.
>   
That's because they aren't talking to the master anymore, so they never
get the command.
> The only way I was able to get replication going again was to drop the
> node.  That, however, while it executed, did not clear the schema (!)
> nor did it stop the slon process from trying to connect; I had to go in
> and drop the slony schema from each database, then re-add the node to
> the replication set.  This worked, BUT it also forced a copy of the
> entire freaking database!
>   
drop node doesn't uninstall the slony schema - it only removes it from
the slony schema on the master node.  The 'uninstall node' command
actually connects to the node and removes the slony schema.  Also, after
the node has been dropped, slony doesn't save any information about the
dropped node so when you add it again, it forces a full sync. 
> I'm wondering what happened here.  It is almost as if the "move set"
> never executed on the other subscribers - an impossibility, no?  They
> WERE all replicating and current just before the shutdown - I checked
> them all.  How does that happen under these circumstances?
>
> Is there a better way for the future?  I'm back up now, but the entire
> point of this exercise was to AVOID having to copy the entire database
> over - while I avoided any material downtime for my users, I was left
> EXPOSED to a failure for the copy period, which was kinda nasty.
>
> Thoughts appreciated.
>
>   

Probably the way to avoid it would have been to issue the store path
changes before switching the ports.  But, if you forget to do it in the
future, you can fix it afterwards by going bare metal and updating the
paths in the _tickerform.sl_path table on the nodes that don't have the
correct information.

-- 
Jeff Frost <jeff at pgexperts.com>
COO, PostgreSQL Experts, Inc.
Phone: 1-888-PG-EXPRT x506
http://www.pgexperts.com/