<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Hello slony1 community,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> We have a head scratcher here. It appears a DROP NODE command was not fully processed. The command was issued and confirmed on all our nodes at approximately 2018-02-21 19:19:50 UTC. When we
went to restore it over two hours later, all replication stopped on an sl_event constraint violation. Investigation showed a SYNC event for the dropped node with a timestamp of just a few seconds before the drop. I believe this is a first for us. The DROP
NODE command is supposed to remove all state for the dropped node. Is that right? Is there a potential race condition somewhere which could leave behind state?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Thanks in advance,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- master log replication freeze error ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 21:38:52 UTC [5775] ERROR remoteWorkerThread_8: "insert into "_ams_cluster".sl_event (ev_origin, ev_seqno, ev_timestamp, ev_snapshot, ev\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">_type ) values ('8', '5002075962', '2018-02-21 19:19:41.958719+00', '87044110:87044110:', 'SYNC'); insert into "_ams_cluster".sl_confirm (con_origi\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">n, con_received, con_seqno, con_timestamp) values (8, 1, '5002075962', now()); select "_ams_cluster".logApplySaveStats('_ams_cluster', 8, '0.139 s'::inter\<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">val); commit transaction;" PGRES_FATAL_ERROR ERROR: duplicate key value violates unique constraint "sl_event-pkey"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">DETAIL: Key (ev_origin, ev_seqno)=(8, 5002075962) already exists.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 21:38:52 UTC [13649] CONFIG slon: child terminated signal: 9; pid: 5775, current worker pid: 5775<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 21:38:52 UTC [13649] CONFIG slon: restart of worker in 10 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- master log replication freeze error ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- master DB leftover event ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ams@ams6.cmb.netmgmt:~$ psql -U akamai -d ams<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">psql (9.1.24)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Type "help" for help.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ams=# select * from sl_event_bak;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ev_origin | ev_seqno | ev_timestamp | ev_snapshot | ev_type | ev_data1 | ev_data2 | ev_data3 | ev_data4 | ev_data5 | ev_data6 | ev_<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">data7 | ev_data8 <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">-----------+------------+-------------------------------+--------------------+---------+----------+----------+----------+----------+----------+----------+----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">------+----------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 8 | 5002075962 | 2018-02-21 19:19:41.958719+00 | 87044110:87044110: | SYNC | | | | | | | <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> | <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">(1 row)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ams=# <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- master DB leftover event ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- master log drop node record ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:50 UTC [22582] CONFIG disableNode: no_id=8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:50 UTC [22582] CONFIG storeListen: li_origin=4 li_receiver=1 li_provider=4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:50 UTC [22582] CONFIG storeListen: li_origin=7 li_receiver=1 li_provider=7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:50 UTC [22582] CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:50 UTC [22582] CONFIG remoteWorkerThread_4: update provider configuration<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:50 UTC [22582] CONFIG remoteWorkerThread_4: connection for provider 4 terminated<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:50 UTC [22582] CONFIG remoteWorkerThread_4: disconnecting from data provider 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:50 UTC [22582] CONFIG remoteWorkerThread_4: connection for provider 7 terminated<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- master log drop node record ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- replica log drop node record ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:51 UTC [22650] WARN remoteWorkerThread_1: got DROP NODE for local node ID<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">NOTICE: Slony-I: Please drop schema "_ams_cluster"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2018-02-21 19:19:53 UTC [22650] INFO remoteWorkerThread_7: SYNC 5001868819 done in 2.153 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">NOTICE: drop cascades to 243 other objects<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">DETAIL: drop cascades to table _ams_cluster.sl_node<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drop cascades to table _ams_cluster.sl_nodelock<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drop cascades to table _ams_cluster.sl_set<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drop cascades to table _ams_cluster.sl_setsync<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drop cascades to table _ams_cluster.sl_table<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drop cascades to table _ams_cluster.sl_sequence<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---- replica log drop node record ----<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Tom </span>
<span style="font-size:11.0pt;font-family:"Apple Color Emoji"">☺</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</body>
</html>