<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style>
<!--
@font-face
{font-family:Helvetica}
@font-face
{font-family:Helvetica}
@font-face
{font-family:Calibri}
@font-face
{font-family:Consolas}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
p
{margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif"}
code
{font-family:"Courier New"}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
span.apple-converted-space
{}
.MsoChpDefault
{font-family:"Calibri","sans-serif"}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
</style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p style="margin:0in; margin-bottom:.0001pt; line-height:13.5pt; background:white; vertical-align:baseline">
<strong><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222; border:none windowtext 1.0pt; padding:0in">Context:</span></strong><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222"></span></p>
<p style="margin-right:0in; margin-bottom:12.0pt; margin-left:0in; line-height:13.5pt; background:white; vertical-align:baseline; word-wrap:break-word; orphans:2; widows:2; word-spacing:0px">
<span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">We run Slony 2.0 with Postgres 8.4 on two CentOS 6 servers--one master, and one slave. Our database is about 30GB in size, which isn't unusual, but we do have a couple of tables
that are more than 5GB each.</span></p>
<p style="margin:0in; margin-bottom:.0001pt; line-height:13.5pt; background:white; vertical-align:baseline; word-wrap:break-word; orphans:2; widows:2; word-spacing:0px">
<span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">Recently, we needed to re-build our Slony cluster. I turned off Slony, restored identical database snapshots on the master and the slave, set up my slony.conf and slon_tools.conf,
started the slons, ran </span><code><span style="font-size:9.0pt; font-family:Consolas; color:#222222; border:none windowtext 1.0pt; padding:0in; background:#F3F3F3">slonik_init_cluster | slonik</span></code><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">,
then<span class="apple-converted-space"> </span></span><code><span style="font-size:9.0pt; font-family:Consolas; color:#222222; border:none windowtext 1.0pt; padding:0in; background:#F3F3F3">slonik_create_set 1 | slonik</span></code><span class="apple-converted-space"><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222"> </span></span><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">(we
only have one replication set), and finally<span class="apple-converted-space"> </span></span><code><span style="font-size:9.0pt; font-family:Consolas; color:#222222; border:none windowtext 1.0pt; padding:0in; background:#F3F3F3">slonik_subscribe_set 1 2 |
slonik</span></code><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">. Everything looked good, and I was able to watch subscription progress in the logs.</span></p>
<p style="margin-right:0in; margin-bottom:12.0pt; margin-left:0in; line-height:13.5pt; background:white; vertical-align:baseline; word-wrap:break-word; orphans:2; widows:2; word-spacing:0px">
<span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">Then the server stopped responding. I rebooted it, and saw "Kernel panic - not syncing: Out of memory and no killable processes" after it had killed everything it could.</span></p>
<p style="margin:0in; margin-bottom:.0001pt; line-height:13.5pt; background:white; vertical-align:baseline">
<span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">It happens during the subscription process when our first “large” (3GB) table is encountered. The logs report “so and so bytes copied for table” for the table in question, then
a few dozen queued SYNC events, and then they detect a child process crash and log a watchdog-initialized restart.
</span></p>
<p style="margin-right:0in; margin-bottom:12.0pt; margin-left:0in; line-height:13.5pt; background:white; vertical-align:baseline">
<span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222"> </span></p>
<p style="margin:0in; margin-bottom:.0001pt; line-height:13.5pt; background:white; vertical-align:baseline; word-wrap:break-word; orphans:2; widows:2; word-spacing:0px">
<strong><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222; border:none windowtext 1.0pt; padding:0in">What I've tried:</span></strong><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222"></span></p>
<p style="margin:0in; margin-bottom:.0001pt; line-height:13.5pt; background:white; vertical-align:baseline; word-wrap:break-word; orphans:2; widows:2; word-spacing:0px">
<span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">First I blew away the database completely, re ran<span class="apple-converted-space"> </span></span><code><span style="font-size:9.0pt; font-family:Consolas; color:#222222; border:none windowtext 1.0pt; padding:0in; background:#F3F3F3">initdb</span></code><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">,
and then restored the identical snapshots again. Same kernel panic. Then I blew it away, uninstalled Postgres and Slony, and reinstalled them. I double-checked all of our memory-based settings in postgresql.conf, and they are all at stock/recommended levels
(i.e.<span class="apple-converted-space"> </span></span><code><span style="font-size:9.0pt; font-family:Consolas; color:#222222; border:none windowtext 1.0pt; padding:0in; background:#F3F3F3">shared_buffers</span></code><span class="apple-converted-space"><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222"> </span></span><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">is
at 1/4 of RAM etc etc). I ran a<span class="apple-converted-space"> </span></span><code><span style="font-size:9.0pt; font-family:Consolas; color:#222222; border:none windowtext 1.0pt; padding:0in; background:#F3F3F3">VACUUM ANALYZE FULL
</span></code><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">on the database before initializing the Slony cluster. As a last-ditch effort, I completely reinstalled the OS and all base software on the db servers and started
from scratch. Same result every time: kernel panic, out of memory. It happens on the slave server, which runs both of the slons.
</span></p>
<p style="margin:0in; margin-bottom:.0001pt; line-height:13.5pt; background:white; vertical-align:baseline">
<span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222"> </span></p>
<p style="margin:0in; margin-bottom:.0001pt; line-height:13.5pt; background:white; vertical-align:baseline; word-wrap:break-word; orphans:2; widows:2; word-spacing:0px">
<strong><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222; border:none windowtext 1.0pt; padding:0in">Question:</span></strong><span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222"></span></p>
<p style="margin-right:0in; margin-bottom:12.0pt; margin-left:0in; line-height:13.5pt; background:white; vertical-align:baseline; word-wrap:break-word; orphans:2; widows:2; word-spacing:0px">
<span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">Why is this happening?</span></p>
<p style="margin-right:0in; margin-bottom:12.0pt; margin-left:0in; line-height:13.5pt; background:white; vertical-align:baseline; word-wrap:break-word; orphans:2; widows:2; word-spacing:0px">
<span style="font-size:10.5pt; font-family:"Helvetica","sans-serif"; color:#222222">Our database has grown fairly linearly over the past few months (at the beginning of the year it was about 23GB, now it's 30), and every other time I have had to re-initialize
the Slony cluster on these same servers, it has worked fine.</span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Zac Bentley</p>
<p class="MsoNormal">Systems Administrator</p>
<p class="MsoNormal">Corporate Reimbursement Services, Inc.</p>
<p class="MsoNormal"><a href="http://www.crsinc.com/"><span style="color:blue">www.crsinc.com</span></a></p>
<p class="MsoNormal">617-467-1949</p>
<p class="MsoNormal"> </p>
</div>
<p><font size="-2">This email message contains information that Corporate Reimbursement Services, Inc. considers confidential and/or proprietary, or may later designate as confidential and proprietary. It is intended only for use of the individual or entity
named above and should not be forwarded to any other persons or entities without the express consent of Corporate Reimbursement Services, Inc., nor should it be used for any purpose other than in the course of any potential or actual business relationship
with Corporate Reimbursement Services, Inc. If the reader of this message is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution, or copying
of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately and destroy the original message.</font></p>
<p><font size="-2">Internal Revenue Service regulations require that certain types of written advice include a disclaimer. To the extent the preceding message contains advice relating to a Federal tax issue, unless expressly stated otherwise the advice is not
intended or written to be used, and it cannot be used by the recipient or any other taxpayer, for the purpose of avoiding Federal tax penalties, and was not written to support the promotion or marketing of any transaction or matter discussed herein.</font></p>
</body>
</html>