<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 12, 2019 at 9:33 AM Christopher Browne <<a href="mailto:cbbrowne@afilias.info" target="_blank">cbbrowne@afilias.info</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Tue, Jun 11, 2019 at 3:44 PM Tory M Blue <<a href="mailto:tmblue@gmail.com" target="_blank">tmblue@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 11, 2019 at 12:26 PM Christopher Browne <<a href="mailto:cbbrowne@afilias.info" target="_blank">cbbrowne@afilias.info</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 11, 2019 at 1:30 PM Tory M Blue <<a href="mailto:tmblue@gmail.com" target="_blank">tmblue@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I'm wondering if there is anything I can do to prevent what appears to be slony replication backup (not replicating) during a pg_dump. I'm using a -N slonschema, so wondering if there is any other techniques to prevent slon replication delay during pg_dumps?<br></div></blockquote><div><br></div><div>Hmm. I remember the "big deal" being to ensure that the Slony schema was left out, as that would definitely lead to locking that would cause replication to fall behind.</div><div><br></div><div>It's possible that something new has come along that would lock in a troublesome way.</div><div><br></div><div>I suggest that you take a look in pg_catalog.pg_locks to see what objects Slony is waiting on, thus, looking for:</div><div><br></div><div> select * from pg_catalog.pg_locks where not granted;</div><div><br></div><div>Some further work needs to be done to trace that to exactly what objects are locked.<br></div></div></div></blockquote><div><br></div><div>Ya will take a look, I also thought that bypassing the slon schema would solve it but apparently not, we can have 2-10million count in the sl_log before it clears and it's apparent that the replication is not happening, so I'll poke around some more. Glad to see folks are still around :)</div></div></div></blockquote><div><br></div><div>Well, growth of sl_log during the backup is fine, and not an indication of anything untowards. Given that you opened a transaction when pg_dump began, and that has not completed, that will lead to Slony not trimming anything out of sl_log during the backup. That's normal, routine stuff. Nothing's wrong (not on the basis of that, anyways).</div><div><br></div><div>On the origin node, you can query the view sl_status, which will show how far behind replication appears to be. (The view is available on all nodes; it does not return terribly interesting results on nodes other than the origin.)</div><div><br></div><div>Here's an example of replication being very far behind:</div><div><br></div><div>usaorigin=# select * from _usa.sl_status limit 3;<br>-[ RECORD 1 ]-------------+------------------------------<br>st_origin | 1<br>st_received | 3<br>st_last_event | 5000002124<br>st_last_event_ts | 2019-04-10 20:02:43.833414+00<br>st_last_received | 5000002124<br>st_last_received_ts | 2019-04-10 20:02:43.885806+00<br>st_last_received_event_ts | 2019-04-10 20:02:43.833414+00<br>st_lag_num_events | 0<br>st_lag_time | 62 days 20:23:14.029816<br>-[ RECORD 2 ]-------------+------------------------------<br>st_origin | 1<br>st_received | 4<br>st_last_event | 5000002124<br>st_last_event_ts | 2019-04-10 20:02:43.833414+00<br>st_last_received | 5000002122<br>st_last_received_ts | 2019-04-10 20:02:33.870577+00<br>st_last_received_event_ts | 2019-04-10 20:02:23.76373+00<br>st_lag_num_events | 2<br>st_lag_time | 62 days 20:23:34.0995<br>-[ RECORD 3 ]-------------+------------------------------<br>st_origin | 1<br>st_received | 5<br>st_last_event | 5000002124<br>st_last_event_ts | 2019-04-10 20:02:43.833414+00<br>st_last_received | 5000002122<br>st_last_received_ts | 2019-04-10 20:02:33.898605+00<br>st_last_received_event_ts | 2019-04-10 20:02:23.76373+00<br>st_lag_num_events | 2<br>st_lag_time | 62 days 20:23:34.0995<br></div><div><br></div><div>That is a replication cluster where I shut down slon processes back in April :-)</div><div><br></div><div>It's only behind by a few events, for the self-same reason :-)</div><div><br></div><div>If the last event received coincides with the time at which you started the pg_dump run, well, that sure sounds like a smoking gun.</div><div><br></div><div>If, on the other hand, SYNC events are being processed, just at slower speed than you'd like, then perhaps the problem is that the I/O induced by pg_dump is delaying replication, and that's more a hardware problem than a software one.</div></div><div class="gmail_quote"><br></div><div class="gmail_quote">And if replication is a few seconds behind, while the system is busy, as indicated by st_lag_time being a few seconds, then it's possible that there is no problem there.<br></div></div></blockquote><div><br></div><div>First postgres 9.x</div><div>CentOS 7.2</div><div>Replication flows:</div><div>primary ->secondary-> 3 query slaves</div><div><br></div><div>Okay here it is, </div><div><br></div><div>My dump started:</div><div><p class="gmail-m_-1330120594842900310gmail-p1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)"><span class="gmail-m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span></p><p class="gmail-m_-1330120594842900310gmail-p1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;margin:0px;color:rgb(0,0,0)"><span class="gmail-m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">Start export of schema clsdb for user postgres at Wed Jun 12 19:00:48 PDT 2019</span></p></div><div><br></div><div>So if I'm reading this correct it's 20:45 but something is showing that the last _received_event_ts was at 19:38? over an hour ago?</div><div>sl_lag_time 1 hour?</div><div><br></div><div>Primary:</div><div>
<p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">select * from _cls.sl_status limit 3;</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>st_origin | st_received | st_last_event | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_event_ts<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>| st_last_receive</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">d |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_received_ts<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>| <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_received_event_ts <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>| st_lag_num_events |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">st_lag_time<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">-----------+-------------+---------------+-------------------------------+----------------</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">--+-------------------------------+-------------------------------+-------------------+---</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">--------------</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>11 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>14 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>5036030946 | 2019-06-12 20:45:29.465785-07 | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>503603013</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 | 2019-06-12 19:38:11.065976-07 | 2019-06-12 19:38:03.311267-07 | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>808 | 01</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">:07:33.039694</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>11 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>15 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>5036030946 | 2019-06-12 20:45:29.465785-07 | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>503603013</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 | 2019-06-12 19:38:12.202539-07 | 2019-06-12 19:38:03.311267-07 | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>808 | 01</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">:07:33.039694</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>11 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>12 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>5036030946 | 2019-06-12 20:45:29.465785-07 | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>503603013</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">8 | 2019-06-12 19:38:08.342637-07 | 2019-06-12 19:38:03.311267-07 | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>808 | 01</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">:07:33.039694</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">
</p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">(3 rows)</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">I have 2.4 million rows in the sl_log_1 table, which looks like it started to hockey stick right around 19:30.</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><br></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">On the secondary (which replicates to my query db's (where the dump is run from (not run from the primary), it includes these lines</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">Secondary:</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>select * from _cls.sl_status limit 3;</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>st_origin | st_received | st_last_event | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_event_ts<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>| st_la</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">st_received |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_received_ts<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>| <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>st_last_received_event_ts <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>| st</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">_lag_num_events | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>st_lag_time<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">-----------+-------------+---------------+-------------------------------+------</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">------------+-------------------------------+-------------------------------+---</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">----------------+-----------------</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>12 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>11 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650405 | 2019-06-12 20:45:35.365757-07 | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650000 | 2019-06-12 19:38:03.815508-07 | 2019-06-12 19:38:03.338486-07 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>405 | 01:07:35.106539</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>12 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>13 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650405 | 2019-06-12 20:45:35.365757-07 | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650403 | 2019-06-12 20:45:17.701455-07 | 2019-06-12 20:45:15.355233-07 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>2 | 00:00:23.089792</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>12 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>14 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650405 | 2019-06-12 20:45:35.365757-07 | <span class="m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>5008650404 | 2019-06-12 20:45:26.642094-07 | 2019-06-12 20:45:25.360167-07 |<span class="m_-1330120594842900310gmail-Apple-converted-space"> </span></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="m_-1330120594842900310gmail-Apple-converted-space"> </span>1 | 00:00:13.084858</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font:14px Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)">
</p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures">(3 rows)</span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="m_-1330120594842900310gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="m_-1330120594842900310gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><br></p></div></div></div>