Unexpected SMTP delay or failure with some domains

[BGSWebDesign]

Hi DW,

If you get several of these in a block of 50 you might notice your counter incrementing very slowly. What we need to do is clean your list with a script such as mentioned here to remove these failing addresses from mailing entirely

I'm not sure this can be possible, if you remember from the very first posting, I would get one of these about every 500 to 1,000 addresses, that doesn't seem to be enough to cause the kind of delay I am seeing, my delay is TIMES TWO, it takes 2 to 3 times longer to send to a list!

TWO TO THREE TIMES LONGER - that would have to mean more bad email addresses than you would ever suspect - either that OR, you are not understanding fully how the SMTP connection is pausing every time it tries to send out an email...

As I stated earlier, it would be MUCH BETTER (ie FASTER and SIMPLER) to just place a simple command in resume.php, watch the reccount() on lm_sendq and if it is STUCK ON ONE RECORD, delete that record and move on - there can't be that many...

If you really want to test this - then write a simple front-end to that script you pointed me to - in that front end, go through a SPECIFIC number (100 or 1000) addresses, and keep a log, every email sent, log it, than the next, and next, and next, and watch the time delay between each address (make this available in your own admin/config of LMP) and we'll see how/why the delays are coming up - you're going to have to do something like this - you can NOT release this to others with large lists - it's going to hog up their resources BIG TIME!  Believe it...

Let me know what you think....

[DW]

Brett,

Please enable the SMTP log either globally or when using "Send Email" and you should be able to see exactly how many delayed/invalid addresses there are.

As I stated earlier, it would be MUCH BETTER (ie FASTER and SIMPLER) to just place a simple command in resume.php, watch the reccount() on lm_sendq and if it is STUCK ON ONE RECORD, delete that record and move on - there can't be that many...

One problem with this is that the lm_sendq table is not sorted.  That is, it has no 'auto-incrementing id' id which can be used to sort processing.  The first email one 'resume' may not be the first email the next.  This was done to prevent having to update the database index every time rows are added or deleted, which causes overhead.

There is one line I see that doesn't need to be in the getsmtpmsg function

Code: [Select]

if(!is_numeric($smtp_timeout)) $smtp_timeout = 9;
This can be moved out of the function and placed in admin.php, for example just above the "function domail" line or near the top of the file.  I also rewrote it like this:

Code: [Select]

if(!$smtp_timeout || !is_numeric($smtp_timeout)) $smtp_timeout = 9;
The call to stream_set_timeout appears correct as per the PHP documentation:
http://ca3.php.net/manual/en/function.stream-set-timeout.php

I tested this on another server and had what I thought to be reasonable success.  If you're reading this, Dean (no I'm not talking to myself.. ), what kind of speed are you experiencing with your email?

Brett, please enable your SMTP log ASAP so we can verify whether or not there are a significant number of invalid domains or if we need to look elsewhere.  A test installation/FTP access on your server I can use to troubleshoot would be amazing...

Regards

[BGSWebDesign]

Hi DW,

Please enable the SMTP log either globally or when using "Send Email" and you should be able to see exactly how many delayed/invalid addresses there are.

Next time I do a test send I'll enable it and let you know what I see... or send you the entire file...

One problem with this is that the lm_sendq table is not sorted. That is, it has no 'auto-incrementing id' id which can be used to sort processing. The first email one 'resume' may not be the first email the next. This was done to prevent having to update the database index every time rows are added or deleted, which causes overhead.

How would this be any different than what I do?  It's not, I just go into lm_sendq and delete the first record, that's it, the mailing continues on from there UNTIL it hits another bad email address - I don't see where this is a problem?

The call to stream_set_timeout appears correct as per the PHP documentation

I'm not so sure about that - look at the 'Ridera' test code posted up on Php.net,  have you ran that test code EMBEDDED in LMP, also have you seen this:

I have found it required to add

"stream_set_blocking($fp, FALSE )"

prior to any fgets(), fread(), etc. to prevent the code from hanging up when remote files are called and the response is slow.

Let us know what you are finding, I'll post a SMTP log probably tomorrow...

[mike2]

Whatever you do, if you do implement something like this, please make it totally optional.  DO NOT SET THIS AS DEFAULT BEHAVIOR.

I don't want some of my mailings NOT reaching some people because of temporary problems and a BROKEN mailer (and I'm not talking about Listmail...I mean your SMTP program).

[DW]

Mike, I hear you - I don't think I will be writing in a multiple-socket SMTP option anytime soon.  On my servers I run about 400 outbound connections at a time using qmail and there's no way I can beat that within the memory limits of PHP.  It could, however, prove useful as an option for welcome messages to avoid PHP mail()... :idea: :lol:

Brett,

How would this be any different than what I do? It's not, I just go into lm_sendq and delete the first record, that's it, the mailing continues on from there UNTIL it hits another bad email address - I don't see where this is a problem?

I may be mistaken but without an auto-incrementing ID we can't be sure of the order of the returned rows during sending, resuming, or with a manual 'Browse' as you do.   I understand that you seem to be able to grab the top row consistently, but I don't think it's guaranteed.  I don't know the specifics but I think MySQL could have a pointer that might be overwritten or lost if it 'expires' or the service is reset...  Items in queue use a random 32 character ID without a MySQL index which allows for sorting.  This is done because an index takes time to update when inserting or deleting rows to maintain quick sorted results, so it would slow down the queuing and sending processes.

If you want a custom script to do a general select and delete of the top row (if the # of entries hasn't changed in X time) I'd still be willing to program it for you.

Still, it would be better if we could check the logs and/or troubleshoot your server to figure out exactly why the system is hanging up...

look at the 'Ridera' test code posted up on Php.net

I had a look and it appears to be written to manually time the connection with blocking set to false.  Blocking off means the read command doesn't wait for data from the server.

ListMail *could* do it this way, but I wonder if it would be more processor-intensive.  Look at all the function calls, variable comparisons, etc. required.

A simple timeout set on the stream with blocking set to true returns a blank result which ListMail can use to know to reconnect to avoid waiting longer for the server.

The other post by ridera is concerning..

I have found it required to add

"stream_set_blocking($fp, FALSE )"

prior to any fgets(), fread(), etc. to prevent the code from hanging up when remote files are called and the response is slow.

This could be applicable here and would mean ridera's solution is the only solution.  I will have to investigate this further possibly by inducing delays etc. to simulate what you experience on your server.  It will be useful to see what's happening in your SMTP log.  While it won't tell us how 'slow' the mailing goes, it will at least rule out the possibility of the majority of delays coming from a very large number of bad addresses / reconnects. Then again, I wonder if by 'hangup' ridera means is a complete stall and not just 'slow'. Are you able to send a whole mailing without crashing, however slowly, with the smtp_timeout configured?

Regards

[BGSWebDesign]

Hi DW,

How would this be any different than what I do? It's not, I just go into lm_sendq and delete the first record, that's it, the mailing continues on from there UNTIL it hits another bad email address - I don't see where this is a problem?

I may be mistaken but without an auto-incrementing ID we can't be sure of the order of the returned rows during sending, resuming, or with a manual 'Browse' as you do.

I'm not sure where your coming from but let me tell you a little about my background, I've got a Bachelor's degree in Computer Science with the concentration in Database Systems, that's the truth, if you want I can scan in my Diploma it's hanging on the wall behind me!

There is NO reason that when you do a MYSQL query that you can't consistently pull the EXACT Same record I'm looking at using PhpMyAdmin, I browse lm_sendq and grab the first record, THAT is the one EVERY TIME that is causing the stalling - how do I know that - because EVERY TIME when I delete that record the mailing resumes as normal - the stalling stops, obviously using PHPMyAdmin the query is:

Code: [Select]

SELECT * FROM `lm_sendq` WHERE 1,  if you do the exact same query (or similar) in LMP - YOU ARE GOING TO GET THE EXACT SAME RECORD!

I understand that you seem to be able to grab the top row consistently, but I don't think it's guaranteed.

You said yourself there is NO indexing, I CAN TELL YOU 100% OF THE TIME - IT IS GUARANTEED....  Why the $%#$% $# do you think it is not guaranteed - if you're curious do the MYSQL query yourself - EVERY TIME it will pull the same record.... do you want me to scan in my diploma?

This is done because an index takes time to update when inserting or deleting rows to maintain quick sorted results, so it would slow down the queuing and sending processes.

I KNOW that, you've got no indexing which is fine, and is faster, BUT that doesn't change the order of the tables - unless I'm completely nuts (which I doubt), YOU ARE GOING TO GET THE EXACT SAME RECORD EVERY TIME - anyone in their right mind would never write a DBMS that would pull different records every time a query is done on an unindexed table, that's pure insanity....

If you want a custom script to do a general select and delete of the top row (if the # of entries hasn't changed in X time) I'd still be willing to program it for you.

This is what I've asked for, for 2-3 weeks now.... of course that's what I want - NOT ONLY THAT, it is THE BEST WAY to solve this problem - and NOT by throwing in 12 second SMTP delays - you've slowed my mailing queue sending tremendously going out SMTP, the solution you have is NOT going to work - I'm about to do a test mailing - I'm going to turn ON SMTP logging for you, and then I'll try to do the same test without the SMTP delay and show you....

Still, it would be better if we could check the logs and/or troubleshoot your server to figure out exactly why the system is hanging up...

I had a look (at Ridera code on PHP.net) and it appears to be written to manually time the connection with blocking set to false.  Blocking off means the read command doesn't wait for data from the server.

ListMail *could* do it this way, but I wonder if it would be more processor-intensive.  Look at all the function calls, variable comparisons, etc. required.

Then PLEASE do it that way, or at least try....  OR, if you do not want to get into it, just provide the function that I mentioned above - and put it in resume.php - watch lm_sendq for xxx seconds, if the same record is there, obviously it's stalled, delete the record (AFTER YOU RECORD the email address in a log for later viewing) and that's it...

To ME, it seems the later choice is the better one for now, it let us all get back to work - I can NOT sit around here and wait, for now I'm stripping OUT your delayed SMTP code, as I said it's slowed my mailings down by a factor of 2 to 3 times and I can NOT wait around for the mailings that long.

I have found it required to add
"stream_set_blocking($fp, FALSE )"
This could be applicable here and would mean ridera's solution is the only solution.  I will have to investigate this further possibly by inducing delays etc. to simulate what you experience on your server.  It will be useful to see what's happening in your SMTP log.  While it won't tell us how 'slow' the mailing goes, it will at least rule out the possibility of the majority of delays coming from a very large number of bad addresses / reconnects.
Regards

Like I said, I'll run a test this afternoon and post the log - PLEASE do something about this, as it's killing my click-through rate and deliverability, the mailings are running VERY SLOW, and when they run, I'm not sure all the mail is being delivered - I'm down to about 10% of what I should be with a mailing?

[DW]

I understand you are frustrated but I need you to work with me with patience - we haven't all traveled the same path.

, if you do the exact same query (or similar) in LMP - YOU ARE GOING TO GET THE EXACT SAME RECORD!

Yep, I get that it works, but obviously didn't have the proper education about works or how/if it can become affected.  I suppose the data in the data file will always be read in the same order and not randomly... I thought I saw or experienced something awhile ago where the order was not predictable, which is the reason behind my confusion...  I'll see what I can find on mysql.com/doc. If you know more about the specifics or can point me toward some information let me know.

anyone in their right mind would never write a DBMS that would pull different records every time a query is done on an unindexed table, that's pure insanity....

I don't think it seems all that insane.. Data could be stored in a number of places, such as RAM, or in a cache.  MySQL could access data in whatever order is more efficient.  Without a sort, and because I didn't write MySQL or have the proper knowledge, I wouldn't assume the order is guaranteed.  I will try to find more information on this.  I can set up the script however you prefer and do/did understand that it does work for you with just "select * from lm_sendq" - I just need to clarify as to whether or not I received or imagined misinformation about the order being unpredictable.

This is what I've asked for, for 2-3 weeks now.... of course that's what I want - NOT ONLY THAT, it is THE BEST WAY to solve this problem - and NOT by throwing in 12 second SMTP delays - you've slowed my mailing queue sending tremendously going out SMTP, the solution you have is NOT going to work

Sorry, I thought we were in agreement it was a workaround instead of a solution... Even so, I should have provided the script just in case the new changes didn't work out.

What we need to do is stop the bad email addresses from getting in in the first place.  Then, the SMTP delays will be available as a backup so the mailing / MySQL does not timeout.

I'll reply soon with the auto-delete script - it will be very simple...  After that I'll try my hand at some email validation / verification.

Regards

[DW]

Ok, a problem with the auto-delete script.  We can only set a cron job to run once a minute. If the $smtp_timeout changes are indeed working as expected and only slow down your mailing when there's a bad address this will not be any faster. I think I've said this before...

How often did you have to resume your mailings to get them delivered?  Wouldn't this indicate how many bad addresses are in your list? Was it dozens of times?  If so, I suppose you could achieve reasonable mailing speed simply by sitting there clicking Resume repeatedly.

Going back to some previous messages...

The change to getsmtpmsg has slowed down my sending considerably - is that supposed to happen DW? I tried it with $smtp_timeout=9, it's slow, so I moved it to $smtp_timeout=4 as you suggested and it's even slower, so now I'm going the other way $smtp_timeout=12 - WHAT am I affecting here, it seems like it's stalling about every 30-40 emails now???

The setting affects how long to wait for a response from the server before returning a blank response, which results in a ListMail SMTP reconnect.

Does your MySQL timeout and provide an error message at all, like Dean's did, when $smtp_timeout is set too high?  I believe this is the reason your mailing process ultimately "stops" and does not continue past bad addresses, no matter how long they happen to take.  Setting $smtp_timeout too high will cause this.  Depending on whether your server mysql_timeout is the default 5 or raised to 10 the proper/safe setting should be 4 or 9.  You could go lower, to 2 or 3... if your server's DNS is performing well most lookups should take less than 1 second.

Dean's SMTP server was taking too long and MySQL returned a connect_timeout error (ListMail should output this).  Setting the SMTP timeout to 9 allowed him to stay within his host's MySQL connect_timeout of 10.

:!: I just tested getsmtpmsg() with microtime().  Note that my server doesn't hang-up on the bad addresses (it checks DNS after queuing - further note: you may be able to disable on-the-fly DNS at the host level by removing a qmail module).  The results were good.  There is definitely no mandatory delay with the changes/timeout to getsmtpmsg():

Code: [Select]

getsmtpmsg() took 0.00039899999999998 ms
getsmtpmsg() took 0.00037000000000004 ms
getsmtpmsg() took 0.002756 ms
getsmtpmsg() took 0.005205 ms

What I'm wondering about is if this new function would be somehow NOT getting some of the mail through because of this:

Code: [Select]

while($str = fgets($xsock,1024)){

This part of the code is unchanged and as mentioned above it doesn't seem there's a problem.  I think this is all due to the bad addresses and the necessary timeout due to your server's mail and MySQL settings.

I'll create the script to run every minute but I thought I'd look into the above because you were worried about the changes to getsmtpmsg().

What I STILL don't understand is how your mailings, when you manually resume/babysit them, could be much faster than with the timeout changes...  Have you sent an entire mailing?  Are you sure it's not just certain blocks of 50 that are being delayed and that ALL messages are?  What about email to a test list - is that fast?

Regards

[BGSWebDesign]

Hi,

Ok, I just ran a test with SMTP_debug on, and I don't see ANY problem here at all, the timing seems to be about what I would expect, there is nothing, that I can tell, out of the ordinary in the SMTP logs - DW, I'm waiting for you to email me your address so I can attach the logs, but I don't think I see any problem here - I even 'seeded' the list with 5 of my own addresses, they were all delivered just fine....

UPDATE
=====
Here's some Bad email addresses I tested, this is what happened, started test, 4 bad email addresses, 2 of my own, first address, chokes, gives me message: Sending error. Check your mail settings. (Server Said: ) - blank nothing in 'Server Said', click on Go To Config, see 4 messages left...  click on Link Tracking, 1 minute later, ALL messages are done, I've got a SMTP log for it too... I'll send you that as well...
=======
END UPDATE

Depending on whether your server mysql_timeout is the default 5 or raised to 10 the proper/safe setting should be 4 or 9. You could go lower, to 2 or 3... if your server's DNS is performing well most lookups should take less than 1 second.

It's at 12 as I said... when I had it 9 things were slow, when I tried it at 4 things were even slower, at 12 everything seems to run fine?  Perhaps it's the same thing as Dean and I'm just above my server's MySQL connect_timeout - could be, anyway you say:

I believe this is the reason your mailing process ultimately "stops" and does not continue past bad addresses, no matter how long they happen to take.quote]  

Not sure what you mean here - 'stops'?  My mailing works fine, it USED to Stall - BEFORE I added the smtp_timeout code into the function you posted, but now it does NOT STALL - sometimes I will get the Server Error, Server Said (WITH NOTHING for the Server Error Message).

I get the feeling it's running slower because, like I said, it seems to take 2-3 times as long to do a mailing - NOW what I believe is happening is this is because it is STALLING at the bad email addresses, and then has to wait 5 minutes for the resume.php to PICK it up again and continue, if there are enough bad email addresses this could slow things considerably, correct?

We can only set a cron job to run once a minute. If

Why, can't it be included in resume.php - oh I see, you want to hit it more frequently - probably a good idea, but as you see above things seem to be working just fine, I'll get you the SMTP logs and you can see, probably what I should do is makeup a test list with all those BAD email addresses and see how those logs look?

Keep me posted on what you do to the code, as I said, I'm running SMTP_timeout at 12, so I think you're going to need to make this a global configuration option...

[mike2]

You'd think with the 2-3 weeks you've waited for a "Fix" to this problem, you would have actually fixed the real problem instead of just trying to use a "Crutch"...

Dean, whatever you do, DO NOT ruin how your program works by fixing this for 1 person, this is my last post unless something else comes up...

[mr.trevor]

I am still amazed how anyone can send mail out to a large list and not get spam complaints, when not using a confirmation mail before adding to list. It seems that by confirming the addresses first a lot of these sending problems would not occur from the bad email addresses.

[DW]

Brett,

my email is dean@listmailpro.com

UPDATE
=====
Here's some Bad email addresses I tested, this is what happened, started test, 4 bad email addresses, 2 of my own, first address, chokes, gives me message: Sending error. Check your mail settings. (Server Said: ) - blank nothing in 'Server Said', click on Go To Config, see 4 messages left... click on Link Tracking, 1 minute later, ALL messages are done, I've got a SMTP log for it too... I'll send you that as well...

This is strange, both the initial mailing and resume should be the same.  I would be interested in seeing both SMTP logs, especially the first.

It's at 12 as I said... when I had it 9 things were slow, when I tried it at 4 things were even slower, at 12 everything seems to run fine?

This is strange too.  With a lower timeout it should be faster... that is, unless it takes several seconds to reconnect, and a number of DNS results are taking between 10 and 12 seconds (?)... it shouldn't take that long to connect, though.

I believe this is the reason your mailing process ultimately "stops" and does not continue past bad addresses, no matter how long they happen to take.

Not sure what you mean here - 'stops'? My mailing works fine, it USED to Stall - BEFORE I added the smtp_timeout code into the function you posted, but now it does NOT STALL - sometimes I will get the Server Error, Server Said (WITH NOTHING for the Server Error Message).

Yes, by "stops" I meant stalled.  If it's not stalling then you must be within your server's MySQL connect_timeout.  I am now curious about what the server is doing when it reports the 'Server Error' message as it might be another issue...  More information will hopefully show up in the ListMail SMTP log.

I get the feeling it's running slower because, like I said, it seems to take 2-3 times as long to do a mailing - NOW what I believe is happening is this is because it is STALLING at the bad email addresses, and then has to wait 5 minutes for the resume.php to PICK it up again and continue, if there are enough bad email addresses this could slow things considerably, correct?

Yes, providing the script is stalling by timeout or the unknown 'Server Said' error it will stop and then need a resume.  The 'Server Said' error means something other than the timeout is now happening.  So this is a new error with ListMail's SMTP communication and not really a stall/crash. From what you state that the mailing no longer stalls as it used to, the 'Server Said' error may now be the only problem and could result in what you describe.

I have completed a large portion of the script you requested, but am now stuck on one thing.  Since the script will run every minute it will run less frequently than auto-resume.  When the script detects that the new count is the same as the old count it is removes the top row, and we then decrease the last count.  The problem is, when the script runs again if the mailing hasn't re-started it will delete the next row because again the queue count matches the last count.  We can work around this by the condition that "if the last row was deleted, don't delete this row" with an additional SQL field, but what if 2 bad addresses appear in succession?  It's not error-proof...

Do not use this script yet, it is affected by what I mention above:

Code: [Select]

<?php
/********************
check lm_sendq every minute, if entries check if changed and delete top row
auto-resume will then pick up the remainder of messages when the mailing has failed for one minute
place this script in your listmail folder and set up a cron task for every minute
ie: * * * * * /usr/bin/wget -O - http://example.com/mail/autodelete.php?pw=1234

A new table and row are required to store the last count:
SQL:
 CREATE TABLE lm_sendq_num (lastcount MEDIUMINT UNSIGNED NOT NULL);
 INSERT INTO lm_sendq_num VALUES ('0');
********************/

include('./config.php');
include('./admin.php');

if($_GET['pw']<>'1234') exit('access denied: wrong password');

$email_notify = 1;
$email_to_notify = 'test@listmailpro.com';

$tbl = 'lm_sendq_num';

$DEBUG = 1;

// leave this
$msg = '';

// retrieve counts
list($oldcnt)=mysql_fetch_row(mysql_query("select lastcount from lm_sendq_num;"));
list($cnt)=mysql_fetch_row(mysql_query("select count(*) from lm_sendq;"));
if($DEBUG) $msg .= "cnt=$cnt, oldcnt=$oldcnt\n";

if($cnt==0){
 // if no items in queue reset the oldcount
 if($DEBUG) $msg .= "no items in queue - resetting oldcount and exiting\n";
 mysql_query("update $tbl set lastcount = '0' where 1;");
} else {
 if($DEBUG) $msg .= "$cnt items queued, checking oldcount\noldcnt = $oldcnt\n";
 // items in queue, check the oldcount
 // if no oldcount update to new count
 if($oldcnt==0){
  if($DEBUG) $msg .= "oldcnt is 0, update to new count ($cnt)\n";
  mysql_query("update $tbl set lastcount = '$cnt' where 1;");
 } elseif($cnt==$oldcnt){
 // otherwise, check if its the same and delete the first row if so
  if($DEBUG) $msg .= "queued items = lastcount, delete top row\n";
  // get data
  list($id,$bat,$battype,$mtype,$uid,$mid,$xtra)=mysql_fetch_row(mysql_query("select * from lm_sendq limit 1"));
  // delete row
  mysql_query("delete from lm_sendq where id = '$id';");
  list($uem)=mysql_fetch_row(mysql_query("select email from lm_users where id = '$uid';"));
  $af = mysql_affected_rows();
  if($af>0){
   $ncnt = $cnt-$af;
   mysql_query("update $tbl set lastcount = '$ncnt' where 1;");
   $msg .= "The following row was deleted from the lm_sendq table:\n\nid=$id bat=$bat battype=$battype mtype=$mtype uid=$uid mid=$mid xtra=$xtra\n\nThe user has the email: $uem\n";
  }
  else $msg .= "Error, row not deleted.";
  // notify
  if($email_notify) mail($email_to_notify,'LM Auto-Delete',$msg,"From: \"LM Auto-Delete\" <noreply@example.com>");
 }
}
if($DEBUG){
 echo "msg=".nl2br($msg);
 if($email_notify) mail($email_to_notify,'LM Auto-Delete (Debug)',$msg,"From: \"LM Auto-Delete\" <noreply@example.com>");
}
?>

[DW]

Got your SMTP logs, Brett - Unfortunately I can't explain why your server is dropping the SMTP connection, seemingly randomly.  If the error always happens at the same time (same command/result pair in SMTP log) I should be able to provide code to correctly detect the blank response and tell ListMail to issue a reconnect.

Regards

[DW]

The first incarnation of an email verification script is now ready:

http://listmailpro.com/forum/index.php?topic=1243.msg5124#msg5124

Regards