ListMailPRO Email Marketing Software Forums
ListMailPRO Email Marketing Software Forums => Development, Suggestions => Topic started by: BGSWebDesign on April 13, 2006, 09:58:45 pm
-
Hi DW,
PLEASE READ ALL OF THIS MESSAGE - 2 updates - something caused resume.php to stall...
Ok, I've upgraded to 1.86, ran into the same problem as others with the 'addopts' - but then restarted the machine to clear the cookie and all is ok - you should tell everyone to make sure they log OUT before uploading the install...
Now, a question on the resume.php script, I'm trying to install it in CRON, so I'm testing it from the command line in SSH, I keep getting strange results and do NOT no if it's running, maybe you can help?
I set debug=true at the top, and tried running with the sample command line you gave:
*/15 * * * * /usr/bin/wget -O /dev/null -T 0 http://example.com/mail/resume.php?pw=YourDailyMailPass
When I do that I keep getting an error:
wget: timeout: Invalid time period `O'
So I took out the -T 0 from the command line giving me this:
*/15 * * * * /usr/bin/wget -O /dev/null http://example.com/mail/resume.php?pw=YourDailyMailPass
When I do that I get:
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
[ <=> ] 42 --.--K/s
00:49:56 (410.16 KB/s) - `/dev/null' saved [42]
Here's another one:
TTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
[<=> ] 0 --.--K/s a [ <=> ] 42 --.--K/s
Any idea what is going on here? Is resume.php running properly or is there an error on line 42? Is this in one of the includes - config.php or admin.php or is it in resume.php?
Let me know what you think?
UPDATE
======
It seems that it may be working correctly, this morning I had many followups go out and they did not stall this morning - so - is it working?
SECOND UPDATE
===========
No, it's not running, I just noticed I stalled at 1802 message left, it's been stalled like that for the last 1-2 hours, unless the system was re-set, the CRON is not running properly - no CRON is still running, I just checked.
Here is what I found in lm_sendp, 2 entries, 1 is for DailyMail Report, the other is my batch which reads:
batid=cab744, qtype=1, formid=2006041401253, started=2006-04-14 01:25:43, lastact=2006-04-14 01:26:09, report=(empty), completed=1
I'm going to reset the completed to '0' to see if it will re-start (resume). Nope, that didn't help - I wonder if it has choked on an Email address of someone that was deleted - I know in the past that could be a problem? Nope that's not it, I tracked down the userid (uid) for the first user in the lm_sendq table and all is fine, then I clicked the 'Resume' button and it started to resume the mailing - very strange - any ideas DW?
When I look in lm_sendq I see 1802 records, with this:
bat=e18c3a, battype=2, mtype=2, mid=85 the user ids are there for the records. This one is stalled...
THIRD UPDATE
==========
Now - here is where it gets really wierd! I just came back and looked at my 'resumed' mailing that I got by Clicking on 'Resume' button, that one had stalled out too with an error - 'Server said goto config', anyway, usually that would mean a stalled queue again - but this time when I clicked on a link to go back to the main LMP menu I see that the mailing has completed - so something must have happened - I'm guessing that the 'resume.php' script must have completed the mailing?
What exactly is going on here DW? It looks like the queue was loaded at 1:25:43, and then finished at 1:26:09, is that right? What about the mailing then - if it stalls doesn't resume.php get it started again? If so, why is completed set to '1'? And what about lastact, does that mean it did NOT have to re-start it at all throughout the early morning? If I sent out as many emails as I think I did, then that would not be true - it must have restarted it many times, then the big question becomes - why did it stall at 1802 left? How did this record get flagged as 'completed' (1)?
Any ideas - DW?
-
Brett,
wget: timeout: Invalid time period `O'
Seems to be a typo. The time period should be set to 0 (Zero) not the letter O. Failing to specify an unlimited time limit (-T 0) could result in unexpected behaviour but shouldn't cause any undesired email.
00:49:56 (410.16 KB/s) - `/dev/null' saved [42]
This looks like it could be the 'normal' result from the script running successfully.
I'm going to reset the completed to '0' to see if it will re-start (resume).
I cannot recommend doing this as, without some testing, I have no idea what behaviour it might cause.
Here is what I found in lm_sendp, 2 entries, 1 is for DailyMail Report, the other is my batch which reads:
The Dailymail entry should contain the report and queue data for the dailymail execution. The other entry (from another mailing, NOT the dailymail resume?) should have everything the same except qtype will be different and the report will be empty.
I notice that you mention 2 different batch ids "cab744" and "e18c3a" - could you have upgraded while a mailing was in progress?
What exactly is going on here DW? It looks like the queue was loaded at 1:25:43, and then finished at 1:26:09, is that right?
I wish I could tell you but I'm a little confused, too. I just verified that the script does set "last active" at the end of mailing when it also sets completed to 1. Completed should only be set to 1 when the mailing is finished without error or interruption by the qfinish() function in admin.php.
I wonder if the confusion is caused by the different batch ids?
I can provide a text file containing several thousand test email addresses you can use if you want to do some further testing. Please let me know if you have any more updates/information.
Regards
-
Hi DW,
Seems to be a typo. The time period should be set to 0 (Zero) not the letter O.
I see that is a problem, I fixed that and the script runs properly, still that doesn't explain why it hung up and then was able to re-start later on?
Also, I don't think you need the -O /dev/null since I don't get any output if I include the 1> /dev/null 2> /dev/null on the end.
Also, I did not update in the middle of a mailing, this is from the mailing following the update, and also, I don't think the batch id's were different I could have made a copying error since I was hand typing in what I saw in the database.
Thanks, I'll keep you posted if I notice anything else...
-
Hi DW,
Ok, it's stalled again - I can't get it to move with over 30,000 messages in the queue! Should I click 'resume'?
When I change 'debug' to 'true' I NEVER see any output here - even if I remove the /dev/null - how can I tell what is going on?
Update
=====
Here's some more information, I looked in lm_sendp, it does NOT have an active (stalled queue) listed in it, ONLY Dailymail reports from the last 4 days, and the last queue from 4/14/2006! Can you tell me WHEN lm_sendp is wiped and when the new queue record is added when a new mailing starts? I don't see any new record in here, AND I have a stalled queue. - Oops check that, I DID find a record for the queue - read below - but there were still OLD records hanging around in here with completed=1 when are these removed?
2nd UPDATE
=========
OK - here's what I see in lm_sendp - there is ONE record that seems to be for this queue, it includes a report 'Dailymail report for 2006-04-17',
so I did not think this was the queue but just the report, it seems to be the queue since it has the same queue ID as the records that are in lm_sendq ('7afbbf'), ANd I notice that lastact is getting updated frequently - as if email is being sent out - BUT - the 'Mail Sent' counter is NOT updating in the header of LMP? Here's what is in this lm_sendp record:
id=5, batid=7afbbf, qtype=2, formid=, started=2006-04-17 04:15:15, lastact=2006-04-17 09:29:27, report=Dailymail Report for 2006-04-17 04:15:15Totals: ..., completed=0
3RD UPDATE
========
Ok, this time I deleted the top record in lm_sendq - since usually when it chokes it's because the top record is bad (bad address/or bad deleted record) and NOW the Mail Sent is being updated, it's running again - BUT, here's something for you DW - it seems that you need to add this check to resume.php - look and see if the NUMBER of records in lm_sendq is the SAME as it was last time you ran - if it is, and it stays that way for 10 minutes or longer you need to DELETE the top record in lm_sendq and then re-set your counter again and see if the number of records change next time in (indicating that the mailing is going out properly).
It seems that a bad record in lm_sendq (previously deleted or bad email address) is stalling out the resume function (resume.php).
What do you think DW? - More info below - but this was BEFORE I knew what seems to be going on - so can probably be disregarded...
-------------------------------------------------------------------------
Any ideas WHAT is going on here, it seems lastact is getting updated but I DO NOT see that any emails are going out - at least the number of records in lm_sendq are always the same - as if NO email is going out??? Is that the way it should be?
I also get this message:
This mailing appears to be in the process of sending normally. It has responded within 1 minute How does it respond, I don't see that the numbers in 'Mail Sent' are going down, so how is it responding - and there is no record in the lm_sendp table?
It seems there is a problem here of some kind? Does the 'resume' automatically updated the records in the queue and delete them as they are sent out - does it updated the 'Messages Left' displayed in the header as I update my LMP page? I don't see that it's doing anything - it's stalled, what can I do to help you figure this out???
-
Brett,
I notice that lastact is getting updated frequently - as if email is being sent out - BUT - the 'Mail Sent' counter is NOT updating in the header of LMP?
This does seem strange. Normally when a queue is processing it will be sending and removing email from the queue (lm_sendq) and updating last active.
Ok, this time I deleted the top record in lm_sendq - since usually when it chokes it's because the top record is bad (bad address/or bad deleted record) and NOW the Mail Sent is being updated, it's running again
Yes, it's starting to make some sense. If ListMail were to be caught in a loop on a single user it would mean no other rows are processed and deleted from the queue table.
Does the 'resume' automatically updated the records in the queue and delete them as they are sent out - does it updated the 'Messages Left' displayed in the header as I update my LMP page?
Each individual email is entered as a row in the lm_sendq table. As the mailing is sent to the server entries are removed one by one from the lm_sendq table. The counter in the ListMail header is based on the row count for each batid in the lm_sendq table.
If deleting a row in the sendq table allows the mailing to run you may want to note the userid so we can cross-reference the user data. If you did this already did you notice anything strange?
I also recommend enabling the $smtp_debug var in admin.php so we can see what's happening with SMTP.
Regards
-
Hi DW,
Yes, it's starting to make some sense. If ListMail were to be caught in a loop on a single user it would mean no other rows are processed and deleted from the queue table.
This must be happening, it just stalled again - I deleted the top record and it's running fine again - here's the email address domain for the user I deleted from lm_sendq:
cprk.com.my
I don't think that's a valid domain address - so that must be what caused it to stall out this time? The question is that you need to write something into resume.php that can handle this - like I said, keep a count of the number of records in lm_sendq, if it stays the same 10 minutes later, or whatever the CRON is set at, if it remains the same - then there is a problem and you'll need to delete the top record in lm_sendq - BUT keep a log of what is going on so we can see how many of these users are bad - write it out as a text file or something.
I also recommend enabling the $smtp_debug var in admin.php so we can see what's happening with SMTP.
Do you still want me to do this? It seems to me it's bad email addresses that are causing the problem, right, or do you still want to see what SMTP is saying? Let me know if you do, next time it chokes I'll put it in and then watch what it says - is that what you want?
-
Presumably you could manually add invalid addresses to your list to check this happening? Maybe this could be a bit dangerous with large mailings but if it was a small 'special' list and you were watching for it then it could be conclusive.
I, for one, appreciate all the extra work that is done by others to clear up these glitches.
Thank you for this work Brett. (and DW of course...)
-
Instead of covering up the issue I would really like to get to the bottom of it. I will try some tests with invalid addresses, but I fear this problem may only be producable on a server running particular mail server software.
I think that yes, Brett, you should enable the SMTP log. It's possible that an SMTP response is not being interpreted properly, causing a loop.
If the logs don't yield any definitive information is there any way you can set me up on a test list on your server? You might set up a 2nd bare-bones installation for this.
Regards
-
Hi DW,
Instead of covering up the issue I would really like to get to the bottom of it. I will try some tests with invalid addresses, but I fear this problem may only be producable on a server running particular mail server software.
I think that yes, Brett, you should enable the SMTP log. It's possible that an SMTP response is not being interpreted properly, causing a loop.
Sure, I'll turn it on - but ONLY when it stalls again - let us know what you find with your own testing...
Regarding the problem with the server causing looping - if this will be a problem for others - why not just write it into resume.php to check for these type of bad addresses and delete them so the mailing can continue? Apparently if I have the problem others will also, but let us know what you find...
-
First I'll need to determine what, exactly, makes the email address 'bad'. I believe it may be a server DNS/mailer issue dealing with non-existent domains. The SMTP logs should give us an indication of what is going on.
Unfortunately I have never had this problem on my own servers or other servers I have had access to. I don't think I'll be able to recreate this one without access to an installation on your particular server. If you set up a 2nd ListMail install (just basic table setup and SMTP settings) on your site and provide access (plus FTP, if possible) I can promise you that I won't email any legitimate addresses that are not my own. I might be able to figure something out with just a couple emails to fake / non-existent domains.
Regards
-
Hi DW,
Ok, it happened again, the domain: goldcountry.bc.ca is also server not found, so I suspect that is the problem...(no, that's not it - it happens every time the email address has 2 periods in it - it did NOT do that in version 1.84 so the problem lies there).
You should be able to test this on your end, or are you saying that if you run email addresses at these bad servers your system runs fine? Just use these two email addresses.
test1@cprk.com.my
test2@goldcountry.bc.ca
Also note how it seems to be addresses that containt two '.'s? Not sure if that means anything...
2ND UPDATE
=========
It just happened again stalling with this domain: col.com.np, here's another test address for you:
test3@col.com.np
3RD UPDATE
========
Here's another one:
test4@net-rosas.com.br
Do you see a pattern here? It always seems to be with addresses with 2 periods in them ('.'). Another thing to keep in mind it did NOT stall like this on addresses like this in version 1.84, so there's something going on here between that version and this one!
I'll be happy to load these two addresses to a test mail list and turn on SMTP debugging and try a mailing - but I've got to get my daily mailing out for today first - and that will take all day from what's left stalled in the queue!
I'll let you know tomorrow what I find, if you don't find something yourself...
Still you should think about providing an option in resume.php to DOUBLE CHECK the reccount() in lm_sendq - if it does NOT change within 15 minutes - obviously something is wrong and the queue is stalled - at that point delete the top record (write the address to a log first) and then go on your merry way - why not? Or tell me how to include this - or I'll hack it myself - since apparently I'm going to need it on this server...
-
This is good information. I do not believe this is as simple as "email addresses with two periods", however. I did some tests and it seems that all domains can have quick or delayed DNS replies. This is probably due to nameserver response time.
I'm remembering something from awhile ago... On some servers I noticed that SMTP communication can be delayed by these types of DNS issues. In my experience it was not fatal, however - after about 10-20 seconds max the queue resumed at a normal speed until another one of these slow addresses was hit. When sending, if you only get one delayed address in a block of 50 (LMP counter update interval) you might be able to notice that this is true.
Another possibility is that the delays are caused by the receiving mail server trying to determine if your domain exists (DNS lookup) before accepting the message. This way your host's delays or the distance between servers might come into play...
The problem seems to be that your server waits for the DNS lookup before accepting the message for delivery. My servers, running qmail and just tested with all of those addresses you provided, accept all messages instantly to the queue, and attempt all delivery in the background.
Shortly after delivery (<10s), I saw this in the logs indicating the destination server could in fact be reached.. Nothing yet for the other addresses, though:
Apr 18 11:08:02 serv2 qmail: 1145383682.898725 delivery 16100729: failure: 219.94.65.171_does_not_like_recipient./
Remote_host_said:_550_<test1@cprk.com.my>:_Recipient_address_rejected:_User_unknown_in_virtual_alias_table/Giving_up_on_219.94.65.171./
The rest returned something like this:
Apr 18 11:11:59 serv2 qmail: 1145383919.164225 delivery 16100735: deferral: Sorry,_I_couldn't_find_any_host_by_that_name._(#4.1.2)/
Either way, my server should send me a bounce so I can remove these users.
Your suggestion to scan the queue to see if it's changed might be a viable quick-fix for you, but I would rather not implement it hastily into ListMail as it could remove legitimate emails that have a chance of delivery. If you give me the exact specs for your custom script (ie. scan queue table every 1 minute while mailings are active within 1 minute) I'll set it up for you then do more research on this.
I am not sure (yet) how to avoid this issue. It could be as simple as a single config file change on your server. Can you tell me what mail software your server uses? This information might be available at the very top of any ListMail-created SMTP log in the 'greeting' for the connection.
Note: I have moved this post to development and changed it's subject.
Regards
-
Hi DW,
If you give me the exact specs for your custom script (ie. scan queue table every 1 minute while mailings are active within 1 minute) I'll set it up for you then do more research on this.
That would be great - let's say scan every 80 seconds while active and if the same reccount() then delete the top record...
Let me know when you have it ready, this is causing me to 'baby-sit' the mailing - exactly what I didn't want to have to do...
-
This does almost sounds like a broken SMTP program to me if indeed this is what is happening.
If you are running Sendmail in interactive mode, I could see this as a problem.
My suggestion for this is run it in background mode. What program are you using if you don't mind me asking?
-
Hi Mike,
No, I'm not running sendmail, I'm running LMP own SMTP
mailer, from LMP Configuration page:
SMTP Server (recommended):
settings - port: 25, reconnect every 249 emails.
Let me know if you have any other ideas - this thing seems to be stalling constantly - about every 30-45 minutes!
Is the 'reconnect every 249 emails' too low? I used to have it at 499, but lowered it when I started having problems.
-
Hi Mike,
No, I'm not running sendmail, I'm running LMP own SMTP
mailer, from LMP Configuration page:
SMTP Server (recommended):
settings - port: 25, reconnect every 249 emails.
Let me know if you have any other ideas - this thing seems to be stalling constantly - about every 30-45 minutes!
Is the 'reconnect every 249 emails' too low? I used to have it at 499, but lowered it when I started having problems.
Right I understand that, but which SMTP software is running on the mail server to which your LMP connects. In the same settings you are talking about where it says "Host", whatever machine that is has to be running an SMTP program that is listening on port 25 (IE> Sendmail, qmail, postfix, etc).
If you know how that is configured it may be helpful to diagnose the problem. If it's something that just started maybe someone changed the config or something.
This really sounds like a broken SMTP, but it's so hard to say since you say you weren't having this problem in V1.84, what about V1.85? If so maybe Dean changed something in the SMTP routines that broke for your mail program.
Posting the SMTP logs would certainly help (not the whole thing, just where it stalls of course).
The best way for a SMTP program to be set up with LMP (IMHO) is basically for LMP to generate the message/headers, etc. and connect to your SMTP and say "Here's the message", your SMTP simply says "Ok, it's queued, Next Please" and on and on... At that point it is the SMTP programs job to deliver the message. Most SMTP programs aren't necessarily set up Stock to do it this way, but can be configured to.
-
Hi Mike,
This really sounds like a broken SMTP, but it's so hard to say since you say you weren't having this problem in V1.84, what about V1.85? If so maybe Dean changed something in the SMTP routines that broke for your mail program.
Funny thing happened - I moved the CRON up on resume.php to every 5 minutes - now it's not doing it at all - with still a large mailing - no stalling when it finished up last night, or on today's mailing?
Also, regarding v1.84 - I remember now I put my own hacks in to one of the modules (domail.php maybe, but I'll have to go back and look), those hacks seemed to get around this type of thing ever happening and they must have worked...
Now - back to the other question - DW - What about Bounce Processing? It doesn't seem that resume.php makes sure that runs - my bounced mailbox is now at 60MB! Is it ok to run Dailymail with ALL options turned off but bounce processing - will that do what I want?
-
Thanks for your help on this one, Mike!
Brett, I must recommend that you have your web host manually delete all of the messages in the bounce mailbox. I fear that the mailbox has become so full that checking messages in it could be disasterous. Due to hard drive slowdowns with several thousand files in a folder it would probably take several days or weeks to process these messages. I once did a test with 11,000 bounced emails and it took upwards of 6 hours. This is why bounce.cgi is strongly recommended for large lists.
resume.php does not process bounced messages - that's dailymail's job. I'm actually a little surprised that dailymail is able to 'get past' bounce processing and queue messages even with such a large number of bounces in the mailbox.
Regards
-
Hi DW,
This is why bounce.cgi is recommended for large lists.
It's not possible - think about it, with a large list - when doing a large mailing 30,000-40,000 think how many bounce requests come in simultaneously!
When they do come in like that - they slow down the entire system and make it unable to handle simple requests - such as link tracking requests... Now which would be worse - taking a little longer to process bounces - or losing sales because no one can click through on your links?
You can see the problem here - also - my question is still unanswered - WHAT happens when resume.php picks up where Dailymail started? Is bounce processing EVER activated in this situation as it seems that it is not - so that means there's a problem with LMP in this situation - it may work with bounce.cgi - but it's impossible doing it the way I do - if resume.php will NOT run the bounce processing!
Regarding bounce.cgi - if you can show me with good testing/results that it's possible to run large lists such as I mention and still be able to process link tracking requests - I may consider switching - but until then - I'll need to find a way to do it on my own - because I absolutely can NOT lose sales when bounce.cgi is running every .5 to 1 second and sometimes more when large lists are mailed.
Let me know what you think and point me to some tests/results so I can determine if it's possible to switch to bounce.cgi for my environment.
-
Bounce processing is never activated by resume.php - I hadn't thought about it being necessary. Under ideal circumstances, especially if only used for smaller lists as recommended, the script will not fail and all bounces will be processed by dailymail. With larger lists bounce.cgi should be used.
so that means there's a problem with LMP in this situation - it may work with bounce.cgi - but it's impossible doing it the way I do - if resume.php will NOT run the bounce processing!
The way you do is with too many bounces to a standard mailbox... it is known that the pogram will not work as expected in this situation. Try checking the mailbox manually and see what happens...
I absolutely can NOT lose sales when bounce.cgi is running every .5 to 1 second and sometimes more when large lists are mailed.
I totally understand. What you can try is having your web host enable a higher 'maximum # of connections' to the web server. Also, some MySQL optimizations can likely be made. If this doesn't help you may want to consider a dedicated server - I know one user running a high-powered dual-processor server with 1 million addresses using bounce.cgi.
Until the process is improved you might want to (or have your host) set up a simple redirect to "null" for your bounce address so bounces can be temporarily discarded.
DW
-
Bounce processing is never activated by resume.php - I hadn't thought about it being necessary.
IMHO it shouldn't have to be, like you said in this case that's dailymails job and dailymail should pick up the next day and handle it.
However, like you said Dean, the mailbox has gotten WAY too full... most email clients would choke on trying to download that much mail. Actually the IPOP (or whatever POP3 program your server uses) process may not even be able to handle it in a shared environment.
Brett, I honestly believe in this case it is almost hopeless to get at that POP mailbox and the longer you let it go the worse it will get. I'd cut my loses and overwrite it now to start fresh.
I totally understand. What you can try is having your web host enable a higher 'maximum # of connections' to the web server. Also, some MySQL optimizations can likely be made. If this doesn't help you may want to consider a dedicated server - I know one user running a high-powered dual-processor server with 1 million addresses using bounce.cgi.
This may or may not help, I believe (and this is from my own experience running a fairly big list of 150K) that the biggest slow down while this is going on is due to the amount of .CGI processes running taking up your resources.
Regarding bounce.cgi - if you can show me with good testing/results that it's possible to run large lists such as I mention and still be able to process link tracking requests - I may consider switching - but until then - I'll need to find a way to do it on my own - because I absolutely can NOT lose sales when bounce.cgi is running every .5 to 1 second and sometimes more when large lists are mailed.
I run a list of 150K and use bounce.cgi with no problems and I mail to the list daily. Now granted I use two fairly low powered linux dedicated machines (on my own T-1 line) to accomplish this does make it a bit different. One way that you could throttle this down is tweaking your SMTP program (which you never told me what it was) to only allow so many incoming connections at a time. This is simple with sendmail...
Brett, with that size of list(s), I HIGHLY suggest running your own dedicated server. It's well worth the extra $100/month or so, but I also understand not wanting to have to manage one.
-
Another user has reported what sounds a lot like the non-existent domain error in this post (http://listmailpro.com/forum/index.php?topic=1200.0). I'm offering tech support in an effort to be able to do some much-needed testing.
Brett, if you can wait a little longer I might be able to solve the problem and avoid the need for the custom hack I offered.
Regards
-
Hi,
Brett, I honestly believe in this case it is almost hopeless to get at that POP mailbox and the longer you let it go the worse it will get. I'd cut my loses and overwrite it now to start fresh.
No, no problems here - I ran it manually - via Dailymail (turning off all other options) and it did just fine processing 7,800 bounces in about 20 minutes! See the last message in this post:
http://listmailpro.com/forum/index.php?topic=1196.0
One thing I did notice - before I ran Dailymail bounce processing - I ran Mailwasher on the mailbox to see what was in there - I did notice that there were a few 'out of office' returns - I didn't think they were EVER supposed to be in there - and also a few virus potential mails - but either way - running Dailymail bounce processing did just fine... in a fair amount of time too.
Brett, if you can wait a little longer I might be able to solve the problem and avoid the need for the custom hack I offered.
That's fine, my own hack to fix admin.php so Dailymail can run Bounce processing by itself is just fine - I can manually start up Bounce Processing any time I want to handle this problem...
Acutally I believe the problem is related to something else - hacks I put into your domail.php module to STOP trying to connect every time when SMTP was returning errors - if you want I can share those with you - it seemed to make things work when I did that in v. 1.84 and apparently is what was causing the problem in this current version: I forgot to put them back in - when I upgraded!
-
This is good information. I do not believe this is as simple as "email addresses with two periods", however. I did some tests and it seems that all domains can have quick or delayed DNS replies. This is probably due to nameserver response time.
DW - Looks to me like DNS also. The one I stalled on today was @ott.net. When I send to the address I get an error back from the SMTP server.
'test@ott.net' on 4/20/2006 2:42 PM
451 DNS temporary failure (#4.3.0)
This website "ott.net" is currently not resolving via HTTP for me either.
:Dean
-
set up test list with two users - one good, one bad
Here's the log:
> MAIL FROM: <email_handler@wesleyan.gotwww.com>
> RCPT TO: <test@ott.net>
250 ok
451 DNS temporary failure (#4.3.0)
LM: Undeliverable. RCPT response: 451 DNS temporary failure (#4.3.0)
. Skipping.
> RSET
451 DNS temporary failure (#4.3.0)
domail.php ends up here:
admin-26-Lost connection to MySQL server during query
-
It seems that the server doesn't respond to the RSET command with success. This must be the problem. I will do some tests with the access info you provided and get back to you soon - thanks Dean!
Regards
-
Hi,
This must be the problem. I will do some tests with the access info you provided and get back to you soon
Please tell us you corrected this, and when it will be released?
I still have the problem, this morning my mailing stalled on someone entering a dummy address:
test5@dsds.com obviously a bad address - but LMP chokes on it, I don't remember v. 1.84 doing that - I must have modified the code, but have looked for over an hour this AM for it and can't find what I changed - there is one thing with the bounce processing that was different but I don't think that is it...
UPDATE
======
I had a further look - I vaguely remember some problem with the '250' error? Have a look at that, also there's that problem with the 'or die' code that was added... apparently I left it commented OUT and did not use it in my vers. 1.84 - here it is - you told me to ADD THIS and it fixed the problem - which I reported to you it did, but you failed to FIX it in the new version - why do I see this happening with you? When someone reports a fix - you should immediately fix it in the PRODUCTION version, this is costing me serious time, here's the problem, line 2540 in admin.php, it needs to have the addslashes, like this:
$cmd = "select id,list,bounces from $utable where email like '".addslashes($email)."'";
$urows = mysql_query($cmd) or die('admin-45-'.mysql_error());
FURTHER UPDATE
===========
Surprise, I found this inside of one of my email texts - it must have been a problem at one time?
pipelining support enabled.
NOOPmsg=250 2.0.0 OK
PIPELINE-FROMmsg=250 2.1.0 ... Sender ok .
RCPT TO: <>
RCPTmsg=553 5.0.0 <>... User address required
DATAmsg=503 5.0.0 Need RCPT (recipient) .
Maybe that can help you DW?
PLEASE fix that in the production version! And then look for '250' errors and fsockopen problems - I know I patched up those on my own too - but can't seem to locate them at this time...
DW - do you have a fix yet?
-
Excuse my ignorance but... how do you get someone entering a dummy address and it being added to a list? If they do not exist they won't get confirmed. Are these not dropped after not being confirmed?
-
how do you get someone entering a dummy address and it being added to a list? If they do not exist they won't get confirmed. Are these not dropped after not being confirmed?
Only if you have (and you should) confirmation turned on.
-
Hi,
how do you get someone entering a dummy address and it being added to a list?
You got it, I don't use confirmation.
DW - please see 2 messages above where I lay out what I think is causing the problem...
Please keep us informed on when you will have a fix for this.
-
These SMTP errors are one of my highest priorities - I've been working on it but haven't been able to fix it yet. I should have something soon. I am not sure if these problems are related to the addslashes() fix for the bounce function, but I will look into this too to make sure it's in the next update.
Using ListMail's methods you should not be able to enter email addresses in a bad format (ie. blank) into the program. This is usually the result of using a custom script or plugin to insert users.
From numerous Google searches I am led to believe the "451 DNS temporary failure" error is related to optional qmail-specific patches.
I am working on allowing SMTP to recover from this error and skip the user during mailout.
Regards
-
Brett, is there any way you can show me your LM SMTP log when the mailing stalls? On Dean's server, which have access to, I'm getting a new type of error "lost connection to MySQL". I am not sure if this is the problem, part of the problem, or a side-effect of the problem. You may need to uncomment the line "$smtp_debug = 1;" in admin.php to enable global SMTP log so the log is written during dailymail, resumes, etc. In the meantime I will continue troubleshooting this with the access I have, of course.
Update re: bounce() addslashes:
This was added to the top of the bounce() function in admin.php to eliminate the need for the change I suggested to you previously:
if(!valid_email($email)) return false;
Regards
-
On Dean's server, which have access to, I'm getting a new type of error "lost connection to MySQL
When this first started happening I asked my host about this. They said it was just the standard time out error. "This happens when a query is using too much MySQL processing time. If a query takes longer than 10 seconds mysql will drop the connection." They said this has been in place for over 6 months and had not previousley effected LM.
Every time I get this error the SMTP log also logs the DNS error.
:Dean
-
I have found some interesting information, such as:
http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
"Lost connection to MySQL server during query"
I've found that this error also will occur if for whatever reason, the client takes longer than <connect_timeout> seconds to connect to the server. The default value is 5 seconds. You can increase this value in your my.cnf file or elsewhere, though it would be ideal to figure out why it's taking so long to connect.
This may be what is happening, caused by the delayed (~10s) response from the DNS lookup.
Brett, it would be interesting to know if you also receive the error message Dean does:
admin-26-Lost connection to MySQL server during query
This message should be displayed on the sending page when the mailing 'stalls'.
I am performing more tests to verify this.
Regards
-
Indeed, when I manually configure admin.php to simply skip to the next email on a certain failing address the MySQL connection error message does not appear and sending continues as normal.
I am not sure what I can do about this. Possible solutions I can think of are all based on server configuration.
1. A line could be added to your /etc/my.cnf (MySQL config file) to change the connect_timeout variable from 5s to 15s (which I think should be long enough).
[mysql]
set-variable = connect_timeout=15
2. The SMTP server could be configured to not perform the DNS lookup on-the-fly during the SMTP "RCPT TO" command.
I am assuming this is qmail-specific problem because searching Google (http://www.google.ca/search?hl=en&q=%22451+DNS+temporary+failure%22&btnG=Search&meta=) for the error "451 DNS temporary failure" results in a number of results containing optional qmail "patches".
Some relevant information may be able to be found here:
http://web.infoave.net/~dsill/lwq.html#smtp-slow
Some possibly irrelevant qmail options can be found here:
http://web.infoave.net/~dsill/lwq.html#config-files
A list of "RCPT TO" patches, one of which could be causing this lookup on-the-fly/problem, seems to be listed here (though, the majority these patches seem to be used to validate local addresses:
http://http.netdevice.com:9080/qmail/rcptck/
Work-arounds might include:
1. LMP recipient email DNS checking before a mailout.
This is a tough one.. :) I hope to be able to find a specific patch that is causing the problem and be able to offer a solution soon.
Regards
-
My host says they have the time out set to 10 but it has been that way for 6 months. The 1.84 version is where this started.
Also, the domains that are causing the problem ARE failing dns look-up (not just at the host) - so they are the problem.
Can't you just "if(timeout) skip" ???
The process has not changed in recent versions. Skipping may be possible but it won't be easy. The delay occurs when waiting for the server's response to the SMTP "RCPT TO" command. I will see if I can place some sort of timer on the read function so you can set how long to wait for response to these requests.
Regards
-
Hi DW,
I will see if I can place some sort of timer on the read function so you can set how long to wait for response to these requests.
This seems the most logical way to do it, please let us know when this is available...
Regarding v. 1.84 - I believe it may have started there - I know that I added/updated the code throughout your loop that sends email to make sure it skipped bad email addresses sometime before this version - I never had this problem in v. 1.84 - I believe it's because of the code I put into your loop, but I can't seem to locate the code - I spent hours looking on Friday, if I find it, I'd be happy to share it with you - I'm pretty sure I was looking for bad addresses (via the error codes - 550 or some other number) and then skipping the re-connect or other code at that point and continuing on - can you locate that code? If so tell me where it is, it will help me track down the changes I made...
UPDATE
=====
I think I found something, I went back to v. 1.74b code and found all of these sections within the send link where it tried to bail out, for example here's one:
if($lastmsg == '503'){
fputs($ssock, "RSET\r\n");
$rsrvmsg = fgets($ssock,1024);
$errmsg .= $srvmsg;
if($bugs){ echo "RSETmsg=$srvmsg<br>"; flush(); }
flush();
$skiptonext = 1;
$reconnect = 1;
if($bugs){ echo "Bad RCPT response ($email), skipping.<br><br>"; flush(); }
} else {
This is inside of this section of code (in v.1.86):
$lastmsg = substr($srvmsg, 0, 3);
if ($lastmsg <> "250"){
Approximately - line 1500, hth, let us know what you find, I also had some code in there to look for if $email= '', do you need that also?
-
Hi DW,
Ok, I had to re-work the code inside the send loop throughout admin.php, I added code for '503' error code and made sure that every time I found an error - or unable to send I set these two:
$skiptonext = 1;
$reconnect = 1;
That's about it, not bad really, but you'll probably want to play around and test it - btw do you have a decent test bed DW? It doesn't sound that way if you're not able to duplicate these types of hiccups?
If you need the code email me... I can post it up here, sometime tomorrow, I'll have to go line by line as I re-worked the send loop in about 3-4 different places.
(*see previous message in this thread also)
UPDATE
=====
Nope, that's not it, it's just stalled on me again - ok DW, since I've been working in that send loop - can you please just tell me WHERE to put some code to delete the top record in lm_sendq - so that I can just delete that record and continue (for example - if skipnext, perhaps look at the record count - reccount, and if it's the same just delete the record)?
Please tell me, or provide the code, that's all we need - this is a pain, it's stalling every day!
2ND UPDATE
========
Ok, it's been 10 days since I first posted the first thread - when are we going to get some code to fix this issue? I'm going to code it myself tomorrow if I don't find something up here - I'll save the reccount() in lm_sendq, if it's the same several minutes later - I'll delete the top record, that should about do it - and I'll put it in the looping code - I don't see what the problem is with providing code to do that - or building it into resume.php - more likely it belongs in the send loop within admin.php...
Skipping may be possible but it won't be easy.
Why? I don't see what's so hard about it, really, and I didn't write this code, you did... Is there something I'm not thinking of, as from what I can tell it only involves a simple reccount() check in the send loop (as I mention above), if it stays the same for several minutes, obviously there's a problem with sending... so delete the top record in the table lm_sendq - can you fill me in DW on anything else?
-
You're right, of course, Brett - there was no need for the pessimism. :)
Ok - I got it! Change the function "getsmtpmsg" in admin.php (search for "ion get") to the following:
function getsmtpmsg($xsock){
global $smtp_timeout;
if(!is_numeric($smtp_timeout)) $smtp_timeout = 9;
$data = "";
stream_set_timeout($xsock,$smtp_timeout);
while($str = fgets($xsock,1024)){
$data .= $str;
if(substr($str,3,1) == " ") break;
}
return $data;
}
To override the default setting of 9 seconds, set the $smtp_timeout var in config.php.
There was some additional cleanup to do in the sending routine (mostly log file writing, etc), but the above change will fix the immediate problem with the same working result. The code in all it's glory will appear in v1.87.
The timeout, unfortunately, renders the socket connection useless and a reconnect must be issued.
I plan to include a tool to weed out these non-existent addresses before they are emailed ASAP.
I also expect to issue a critical release very soon in light of the recent bounce mailbox bug discovery.
Ok, it's been 10 days since I first posted the first thread - when are we going to get some code to fix this issue? btw do you have a decent test bed DW? It doesn't sound that way if you're not able to duplicate these types of hiccups?
Since individual servers can vary so greatly testing can require that a user give me access to their installation. If you could have provided a test installation for me to work on I might have been able to start sooner. Thankfully, another user having the same problem (dean) came to the rescue by submitting his info. This fix was made possible by having direct access to his installation and files.
I hope this fixes your troubles, Brett! If it doesn't, try setting the timeout lower, with a line such as follows in config.php:
$smtp_timeout = 4;
Regards
-
I plan to include a tool to weed out these non-existent addresses before they are emailed ASAP.
Hi DW, I have been following this thread (as far as I can understand it...) with interest.
As well as a 'tool' for cleaning up existing lists (if I understood that correctly), will something be included in the subscribe code to check and prevent these from being added to a list in the first place, maybe.? (possibly even checking any added manually...)
(My understanding of a 'tool' is something that you use when needed, as apposed to something inherent in the system that you don't have to remember to use.)
Regards.
-
I was thinking the routine could be run manually and/or before dailymail each day. I like the idea of adding it to the subscribe process - I'll definitely set that up along with a new error message. This should probably also be added to the import process. It might be better to run the verification after import rather than on-the-fly... either way it should work.
Check out this very useful PHP code snippet:
http://php.net/manual/en/function.getmxrr.php#64117
It seems from the resulting SMTP responses we might even learn the exact status of mailboxes, such as "Mailbox Full". That is, if there's a numeric SMTP code for each status, as the text messages are mailer-configurable.
The question is, do we want to do this on EVERY mailing for EVERY message and ultimately take ALL unnecessary load off of the SMTP server and bounce routines? Sounds like a good idea to me. :)
While I'm at it, why even send to an SMTP server at all... I could include a new mailing option for ListMail to be it's "own" SMTP server.
The most concerning problems, I think, are 1) the number of sockets we are able to open up at a time and 2) achieving some sort of efficiency/speed. I think common SMTP software has come a long way in this area... Then again, it wouldn't necessarily be slow overall, because ListMail needs to take the time to process the queue, message codes, etc. for each email anyways.
I'm going to move this to a new thread soon for more discussion.
UPDATE: I'm done editing this post now. :)
Regards
-
Hi DW,
Ok, back to this - thanks a million - it did solve the problem, oops I think I spoke to soon...
UPDATE
======
The change to getsmtpmsg has slowed down my sending considerably - is that supposed to happen DW? I tried it with $smtp_timeout=9, it's slow, so I moved it to $smtp_timeout=4 as you suggested and it's even slower, so now I'm going the other way $smtp_timeout=12 - WHAT am I affecting here, it seems like it's stalling about every 30-40 emails now???
Another thing I still wonder about though, I seem my mail going through, but recently using the new function you wrote (getsmtpmsg($xsock)) - it seems that I'm getting fewer clicks through, and fewer purchases...
What I'm wondering about is if this new function would be somehow NOT getting some of the mail through because of this: while($str = fgets($xsock,1024)){?
WHAT if the 9 second delay is not enough to get 1024 bytes through? I'm guessing that it would SKIP this email address and go on to the next is that right?
-----
Regarding your other interesting find: It might be better to run the verification after import rather than on-the-fly... either way it should work.
YES! Please provide a way to run this separately AFTER import, and also give anyone a way to turn this Off/On when they want to, I'm not 100% sold on this idea - as I would guess other large list members up here are also not...
Here's why - this needs to be fully tested, as does the little getsmtpmsg function described above, in a full production environment with a LARGE list to be absolutely sure - 100% POSITIVE - that the same number of emails will get through with these new function calls in place.
The question is, do we want to do this on EVERY mailing for EVERY message and ultimately take ALL unnecessary load off of the SMTP server and bounce routines? Sounds like a good idea to me.
I'm skeptical, as I've seen these type of things before and they never seem to work completely, or at least it may LOOK like they are working, but try them out on a large list and you lose 1/2 of your messages going through! Yes, it can happen! So, please see to it that you full test these functions on a very large list and make sure you are getting the correct numbers, for all of us, then please post the results so we can all rest easily before this type of thing is put into the code in the current version...
While I'm at it, why even send to an SMTP server at all... I could include a new mailing option for ListMail to be it's "own" SMTP server.
Now that sounds real promising - but I'm guessing if you're going to do something like that - it ain't gonna work in Php, it's just not fast enough, you're probably going to need Perl, or a compiled 'C' program as a base module to pull that off??? Best of luck, interested to hear what you find out and further information on this moving forward...
-
If you want my $0.02 as a somewhat large list owner...
I'd much rather Listmail NOT do the SMTP work at all... thats what (~INSERT YOUR SMTP PROGRAM HERE~)'s job is.
IMHO all the problems you are having with this is probably more easily fixed by tweaking your (SMTP PROGRAM) instead of Listmail.
Although I understand, Dean, that you want Listmail to "Work" with everything.
As I stated before, it should be this easy:
Listmail generates the email, connects to SMTP, sends it, Email program says ok, next, etc, etc. THEN the SMTP program tries to deliver the message and "bounce" processing removes any bad messages.
Of course on signup/import, etc. I think the rules in place already for simply not allowing badly formed email addresses is probably enough. I think it probably would be a BAD idea for some kind of super complicated routine to make sure an email address is actually working when imported.
-
Hi DW,
Please reply to this - you never responded to my previous message (above Mike's message)....
It seems that adding this new function (or adding the delay) to the smtp calls is helping get the mail delivered but it SLOWED down my emails tremendously, it's taking 2-3 times longer to get my mail sent out!
I now have $smtp_timeout = 12; set like this, I tried original suggestion of 9 and had problems, so went to 4 - that did NOT work and still hung up - so I went back the other way to 12 - but now my mailings are very, very slow....
What is going on here DW? Is there any way to get this under control, I still think the best way to handle this is to do what I suggested, you put a SQL call in resume.php - it looks at the reccount() in lm_sendq, if it does NOT change in 30-40 seconds, delete the top record - that way the mail can still go through quickly - instead of throwing this LONG DELAY into every single SMTP call - think about it.... let me know what you plan to do, but I know I can NOT tolerate this slowness of mailing, I'll have to go back and write my own function call in resume.php if you can't supply the code for it... or at least give me an idea on how to do it - how much time to wait, etc...
-
It seems that adding this new function (or adding the delay) to the smtp calls is helping get the mail delivered but it SLOWED down my emails tremendously, it's taking 2-3 times longer to get my mail sent out!
This is somewhat expected as it will take up to the $smtp_timeout # of seconds for each invalid address to fail and ListMail to detect it, skip the user, and reconnect. If you get several of these in a block of 50 you might notice your counter incrementing very slowly. What we need to do is clean your list with a script such as mentioned here (http://php.net/manual/en/function.getmxrr.php#64117) to remove these failing addresses from mailing entirely.
if it does NOT change in 30-40 seconds, delete the top record, instead of throwing this LONG DELAY into every single SMTP call
We should be able to continue within 5-10 seconds with the changes made. There is no mandatory delay for every call, a 'timeout' was simply added for when the server doesn't respond sooner.
Regards
-
Hi DW,
If you get several of these in a block of 50 you might notice your counter incrementing very slowly. What we need to do is clean your list with a script such as mentioned here to remove these failing addresses from mailing entirely
I'm not sure this can be possible, if you remember from the very first posting, I would get one of these about every 500 to 1,000 addresses, that doesn't seem to be enough to cause the kind of delay I am seeing, my delay is TIMES TWO, it takes 2 to 3 times longer to send to a list!
TWO TO THREE TIMES LONGER - that would have to mean more bad email addresses than you would ever suspect - either that OR, you are not understanding fully how the SMTP connection is pausing every time it tries to send out an email...
As I stated earlier, it would be MUCH BETTER (ie FASTER and SIMPLER) to just place a simple command in resume.php, watch the reccount() on lm_sendq and if it is STUCK ON ONE RECORD, delete that record and move on - there can't be that many...
If you really want to test this - then write a simple front-end to that script you pointed me to - in that front end, go through a SPECIFIC number (100 or 1000) addresses, and keep a log, every email sent, log it, than the next, and next, and next, and watch the time delay between each address (make this available in your own admin/config of LMP) and we'll see how/why the delays are coming up - you're going to have to do something like this - you can NOT release this to others with large lists - it's going to hog up their resources BIG TIME! Believe it...
Let me know what you think....
-
Brett,
Please enable the SMTP log either globally or when using "Send Email" and you should be able to see exactly how many delayed/invalid addresses there are.
As I stated earlier, it would be MUCH BETTER (ie FASTER and SIMPLER) to just place a simple command in resume.php, watch the reccount() on lm_sendq and if it is STUCK ON ONE RECORD, delete that record and move on - there can't be that many...
One problem with this is that the lm_sendq table is not sorted. That is, it has no 'auto-incrementing id' id which can be used to sort processing. The first email one 'resume' may not be the first email the next. This was done to prevent having to update the database index every time rows are added or deleted, which causes overhead.
There is one line I see that doesn't need to be in the getsmtpmsg function
if(!is_numeric($smtp_timeout)) $smtp_timeout = 9;
This can be moved out of the function and placed in admin.php, for example just above the "function domail" line or near the top of the file. I also rewrote it like this:
if(!$smtp_timeout || !is_numeric($smtp_timeout)) $smtp_timeout = 9;
The call to stream_set_timeout appears correct as per the PHP documentation:
http://ca3.php.net/manual/en/function.stream-set-timeout.php
I tested this on another server and had what I thought to be reasonable success. If you're reading this, Dean (no I'm not talking to myself..:) ), what kind of speed are you experiencing with your email?
Brett, please enable your SMTP log ASAP so we can verify whether or not there are a significant number of invalid domains or if we need to look elsewhere. A test installation/FTP access on your server I can use to troubleshoot would be amazing...
Regards
-
Hi DW,
Please enable the SMTP log either globally or when using "Send Email" and you should be able to see exactly how many delayed/invalid addresses there are.
Next time I do a test send I'll enable it and let you know what I see... or send you the entire file...
One problem with this is that the lm_sendq table is not sorted. That is, it has no 'auto-incrementing id' id which can be used to sort processing. The first email one 'resume' may not be the first email the next. This was done to prevent having to update the database index every time rows are added or deleted, which causes overhead.
How would this be any different than what I do? It's not, I just go into lm_sendq and delete the first record, that's it, the mailing continues on from there UNTIL it hits another bad email address - I don't see where this is a problem?
The call to stream_set_timeout appears correct as per the PHP documentation
I'm not so sure about that - look at the 'Ridera' test code posted up on Php.net, have you ran that test code EMBEDDED in LMP, also have you seen this:
I have found it required to add
"stream_set_blocking($fp, FALSE )"
prior to any fgets(), fread(), etc. to prevent the code from hanging up when remote files are called and the response is slow.
Let us know what you are finding, I'll post a SMTP log probably tomorrow...
-
Whatever you do, if you do implement something like this, please make it totally optional. DO NOT SET THIS AS DEFAULT BEHAVIOR.
I don't want some of my mailings NOT reaching some people because of temporary problems and a BROKEN mailer (and I'm not talking about Listmail...I mean your SMTP program).
-
Mike, I hear you - I don't think I will be writing in a multiple-socket SMTP option anytime soon. On my servers I run about 400 outbound connections at a time using qmail and there's no way I can beat that within the memory limits of PHP. It could, however, prove useful as an option for welcome messages to avoid PHP mail()... :idea: :lol:
Brett,
How would this be any different than what I do? It's not, I just go into lm_sendq and delete the first record, that's it, the mailing continues on from there UNTIL it hits another bad email address - I don't see where this is a problem?
I may be mistaken but without an auto-incrementing ID we can't be sure of the order of the returned rows during sending, resuming, or with a manual 'Browse' as you do. I understand that you seem to be able to grab the top row consistently, but I don't think it's guaranteed. I don't know the specifics but I think MySQL could have a pointer that might be overwritten or lost if it 'expires' or the service is reset... Items in queue use a random 32 character ID without a MySQL index which allows for sorting. This is done because an index takes time to update when inserting or deleting rows to maintain quick sorted results, so it would slow down the queuing and sending processes.
If you want a custom script to do a general select and delete of the top row (if the # of entries hasn't changed in X time) I'd still be willing to program it for you.
Still, it would be better if we could check the logs and/or troubleshoot your server to figure out exactly why the system is hanging up...
look at the 'Ridera' test code posted up on Php.net
I had a look and it appears to be written to manually time the connection with blocking set to false. Blocking off means the read command doesn't wait for data from the server.
ListMail *could* do it this way, but I wonder if it would be more processor-intensive. Look at all the function calls, variable comparisons, etc. required.
A simple timeout set on the stream with blocking set to true returns a blank result which ListMail can use to know to reconnect to avoid waiting longer for the server.
The other post by ridera is concerning..
I have found it required to add
"stream_set_blocking($fp, FALSE )"
prior to any fgets(), fread(), etc. to prevent the code from hanging up when remote files are called and the response is slow.
This could be applicable here and would mean ridera's solution is the only solution. I will have to investigate this further possibly by inducing delays etc. to simulate what you experience on your server. It will be useful to see what's happening in your SMTP log. While it won't tell us how 'slow' the mailing goes, it will at least rule out the possibility of the majority of delays coming from a very large number of bad addresses / reconnects. Then again, I wonder if by 'hangup' ridera means is a complete stall and not just 'slow'. Are you able to send a whole mailing without crashing, however slowly, with the smtp_timeout configured?
Regards
-
Hi DW,
How would this be any different than what I do? It's not, I just go into lm_sendq and delete the first record, that's it, the mailing continues on from there UNTIL it hits another bad email address - I don't see where this is a problem?
I may be mistaken but without an auto-incrementing ID we can't be sure of the order of the returned rows during sending, resuming, or with a manual 'Browse' as you do.
I'm not sure where your coming from but let me tell you a little about my background, I've got a Bachelor's degree in Computer Science with the concentration in Database Systems, that's the truth, if you want I can scan in my Diploma it's hanging on the wall behind me!
There is NO reason that when you do a MYSQL query that you can't consistently pull the EXACT Same record I'm looking at using PhpMyAdmin, I browse lm_sendq and grab the first record, THAT is the one EVERY TIME that is causing the stalling - how do I know that - because EVERY TIME when I delete that record the mailing resumes as normal - the stalling stops, obviously using PHPMyAdmin the query is: SELECT * FROM `lm_sendq` WHERE 1, if you do the exact same query (or similar) in LMP - YOU ARE GOING TO GET THE EXACT SAME RECORD!
I understand that you seem to be able to grab the top row consistently, but I don't think it's guaranteed.
You said yourself there is NO indexing, I CAN TELL YOU 100% OF THE TIME - IT IS GUARANTEED.... Why the $%#$% $# do you think it is not guaranteed - if you're curious do the MYSQL query yourself - EVERY TIME it will pull the same record.... do you want me to scan in my diploma?
This is done because an index takes time to update when inserting or deleting rows to maintain quick sorted results, so it would slow down the queuing and sending processes.
I KNOW that, you've got no indexing which is fine, and is faster, BUT that doesn't change the order of the tables - unless I'm completely nuts (which I doubt), YOU ARE GOING TO GET THE EXACT SAME RECORD EVERY TIME - anyone in their right mind would never write a DBMS that would pull different records every time a query is done on an unindexed table, that's pure insanity....
If you want a custom script to do a general select and delete of the top row (if the # of entries hasn't changed in X time) I'd still be willing to program it for you.
This is what I've asked for, for 2-3 weeks now.... of course that's what I want - NOT ONLY THAT, it is THE BEST WAY to solve this problem - and NOT by throwing in 12 second SMTP delays - you've slowed my mailing queue sending tremendously going out SMTP, the solution you have is NOT going to work - I'm about to do a test mailing - I'm going to turn ON SMTP logging for you, and then I'll try to do the same test without the SMTP delay and show you....
Still, it would be better if we could check the logs and/or troubleshoot your server to figure out exactly why the system is hanging up...
I had a look (at Ridera code on PHP.net) and it appears to be written to manually time the connection with blocking set to false. Blocking off means the read command doesn't wait for data from the server.
ListMail *could* do it this way, but I wonder if it would be more processor-intensive. Look at all the function calls, variable comparisons, etc. required.
Then PLEASE do it that way, or at least try.... OR, if you do not want to get into it, just provide the function that I mentioned above - and put it in resume.php - watch lm_sendq for xxx seconds, if the same record is there, obviously it's stalled, delete the record (AFTER YOU RECORD the email address in a log for later viewing) and that's it...
To ME, it seems the later choice is the better one for now, it let us all get back to work - I can NOT sit around here and wait, for now I'm stripping OUT your delayed SMTP code, as I said it's slowed my mailings down by a factor of 2 to 3 times and I can NOT wait around for the mailings that long.
I have found it required to add
"stream_set_blocking($fp, FALSE )"
This could be applicable here and would mean ridera's solution is the only solution. I will have to investigate this further possibly by inducing delays etc. to simulate what you experience on your server. It will be useful to see what's happening in your SMTP log. While it won't tell us how 'slow' the mailing goes, it will at least rule out the possibility of the majority of delays coming from a very large number of bad addresses / reconnects.
Regards
Like I said, I'll run a test this afternoon and post the log - PLEASE do something about this, as it's killing my click-through rate and deliverability, the mailings are running VERY SLOW, and when they run, I'm not sure all the mail is being delivered - I'm down to about 10% of what I should be with a mailing????
-
I understand you are frustrated but I need you to work with me with patience - we haven't all traveled the same path. :)
, if you do the exact same query (or similar) in LMP - YOU ARE GOING TO GET THE EXACT SAME RECORD!
Yep, I get that it works, but obviously didn't have the proper education about works or how/if it can become affected. I suppose the data in the data file will always be read in the same order and not randomly... I thought I saw or experienced something awhile ago where the order was not predictable, which is the reason behind my confusion... I'll see what I can find on mysql.com/doc (http://mysql.com/doc). If you know more about the specifics or can point me toward some information let me know.
anyone in their right mind would never write a DBMS that would pull different records every time a query is done on an unindexed table, that's pure insanity....
I don't think it seems all that insane.. Data could be stored in a number of places, such as RAM, or in a cache. MySQL could access data in whatever order is more efficient. Without a sort, and because I didn't write MySQL or have the proper knowledge, I wouldn't assume the order is guaranteed. I will try to find more information on this. I can set up the script however you prefer and do/did understand that it does work for you with just "select * from lm_sendq" - I just need to clarify as to whether or not I received or imagined misinformation about the order being unpredictable.
This is what I've asked for, for 2-3 weeks now.... of course that's what I want - NOT ONLY THAT, it is THE BEST WAY to solve this problem - and NOT by throwing in 12 second SMTP delays - you've slowed my mailing queue sending tremendously going out SMTP, the solution you have is NOT going to work
Sorry, I thought we were in agreement it was a workaround instead of a solution... Even so, I should have provided the script just in case the new changes didn't work out.
What we need to do is stop the bad email addresses from getting in in the first place. Then, the SMTP delays will be available as a backup so the mailing / MySQL does not timeout.
I'll reply soon with the auto-delete script - it will be very simple... After that I'll try my hand at some email validation / verification.
Regards
-
Ok, a problem with the auto-delete script. We can only set a cron job to run once a minute. If the $smtp_timeout changes are indeed working as expected and only slow down your mailing when there's a bad address this will not be any faster. I think I've said this before...
How often did you have to resume your mailings to get them delivered? Wouldn't this indicate how many bad addresses are in your list? Was it dozens of times? If so, I suppose you could achieve reasonable mailing speed simply by sitting there clicking Resume repeatedly.
Going back to some previous messages...
The change to getsmtpmsg has slowed down my sending considerably - is that supposed to happen DW? I tried it with $smtp_timeout=9, it's slow, so I moved it to $smtp_timeout=4 as you suggested and it's even slower, so now I'm going the other way $smtp_timeout=12 - WHAT am I affecting here, it seems like it's stalling about every 30-40 emails now???
The setting affects how long to wait for a response from the server before returning a blank response, which results in a ListMail SMTP reconnect.
Does your MySQL timeout and provide an error message at all, like Dean's did, when $smtp_timeout is set too high? I believe this is the reason your mailing process ultimately "stops" and does not continue past bad addresses, no matter how long they happen to take. Setting $smtp_timeout too high will cause this. Depending on whether your server mysql_timeout is the default 5 or raised to 10 the proper/safe setting should be 4 or 9. You could go lower, to 2 or 3... if your server's DNS is performing well most lookups should take less than 1 second.
Dean's SMTP server was taking too long and MySQL returned a connect_timeout error (ListMail should output this). Setting the SMTP timeout to 9 allowed him to stay within his host's MySQL connect_timeout of 10.
:!: I just tested getsmtpmsg() with microtime(). Note that my server doesn't hang-up on the bad addresses (it checks DNS after queuing - further note: you may be able to disable on-the-fly DNS at the host level by removing a qmail module). The results were good. There is definitely no mandatory delay with the changes/timeout to getsmtpmsg():
getsmtpmsg() took 0.00039899999999998 ms
getsmtpmsg() took 0.00037000000000004 ms
getsmtpmsg() took 0.002756 ms
getsmtpmsg() took 0.005205 ms
What I'm wondering about is if this new function would be somehow NOT getting some of the mail through because of this:
while($str = fgets($xsock,1024)){
This part of the code is unchanged and as mentioned above it doesn't seem there's a problem. I think this is all due to the bad addresses and the necessary timeout due to your server's mail and MySQL settings.
I'll create the script to run every minute but I thought I'd look into the above because you were worried about the changes to getsmtpmsg().
What I STILL don't understand is how your mailings, when you manually resume/babysit them, could be much faster than with the timeout changes... Have you sent an entire mailing? Are you sure it's not just certain blocks of 50 that are being delayed and that ALL messages are? What about email to a test list - is that fast?
Regards
-
Hi,
Ok, I just ran a test with SMTP_debug on, and I don't see ANY problem here at all, the timing seems to be about what I would expect, there is nothing, that I can tell, out of the ordinary in the SMTP logs - DW, I'm waiting for you to email me your address so I can attach the logs, but I don't think I see any problem here - I even 'seeded' the list with 5 of my own addresses, they were all delivered just fine....
UPDATE
=====
Here's some Bad email addresses I tested, this is what happened, started test, 4 bad email addresses, 2 of my own, first address, chokes, gives me message: Sending error. Check your mail settings. (Server Said: ) - blank nothing in 'Server Said', click on Go To Config, see 4 messages left... click on Link Tracking, 1 minute later, ALL messages are done, I've got a SMTP log for it too... I'll send you that as well...
=======
END UPDATE
Depending on whether your server mysql_timeout is the default 5 or raised to 10 the proper/safe setting should be 4 or 9. You could go lower, to 2 or 3... if your server's DNS is performing well most lookups should take less than 1 second.
It's at 12 as I said... when I had it 9 things were slow, when I tried it at 4 things were even slower, at 12 everything seems to run fine? Perhaps it's the same thing as Dean and I'm just above my server's MySQL connect_timeout - could be, anyway you say: I believe this is the reason your mailing process ultimately "stops" and does not continue past bad addresses, no matter how long they happen to take.quote]
Not sure what you mean here - 'stops'? My mailing works fine, it USED to Stall - BEFORE I added the smtp_timeout code into the function you posted, but now it does NOT STALL - sometimes I will get the Server Error, Server Said (WITH NOTHING for the Server Error Message).
I get the feeling it's running slower because, like I said, it seems to take 2-3 times as long to do a mailing - NOW what I believe is happening is this is because it is STALLING at the bad email addresses, and then has to wait 5 minutes for the resume.php to PICK it up again and continue, if there are enough bad email addresses this could slow things considerably, correct?
We can only set a cron job to run once a minute. If
Why, can't it be included in resume.php - oh I see, you want to hit it more frequently - probably a good idea, but as you see above things seem to be working just fine, I'll get you the SMTP logs and you can see, probably what I should do is makeup a test list with all those BAD email addresses and see how those logs look?
Keep me posted on what you do to the code, as I said, I'm running SMTP_timeout at 12, so I think you're going to need to make this a global configuration option...
-
You'd think with the 2-3 weeks you've waited for a "Fix" to this problem, you would have actually fixed the real problem instead of just trying to use a "Crutch"...
Dean, whatever you do, DO NOT ruin how your program works by fixing this for 1 person, this is my last post unless something else comes up...
-
I am still amazed how anyone can send mail out to a large list and not get spam complaints, when not using a confirmation mail before adding to list. It seems that by confirming the addresses first a lot of these sending problems would not occur from the bad email addresses.
-
Brett,
my email is dean@listmailpro.com
UPDATE
=====
Here's some Bad email addresses I tested, this is what happened, started test, 4 bad email addresses, 2 of my own, first address, chokes, gives me message: Sending error. Check your mail settings. (Server Said: ) - blank nothing in 'Server Said', click on Go To Config, see 4 messages left... click on Link Tracking, 1 minute later, ALL messages are done, I've got a SMTP log for it too... I'll send you that as well...
This is strange, both the initial mailing and resume should be the same. I would be interested in seeing both SMTP logs, especially the first.
It's at 12 as I said... when I had it 9 things were slow, when I tried it at 4 things were even slower, at 12 everything seems to run fine?
This is strange too. With a lower timeout it should be faster... that is, unless it takes several seconds to reconnect, and a number of DNS results are taking between 10 and 12 seconds (?)... it shouldn't take that long to connect, though.
I believe this is the reason your mailing process ultimately "stops" and does not continue past bad addresses, no matter how long they happen to take.
Not sure what you mean here - 'stops'? My mailing works fine, it USED to Stall - BEFORE I added the smtp_timeout code into the function you posted, but now it does NOT STALL - sometimes I will get the Server Error, Server Said (WITH NOTHING for the Server Error Message).
Yes, by "stops" I meant stalled. If it's not stalling then you must be within your server's MySQL connect_timeout. I am now curious about what the server is doing when it reports the 'Server Error' message as it might be another issue... More information will hopefully show up in the ListMail SMTP log.
I get the feeling it's running slower because, like I said, it seems to take 2-3 times as long to do a mailing - NOW what I believe is happening is this is because it is STALLING at the bad email addresses, and then has to wait 5 minutes for the resume.php to PICK it up again and continue, if there are enough bad email addresses this could slow things considerably, correct?
Yes, providing the script is stalling by timeout or the unknown 'Server Said' error it will stop and then need a resume. The 'Server Said' error means something other than the timeout is now happening. So this is a new error with ListMail's SMTP communication and not really a stall/crash. From what you state that the mailing no longer stalls as it used to, the 'Server Said' error may now be the only problem and could result in what you describe.
I have completed a large portion of the script you requested, but am now stuck on one thing. Since the script will run every minute it will run less frequently than auto-resume. When the script detects that the new count is the same as the old count it is removes the top row, and we then decrease the last count. The problem is, when the script runs again if the mailing hasn't re-started it will delete the next row because again the queue count matches the last count. We can work around this by the condition that "if the last row was deleted, don't delete this row" with an additional SQL field, but what if 2 bad addresses appear in succession? It's not error-proof...
Do not use this script yet, it is affected by what I mention above:
<?php
/********************
check lm_sendq every minute, if entries check if changed and delete top row
auto-resume will then pick up the remainder of messages when the mailing has failed for one minute
place this script in your listmail folder and set up a cron task for every minute
ie: * * * * * /usr/bin/wget -O - http://example.com/mail/autodelete.php?pw=1234
A new table and row are required to store the last count:
SQL:
CREATE TABLE lm_sendq_num (lastcount MEDIUMINT UNSIGNED NOT NULL);
INSERT INTO lm_sendq_num VALUES ('0');
********************/
include('./config.php');
include('./admin.php');
if($_GET['pw']<>'1234') exit('access denied: wrong password');
$email_notify = 1;
$email_to_notify = 'test@listmailpro.com';
$tbl = 'lm_sendq_num';
$DEBUG = 1;
// leave this
$msg = '';
// retrieve counts
list($oldcnt)=mysql_fetch_row(mysql_query("select lastcount from lm_sendq_num;"));
list($cnt)=mysql_fetch_row(mysql_query("select count(*) from lm_sendq;"));
if($DEBUG) $msg .= "cnt=$cnt, oldcnt=$oldcnt\n";
if($cnt==0){
// if no items in queue reset the oldcount
if($DEBUG) $msg .= "no items in queue - resetting oldcount and exiting\n";
mysql_query("update $tbl set lastcount = '0' where 1;");
} else {
if($DEBUG) $msg .= "$cnt items queued, checking oldcount\noldcnt = $oldcnt\n";
// items in queue, check the oldcount
// if no oldcount update to new count
if($oldcnt==0){
if($DEBUG) $msg .= "oldcnt is 0, update to new count ($cnt)\n";
mysql_query("update $tbl set lastcount = '$cnt' where 1;");
} elseif($cnt==$oldcnt){
// otherwise, check if its the same and delete the first row if so
if($DEBUG) $msg .= "queued items = lastcount, delete top row\n";
// get data
list($id,$bat,$battype,$mtype,$uid,$mid,$xtra)=mysql_fetch_row(mysql_query("select * from lm_sendq limit 1"));
// delete row
mysql_query("delete from lm_sendq where id = '$id';");
list($uem)=mysql_fetch_row(mysql_query("select email from lm_users where id = '$uid';"));
$af = mysql_affected_rows();
if($af>0){
$ncnt = $cnt-$af;
mysql_query("update $tbl set lastcount = '$ncnt' where 1;");
$msg .= "The following row was deleted from the lm_sendq table:\n\nid=$id bat=$bat battype=$battype mtype=$mtype uid=$uid mid=$mid xtra=$xtra\n\nThe user has the email: $uem\n";
}
else $msg .= "Error, row not deleted.";
// notify
if($email_notify) mail($email_to_notify,'LM Auto-Delete',$msg,"From: \"LM Auto-Delete\" <noreply@example.com>");
}
}
if($DEBUG){
echo "msg=".nl2br($msg);
if($email_notify) mail($email_to_notify,'LM Auto-Delete (Debug)',$msg,"From: \"LM Auto-Delete\" <noreply@example.com>");
}
?>
-
Got your SMTP logs, Brett - Unfortunately I can't explain why your server is dropping the SMTP connection, seemingly randomly. If the error always happens at the same time (same command/result pair in SMTP log) I should be able to provide code to correctly detect the blank response and tell ListMail to issue a reconnect.
Regards
-
The first incarnation of an email verification script is now ready:
http://listmailpro.com/forum/index.php?topic=1243.msg5124#msg5124
Regards