ListMailPRO Email Marketing Software Forums
ListMailPRO Email Marketing Software Forums => General Help & How-To => Topic started by: BGSWebDesign on December 26, 2014, 11:55:10 am
-
Hi,
I've researched this on the forum, yet, I still cannot find a good way to track these down? My Send Queue keeps choking, and I need to find out WHICH email address is causing it? I'd like to find out for good, so I could prevent any email address like that from getting into the system, but for now if I could just find out which row is bad, and remove it (and then find a way to get at the email address of that row)?
Here's what my top row of the table sendq looks like:
--
ID: BAT BATTYPE NTYPE UID MID XTRA
==============================================================
0dbae0e52d312ac7440f8a9aac542ff2 11c1e3 2 2 1138039 3421 fup
---
Can you give me some idea, of how I would track down the USER this row is trying to access? Also, is THIS the row I should be concerned with (the TOP row of the table), or should I be looking at the BOTTOM row of this table? Which looks like this:
--
ID: BAT BATTYPE NTYPE UID MID XTRA
==============================================================
b8c90e8e37e935ffd0c29e5ce6f6e9cd 11c1e3 2 2 1138192 3518 fup
---
So, What does all this mean? How can I check the UID, and run it back in the USER table to find that user?
-
Hi,
The SMTP log will likely indicate which email address is holding things up. It may end up eventually timing out and skipping the user. Such things sometimes happen when a server does a long or failing DNS lookup. The mail server may be able to be configured to delay lookups until sending time and not do them during queuing which could eliminate the symptoms.
Regards
-
Hi DW,
The SMTP log will likely indicate which email address is holding things up.
I do not have SMTP logging on, but I just now turned it on. When the queue resumes, will it write the log, or am I going to have to Cancel all of the queue entries, and let it re-start over, and wait for it to hit the bad address again? It doesn't appear to be writing the log, now that I've turned it on... so seems I will have to delete all the Dailymail queue entries and watch for the next one to choke?
server does a long or failing DNS lookup. The mail server may be able to be configured to delay lookups until sending time and not do them during queuing which could eliminate the symptoms.
What is the preferred way to turn off the DNS lookup, is that in the code? You say configure the mail server, what is the best way to do that? I assume you mean changing the configuration of the Courier mail server? Is that right? Where would I look for those configurations?
Is it only possible to make those changes in the mail server, or is it possible to add code (or update code) that is already in ListMailPRO now to limit DNS lookups when queuing is taking place?
-
Hi Brett,
When the queue resumes, will it write the log
Yes, you should not have to cancel your queues. The logs should appear in the /attach folder, which must have the correct permissions (777 or read/write/execute for all).
What is the preferred way to turn off the DNS lookup ... Is it only possible to make those changes in the mail server
Yes, if this is the problem, and I've only heard of it being a problem on sendmail systems (http://listmailpro.com/forum/server-optimization-tweaks/sendmail-fix-sending-error-check-your-mail-settings/), the mail server config could be updated.
Regards
-
Hi DW,
Thanks... unfortunately I cancelled all of my jammed queues! I probably lost 50,000-80,000 emails that were scheduled to go out...
After turning on SMTP logging, I see why I do not always have it on, it takes 2-3 times longer to send mail out when it is turned on. This is why I would prefer it if there was a way to track down the bad emails that in the current Queue that is stuck? I thought in the past I was able to do that, by converting the UID to HEX and looking it up that way in the Users table, is that correct, or will it not work that way?
I believe I'm running on Sendmail, can you tell me what the suggested configuration updates are on a Sendmail system to turn off the DNS lookup, if I find that's the problem, I'm going to have to do it, I continue to get blocked up queues?
-
Hi DW,
I once again have blocked up queue with 15,000 emails. I do not have logging turned on, because it slows sending, and creates large log files, so I'm asking again, is there some way to interpret the queue database entries in lm_sendq and convert the uid field into the same uid field that appears in lm_users because they do not seem to match?
How do I make that conversion? I think I tried getting the uid field from lm_sendq and converting it to HEX, but not sure that worked either?
1.) Can you please tell me - what is the proper way to convert lm_sendq/uid field into the uid field that appears in lm_users?
2.) Are the queued entries that are stuck/jammed, coming from the TOP of the lm_sendq database, or the BOTTOM of the lm_sendq database? In other words, when I am looking up 'stuck queue entries', do I assume that it is RECORD #1 that is stuck and cannot send, or is it the LAST RECORD in lm_sendq that is stuck? Does it go sequentially after that?
-
Hi Brett,
1.) Can you please tell me - what is the proper way to convert lm_sendq/uid field into the uid field that appears in lm_users?
The uid field in lm_sendq corresponds with the id field in lm_users, so you could do the following, where "123" is the UID from the lm_sendq table.
SELECT uid FROM lm_users WHERE id = '123';
There is no other way to convert, per se, as the lm_users uid is simply randomly generated.
2.) Are the queued entries that are stuck/jammed, coming from the TOP of the lm_sendq database, or the BOTTOM of the lm_sendq database? In other words, when I am looking up 'stuck queue entries', do I assume that it is RECORD #1 that is stuck and cannot send, or is it the LAST RECORD in lm_sendq that is stuck? Does it go sequentially after that?
The queue-reading process does not use an ORDER BY clause so the order cannot be guaranteed...
Messages may be "timing out", causing confusion in the sending process. You could try increasing the SMTP timeout per this (http://listmailpro.com/forum/general-help-how-to/sending-error-server-said-354-start-mail-input-end-with/msg8870/#msg8870) and other posts.
The best way to see what's happening is with the SMTP log.
Regards
-
Hey Dean,
The uid field in lm_sendq corresponds with the id field in lm_users, so you could do the following, where "123" is the UID from the lm_sendq table.
Are you certain about that? I've tried that SQL query over and over, for multiple UID's here's a query I use:
select * from lm_users where uid='1063330';
AHHH - never mind that part, I see the problem, I was looking for uid in lm_users, when it's the id field I am matching!
Not sure why though, you chose to name the field uid in lm_sendq, when it's matching the id field in lm_users? Usually you want to keep the field names the same, especially when you have a id and uid field both in lm_users, and in lm_sendq!
Now, to resolve why I've got so many hung up in my queue, looking now at SMTP delay... just a min...
-
Hi,
See my previous post, I was pulling the wrong uid field, anyway, all those email addresses (I looked up from the UID's) in the front & back of lm_sendq look fine, so I looked to update my smtp_timeout variable, what I found is that it was missing from my config.php, even though previously I had one set at 30:
$smtp_timeout = 30;
If it is missing and then I add it, does the queue continue on, as it appears to still be locked? Do I need to cancel all those mailings that have failed, and re-start them (which is impossible as they are Dailymail sendings)?
-
Hi,
My queue is stuck at 23075. I have sent to large lists before without any problem, I have resume running every 2 minutes and this has worked for years, but now for some reason, my queue is stuck and does not move?
I've confirmed now that the email addresses are good in the lm_sendq database, I've tried removing 50 records at a time (25 from top, 25 from bottom), but the queue still does not move? I've also changed my SMTP timeout to 45, as indicated. Since this is a Dailymail sending, there's no way I know of to delete this and re-start it... so
How am I to get the queue moving again, and what else can I try?
-
Hi Brett,
The best place for more clues is the SMTP log. What does the end of the log say when it won't send?
The $smtp_timeout adjustment is only effective if the problem is identified as individual emails timing out when queuing to the server due to DNS delays or other. You may see evidence of this in the SMTP log.
It is difficult and time consuming to make further guesses at a possible cause or solution.
If you still can't figure it out please create a support ticket of the type Other/Bug Investigation at https://listmailpro.com/support
Regards
-
Hi DW,
How would I get the SMTP log to write, when this is a Dailymail send that is stuck right now? If I turn on SMTP logging, that will not record anything for that (Dailymail) send, correct? So, I would be stuck to looking at the SMTP log on the next one, right?
When I did have SMTP logging on, I was getting 0 byte log files, I just deleted hundreds of them. What would cause that?
-
Hi Brett,
How would I get the SMTP log to write, when this is a Dailymail send that is stuck right now? If I turn on SMTP logging, that will not record anything for that (Dailymail) send, correct? So, I would be stuck to looking at the SMTP log on the next one, right?
When you Resume any queue, including a Dailymail queue, the SMTP log is written, if enabled.
When I did have SMTP logging on, I was getting 0 byte log files, I just deleted hundreds of them. What would cause that?
Are you out of disk space?
Regards
-
Hi DW,
When you Resume any queue, including a Dailymail queue, the SMTP log is written, if enabled.
It does not seem I can Resume a Dailymail queue, as I have auto-resume firing off every 2 minutes. Even then, if I switch my configuration to Always write SMTP logfile, it doesn't make any difference, nothing gets written, no file at all.
Are you out of disk space?
It seems these zero byte files are written over and over, when I had resume running and had the: Always write SMTP log file set to - On, before the Dailymail.php ran that stuck the queue.
Here's the one larger file that was written, but I don't think this queue got stuck, does this look like anything to worry about? I notice that before every message, there is that
SMTP message: '354 Enter message, ending with "." on a line by itself'?
--
250 OK
250 Accepted
> DATA
354 Enter message, ending with "." on a line by itself
> SENT DATA
250 OK id=1YQd6m-0000cf-NT
--
-
Hi DW,
See my previous post... then what I found below...
It seems these zero byte files are written over and over, when I had resume running
Check that, it appears that every SMTP logfile written is zero bytes, even when it first starts Dailymail, and continues running it (with resume). I've got a directory full of zero byte files, and I do not have a disk that is full.
At some point though I did get a SMTP logfile written (see below), but that file may have been written from a Send an Email send (instead of a Dailymail send), is there some problem with Dailymail writing logfiles?
-
Hi Brett,
The section of SMTP log you posted doesn't suggest a problem.
I am unfortunately out of guesses and unable to determine why zero byte files are being written without further hands-on access. Please create a support request (https://listmailpro.com/support). If absolute privacy is desired you could set up an extra LMP installation and FTP account where the issue can be recreated for me to work on.
Regards
-
Hi DW,
Thanks for letting me know. I've found what I think may have caused some of the problem. My mail queue was overloaded with 'Delayed sending messages', actually it was filling up near the top of my capacity setting for my 'bounced email address' mailbox. I've had that cleared to zero, and wiped everything else, and now I'm not getting a stuck queue anymore.
I am still however getting the zero byte files, every time Resume runs. I'll let you know if that becomes a problem.
It appears that Resume.php is creating zero byte logfiles, however normal sending from within LMP works fine, and creates a normal logfile.
-
DW,
Can you please tell me the mechanism (code) that sets the lm_sendp (queue protection table) completed field (lm_sendp->completed) to '0', so that it can start sending? Is that done in Dailymail.php?
The reason I'm asking is that I think I found my problem.... I know it makes no sense, but none of my Dailymail followups send, BUT, every time I do a Selection Send (even to the same email addresses), it works just fine, so I suspected there was some other problem.
What I found is that my lm_sendp table had all of my Dailymail followup batches listed in it with completed field set to 'q', which meant that it would NEVER try to send those, even though they had just been loaded up?
Why that is occurring I don't know, so I'm trying to locate the code that should set that value (lm_sendp->completed) to '0' (ok to send) so that my Dailymail followups can continue to send properly?
If I can locate that code, and make sure it is being run - after the queue of followups is loaded, then it's likely I can get everything running properly again.
-
Hi,
One last piece of the puzzle. For some unknown reason my Listmail installation has stopped working normally. What I've found is that the queue protection table lm_sendp gets loaded with the batch when Dailymail runs every day, and the 'completed' field ( lm_sendp->completed ) always gets initialized to 'q', which prevents the resume function from ever sending that mail, until apparently it gets set to '0'?
Here is the code in Admin (line 2831) that sets it to 'q':
mysql_query("insert into $ptable values('','".addslashes($bat)."','".addslashes($qtype)."','".addslashes($doformid)."','$now','$now','','q');") or die(mysql_error());
I'm trying to locate the of code that sets the value of lm_sendp->completed to '0' so the sending can start?
Can anyone let me know where that code is - or if there is some other way that this value in lm_sendp gets set to '0' so that the sending can start? I've noticed if I manually set that to '0' everything is fine and the sending commences, but if I do not, all that happens is the queue gets loaded with all the Followups and other Dailymail mail, but never gets sent - it just sits there?
I'm not sure why this just started happening, but my guess is it might have something to do with PHP code running differently? I cannot think of anything else that would cause it, I've been running ListmailPro just fine for 12 years now!
Thanks in advance...
-
An SMTP log of zero bytes or a queue that won't send could be because the queuing script dies or is killed by the server before sending.
A script like this could be used to see if your server ignores set_time_limit:
<?php
set_time_limit(0); // unlimited
$time_to_run = 400; // seconds to run
$start=time();
$last=$start;
while(1){
$diff=$now-$last;
$now=time();
if($diff>0){
$last=$now;
echo ($now-$start).'.. ';
flush();
}
if($now-$start>=$time_to_run) break;
}
?>
A crude band-aid that would at least see that messages that were able to be queued are sent out would be a script like this:
<?php
// fix LMP queues that stall. include this script in resume.php after config.php and admin.php, or run manually from separate cron job. script requires writable queuefix.dat in same folder
include_once('./config.php');
include_once('./admin.php');
if($file_version>'1.88') exit("Fatal Error: queuefix at http".(!empty($_SERVER['HTTPS'])?"s":"").
"://".$_SERVER['SERVER_NAME'].$_SERVER['REQUEST_URI']." needs to be updated to use mysqli");
$log='';
$echolog=false; // set to true for output
// read last recorded data
$fdata=file_get_contents('./queuefix.dat');
if($fdata) $log .= "existing data:\n".nl2br($fdata)."\n\n"; else $log .= "no existing data\n";
$lines=explode("\n",$fdata);
$data=array();
if(!empty($lines)){
foreach($lines as $line){
list($batid,$msgcnt)=explode(':',$line);
$data[$batid]=$msgcnt;
}
}
// read current data
$out='';
$now=time();
// v1.88 mysql_
if($file_version<='1.88'){
$rows=mysql_query("select batid from lm_sendp where completed = 'q';");
if(mysql_num_rows($rows)>0){
$log .= "found queued messages, processing..\n";
while(list($batid)=mysql_fetch_row($rows)){
$log .= "batid $batid is queuing.. checking.. ";
list($count)=mysql_fetch_row(mysql_query("select count(*) from lm_sendq where bat = '$batid';"));
$log .= "count=$count\n";
if($count>0){
if(!isset($data[$batid])){ // no existing data, record it
$log .= "no existing data for batid $batid, recording:\n";
$log .= "out.=$batid:$count\\n\n";
$out.="$batid:$count\n";
} else {
// check if count is the same as last time, if it is fix the queue
if($count==$data[$batid]){
$log .= "count is same as last time, queuing stalled, running fix: ";
$log .= "update lm_sendp set completed = '0' where batid = '$batid';\n";
mysql_query("update lm_sendp set completed = '0' where batid = '$batid';");
$echolog=true;
} else {
$log .= "count changed, still queuing, ignore.\n";
// if count not same, still queueing up or down or now sending, so ignore it and start again
}
}
}
}
} else {
$log .= "no messages are queuing\n";
}
// End v1.88
} else {
// update for mysqli
}
// write out new data
if($out){
$log .= "writing out data:\n".nl2br($out)."\n";
file_put_contents('./queuefix.dat',trim($out));
} elseif($fdata){
$log .= "clearing out data\n";
file_put_contents('./queuefix.dat','');
}
//file_put_contents('./queuefix.dat','test:test');
if($echolog){
if(strstr($_SERVER['HTTP_USER_AGENT'],'Wget')===false) $log=nl2br($log);
echo $log;
}
?>
That script should also answer you question about the command to use to get such a stuck queue going.
Regards
-
Hi DW,
Does this code: $file_version>1.88 indicate ListmailPRO version 1.88, or is that related to mysql? I am running version 1.85 of ListmailPRO.
A few things.
1.) I do have the Zero Byte SMTP logs all the time, so I'm guessing the script is timing out, but why would it start doing that now when it's been running fine for 12 years? I will run your test scripts and see
what I come up with.
2.) Thanks a million for the Band-Aid, is it better to run that by itself, or to put that code directly in resume.php? If I run it directly with a cronjob, can I run it every 5 minutes, or so? It appears it would do what I'm doing manually, checking that there is a 'q' in completed, change it to '0' and let that queue start sending.
I know that once I set completed to '0', the emails go out, as I can see them sending out as the queue size goes down as sending proceeds - are you suggesting that those emails may not be sending out - if the script times out?
That's a huge enlightment, as that would mean that anything I'm sending out, even large broadcasts, that I send to 70,000 or more would NOT be going out if the script times out, is that correct? The reason I ask, is that last week I've sent out several times to over 70,000 and I've seen a few times, where I did actually get the 'last screen' showing that ALL Messages sent, how fast they sent, etc... in other words, it appears those very large sends were not timing out, would that be correct? While often I will see the screen that says SMTP error, but then the Resume script keeps them going out... at least that's what I thought.
I will post here next with the results from the timeout script, to let you know what that shows.
Thanks a ton!
-
Hi DW,
See my previous reply...
Here are the results of the Timeout script, does this tell you anything?
--
1.. 2.. 3.. 4.. 5.. 6.. 7.. 8.. 9.. 10.. 11.. 12.. 13.. 14.. 15.. 16.. 17.. 18.. 19.. 20.. 21.. 22.. 23.. 24.. 25.. 26.. 27.. 28.. 29.. 30.. 31.. 32.. 33.. 34.. 35.. 36.. 37.. 38.. 39.. 40.. 41.. 42.. 43.. 44.. 45.. 46.. 47.. 48.. 49.. 50.. 51.. 52.. 53.. 54.. 55.. 56.. 57.. 58.. 59.. 60.. 61.. 62.. 63.. 64.. 65.. 66.. 67.. 68.. 69.. 70.. 71.. 72.. 73.. 74.. 75.. 76.. 77.. 78.. 79.. 80.. 81.. 82.. 83.. 84.. 85.. 86.. 87.. 88.. 89.. 90.. 91.. 92.. 93.. 94.. 95.. 96.. 97.. 98.. 99.. 100.. 101.. 102.. 103.. 104.. 105.. 106.. 107.. 108.. 109.. 110.. 111.. 112.. 113.. 114.. 115.. 116.. 117.. 118.. 119.. 120.. 121.. 122.. 123.. 124.. 125.. 126.. 127.. 128.. 129.. 130.. 131.. 132.. 133.. 134.. 135.. 136.. 137.. 138.. 139.. 140.. 141.. 142.. 143.. 144.. 145.. 146.. 147.. 148.. 149.. 150.. 151.. 152.. 153.. 154.. 155.. 156.. 157.. 158.. 159.. 160.. 161.. 162.. 163.. 164.. 165.. 166.. 167.. 168.. 169.. 170.. 171.. 172.. 173.. 174.. 175.. 176.. 177.. 178.. 179.. 180.. 181.. 182.. 183.. 184.. 185.. 186.. 187.. 188.. 189.. 190.. 191.. 192.. 193.. 194.. 195.. 196.. 197.. 198.. 199.. 200.. 201.. 202.. 203.. 204.. 205.. 206.. 207.. 208.. 209.. 210.. 211.. 212.. 213.. 214.. 215.. 216.. 217.. 218.. 219.. 220.. 221.. 222.. 223.. 224.. 225.. 226.. 227.. 228.. 229.. 230.. 231.. 232.. 233.. 234.. 235.. 236.. 237.. 238.. 239.. 240.. 241.. 242.. 243.. 244.. 245.. 246.. 247.. 248.. 249.. 250.. 251.. 252.. 253.. 254.. 255.. 256.. 257.. 258.. 259.. 260.. 261.. 262.. 263.. 264.. 265.. 266.. 267.. 268.. 269.. 270.. 271.. 272.. 273.. 274.. 275.. 276.. 277.. 278.. 279.. 280.. 281.. 282.. 283.. 284.. 285.. 286.. 287.. 288.. 289.. 290.. 291.. 292.. 293.. 294.. 295.. 296.. 297.. 298.. 299.. 300.. 301.. 302.. 303.. 304.. 305.. 306.. 307.. 308.. 309.. 310.. 311.. 312.. 313.. 314.. 315.. 316.. 317.. 318.. 319.. 320.. 321.. 322.. 323.. 324.. 325.. 326.. 327.. 328.. 329.. 330.. 331.. 332.. 333.. 334.. 335.. 336.. 337.. 338.. 339.. 340.. 341.. 342.. 343.. 344.. 345.. 346.. 347.. 348.. 349.. 350.. 351.. 352.. 353.. 354.. 355.. 356.. 357.. 358.. 359.. 360.. 361.. 362.. 363.. 364.. 365.. 366.. 367.. 368.. 369.. 370.. 371.. 372.. 373.. 374.. 375.. 376.. 377.. 378.. 379.. 380.. 381.. 382.. 383.. 384.. 385.. 386.. 387.. 388.. 389.. 390.. 391.. 392.. 393.. 394.. 395.. 396.. 397.. 398.. 399..
--
I will also run the other script queuestuck next.
-
DW,
Thank you for that code to unstick my Queue, I'm still not sure why it happens (queue sticks every day), after years of running just fine, but your code runs perfectly and gets things running again!
Most likely then, is it because the script is timing out, is that the problem?