ListMailPRO Email Marketing Software Forums

ListMailPRO Email Marketing Software Forums => Server Optimization, Tweaks => Topic started by: gwill23 on August 20, 2004, 10:00:12 am

Title: Mailing list 100,000 and up
Post by: gwill23 on August 20, 2004, 10:00:12 am
I have a large mailing list.  The hosting company I am using is mad because I was using a shared server and the bounce.cgi script was crashing the server.  I guess I get too many bounces.  Stinking free email companies.  

Anyway I am working on a solution.  Any help from anyone having similar problems your advise would be appreciated.  

I believe this is why my dailymail is stopping every 5-10k messages or so.

just post a reply to this post with this is me and that way you will be notified any time I update this post.  I'll will let everyone know what is going on.
Title: Mailing list 100,000 and up
Post by: gwill23 on August 24, 2004, 04:37:33 pm
I am now up and running on a virtual private server.  Not as expensive as a dedicated server.  However, I may upgrade to a full dedicated server soon.  

I am testing dailymail for a few days and might hav a bounce issue but we will see soon.  I will update this post soon.

A word about migrating from one system to the next.  It can be hard.  Finding out how to dump the mysql database to a .sql file seemed to speed things up for me.  It was an option on my server so I don't know the code.  It would be the same as using a backup feature I think.
Title: Mailing list 100,000 and up
Post by: gwill23 on August 24, 2004, 06:23:45 pm
Thank you thank you thank you.  I pegged my server during dailymail.  This caused dailymail to stop processing.  My server statistics are below.  I can almost be certain that anyone sending out more than 20k email on a shared server is pegging the processors.  

No I will admit I think I get a large number of bounces.  While the emails are good.  I have several spammer addresses in my list.  By sending my list out FROM my bounce address all the autoreplys are cleaned off my list.  With a list my size I can't keep up with all the autoreplys and most are just garbage anyway.

First try server
Transfer (MB) 26,000
Burstable Transfer (MBps) 1.5
Disk Space (MB) 4500
Reserved CPU Processes 175
Burstable CPU Speed 2800+  
Reserved Memory (MB) 256

Second try with over 14k email to send
Transfer (MB) 26,000
Burstable Transfer (MBps) 1.5
Disk Space (MB) 4500

Reserved CPU Processes 350
Burstable CPU Speed 2800+
Reserved Memory (MB) 512

I will continue to keep you all updated.
Title: 16000+ emails sent no problem
Post by: gwill23 on August 25, 2004, 09:10:27 am
Dailymail didn't die and I recieved a confirmation email stating the correct number of followups had been sent out.

I still have an issue editing the database.  I hope DW will help me with that soon.

I also, will try sending out to the entire list sometime next week.  This way I can see what happens when I max out my resources.
Title: good job
Post by: BGSWebDesign on August 26, 2004, 07:52:26 am
Hi Gunther,

Thanks for keeping us posted here, this is valuable information for anyone running a large list (over 50,000).  I have 120,000 and know that an issue with bounces is going to come up.

If you haven't already seen it, there's one other option, but you might not be willing to use it, as it will slow your send process by hours: use throttle code to throttle the sending procedure.  I've done this with some good luck, what it does is also limit the speed of the bouncebacks.

See this post for more info:
http://listmailpro.com/forum/index.php?topic=32.0

See my Follow Up and Stats post (May 11, 2004), and good luck!
Keep us posted on how it goes.
Title: Mailing list 100,000 and up
Post by: gwill23 on August 26, 2004, 08:51:59 am
35000+ followups sent out last night without any issues.  Dailymail did not stop.  It looks like the issue with dailymail stopping and leaving messages in queue is due to pegging the processes allotted to you.  The throddeling code should work to fix that but I think it might cause dailymail reports to be sent out before the process is finished, and you must be willing to wait for the log process.  My server right now will send out 20000 emails in under an hour.  

DW is looking at a database error I am having as well as possible issues with the bounce.cgi.  We will have to see how it preforms once these two things are fixed for sure.

FYI if you have SSH console access to your system you can execute the TOP C command to watch the processes during a mailing.  This is helpful to see what exactly is happening during the dailymail process.

Gunther Williams
http://www.opportunityone.net
Title: Mailing list 100,000 and up
Post by: porl on November 22, 2004, 12:43:51 am
You want to push your server then it will bog down and spit the dummy. Slow it down a bit. I am pushing lists of 25,000 at a time and taking 4 hours per list. OK - its slow - but I can leave it and the server will not crash.
Title: Mailing list 100,000 and up
Post by: DW on November 22, 2004, 10:29:25 pm
You should be able to achieve at least 750,000 messages per day but yes, your server can be bogged down.  Performance with very large lists is still being tweaked to prevent this.  I will likely have to rewrite bounce.cgi to be a complete Perl solution, as opposed to passing data to a URL, to offer better performance.  This might mean 2 config files.  config.php and config.pl.  I believe this is where a lot of the problems occur.  I am also considering that there may be MySQL processes that don't die or are increasingly using more memory.
Title: Large lists
Post by: KB on November 25, 2004, 09:42:18 am
I have been watching the posts re large lists. I run upwards of 1 mill now with listmailpro. Lots of trouble with even a dedicated server in the beginning. Require 2 things to fix. Clean supply of lists - that reduces the bounces - and a dual processor.  I was with ev1 - waste of time  - now with a new server supplier and no crashes even if I push the server to way past it's operating limit. Can still only send about 30k ( merging ) an hour. i'd like to hear if anyone is maxing that and how.

Kevin
Title: Mailing list 100,000 and up
Post by: mike2 on December 02, 2004, 08:08:38 am
My List is only around 150K, but I run it normally at 60-80K / hour.  Done with 2 machines, one hosts the DB, the other is a dedicated Mailer running sendmail.

Mike Rogers
Get Extreme Marketing For Your Website:
http://ExtremeMarketingProgram.com
Title: wow.... didn't know it was possible..
Post by: BGSWebDesign on December 08, 2004, 08:05:09 am
Hi Kevin and Mike,

Quote
but I run it normally at 60-80K / hour.


Those are amazing numbers, also Kevin the 1Million addresses, I had no idea LMP could handle a list of this size, if either of you could share which host is best for a list of that size I'd like to know (Kevin, you mention that EV1 is not good anymore, which host do you use now?)

Mike, using two boxes and splitting db and sendmail up is great, I never heard of doing that with LMP but sounds great too - surely you're running dedicated?  or leased?  

Kevin, just curious how you do handle bounce processing???  I'm running about 120K bouncing is the next issue I'll have to deal with..

Any info you could provide would be helpful...
Title: I need a host recommendation??
Post by: BGSWebDesign on December 27, 2004, 04:39:24 pm
Hi everyone,

I'm over 120K list size now and need a recommendation on a decent host that can handle this size?  Colocate, shared, virtual private, can someone please tell me which one is best configured for a large list?  

Basically I want to be able to mail to large list sizes 12K-25K and let it go all day, if it has to - it would be best if I could shut down the browser window and have it still run in the background - but I don't even know if that's an option, can someone tell me how they are doing that:

Porl -
Quote
25,000 at a time and taking 4 hours per list. OK - its slow - but I can leave it and the server will not crash.


Gunther (gwill23) -
Quote
I am now up and running on a virtual private server. Not as expensive as a dedicated server. However, I may upgrade to a full dedicated server soon.


Gunther - do mind me asking which host you are using, and if you're still happy?

Mike2 -
Quote
My List is only around 150K, but I run it normally at 60-80K / hour. Done with 2 machines, one hosts the DB, the other is a dedicated Mailer running sendmail.


Which host are you using for something like that, or are you doing it all on your own server - in house?

bauer -
Quote
I was with ev1 - waste of time - now with a new server supplier and no crashes even if I push the server to way past it's operating limit.


Kevin, can you please tell me who you would recommend?  I've got to get this server issue handled as I'm spending HOURS baby-sitting LMP and clicking on the 'resume' link!

I can't do that much longer - and DW (Dean) when will see an update/new version with real background processing on sending email?  Or, can you please tell me a way to code that with a PERL or Php task that I can run with CRON?

Getting frustrated 'resuming' all day long!!!!

Sure could use some input...
Title: Mailing list 100,000 and up
Post by: mike2 on December 28, 2004, 11:42:04 am
I am using my own machines on a Dedicated T-1 Line in my own office... :D

The machines are both relatively low-end Redhat Linux.  The mailer itself, which is probably most important is like a 1.2Ghz P4, with 2 regular 7200RPM IDE drives with a somewhat self-tweaked sendmail.

BTW, I love LMP and Dean has been tremendous, but I've been looking for an update now for almost 1 year with no luck... so lets hope I guess on that update.
Title: Mailing list 100,000 and up
Post by: BGSWebDesign on December 29, 2004, 03:34:38 pm
Hi Mike,

THanks for the follow-up... unfortunately I'm not at the point where I can afford my own T-1 LINE!!!   I need another way to handle the problem and I'm looking at creating a module that will jump start the QUEUE from a PERL module called by CRON.

If anybody else has ideas on this I'd like to know...
Title: Mailing list 100,000 and up
Post by: DW on December 30, 2004, 02:30:31 am
Most servers do not time out in the middle of sending.  Your host may have a restriction on PHP script loading which would still be reached even with the upcoming background sending option.  A dedicated server should handle over 720,000 messages per day so even a shared host (configured correctly) with 10-20 users should be ok.
Title: Mailing list 100,000 and up
Post by: BGSWebDesign on December 30, 2004, 07:48:43 am
Dean,

Quote from: "DW"
Most servers do not time out in the middle of sending.  
Your host may have a restriction on PHP script loading which would still be reached even with the upcoming background sending option.


I don't believe this is the case, as sometimes it can complete all the way through the mailing without showing the RESUME option, how could that be if the problem were related to PHP script loading?  

Also, I use SMTP server mailing (reconnect every 2500 emails), not PHP, is that part of the problem, should I set RECONNECT every 2500 emails to a lower number, say 500?
Title: Mailing list 100,000 and up
Post by: chufford on December 30, 2004, 06:49:48 pm
I was loosing messages a while back. I found that my server only allowed 1000 messages per connection. I changed the "reconnect" setting to 999 and that made all the messages work.

There are some other interesting optimizations discussed at
http://listmailpro.com/forum/index.php?topic=49.0
Title: Requests about where i host
Post by: KB on February 01, 2005, 01:45:00 am
Hi Guys

I've just seen some of your queries about where I host. I've mailed my own server guys to see if they want more mailing business.They will respond shortly - if they are happy I'll post their details.

Regards

Kevin
Title: options for hosting...
Post by: BGSWebDesign on February 01, 2005, 06:54:09 am
Hi Kevin,

Thanks, I'd like to know if they are available, to have an option for emailing larger lists with a host that knows how LMP works  :P

Keep us posted please...
Title: Server for big lists
Post by: KB on February 01, 2005, 07:53:00 am
Hi guys.

I have been in contact with the guy who hosts my listmail stuff. He provides a dedicated server that never falls over and gives me between 500 and 600k a day. Also helped me move my stuff from ev1
My server runs $300 a month
He's happy to talk with you - phoned me in South Africa to answer some of my questions. Has reverse dns so so you can whitelist at aol. If you want his details then mail me at bauer@mweb.co.za with your details and I'll pass them on.

Regards

Kevin
ps. He does have smaller systems so maybe they would be suitable for smaller lists
Title: Mailing list 100,000 and up
Post by: neil on February 08, 2005, 11:55:25 pm
I don't have a list anywhere close to those numbers, but I can't even send out to 5000 a day on H4P.

What I do have is my own server for website hosting, so I'll be moving from H4P ASAP.  What is the best way to move my subscribers?

I have a lot of space and bandwidth to offer for a GREAT fee, for LMP owners only.  If you will send me a PM and tell me what you are looking for, I will reply with my Special LMP Price.

Neil
Title: Throttle Code Use - when?
Post by: dane on February 14, 2005, 07:23:42 pm
Dean,

Is the throttling code still recommended int he newest version of LM to assist in the sending of lists larger than 10K?

Please provide as much detail as possible regarding when and if the code should be used.

Thanks,

Dane

Quote from: "DW"
Most servers do not time out in the middle of sending.  Your host may have a restriction on PHP script loading which would still be reached even with the upcoming background sending option.  A dedicated server should handle over 720,000 messages per day so even a shared host (configured correctly) with 10-20 users should be ok.
Title: Mailing list 100,000 and up
Post by: DW on February 14, 2005, 09:42:50 pm
Dane,

I don't recall why this was done but I do not recommend throttling code be implemented with SMTP.  I have not found it to be necessary at all.  If your list is over 25,000 or so I recommend that you ask your host to make sure the server is optimized so your messages do not slow down the server.  (I would also reassure them that you run a very professional list). You can already delay "internal PHP mail" if you use that for your list email.
Title: Mailing list 100,000 and up
Post by: DW on February 14, 2005, 09:43:55 pm
Neil,

The best way to move your subscribers - in fact, your entire ListMail database, is through PhpMyAdmin. Do an "Export" of all of your tables with the "Complete Inserts" option enabled. Save it to your local computer and re-import it into a new database on the new host via PhpMyAdmin or the command-line.

Regards
Title: Large List Set-up Config
Post by: dane on February 14, 2005, 09:55:54 pm
Dean,

Are there any special set-up features to run larger lists, 10K - 250K in size?

Thanks,

Dane

Quote from: "DW"
You should be able to achieve at least 750,000 messages per day but yes, your server can be bogged down.  Performance with very large lists is still being tweaked to prevent this.  I will likely have to rewrite bounce.cgi to be a complete Perl solution, as opposed to passing data to a URL, to offer better performance.  This might mean 2 config files.  config.php and config.pl.  I believe this is where a lot of the problems occur.  I am also considering that there may be MySQL processes that don't die or are increasingly using more memory.
Title: Mailing list 100,000 and up
Post by: DW on February 15, 2005, 12:54:30 am
I have experience with qMail (Plesk) and Exim (cPanel, Ensim) and have posted a few optimizations for these mailers.  Your server administrator could make use of what is mentioned in these posts.  You may want to forward the information to them.

There is one further optimization that may not be in effect on your installation: Try adding an index on the id field of the lm_sendq table with PhpMyAdmin.

Regards
Title: 405,000+
Post by: andrew on August 12, 2005, 08:29:31 pm
Great topic.
Good input from all.

Our primary list topped 405K this week. Big. Very big.
I also just moved to a new server w/Cpanel- so great timing DW with
the LMP import fix.

It seems like everyone's working pretty hard to manage the bigger lists, between throttling, primary key fixes (maybe from another post), running on two machines...

We're not very technical, so we needed a clean, simple solution. Knowing that we've run successfulyl with lists in the 30-40K range, I decided to take our master list and split it by 50K each, done alphabetically with Mailloop's listp processor, and catalogued alphabetically to keep track of where we were within the database.

So far so good. Looking at it now we sitting at 2.11 emails per second on the first list. I plan to queue them up seperately and run each list individually.

Just wanted to share what we've done, thought I would sure be interested in any easy to implement solutions to tweak, modify or seriously expedite things to reach the 10-12K per-hour mark.

Thanks for all your input.

Andrew
Title: Mailing list 100,000 and up
Post by: Stefaans on August 13, 2006, 01:41:10 pm
Quote from: "DW"
There is one further optimization that may not be in effect on your installation: Try adding an index on the id field of the lm_sendq table with PhpMyAdmin.

My apologies for resurrecting an old thread, but I need clarification.

I am using ListMailPro version 1.87. Looking at the lm_senq table, it seems like there is already a primary key for the id column set up. So, do I take it to be optimized already?
Title: Mailing list 100,000 and up
Post by: DW on August 13, 2006, 03:13:36 pm
Stefaans,

Yes, that's right. :)

Regards