SMTP with Amazon SES Problem

[franklinfick]

I can not get the SMTP Setting with Amazon SES to work

I am getting this error message on the log
[23-Nov-2014 01:39:34 UTC] PHP Warning:  fsockopen(): unable to connect to      email-smtp.us-east-1.amazonaws.com:465 (php_network_getaddresses: getaddrinfo failed: Name or service not known) in /home/shenlong/public_html/connect/editconf.php on line 1894

and this on the test mail settings:
[SMTP Server]
Is enabled?  Yes
Connecting to SMTP server.. Connected!
Verifying username and password... Failed!
The SMTP server is not configured correctly. Check your settings.


I have the
host
port
Auth - login ID and Password

all filled out correctly

and I have tried every port that Amazon has in their help file on connecting to the SMTP server

///////////////////////
STARTTLS

STARTTLS is a means of upgrading an unencrypted connection to an encrypted connection. There are versions of STARTTLS for a variety of protocols; the SMTP version is defined in RFC 3207.

To set up a STARTTLS connection, the SMTP client connects to the Amazon SES SMTP endpoint on port 25, 587, or 2587, issues an EHLO command, and waits for the server to announce that it supports the STARTTLS SMTP extension. The client then issues the STARTTLS command, initiating TLS negotiation. When negotiation is complete, the client issues an EHLO command over the new encrypted connection, and the SMTP session proceeds normally.
TLS Wrapper

TLS Wrapper (also known as SMTPS or the Handshake Protocol) is a means of initiating an encrypted connection without first establishing an unencrypted connection. With TLS Wrapper, the Amazon SES SMTP endpoint does not perform TLS negotiation: it is the client's responsibility to connect to the endpoint using TLS, and to continue using TLS for the entire conversation. TLS Wrapper is an older protocol, but many clients still support it.

To set up a TLS Wrapper connection, the SMTP client connects to the Amazon SES SMTP endpoint on port 465 or 2465. The server presents its certificate, the client issues an EHLO command, and the SMTP session proceeds normally.
///////////////////////



I previously had my listmap pro installation set up to run STMP through a VPS that I had

I no longer have the VPS

and have decided to try Amazon SES

but can't get to work


Here is my SMTP log File that was written

> EHLO localhost

250-email-smtp.amazonaws.com

250-8BITMIME

250-SIZE 10485760

250-STARTTLS

250-AUTH PLAIN LOGIN

250 Ok

LM: AUTH LOGIN Detected

> AUTH LOGIN

530 Must issue a STARTTLS command first


thanks for any help in this matter

Franklin

[DW]

Hi Franklin,

It appears there are two separate issues in your post.

1. "php_network_getaddresses: getaddrinfo failed: Name or service not known" suggests a DNS error. The PHP script is not able to do a hostname lookup to determine the IP to connect to. You might talk to your host about fixing that or try connecting by IP after looking up the host yourself.

2. LMP doesn't currently have any support for STARTTLS connections, however I will consider it for the future. Until then, please use an unencrypted service if available.

Regards,

[franklinfick]

Hi Franklin,

It appears there are two separate issues in your post.

1. "php_network_getaddresses: getaddrinfo failed: Name or service not known" suggests a DNS error. The PHP script is not able to do a hostname lookup to determine the IP to connect to. You might talk to your host about fixing that or try connecting by IP after looking up the host yourself.

2. LMP doesn't currently have any support for STARTTLS connections, however I will consider it for the future. Until then, please use an unencrypted service if available.

Regards,

for 1-  i think that was an error i got when trying other ports to connect through
and could not connect


2- then amazon ses is not an option?
i don't think they support unencrypted connections
(at least i can not find anything in their docs about it)

please consider in the future to make LMP comparable with Amazon SES
that would be wonderful
thanks


Franklin

[DW]

Hey Franklin,

I have heard of other clients using LMP with SES so I wonder if the encryption requirement is new.

We may be in luck pursuant to this and this.

Try using the hostname "tls://email-smtp.us-east-1.amazonaws.com" on port 465.

Regards

[franklinfick]

thanks

still getting an error message when trying to connect to: tls://email-smtp.us-east-1.amazonaws.com

with either port 465 and 2465


-----------------------
[SMTP Server]
Is enabled?  Yes
Connecting to SMTP server..
Warning: fsockopen(): unable to connect to tls://email-smtp.us-east-1.amazonaws.com:465 (Connection refused) in /home/shenlong/public_html/connect/editconf.php on line 1894
Could not find server!

The SMTP server is not configured correctly. Check your settings.
-----------------------------------


Franklin

[DW]

Hi Franklin,

On two separate systems, the following PHP test script works:

Code: [Select]

<?php
error_reporting(E_ALL);
$sock=fsockopen('tls://email-smtp.us-east-1.amazonaws.com',465);
if($sock) echo "Socket established\n"; else exit("Socket could not be established\n");
echo fgets($sock,1024);
echo "> EHLO localhost\n";
fputs($sock,"EHLO localhost\r\n");
$line='';
while(substr($line,3,1)!==' '){
        $line=fgets($sock,1024);
        echo $line;
}
The result when run from browser or commandline is e.g.:

Code: [Select]

Socket established
220 email-smtp.amazonaws.com ESMTP SimpleEmailService-908880847 2G85c3kZxdN3aiX48I3
> EHLO localhost
250-email-smtp.amazonaws.com
250-8BITMIME
250-SIZE 10485760
250-AUTH PLAIN LOGIN
250 Ok
For me, everything seems to be working as expected. The SMTP conversation should be able to continue as normal. LMP does it the same way.

If this test doesn't return similar results for you you will need to speak with your host as they may have limited or restricted such PHP socket connections or be experiencing a technical issue.

Regards

[franklinfick]

i tested the php file on my server

here is what it returned:

------------------------
Warning: fsockopen(): unable to connect to tls://email-smtp.us-east-1.amazonaws.com:465 (Connection refused) in /home/shenlong/public_html/connect/testphp.php on line 3
Socket could not be established
---------------------------------------------------------------


is this caused because the port 465 is not open
or something else

I need to know what to contact my host about...
and I have no clue what could be causing this


thanks
Franklin

[franklinfick]

on the phone with my hosting support they confirmed that port 465 is open

and now I am getting 2 errors when trying to connect throguh LMP

--------------------
[SMTP Server]
Is enabled?  Yes
Connecting to SMTP server..
Warning: fsockopen(): php_network_getaddresses: getaddrinfo failed: Name or service not known in /home/shenlong/public_html/connect/editconf.php on line 1894

Warning: fsockopen(): unable to connect to tls://email-smtp.us-east-1.amazonaws:465 (php_network_getaddresses: getaddrinfo failed: Name or service not known) in /home/shenlong/public_html/connect/editconf.php on line 1894
Could not find server!

The SMTP server is not configured correctly. Check your settings.

______________________

[DW]

on the phone with my hosting support they confirmed that port 465 is open

and now I am getting 2 errors when trying to connect throguh LMP

Please mention to your host that we need to connect TO email-smtp.us-east-1.amazonaws.com on port 465 using PHP fsockopen() with TLS encryption enabled and it's not working due to DNS or other. You can send them a link to this thread. I recommend starting a support ticket if they offer that rather than phoning as it may take them some time to diagnose and fix.

Required of the host is:

• The unrestricted ability to connect via PHP sockets
• PHP support for SSL/TLS connections
• A working DNS system (Your host seems to be giving DNS errors)

Regards

[franklinfick]

thanks for your help Dean

seems I need to change hosts

here is the jist of the conversation I just had with Godaddy representative:
--------------------------
Amy -
    I apologize for any confusion and incorrect information you had received, however, within our shared hosting environments we intentionally restrict SMTP traffic. You will not be able to use a 3rd party mail server. This is done to prevent our systems from being used to send spam, or otherwise malicious messages.

franklin -
    so you are saying it is immpossible to have a web aplication that connects with amazon ses with your shared hosting plan
    franklin -
    ?

Amy -
    That is correct
   
franklin -
    and it would also be impossible to connect with any other smtp provider as well
   
franklin -
    like sendgrid
   
franklin -
    or madrill
   
Amy -
    Correct, we do provide a relay server in order to use, that would be the only way to connect through SMTP with our shared hosting.



-------------------------------------

[franklinfick]

on phone support with them...

the only option they could offer me was to connect without any encryption on port 25


so it looks like my host is not able to connect to amazon ses


any recommendations on affordable hosting that will work with tls:// on port 465?


thanks

Franklin

[DW]

Hi Franklin,

There are a lot of hosts out there, many more permissive than GoDaddy. You'll have to ask to be sure if they support PHP connections to external servers like Amazon SES.

I just updated our Email Hosting page. Perhaps you will find something there.

Regards

[BGSWebDesign]

Hi Franklin,

I have heard of other clients using LMP with SES so I wonder if the encryption requirement is new.

No, I'm not using LMP with Amazon SES because, as I mentioned in that post, LMP must support a method to retrieve SNS messages from Amazon to retrieve Complaints and Bounced emails, and immediately unsubscribe those users from the database.  At this time, LMP does not provide an option for that, so it would be impossible to use LMP with SES because it could end up causing problems with Amazon SES and get you banned from using that service. 

I'm hoping that Dean (the author of LMP), can soon provide us with a method to setup/configure Amazon SES with Amazon SNS to track complaints/bounces of messages, as well as the allowed message send queue size in real-time.  I've discussed this a few times and without those capabilities, it is simply not safe to use LMP with Amazon SES, unless you are willing to possibly be banned from Amazon SES, especially if you are mailing large lists, but possibly with lists of any size.  The methods that Dean has described for using LMP with SES are those that are used for transactional emails only, and not for sending broadcasts to large lists.

There are other services I use, contact me directly if you wish to discuss them (my contact page is in my signature - below).

[DW]

Hi Brett,

I'm hoping that Dean (the author of LMP), can soon provide us with a method to setup/configure Amazon SES with Amazon SNS to track complaints/bounces of messages

Try this:

Code: [Select]

<?php
// LMP Amazon SES SNS Complaint & Bounce Processing
// based on https://github.com/npflood/AWS-SNS-HTTP-PHP-ENDPOINT/blob/master/receiver.php
//
// Name this file ses_bounce.php or similar, put it in your LMP folder, and (after configuring below) set its URL as an SNS HTTP endpoint (also configure SES to publish to the SNS topic)
// Change $allowedTopic below to your full SNS topic
// Change $sourceDomain below to suit your region
// see bottom for LMP

//////
//// CONFIGURATION
//////
//For Debugging.
$logToFile = false;
//Should you need to check that your messages are coming from the correct topicArn
$restrictByTopic = true;
$allowedTopic = "arn:aws:sns:us-west-2:100885381234:SES_Complaints_and_Bounces";
//For security you can (should) validate the certificate, this does add an additional time demand on the system.
//NOTE: This also checks the origin of the certificate to ensure messages are signed by the AWS SNS SERVICE.
//Since the allowed topicArn is part of the validation data, this ensures that your request originated from
//the service, not somewhere else, and is from the topic you think it is, not something spoofed.
$verifyCertificate = true;
$sourceDomain = "sns.us-west-2.amazonaws.com";
 
//////
//// OPERATION
//////
$signatureValid = false;
$safeToProcess = true; //Are Security Criteria Set Above Met? Changed programmatically to false on any security failure.
if($logToFile){
////LOG TO FILE:
$dateString = date("Ymdhis");
$dateString = $dateString."_r.txt";
$myFile = $dateString;
$fh = fopen($myFile, 'w') or die("Log File Cannot Be Opened.");
}
//Get the raw post data from the request. This is the best-practice method as it does not rely on special php.ini directives
//like $HTTP_RAW_POST_DATA. Amazon SNS sends a JSON object as part of the raw post body.
$json = json_decode(file_get_contents("php://input"));
//Check for Restrict By Topic
if($restrictByTopic){
if($allowedTopic != $json->TopicArn){
$safeToProcess = false;
if($logToFile){
fwrite($fh, "ERROR: Allowed Topic ARN: ".$allowedTopic." DOES NOT MATCH Calling Topic ARN: ". $json->TopicArn . "\n");
}
}
}
//Check for Verify Certificate
if($verifyCertificate){
//Check For Certificate Source
$domain = getDomainFromUrl($json->SigningCertURL);
if($domain != $sourceDomain){
$safeToProcess = false;
if($logToFile){
fwrite($fh, "Key domain: " . $domain . " is not equal to allowed source domain:" .$sourceDomain. "\n");
}
}



//Build Up The String That Was Originally Encoded With The AWS Key So You Can Validate It Against Its Signature.
if($json->Type == "SubscriptionConfirmation"){
$validationString = "";
$validationString .= "Message\n";
$validationString .= $json->Message . "\n";
$validationString .= "MessageId\n";
$validationString .= $json->MessageId . "\n";
$validationString .= "SubscribeURL\n";
$validationString .= $json->SubscribeURL . "\n";
$validationString .= "Timestamp\n";
$validationString .= $json->Timestamp . "\n";
$validationString .= "Token\n";
$validationString .= $json->Token . "\n";
$validationString .= "TopicArn\n";
$validationString .= $json->TopicArn . "\n";
$validationString .= "Type\n";
$validationString .= $json->Type . "\n";
}else{
$validationString = "";
$validationString .= "Message\n";
$validationString .= $json->Message . "\n";
$validationString .= "MessageId\n";
$validationString .= $json->MessageId . "\n";
if($json->Subject != ""){
$validationString .= "Subject\n";
$validationString .= $json->Subject . "\n";
}
$validationString .= "Timestamp\n";
$validationString .= $json->Timestamp . "\n";
$validationString .= "TopicArn\n";
$validationString .= $json->TopicArn . "\n";
$validationString .= "Type\n";
$validationString .= $json->Type . "\n";
}
if($logToFile){
fwrite($fh, "Data Validation String:");
fwrite($fh, $validationString);
}

$signatureValid = validateCertificate($json->SigningCertURL, $json->Signature, $validationString);

if(!$signatureValid){
$safeToProcess = false;
if($logToFile){
fwrite($fh, "Data and Signature Do No Match Certificate or Certificate Error.\n");
}
}else{
if($logToFile){
fwrite($fh, "Data Validated Against Certificate.\n");
}
}
}
if($safeToProcess){
//Handle A Subscription Request Programmatically
if($json->Type == "SubscriptionConfirmation"){
//RESPOND TO SUBSCRIPTION NOTIFICATION BY CALLING THE URL

if($logToFile){
fwrite($fh, $json->SubscribeURL);
}

$curl_handle=curl_init();
curl_setopt($curl_handle,CURLOPT_URL,$json->SubscribeURL);
curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,2);
curl_exec($curl_handle);
curl_close($curl_handle);
}


//Handle a Notification Programmatically
if($json->Type == "Notification"){
//Do what you want with the data here.
//fwrite($fh, $json->Subject);
//fwrite($fh, $json->Message);
}
}
//Clean Up For Debugging.
if($logToFile){
ob_start();
print_r( $json );
$output = ob_get_clean();
fwrite($fh, $output);
////WRITE LOG
fclose($fh);
}
//A Function that takes the key file, signature, and signed data and tells us if it all matches.
function validateCertificate($keyFileURL, $signatureString, $data){

$signature = base64_decode($signatureString);


// fetch certificate from file and ready it
$fp = fopen($keyFileURL, "r");
$cert = fread($fp, 8192);
fclose($fp);

$pubkeyid = openssl_get_publickey($cert);

$ok = openssl_verify($data, $signature, $pubkeyid, OPENSSL_ALGO_SHA1);


if ($ok == 1) {
    return true;
} elseif ($ok == 0) {
    return false;
    
} else {
    return false;
}
}
//A Function that takes a URL String and returns the domain portion only
function getDomainFromUrl($urlString){
$domain = "";
$urlArray = parse_url($urlString);

if($urlArray == false){
$domain = "ERROR";
}else{
$domain = $urlArray['host'];
}

return $domain;
}

// LMP - Process Bounce or Complaint
// per http://docs.aws.amazon.com/ses/latest/DeveloperGuide/notification-examples.html
include('./config.php');
include('./admin.php');
if($json->notificationType=='Bounce'){
if($json->bounce->bounceType=='Permanent'){
foreach($json->bounce->bouncedRecipients as $r){
// set user to bounced
mysql_query("UPDATE $utable SET cnf = '3' WHERE email LIKE '".addslashes($r->emailAddress)."';") or die(mysql_error());
}
}
} elseif($json->notificationType=='Complaint'){
foreach($json->complaint->complainedRecipients as $r){
// set user to removed
mysql_query("UPDATE $utable SET cnf = '2' WHERE email LIKE '".addslashes($r->emailAddress)."';") or die(mysql_error());
}
}

?>
I'll have to consider the queue size issue further.

Regards

[BGSWebDesign]

Hi Dean,

Thanks!  This looks like just what I need.  I'm not that concerned with Quota (Queue size) now as mine is set really high on Amazon, but others will need that in place, so - some type of throttling to slow down sending, or leave it wide-open and somehow cut the user off after they hit their Amazon SES Quota for the day?

This is just great, I will play around with that code, and turn on the logfile to see what I get out of it.  It shouldn't be too hard to get this going, and that would really change my use of LMP making it
much more usable for sending via SES.  I still might like some type of integration step-by-step guide, as to what else I change, is it just the SMTP information pointing to Amazon SES, and then read below
about turning off, or changing how bounce processing is currently setup...

A few questions about the code. 

1.) At the top it says:

Code: [Select]

$allowedTopic = "arn:aws:sns:us-west-2:100885381234:SES_Complaints_and_Bounces";
//For security you can (should) validate the certificate, this does add an additional time demand on the system.
//NOTE: This also checks the origin of the certificate to ensure messages are signed by the AWS SNS SERVICE.
//Since the allowed topicArn is part of the validation data, this ensures that your request originated from
//the service, not somewhere else, and is from the topic you think it is, not something spoofed.
$verifyCertificate = true;
$sourceDomain = "sns.us-west-2.amazonaws.com";
I think something has to change here, as the $allowedTopic includes a sourceDomain in it?  In other words, my source domain is really USA East, not West, so I either have to change this in both places above, or maybe you move sourceDomain up top and the allowedTopic variable is setup to include the proper sourceDomain in it?  Also, I can look it up, but maybe you have the string for USA East handy?

2.) I'm not sure how this fits in with current/standard bounce processing?  What would I do with LMP's current bounce processing if I setup to send via Amazon SES, just turn it off, and what is the best way to turn off the other Bounce Processing that I currently have setup? 

Other then those two things, this is perfect and should allow me to send via SES - using LMP, something I've been wanting to do for some time now.  I'll probably setup an alternate LMP installation to test this on.

Could you please provide us with an ETA on the next version of LMP?  If you're able to crank out code like this in a few minutes, I suspect you must be ready with a new version of LMP???