[Dev] Split Testing

[erik]

A feature that I would love to see is the ability to "a/b split test" mailings.  Basically, send one version of an email to part of the list (say 50%), and the other part to the other half (or have 3 or 4 messages and be able to assign "weights" - 10 = 10% etc...).

This way we could track which letter produces a great response...

Additionally, it would be great to be able to send a "test message" out to only part of the list - just another way to test responsivness to the offer...

Thanks!
Erik

PS  The recipients of each message should be chosen at random from the DB, this way tests would provide more accurate results. (people who signup at different times may respond differently to various offers, and this would skew the resluts if the sample were not random...

[DW]

Thank you for your suggestion.  This is a tricky one, but it will be considered.

Regards

[DW]

> is it possible to send a mailing to, say, just the back half of the list?

That's twice that this has come up now.  I have an idea :idea: .  What about a small custom script to insert a "1" value into a specific custom field for half of your list members.  Once the script is run, you can use the new "User Selection" feature to search for the "1" value and send to half of your list.  Later, you can search for "not equal to 1" to email the rest of your list.

The script would be easy.  All I need to know is what percentage of what list you would like to flag and what custom field to flag them with.

Regards

[erik]

> is it possible to send a mailing to, say, just the back half of the list?

That's twice that this has come up now.  I have an idea :idea: .  What about a small custom script to insert a "1" value into a specific custom field for half of your list members.  Once the script is run, you can use the new "User Selection" feature to search for the "1" value and send to half of your list.  Later, you can search for "not equal to 1" to email the rest of your list.

The script would be easy.  All I need to know is what percentage of what list you would like to flag and what custom field to flag them with.

Regards

How about maxing those form fields - then we can choose our own sample size.. .Set the default to 50%.

[DW]

Sorry, Erik, I'm not exactly sure what you mean by "maxing the form fields".  I could provide a simple script that lets you choose the percentage, sure.

Regards

[erik]

Sorry, Erik, I'm not exactly sure what you mean by "maxing the form fields".  I could provide a simple script that lets you choose the percentage, sure.

Regards

typo - maxing = making, sorry

[DW]

By "making them form fields" are you suggesting I try to implement the percentage-sending feature into User Selection?

I'm not sure there's an easy way to go about this.  I think we should save which part of the list was sent it to so we can send a message later to those who didn't receive it.  To do this without using a new data table we simply set a value in a custom field for a user for those last mailed.  Using a single custom field, or a new field, could limit us to one set of percentages at a time and risk emailing the wrong users.

To save sets of percentages would require a new data table, which I'm not opposed to implementing, either.  This feature could be a bit tricky but would end up being very useful.

I'll consider this for future updates.  For the meantime, would you like a simple script to flag any percentage members based on certain fields (ie. date added)?

Regards

[BGSWebDesign]

Hey DW,

I think we should save which part of the list was sent it to so we can send a message later

THAT is exactly what a message History would do, and I believe you would confuse how these two are separate... read below....

Nah, saving part of the list can pretty much be done already, that won't do enough for us.... we need split testing capabilities too... look at the suggestions...  Split testing would be incredibly powerful!  I can split test the clicks coming into my website, but I can't (YET) split test the Emails going out, which is what I would like to do!

For example, list of 10,000.  I'd like to send 4 different emails to the list - split them up, 1, 2, 3, 4, then start over again.... a simple send page would let me enter MORE than 1 Email to send (I don't need HTML so that's no big deal, but others might)....  this page let's me enter up to 6 messages which are DIFFERENT (maybe one word different, maybe sentences/paragraphs).

Then, it also let's me customize a LINK that is EMBEDDED in that mesage, by adding a NUMERIC (incremented identifier) onto it, so that when the link is displayed it appends either 1, 2, 3, or 4 to the link - that way I can setup my Tracking system as it is now - with subcampaigns.

We enter the link (beginning part), and you tack on the identifier every time the new message is queued....  in the end like this:
MSG #1 - includes different text.... AND link with #1 appended.
MSG #2 - includes different text.... AND link with #2 appended.
MSG #3 - includes different text.... AND link with #3 appended.
MSG #4 - includes different text.... AND link with #4 appended.

to those who didn't receive it.  To do this without using a new data table we simply set a value in a custom field for a user for those last mailed.  Using a single custom field, or a new field, could limit us to one set of percentages at a time and risk emailing the wrong users.

Honestly I don't see where this new data table comes in, you wouldn't even need it for split testing, just another Send Email page that allows entry of multiple messages (1,2,3,4,5,6), and ONE link that has the identifier tacked on the end.... when I hit send, the QUEUEing does all the work and loads all the messages up in the queue appropriately.

I'll consider this for future updates.  For the meantime, would you like a simple script to flag any percentage members based on certain fields

That's a possibility too...  date added, but still it's not enough... I have to manually setup User Selection every time this changes, that's a pain...  

Better option is to do as indicated above... and as you stated, ok, I see where sending to a part of your list would be interesting, but you can do that now with user selection, just guess at an alphabetic cutoff for the Email address and use that?  Search >= in Email Address would do it...

The tracking is MORE useful, and the split testing to me.... just sending to a part of the list can already be done with User Selection.... what do you think DW?   Will this (Split Testing) be implemented soon, or as an 'extra feature' as we talked about you offering?

[nednorris]

Split testing would be a really cool addition.

The way Webshaman is proposing sounds like it could work.

If the user enters two different e-mails it would just split the list 50/50. If they enter three e-mails then it would be 33.33/33.33/33.33 and so on.

The earlier proposal of allocating a field that has a number in could work too. Even if it was just 1 and 2 to would make a split test easy to do using the Selection feature. Maybe even extend it to 1,2,3 and 4. Then people could send up to four mailings out.

Do keep us posted on this as it could be very exciting.

Keep up the good work

Ned

[DW]

Honestly I don't see where this new data table comes in, you wouldn't even need it for split testing, just another Send Email page that allows entry of multiple messages (1,2,3,4,5,6), and ONE link that has the identifier tacked on the end.... when I hit send, the QUEUEing does all the work and loads all the messages up in the queue appropriately.

The reason I envision a new data table is that I figured we would want to "save" the groups of users who were sent each message.  his way we could later on send further messages that could rely on split changes in the previous ones.  Doing this could also lead the way to a history of which messages have been sent to which users.

I get now how split testing could be useful for you even without storing which users received which message, with a modification to link tracking.  I think this is an important part of it but my idea should be implemented too.

I'm getting more interested and experienced with tracking ROI, conversions, etc. and agree these features would be very powerful and an excellent selling point for ListMail.  I am confident that I will be able to implement them soon.

Regards

[BGSWebDesign]

Hi DW,

The reason I envision a new data table is that I figured we would want to "save" the groups of users who were sent each message. his way we could later on send further messages that could rely on split changes in the previous ones. Doing this could also lead the way to a history of which messages have been sent to which users.

Yeah, I see, that's an interesting idea     - I'm not sure how necessary it will be, as I don't envision sending split tested emails to the same group with a 'different set' of split test messages.   I suppose there may be some value in it...

If it helps you setup the Message Sent history, then go for it, but really, it may be more information then I need.... for example, I can't imagine needing to know EXACTLY which message was sent to which person, when it's more important that I know that 1,000 messages went to GROUP1, and 200 CLICKED, and then eventually, WHO BOUGHT, so, you can see the level of detail needed.

I would like to see something like this AFTER I do a split test mailing:
-----------------------------------------------------------------------------

Code: [Select]


GROUP    SENT    CLICKED   PURCHASED   CTR     CPR
--------------------------------------------------------------
GROUP1   1000       200     5         .20      .005
GROUP2   1000       150     3         .15      .003
GROUP3...  etc, etc, etc...


So, DW what you need to figure out is HOW to link up (for tracking purposes), the clicks through, using your current Link Tracking link - which would have to be ADDED TO - so somehow support the additional SPLIT TESTING information, for example, each Tracking link would have to have additional data appended to it to tell you which GROUP they came from in the split test, and also, you would have to have a way to provide an additional LINK tracked counter (on the Purchase page, something like an invisible gif or something) that could provide information that this person made the purchase...   I can then get detailed stats on how  purchases were made...  

That would be perfect, as I could tell which changes in the message led to more clicks - and possibly - more purchases down the road, though most likely it would be the Landing Page that they are sent to that is going to lead to more purchases....

Can you see where this is going, SO, if you combine the methods above for detailed Split Test Mailings with the methods needed for Split Test Link Tracking, which we have not fully discussed yet, but that method would allow us to setup Split Test Links, in other words, I enter a message code, reference code, title and a LIST of LINKS to send the user to at random (using cookies you make sure that once the user visits the link they ALWAYS get the same link, or you can setup an option allowing the ADMIN to not use the cookie and always go to a random link) - and then you setup LMP so that it can track this link specifically, ALSO based on which Split Test mailing group it came from  :wink:

That would be perfect, and I'm sure would provide you with a very great product feature/benefit that others cannot provide (that I know of).

The final table might look something like:
-----------------------------------------------

Code: [Select]


GROUP  STLNK  SENT    CLICKED   PURCHASED CTR     CPR     PCR
----------------------------------------------------------------------------
GROUP1         1000     187         5     .187   .0020      
         1      333      50         2     .15    .0006   .040
         2      333      75         2     .22    .0006   .026
         3      334      62         1     .18    .0003   .016

GROUP2...  etc, etc, etc...


This provides the full details, since I get, CTR - Click Thru rate (clicks/#sent), CPR (purchases/sent), and the most important PCR (purchases/clicks).    That way, the user can see exactly which mailed message (of the Split Test group) led to the most clicks (would need to see all the groups in the details above to determine this), and which Split Link within each group (STLNK 2 in this example) led to the most clicks, and which led to the most purchases (STEST 1,2 in the example) and which led to the most purchases based on the number of clicks (STEST 1 in the example).

Hope that helps and gives you enough information to complete this.... when can we expect it, any ideas, how's it coming along?

[BGSWebDesign]

Hi,

A quick followup to the above message... please read my previous message in this thread, and combine it with the ability to do Subject Line testing when sending emails (info below)...

--------------

Please, please also provide a way that the SUBJECT of the Email can be entered as part of the split tested message. That way if we do NOT want to change the BODY of the Email, we can just type in new Subject lines in the block of Subject/Body provided for Split Testing and get an Email that will have the SAME body, but different subject lines!

Right now this is a big pain, as I have to setup Link Tracking, and try to send to small groups on my list, to determine the best Subject Line...

Subject Line testing is an integral part of Split Testing.

[jonisolis]

Hi, I just want to say that I would love to see split testing added to ListMail too and soon. But I would also like to see a very good help page on how it works and how to work it also. --Joni Solis

[DW]

Brett,

I think you have laid it out completely here.  Is 6 the most messages you would ever need?  It would be easiest to split uniformly, ie: 50% or 3 sets of 33%, as opposed to 20% vs 80%.  Is there a great value to the latter?  Anything's possible.

In regards to what you said about a link being automatically appended with a "1" or "2" to allow for different links, could you not simply append this value yourself when creating the messages?  I am thinking to provide split testing within the Link Tracking feature itself (post is here), so this seems to tie-in with what you want to see in the report.  Links could be split separately from mailings, but we would need to ensure that EACH group is cycled separately.

Regarding quick saving and sending.  The browser *could* store the up to 6 messages in JavaScript memory (so you might send without saving..)  I like the idea of toggling between the various messages (1|2|3|4) but don't see 6 messages being saved in a single save/load..  If I were to implement quick save GROUPS and that message "TextID" you've talked about we might be able to give split tested mailings the same ID and make a 'group' for them under quick save...  You could then 'quick save' any message you want to add to the split test and see a listing of all messages in the quick save/load menubox... not sure if this is the best way to go about this as I haven't given it a lot of thought yet.

Please, please also provide a way that the SUBJECT of the Email can be entered as part of the split tested message. That way if we do NOT want to change the BODY of the Email, we can just type in new Subject lines in the block of Subject/Body provided for Split Testing and get an Email that will have the SAME body, but different subject lines!

Would it be ok if you could simply load the other message and change the subject?  Alternately, I might be able to provide a JavaScript link that will insert the text of any other message being split with instantly.  I'll be giving this some more thought, too.

I can very likely improve link tracking to include setting of a cookie so that you can track future goals, such as purchase.  I will have to consider further how I might provide the style of report you wish for.  This will be a pretty major overhaul of Sent Messages and the message sending process in general.

Great idea and great input as usual, Brett.  This would be one heck of a feature.  I hope to be able to make this one a reality for you shortly after I get the new site up.

Regards

[BGSWebDesign]

Hi DW,

Is 6 the most messages you would ever need?

6 is plenty enough for me.

It would be easiest to split uniformly, ie: 50% or 3 sets of 33%, as opposed to 20% vs 80%. Is there a great value to the latter? Anything's possible

EVEN SPLIT is fine for me, don't overthink it...  there is NO value that I see in splitting unevenly - the whole PURPOSE of the Split Test is to determine which item being tested results in the most clicks - the ONLY way to do that is with EVENLY SPLIT BATCHES.

In regards to what you said about a link being automatically appended with a "1" or "2" to allow for different links, could you not simply append this value yourself when creating the messages?

No, YOU need to automatically append the value - since this is HOW YOU WOULD Track the Split Tested click - there are 2 ways to go with this:

1.) YOU provide us a way to enter a Link for Tracking (that is from LMP's Track Link feature - there are many Split Testers on the market now, try a few out to see how they work) that is a Split Test Link - when you see that link in the outgoing messages that you queue YOU will need to append the correct value 1, 2,3,4,5 or 6 depending on which Split Test group it's going out to, got it?

2.) YOU provide us a way to enter the beginning part of a link (this link is one that uses our own custom link tracking that way we can either use our own tracking OR your provided tracking as defined in 1 above) which YOU need to append to the correct value 1-6 depending on which Split Test group it's in.

I like the idea of toggling between the various messages (1|2|3|4) but don't see 6 messages being saved in a single save/load..

Don't over-think it, I do NOT need (nor does anyone else) a way to Quick Load 6 entire messages - ok, on second thought - saving those as a GROUP would be helpful in that we could go back and look at them, but it's not THAT important, as anyone doing this will be meticulous about saving their 6 different messages on their own local machine... and then using those saved messages for comparison - the only place this is helpful is for your purposes - IF YOU want to have a way to let the user click a link to see which message was assigned to which link (1-6)... that could be interesting...

'group' for them under quick save... You could then 'quick save' any message you want to add to the split test and see a listing of all messages in the quick save/load menubox...

That could be highly cool - you provide us a Split Test GROUP Message page where we enter and save GROUPS of messages (1 through 6 each with different text in them - and the custom Split Test link - which you append the correct value on to when the message are queued) - the GROUP would also have to provide a message SUBJECT that can be separate or different for each of the messages - as someone might just be split testing a subject line...  Then when you are on the SEND Split Test page - you could LOAD a Split Test GROUP Message using a pulldown to select a NAMED Message Group that was saved previously - do NOT get any more sophisticated than that - it will take you too long to program if you do...

Would it be ok if you could simply load the other message and change the subject?

NO, well, it's cumbersome to do it that way - the better way is to ALLOW us to save Group Messages (and Subjects) as a group (with a name) and then load them by name using a pulldown.

I can very likely improve link tracking to include setting of a cookie so that you can track future goals, such as purchase.

Have a look at AdTrakz: http://adtrackz.com/ I've already suggested previously you work with them... the method used for that tracker is excellent - either a cookie, OR an invisible .gif image on the checkout page - not sure off-hand which it is....  

I will have to consider further how I might provide the style of report you wish for. This will be a pretty major overhaul of Sent Messages and the message sending process in general

The style of report I laid out for you above - do it just like that... that is important - as a report that does NOT contain the right information will prove to be worthless....  if you start out with what I've shown above, then provide us a sample to lookat I could give input... the basic one I describe above is a good start - from there I could see where THIS COULD GO INTO incredible possibilities like tracking how many 'Interested' people (those that had previously clicked on a link) were more likely to purchase, etc, etc...  the possibilities here are endless and you could provide plug-ins and additional options here for more money in the future?

Good luck, let me know how it's going - I truly believe if you can complete this it will put LMP way above other Mailing List managers... if you do it the right way...