Sunday, June 29, 2008

Emailing and Texting with Scala & lift

This blog post will walk you through the process of configuring and using lift's Mailer, a utility for sending email from your web application. By the end of it, your webapp will be able to

  • Send email account activations on user signups
  • Support forgotten password resets
  • Send plain text emails
  • Broadcast emails to users

Getting Started: Setting Up a New lift Project

As a prerequisite, you must have JDK 1.5 or greater along with Maven2 installed on your computer. lift uses Maven to manage dependencies (like scala and lift itself), generate new projects, run and deploy your web app, and, in general, make your life easier. To generate your lift project, you can use one of lift's "archetypes." In Maven, archetypes are templates for new projects. For more information on lift and Scala archetypes, you can consult the relevant page hosted on our wonderful wiki. Feel free to contribute!

The archetype we will be using is the "basic" archetype. This sets up a User model, login page, a sign up page, an edit profile page, a logout link, various other user management functionalities, a generic layout template built with Blueprint CSS , and a Sitemap that pulls it all together.

mvn archetype:create -U \
-DarchetypeGroupId=net.liftweb \
-DarchetypeArtifactId=lift-archetype-basic \
-DarchetypeVersion=0.9 \
-DremoteRepositories=http://scala-tools.org/repo-releases \
-DgroupId=org.example.mailer -DartifactId=mailer-example

The "groupId," in this case "org.example.mailer" will be the package that you intend to keep your lift code in. The "artifactId," in this case "mailer-example" is the name of the project to be created. There you go. Change directories into the project and mvn jetty:run to start an instance of Jetty and deploy your app to it, then head over to http://localhost:8080 and explore what you have created.

At this point the app is configured to use the SMTP server hosted at localhost, so unless you happen to be sitting on an SMTP server that does not require authorization, the mailing wont work.

After you've played around with the app a bit (sign up with a fake email address, as you'll need to sign up with a second real one later for testing), return back to the console and cntrl + c to shut down the jetty server. If you are just discovering lift and want to get the most out of this tutorial, I suggest that you read through the Hello Darwin. It is a superb overview of lift's features and how to utilize them.

Getting Started: Your Choice of Editor

You can develop lift with a number of editors. In my experience with the lift community, I've encountered people using Eclipse, NetBeans(nightly), JEdit, Emacs, VIm, and some stranger ones that I have forgotten. In fact, if you go into your Scala home directory and run bin/sbaz install scala-tool-support and then look in the "misc" directory. You will find "scala-tool-support/" which will contain plugins for developing Scala in a2ps, bluefish, context, emacs, enscript, gedit, gentoo, geshi, intellij, jedit, kate, latex, notepad-plus, scite, textmate, textpad, textwrangler, ultraedit, vim, and xcode. Have your pick! sbaz is a very useful tool. With it you can upgrade your scala installation, install a local copy of the Scala API, install aforementioned plugins, and some other cool stuff. Check it out.

To generate the needed settings for Eclipse, run mvn eclipse:eclipse. To take advantage of them, you will need the Eclipse Scala Plugin and the Eclipse Maven Integration plugin called M2Eclipse or some other way of having dependencies managed by Maven. After that, just import the project, right click on it and select "enable maven dependency management." At first, you may need to do a clean build (project -> clean ) so that the dependencies resolve for eclipse usage. The scala plugin for eclipse can be finicky at times. If you suddenly have problems editing a file cntrl+s to save and then close it and reopen. Note, if a scala source file stops being able to be opened at all, try restarting eclipse.

For unsupported development with NetBeans, check out the new NetBeans Scala plugin.

Configuring Mailer

For this tutorial we'll be using Google's smtp server to send emails, so if you want to follow this verbatim, you'll need an account from GMail.com. However, another SMTP server will also work.It's time to configure Mailer. At the heart of Mailer is JavaMail, so if you're familiar with that some of this may be familiar to you. Mailer looks to System's properties for configuration settings, so that's what we'll be changing. With whatever tool you'd like to use, start editing bootstrap.liftweb.Boot. For those unfamiliar with the standard Maven2 project layout, this will be in "src/main/scala/bootstrap/liftweb." For a lift specific page on your project layout, see the Archetype page of the lift wiki

To this source file you want to add the needed imports and then a method to the Boot class for configuring Mailer, as shown below.

import javax.mail._
import javax.mail.internet._

/*... */

// inside the Boot class add a method to configure the mailer:
// you will pass this the name of the
// smtp server and the login credentials
def configMailer(host: String, user: String, password: String) {
// Enable TLS support
System.setProperty("mail.smtp.starttls.enable","true");
// Set the host name
System.setProperty("mail.smtp.host", host) // Enable authentication
System.setProperty("mail.smtp.auth", "true") // Provide a means for authentication. Pass it a Can, which can either be Full or Empty
Mailer.authenticator = Full(new Authenticator {
override def getPasswordAuthentication = new PasswordAuthentication(user, password)
})
}

So that the mailer is configured when the application starts, add the following call to the boot method (using your own credentials):

configMailer("smtp.gmail.com", "yourusername@gmail.com", "yourpassword")

Now at this point the mailer should be configured and working properly. However, it is not doing anything interesting. Let's aright that.

Account Activations and Password Resets

You may want to require that user's activate their accounts after signup inorder to confirm the validity of their email address. This process is simple; open org.example.mailer.model.User in your editor and then change thefollowing line, or simply comment it out:

override def skipEmailValidation = true

to:

override def skipEmailValidation = false

Then return to the console and run mvn jetty:run.


Henceforth, when a user signs up for an account, a request for activation will be sent by email to their address. At this point, the password reset functionality should also work. Sign up for a new account and give them a try.

Open and Configure the lift Console

In the next two sections we will use the scala console to experiment. lift comes with a handy lift console that starts up your web app and then gives you a REPL to play around with while it is running. This is quite convenient for experimentation. As the liftConsole is a class in the test sources, you will need to do a full project build before you can use it. To build everything and then run the tests, run mvn install. Next, start up the console with mvn scala:console -DmainConsole=LiftConsole

Once at the Scala prompt, import Mailer and all of its inner classes and methods. Mailer is an object singlton so you can do that.

scala> import net.liftweb.util.Mailer
import net.liftweb.util.Mailer
...
scala> import Mailer._
import Mailer._
...

Keep this open for the next few examples, if you'd like to experiment with the code I give you.

Sending Emails

Now that your console is open and set up, I will briefly run you through the process of sending emails manually using Mailer. After you learn how to do it, you will be ready to incorporate it into your project. To send email we will utilize the sendMail method, passing it a series of case classes (MailTypes) that describe its functionality.

The method is defined as

def sendMail(from : From, subject : Subject, rest : MailTypes*)

The essential MailTypes are:

From(address: String) // The address that the email will seem to be sent from (often overwritten)

To(address: String) //This specifies the intended recipient

Subject(text: String) // Specifies the Subject line of the email

PlainMailBodyType(body: String) // The body of the email

For example, say I want to send an email from "sender@example.org" to "recipient@example.org" with the subject "Test," and body "Woo! It worked :-)" you would do the following (try putting in your own details):



sendMail(From("sender@example.org"),
Subject("Test"),
To("recipient@example.org"),
PlainMailBodyType("Woo! It worked :-)"))

Here are some other useful MailTypes:

CC(address: String) // As they said in accient times, the "Carbon Copy"

BCC(address: String) // An unseen recipient; "Blind Carbon Copy"

ReplyTo(address: String) // for email clients that support it, this is the address that a reply will be sent to

If you want to get more tricky and send xhtml and xhtml with images, try HTMLMailBodyType and XHTMLPlusImages, respectively, with Scala's support for inline xml markup.

Sending Text Messages

Surprisingly, certain major carriers support text messaging over SMTP. An email address is exposed for each phone number of that carrier. By way of an SMS gateway, emails sent to that address are forwarded as text messages to the phone they are asigned to. This makes text messaging from your web application a breeze. There is an extensive list of email schemas for various local and international carriers available here. However, if you are unaware of the phone number's carrier, which unfortunately will most likely be the case, you can use a service such as Teleflip.

TeleFlip is free to use for up to 100 messages per month. If you want to send more and exclude their signature from the messages, you will need to subscribed to their services, which start at $5 per month. Another drawback is that it is only compatable with SMS-enabled phones in the U.S. and Canada. If you use a carrier from elsewhere, you will have to consult the aforementioned list or do some digging on the net. Let us know of your findings! Perhaps there is a similar service in other corners of the world.

The email address pattern for using teleflip is simple; ###-###-####@teleflip.com.

For example, the email address for the phone number 415-555-1212 would be

415-555-1212@teleflip.com.

Let's put this theory into action. Using the console and Mailer, give it a try (substitute with the appropriate phone number or email address):


sendMail(From("sender@example.org"),
Subject("Text!!!"),
To("415-555-1212@teleflip.com"),
PlainMailBodyType("Woo! I can text :-)"))

Beep! Each call of sendMail intended to send a text will most likely be in the same form as the above, so to save some keystrokes we'll build a utility object for phone related operations. You can close the console by typing "exit."

Create package org.example.mailer.util and therein create a new Scala object singleton PhoneUtil. The following is the code for my PhoneUtil.scala:


package org.example.mailer.util

import net.liftweb.util.Mailer
import net.liftweb.util.Mailer._

object PhoneUtil {

}

First we want a method that when given a phone number will produce the proper email address:

// So far only supports 10 digit numbers, therefor discarding the rest.
def phoneToEmail(number: String): String =
// convert to a list, remove non digits, drop all but the last 10 digits, convert it back into a string, append email suffix
number.toList.filter(_.isDigit).takeRight(10).mkString + "@teleflip.com"

and then the convenience method:

def sendText(from: String, to: String, subject: String, content: String) =
Mailer.sendMail(From(from), Subject(subject), To(phoneToEmail(to)), PlainMailBodyType(content))

While I wont be using the following hereafter, if you plan for this application to not rely on teleflip, you could set something up like thefollowing (perhaps in a Phone.scala somewhere)

// Represents the phone itself
case class Phone(number: String, carrier: Carrier)

// "sealing" this ensures that only this file can create additional Carriers
sealed trait Carrier

// a case singlton representing Verizon Wireless
case object Verizon extends Carrier

// a carrier you don't know about
case object Unknown

In this case your phoneToEmail might look something like

// Represents the phone itself
case class Phone(number: String, carrier: Carrier)

// "sealing" this ensures that only this file can create additional Carriers
sealed trait Carrier

// a case singlton representing Verizon Wireless
case object Verizon extends Carrier

// a carrier you don't know about
case object Unknown

However, for the sake of my convenience, I will only be using the simplified version for this tutorial. You're welcome to do otherwise.

Building The lift App (SiteMap and Pages)

I will do my best to explain along the way, but if something is unclear I suggest consulting the Hello Darwin tutorial for clarity. Also, feel free to post on our Google Group. First we must ask ourselves the question, "what do we want our users to be able to do?" Here come the numbered use cases!

  1. Send emails
  2. Send text messages
  3. Broadcast email to other users

Analyzing the use cases, three pages come to mind

  1. sendEmail.html
  2. sendText.html
  3. broadcastEmail.html

Create blank files of the above names in src/main/webapp. While it is logical to combine these three into one form, breaking them up is allot easier and has less of a learning curve. Now in each of these files, in order toutilize the default template, add the following:


<lift:surround with="default" at="content">

</lift:surround>

Basically this will surround the children with everything outside of the

<lift:bind name="content" />
tag in templates-hidden/default.html where the template is defined. For more information on the topic of tags, I refer you to the lift Tags article in the lift wiki. If you'd like to style the pages as we go along, feel free; any html tags will work. I'm leaving that duty up to you.

However, before these pages are accessible, they must first be declared in your lift app's SiteMap. You must do this in bootstrap.liftweb.Boot. Observe the following line:

 val entries = Menu(Loc("Home", "/", "Home")) ::User.sitemap 

The above creates a list of Menus that summarizes the availability and location of your current lift app's pages and provides a means of building a menu.

Loc is formally defined as:

  
/**
* Create a Loc (Location) instance
*
* @param name -- the name of the location. This must be unique across your entire sitemap.
* It's used to look up a menu item in order to create a link to the menu on a page.
* @param link -- the Link to the page
* @param text -- the text to display when the link is displayed
* @param params -- access test, title calculation, etc.
*
*/
def apply(name: String, link: Link, text: LinkText, params: LocStuff*): Loc

Remember, you only want users that are already logged in to be able to view these pages. This is easy for in the LocStuff vararg you can pass a series of conditional statements (that you build in a DSL like fashion) that determine whether or not a page is available. This reduces the complexity of setting up roles and permissions. Of significant relevance to our work is User.testLogginIn which will detect whether or not a user is loggedin, and therefore doing allot of work for us.

With all this in mind, we produce the following SiteMap, using the "::" method to create a List:

// Build SiteMap
val entries =
Menu(Loc("Home", "/", "Home")) ::
Menu(Loc("Send Email", "/sendEmail", "Send Email", User.testLogginIn)) ::
Menu(Loc("Send Text", "/sendText", "Send Text", User.testLogginIn)) ::
Menu(Loc("Broadcast Email", "/broadcastEmail", "Broadcast Email", User.testLogginIn)) ::
User.sitemap

Next start your lift app back up and turn to localhost:8080 to check your changes. Everything should work. After all, "it works on my computer!" ;-)

Building The lift App (Send Text)

As we've been talking about text messaging, let's start out with something cool; sending text messages from your webapp. To build the form, we'll first start out by writing a stateful snippet that defines the components and functionality of the form. Unlike a regular snippet, which is discarded once rendering has completed, a stateful snippet is persisted between form submits. This makes writing forms easy. In order for lift to find your snippet, you must put it in your "snippet" package. Create a new scala source file named SendTextForm.scala in that package. It should have the following contents that I have excessively commented with the individual purpose and explanation of each line. In this particular tutorial, I will not cover validation. However, there will be a follow up that will dig into that subject.
package org.example.mailer.snippet

// Represents a sequence of xml nodes
import scala.xml.NodeSeq
// handy methods for generating html
import net.liftweb.http.SHtml
// extends this to make your snippet stateful
import net.liftweb.http.StatefulSnippet
// implicit conversions and "bind" which we will later use
import net.liftweb.util.Helpers._
// the phone utility we wrote earlier
import org.example.mailer.util.PhoneUtil

// Extending StatefulSnippet lets /lift/ know it should be persisted
class SendTextForm extends StatefulSnippet {
/**
* Normally look up is done by method name, but with stateful snippets
* you define a partial function that matches on the tag name and returns a function
* of type NodeSeq => NodeSeq
*/
def dispatch: DispatchIt = {
/**
* We want the method show to be executed when "show" is called
* so we return a reference to the method as a partially applied function
*/
case "show" => show _
}

// define the fields you'll be using
var to: String = ""
var subject: String = ""
var content: String = ""

// will be called and passed the children of <lift:sendtextform.show>...</lift:sendtextform.show>
def show(children: NodeSeq) =
// when rendering children, the fields prefixed with "sendText" are bound to these definitions
bind("sendText", children,
// called with <sendtext:to/>,
// SHtml.text(defaultValueOfField, (valueEnteredByUser) => what to execute upon submission
// Function updates the values of our variables
"to" --> SHtml.text("", value => to = value),
"subject" --> SHtml.text("", s => subject = s),
"content" --> SHtml.textarea("" c => content = c),
// When the submit button is clicked, we send a text message
"submit" --> SHtml.submit("Send", something => sendText(to, subject, content)))

// This is where we use the phone utility we defined earlier
def sendText(to: String, subject: String, content: String) = try {
PhoneUtil.sendText("texter@example.org", to, subject, content)
} catch {
case e => println(e)
}
}

Open sendText.html. Keep in mind that adding the "form" attribute to a snippet causes it to be wrapped in a <form>


<lift:surround with="default" at="content">
<lift:sendTextForm.show form="POST">
<label for="toField">Phone Number: </label>
<sendText:to id="toField" />
<label for="subjectField">Subject: </label>
<sendText:subject id="subjectField" />
<label for="contentField">Content: </label>
<sendText:content id="contentField" />
<sendText:submit />
</lift:sendTextForm.show>
</lift:surround>

Feel free to style this to your liking. Building The lift App (Send Email) For this next section we'll add a form to sendEmail.html that will allow users to send emails to an address that they specify. This time around we will begin to explore the User model. We want sent messages to appear as if they were sent from the user's email account. However, this wont actually work on the GMail SMTP server as it overrides the From field. If you were to serve your own personal SMTP server or use one that was less restrictive, I believe you could overcome this limitation. At least you should be able to use the domainof your SMTP server: username@mailer.example.org

package org.example.mailer.snippet

import scala.xml.NodeSeq
import net.liftweb.http.SHtml
import net.liftweb.http.StatefulSnippet
import net.liftweb.util.Helpers._
import net.liftweb.util.Mailer
import net.liftweb.util.Mailer._
// The User model and acompanying object singleton
import org.example.mailer.model.User

class EmailForm extends StatefulSnippet {
def dispatch: DispatchIt = {
case "show" => show _
}

// define the fields you'll be using
var to: String = ""
var subject: String = ""
var content: String = ""

def show(children: NodeSeq) =
bind("email", children,
"to" --> SHtml.text("", t => to = t),
"subject" --> SHtml.text("" s => subject = s),
"content" --> SHtml.textarea("", c => content = c),
"submit" --> SHtml.submit("Send", something => sendEmail))

// back to our good friend Mailer
def sendEmail = try {
// grab a can of the current user, assume it's full and open it, retrieve email address
val email = User.currentUser.open_!.email
Mailer.sendMail(From(email), Subject(subject), To(to))
} catch {
case e => println(e)
}
}

Open sendEmail.html

<lift:surround with="default" at="content">
<lift:emailForm.show form="POST">
<label for="toField">To: </label>
<email:to id="toField" />
<label for="subjectField">Subject: </label>
<email:subject id="subjectField" />
<label for="contentField">Content: </label>
<email:content id="contentField" />
<email:submit />
</lift:emailForm.show>
</lift:surround>

mvn jetty:start
and give it a try.

Building The lift App (Working with Users and Broadcasting Email)

The email broadcaster is roughly the same concepts reapplied, but introduces the some basics of the User model. The messages will support keywords to "customize" the emails for the users. In this example only firstName and lastName are given, but others could be support with the same technique. Start a new Scala source file in the snippets package again. Name it broadcastForm.scala. This time through my comments will be sparse, only addressing new concepts.

package org.example.mailer.snippet

// Import the User model
import org.example.mailer.model.User
import scala.xml.NodeSeq
import net.liftweb.http.SHtml
import net.liftweb.http.StatefulSnippet
import net.liftweb.util.Helpers._
import net.liftweb.util.Mailer
import net.liftweb.util.Mailer._

/**
* A snippet backing a form that mass broadcasts email to users
*/
class BroadcastForm extends StatefulSnippet {
var from: String = ""
var subject: String = ""
var content: String = ""

// resolves keywords to their relevant values
def resolveKeywords(text: String, u: User) =
text.replaceAll("%user.first_name", u.firstName).
replaceAll("%user.last_name", u.lastName)
// you could also do cool stuff with the locale
// to get it: u.locale.isAsLocale

def dispatch: DispatchIt = {
case "show" => show _
}

def show(children: NodeSeq) =
bind("broadcast", children,
"from" --> SHtml.text("", v => from = v),
"subject" --> SHtml.text("", v => subject = v),
"content" --> SHtml.textarea("", v => content = v),
"submit" --> SHtml.submit("Send", something => broadcastEmail))
// will send an email out to all users
def broadcastEmail = {
// grab a list of all the users currently registered
val users = User.findAll
// Send the email to each user's email address one
users.foreach(u => Mailer.sendMail(From(from), Subject(subject),
// u.email returns the user's address
To(u.email), PlainMailBodyType(resolveKeywords(content, u))))
}
}
And now its use in broadcastEmail.html:

<lift:surround with="default" at="content">
<lift:broadcastForm.show form="POST">
<label for="fromField">From: </label>
<broadcast:to id="fromField" />
<label for="subjectField">Subject: </label>
<broadcast:subject id="subjectField" />
<label for="contentField">Content: </label>
<broadcast:content id="contentField" />
<broadcast:submit />
</lift:broadcastForm.show>
</lift:surround>

If you are interested in all the evil things you can do to users, take a look at MetaMegaProtoUser. Singleton object User subclasses it, so you can make calls like we did above.

Wandering off

As I'm sure you've noticed, these three samples share a lot of repeated code. However, for the purpose of this tutorial I thought it would be distracting if I wandered too far off the beaten path with additional problem/solutions. There is, however, a much cooler and useful way to keep this dry; combine all three forms into one on a single page; a master message control panel :-). This may actually be of use to you locally. The problem then becomes locking things down so that no new users can sign up. That, however, is also out of the scope of this tutorial. If I get enough positive feedback, requests, and suggestions on this tutorial I will gladly write a follow up.

The last critical piece missing from all of this is validation and security, both client and server side. I am currently working on my next tutorial to post here which will cover these issues and revisit and approve upon what we have covered in this session. In the meantime, feel free to comment with questions, suggestions orconcerns. Until next time, keep the foo counters turning.

- Dan

7 comments:

Alexander Kosenkov said...

Please check 'Broadcasting Email' block - resolveKeywords function seems to be unused!

I suppose you should change
PlainMailBodyType((content))
to
PlainMailBodyType(resolveKeywords(content, u))

Daniel Green said...

Whoops! Thanks. Did you get it working?

Alexander Kosenkov said...

It seems to ignore id attribute of broadcast:subject and other custom tags. Trying to find out how to push them

philip said...

Hi, this is great. Could you maybe write some articles on the simple topics to help beginner ease-in?

1) Forms, validation of form and simple CRUD to database.
2) Comet added to form for form validation as you type or some other example of comet usage.
3) Model/View/Controller, how best to keep them apart

Again thanku this is very helpful.

bradford said...

Very nice tutorial. I did notice that the wrong code was pasted into the "In this case your phoneToEmail might look something like" section.

Anyways, I learned a lot from this. Thanks.

Dave said...

This one really helped me out to do a simple contact form... Thank you.

bantic said...

Thanks for writing this. It's very helpful.
In working through it, I got the following error in the scala console after trying to make a call to the 'sendMail' method:

javax.mail.MessagingException: IOException while sending message;
nested exception is:
javax.activation.UnsupportedDataTypeException: no object DCH for MIME type multipart/alternative;

Here's a full stacktrace:
http://pastie.org/732597