Wednesday, December 5, 2007

Dynamic web applications with lift and Scala

This is a tutorial introduction to building dynamic web applications with lift and Scala. We'll be building a linksharing site, much like Digg or reddit. However, lift's advanced support for Comet-enabled web applications will allow our site to update live as new links get submitted or old links get voted up or down, without our visitors having to refresh the page.



This is a short (50 seconds) screencast of what we're trying to build. The quality on the YouTube version isn't great, so you can also download the full-quality QuickTime version (~800K). I'm running two different browsers, Safari and Camino, and loading the finished site with both of them. Notice how the link submissions and votes made in one browser get automatically updated in the other browser. We accomplish this using Comet, and we'll see how lift makes Comet-enabled web applications a breeze. Let's get started.

I'll assume that you have the most recent versions of Scala, Eclipse, the Scala Plugin for Eclipse, and Maven installed. I'll also assume you have some rudimentary knowledge of Scala.

(I've you have used lift before the 0.3.0 release on Dec. 7, there is a bug in our repository that might cause you to use a pre-release version of the lift libraries. To fix this problem, delete the net/liftweb directory in your local Maven repository before continuing. I believe the default location is: ~/.m2/repository/net/liftweb)

From Eclipse's workspace directory, lets create a new project called "linkshare" in the "com.test" package by typing the following into the command line:

mvn org.apache.maven.plugins:maven-archetype-plugin:1.0-alpha-7:create -U \
-DarchetypeGroupId=net.liftweb \
-DarchetypeArtifactId=lift-archetype-blank \
-DarchetypeVersion=0.3.0 \
-DremoteRepositories=http://scala-tools.org/repo-releases \
-DgroupId=com.test -DartifactId=linkshare

That's all we need to do to set up a skeleton lift site. (If you're using Eclipse and want to use Q4E, see: http://liftweb.net/index.php/Archetypes) Now we'll go into the directory that contains our project and run it with Jetty. Maven should automatically download Jetty for you if you don't already have it.

cd linkshare
mvn jetty:run

If everything goes well, you should be able to see your skeleton lift site running on http://localhost:8080/. We see a nice welcome screen that shows the current time. Now let's do something with the site. Leave Jetty running, and open the index.html file in src/main/webapp. Edit the file so that it contains a more topical message, like "Welcome to your linksharing site!". Save the file and reload your browser. Your changes should have been picked up by Jetty.

Jetty also automatically picks up on changes to your compiled class files and redeploys your webapp when necessary, so you can leave the Jetty process running throughout this tutorial and see the changes in your browser. We'll still want to do some stuff on the command line, so open up another command line window. Then create an eclipse project with the following command:

mvn eclipse:eclipse

Now you can import the lift project into Eclipse. There are two directories you should mostly be concerned with. The src/main/scala directory contains all the Scala code for your lift project, while the src/main/webapp directory contains the html, css, javascript, etc.

You'll probably want to enable Eclipse's incremental compilation. That way, whenever you save changes to your code, Eclipse will recompile your .class files, and Jetty will pick up the new code. If you're not using Eclipse, or not using incremental compilation, you'll want to issue a 'mvn compile' command to see any changes to your code reflected in Jetty.

You'll notice that our app's page has a "Home" link which points right back to the main page. This is actually a part of lift's advanced Sitemap feature, but for now we won't be using it, so let's disable it.

The Sitemap is defined in our bootstrap. Our site's bootstrap code gets run by lift as soon as our app gets deployed. This is a great place to make sure we set up our databases correctly, among other things. It's also where our Sitemap is defined, so let's go take a look.

You'll find the bootstrap code in src/main/scala/bootstrap/liftweb/Boot.scala. It's fairly lonely in here, as our app doesn't do much yet. Let's modify the appropriate line to hide the "Home" link from our Sitemap. We do this by adding the "Hidden" argument, like so:

    val entries = Menu(Loc("Home", "/", "Home", Hidden)) :: Nil

Save the file, then wait a few seconds for Jetty to pick up the changes. (If you're not using Eclipse's incremental compilation, then you'll have to issue a 'mvn install' command.) When you refresh your site's page, you should see that the "Home" link has disappeared. Awesome! Now let's do something useful.

Our first order of business is to allow our visitors to submit links. If we want to receive links from our visitors, then we also want somewhere to keep those links. At this point, most people would start defining database connections, database schemata, model objects, object-relational mapping... Doing all of this is very easy with lift. The included Mapper classes provide a great way of easily, securely, and meaningfully storing your bytes.

However, I'm going to make a controversial decision: we won't use a database. lift, you see, is a highly stateful web framework. It keeps state around between requests, so we can get very far without a database. Of course, this means that if our app gets redeployed, or if it goes down for maintenance, or if it crashes, then we'll lose all of our state and we'll have to start collecting links from scratch. This is clearly unacceptable in a production environment, where we would want to persist our important information to a database. Like I said, lift makes this easy with the included Mapper classes. However, for the purposes of this demo, we can deal with losing all of our links every now and then.

So if we're not keeping our links in a database, then where are we keeping them? We're going to use a long-lived Scala Actor. Scala Actors are lightweight threads, similar to Erlang's Actors. They can send and receive messages to other Actors, which will come in very handy. Let's make a LinkStore object in our com.test.controller package:

package com.test.controller;

import scala.collection.mutable.{HashMap, HashSet}
import scala.actors.Actor
import scala.actors.Actor._

import net.liftweb.util.Helpers._

// Messages
case class AddListener(listener: Actor)
case class RemoveListener(listener: Actor)
case class UpdateLinks(topLinks: List[Link])
case class AddLink(url: String, title: String)
case class VoteUp(linkId: String)
case class VoteDown(linkId: String)

// Data Structures
case class Link(id: String, entry: LinkEntry)
case class LinkEntry(url: String, title: String, var score: Int)

object LinkStore extends Actor {
val linkMap = new HashMap[String, LinkEntry]
val listeners = new HashSet[Actor]
var topLinks: List[Link] = Nil

def notifyListeners = {
topLinks = linkMap.toList.map(p => Link(p._1, p._2)).
sort(_.entry.score > _.entry.score).take(20)

listeners.foreach(_ ! UpdateLinks(topLinks))
}

def act = {
loop {
react {
case AddListener(listener: Actor) =>
listeners.incl(listener)
reply(UpdateLinks(topLinks))
case RemoveListener(listener: Actor) =>
listeners.excl(listener)
case AddLink(url: String, title: String) =>
linkMap += randomString(12) -> LinkEntry(url, title, 1)
notifyListeners
case VoteUp(linkId: String) => try {
linkMap(linkId).score += 1
notifyListeners
}
case VoteDown(linkId: String) => try {
linkMap(linkId).score -= 1
notifyListeners
}
}
}
}

start // This starts our singleton Actor
}


The central data structure here is the LinkEntry. It stores the url, title, and current score of a link. Our LinkStore is a singleton Actor which is started up when the class is loaded. The LinkStore keeps a HashMap of LinkEntries, indexed by a unique ID of 12 characters. The LinkStore responds to three messages that can modify the LinkEntries: AddLink, VoteUp, and VoteDown. Each does what its name suggests. The LinkStore also keeps a HashSet of listeners. Listeners are other actors that want to be notified when the LinkStore changes. Registered listeners are sent the newest list of Top 20 Links whenever there is an update.

Now that we have a place to store our links, we want to let our visitors send them to us. If you take a look at index.html in src/main/webapp, you'll notice several <lift:...> tags. When a user requests a page, lift will try to find it in your src/main/webapp folder. If the page has any <lift:...> tags, they'll get processed before the response is sent back to the user. The first such tag we see is <lift:surround>. This tells lift to surround the enclosed xhtml with the "default" template. Templates can be found in src/main/webapp/templates-hidden. The second such tag is a <lift:snippet> tag. This tag instructs lift to call the "helloWorld:howdy" snippet, and replace the <lift:snippet> tag with the results of the snippet call. Snippets can be found in src/main/scala/com/test/snippet. Let's take a look at the HelloWorld.scala snippet. HelloWorld is a class which defines one method, "howdy". All that howdy does is return some xhtml. This is the xhtml that will replace the <lift:snippet> tag before the whole page gets sent back to the user. Notice that Scala allows XML literals right in the syntax. It also allows arbitrary Scala expressions inside the XML when escaped by {} brackets. As you can imagine, embedding xhtml straight into your code can be very useful for web development.

Let's get rid of the HelloWorld snippet and replace it with our own. Make a new class in com.test.snippet called "Submission", and define a method "form":

package com.test.snippet;

import net.liftweb.http.S
import net.liftweb.util.Helpers._

import com.test.controller._

class Submission {
def form = {
var url = ""
var title = ""

<span>
{ S.text("URL", u => url = u) }
{ S.text("Title", t => title = t) }
{ S.submit("Submit", ignore => LinkStore ! AddLink(url, title)) }
</span>
}
}


Phew! There's a lot going on in these few lines. Let's try to break it down piece by piece. We're taking full-advantage of Scala's support for XML in the syntax to send back an xhtml form. We're using {} brackets to escape the XML syntax and insert arbitrary Scala expressions. The first and second Scala expressions reference "S.text". This is a method that will return a text input field. The first argument to the method is the default text in the field. The second argument is a function that will be executed on the contents of the text input field when the form is submitted. Take a moment to let that sink in. So when the form gets submitted, the stuff in the url field will be assigned to the "url" variable and the stuff in the title field will be assigned to the "title" variable.

"But how?!", you might ask. After all, Submission's "form" method gets called in one request-response cycle, and the html form doesn't get submitted until the next request-response cycle! Well, this is part of lift's magic. Between requests, lift keeps around a closure that knows about the "url" variable and about the "title" variable. Even better, the closure knows what to do with these variables once the form submission comes in. It works almost like magic.

Which leaves us one last thing to analyze. As you might have guessed, "S.submit" returns a submit input field (a submit button). The first argument to "S.submit" is the label on the button. The second argument is a function to be called once the form is submitted. This function sends a message (using "!", a method defined on Actors) to the LinkStore, telling it that we want to add a new link.

Now let's go back to index.html and change the "helloWorld:howdy" snippet to a "submission:form" snippet, like so:

    <p><lift:snippet type="submission:form" form="post" /></p>

Notice that we added a "form" attribute to let lift know that this snippet should be rendered as a POST form.

Reload your page and you should now see two form fields and a submission button. Entering information and pressing submit doesn't seem to do anything, but if we add a "Console.println" to LinkStore we'll notice that it is indeed receiving and storing links.

Now that we have links, how do we share them with the world? Something needs to be responsible for showing these links to our visitors and updating them whenever necessary. This sounds like just the job for lift's CometActor. CometActors know how to render themselves onto an xhtml page. They are also long-lived; a visitor can navigate away from the page and come back to find the same CometActor servicing his requests. Since they're long-lived, they can interact asynchronously with the visitor, outside of the normal http request/response cycle. We'll use these capabilities to make "live" updates to the page a visitor is viewing, without the visitor having to refresh the page. Let's get cranking. Add a new "com.test.comet" package, and in it define a new LinkActor class:

package com.test.comet;

import scala.collection.mutable.HashMap
import scala.xml.{NodeSeq, Text}

import net.liftweb.http._
import net.liftweb.http.js.JsCmds.Noop
import net.liftweb.util.Can

import com.test.controller._

class LinkActor(theSession: LiftSession, name: Can[String],
defaultXml: NodeSeq, attributes: Map[String, String])
extends CometActor(theSession, name, defaultXml, attributes) {
var topLinks: List[Link] = Nil

def defaultPrefix = "links"

def render = {
def linkView(link: Link): NodeSeq = {
<li>
<a href={link.entry.url}>{link.entry.title}</a>
[{link.entry.score} {if (link.entry.score == 1) "vote" else "votes"}]
{S.a(() => {LinkStore ! VoteUp(link.id); Noop}, Text("[up]"))}
{S.a(() => {LinkStore ! VoteDown(link.id); Noop}, Text("[down]"))}
</li>
}

bind("view" -> <ol>{topLinks.flatMap(linkView _)}</ol>)
}

override def localSetup {
LinkStore !? AddListener(this) match {
case UpdateLinks(links) => topLinks = links
}
}

override def localShutdown {
LinkStore ! RemoveListener(this)
}

override def lowPriority : PartialFunction[Any, Unit] = {
case UpdateLinks(newLinks) => topLinks = newLinks; reRender(false)
}
}


Let's start with the easy stuff. The "localSetup" method is called when the CometActor is created. This method sends a message to LinkStore (to make sure it gets added as a listener) and waits for a reply. The reply will contain the initial list of top links, which we store in the "topLinks" variable. The "localShutdown" method is called when the CometActor gets retired. This just cleans up and makes sure we're no longer listening to the LinkStore.

The "render" method does most of the interest work. This is how the CometActor renders itself to xhtml. It's going to create some xhtml with an ordered list of links. The links themselves get rendered by the inner "linkView" method. This method will render the link with the appropriate title, as well as showing the score beside the link, and "[up]" and "[down]" links to vote the story up or down. The "S.a" method takes two parameters, and returns the xhtml code for an anchor tag (link). The first parameter is the code that will be executed when the link is pressed. This code sends a message to the LinkStore, telling it whether the vote was an up-vote or a down-vote. (The "Noop" at the end of the block is just to let Scala know that this is indeed a JsCmd block that is safe to send to the browser.) The second parameter is an xhtml node that will be wrapped inside the anchor tag. In this case, it's just a simple text node that indicates whether the vote is up or down. With these parameters, lift creates an anchor tag that, when clicked, will send an AJAX request to the server to execute the provided code. Neat stuff.

Finally, the "lowPriority" method is where we listen for updates from the LinkStore. As you might imagine, CometActors have three similar methods: "lowPriority", "mediumPriority", and "highPriority". The only differences are the order in which they get applied. We're not doing anything too complicated with CometActors here, so "lowPriority" will work fine for our purposes. Whenever we get an UpdateLinks message from the LinkStore, we want to update our local copy of the links. We also call "reRender", which pushes the latest result of "render" to the page containing the corresponding <lift:comet> tag for this CometActor.

The last thing we have to do is place the <lift:comet> tag on a page. This will render the CometActor we've just created to that page, and will listen to the CometActor for updates. Let's go back to index.html and add the following right below our form submission:

    <p>
<lift:comet type="LinkActor">
<links:view>Loading...</links:view>
</lift:comet>
</p>


Now you should have a site that functions exactly like the site in the screencast. If you open two different browsers, or two different windows in the same browser, actions taken in one browser window will be automatically reflected in the second browser window.

This introduction barely touches the full power of lift. We didn't really explore lift's Sitemap, ORM Mapper classes, extensive type-safety, semantic models, security-by-default, or advanced templating system. Hopefully, however, this gives you a taste of web development with lift, and showcases just one of lift's many features: advanced support for Comet-style web development.

This code borrows heavily from Steve Jenson's DynamicBlog, a part of the "hellolift" example project. Check it out at the lift source tree (http://liftweb.googlecode.com/svn/trunk/). Thanks to David Bernard for his help with syntax highlighting and with his feedback on a draft version of this blog post. And many, many thanks to David Pollak for his feedback on this blog post, but most of all for making such an awesome web framework.

16 comments:

TylerWeir said...

This is great, thanks for your effort.

jeortiz said...

Great demo!!!

Bjarte said...

Great post!

I tried to run it using scala-2.6.0-final and I get an error in the last step saying

[WARNING] .../linkshare/src/main/scala/com/test/comet/LinkActor.scala:43: error: wrong number of arguments for method reRender: ()net.liftweb.http.AnswerRender
[WARNING] case UpdateLinks(newLinks) => topLinks = newLinks; reRender(false)
[WARNING] ^
[WARNING] one error found

Anybody got a hint on how I can solve this or what I did wrong?

Bjarte said...

If i remove the argument to reRender it works. Looks like this change was made in r387 of lift. Just writing this so that if anybody else has the same problem there is a sollution :)

Jorge Ortiz said...

Bjarte,

Thanks for your comments!

I reproduced your problem. I think your Maven repository has a pre-release copy of the lift 0.3.0 core libraries. As you pointed out, r387 (well before the 0.3.0 release) added a boolean parameter to reRender.

To fix this, clear the outdated version of the 0.3.0 libraries from your Maven repository. (I believe the default location is ~/.m2/repository/net/liftweb/lift-core/0.3.0/. Just delete the whole directory.) Then do a 'mvn install' from the linkshare project. Maven will notice that the 0.3.0 libraries are missing and will automatically download the release version.

Thanks for reading!

Brett Morgan said...

Great tutorial Jorge. Hopefully i can have a hack'n'slash at this tonight =)

Jörn Zaefferer said...

Thanks for the tutorial. Easy to follow and a good introduction to lift.

The only thing that doesn't work are dynamic updates in IE6. IT sends the correct request (and the application in FF gets updated), but the reRender seems to fail. I'm completely lost at debugging lift. Perhaps you could add a few lines here and there what to do when something fails, like in this case.

Enrico said...

Great tutorial. Thanks for your effort!

Marius said...

Yes everything looks great although trying the comet chat example from lifweb site I could not notice the fact that polling is used. Does lift uses server push feature from tomcat 6.0 for example using CometControler?

If so, how?

Coughlin said...

Love this tutorial! Great!

Corbin said...

Thanks for the great demo!

Hints for myself 2 hrs ago and future newbies:

*upgrade to scala 2.7.1 and lift 0.9 in pom.xml
*change LiftServlet to LiftRules in Boot.scala
*change S to SHtml in Submission.java and LinkActor.java

Thomas Sant'ana said...

I'm new at lift, and have done this very nice demo using Eclipse. However I failed to understand who to configure Eclipse for Incremental update. Using Project Clean and build make Maven install fail. What should I do to have Eclipse compile and publish directly?

Ryan said...

I recently came accross your blog and have been reading along. I thought I would leave my first comment. I dont know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.


Sarah

http://www.lyricsdigs.com

chaecker said...

Thank you for the blog!
My problem is that for

<p><lift:comet type="LinkActor">
<links:view>Loading...</links:view>
</lift:comet></p>

in index.html the following is sent to the browser:

</span></form></p>
<p><!--FIXME - comet type: Full(LinkActor) name: Empty Not Found -->
<links:view>Loading...</links:view>
</p>

Any ideas? Thanks in advance,
Claudius

james said...

can you pls send me structure and of the src/scala like this based on i can make iam new to lift framework

verbatim said...

Thanks for this tutorial. Liftweb has changed a bit in the past few years, so I updated it for 2.3.

You can find a complete update of this on github.