XForms Patterns: Incremental and 'Google Suggest'

As promised in my last blog (Ajax, Hard Facts, Brass Tacks ... and Bad Slacks), I'm going to show how to construct 'Google Suggest', using XForms. We'll take Google Suggest as our case study since it is often discussed in Ajax-related articles. (I'll look at others over the coming weeks.)

Google Suggest contains a number of neat features, and certainly shows what's possible with scripting. But if you look at the code you'll see that it most likely took a lot of people a lot of time to implement. That's great for Google -- they have some incredibly talented people, and plenty of them -- but it's not so great for the rest of us. We want to put the same features into our applications and we want to give our users the same experiences, but we may not have the time or expertise to do it using complicated script.

I described this issue of trying to capture these patterns and make their functionality available to everyone, in my previous blog. I talked about how easy it has been for people to take part in the world of hypertext since all it involved was mastery of a few key mark-up tags -- in particular <a>. Whilst programmers could easily connect documents together over a network using C and other language, it took the advent of the mark-up language HTML to make 'networked documents' available to the rest of us. The main argument in my previous blog was that although the techniques used in Ajax are complicated, they are actually a rich source of 'patterns' crying out to be codified, and that the main purpose of XForms was to try to capture as much of this functionality as possible and convert it into easily re-used mark-up.

But since this whole process begins by spotting patterns and trying to generalise them, we're going to do exactly that with Google Suggest as our case study.

Anatomy of Google Suggest

Let's look at what Google Suggest does, before we try to reproduce it in XForms. After all, we're primarily concerned with design patterns rather than clever trickery in scripting.

There are two major parts to Google Suggest. The first is obviously that a user can type into an input box and see suggested search terms based on what they have entered. The suggestions seem to be based on searches that users of Google have carried out, rather than what is in the database (we know this because they include misspellings, and don't include all permutations of a suggestion that contains more than one word).

The second part of Google Suggest is that once a choice has been made -- or a completely new term entered -- a search is made on Google.

We can model these two pieces, as follows:



Let's drill further into these two halves.

Obtaining Suggestions

As we know then, Google Suggest provides a simple input box that gives the user a series of search terms as they type. This is achieved by taking the characters that the user enters and sending them to a Google server, and recieving back a list of 10 suggestions. As it happens, the server sends back some JavaScript that contains a string array, filled with suggestions, and used as the parameter to a function call! But let's ignore the implementation details as we try to pull out what is common and reusable.

Retrieving the Data

The actual request to Google for suggestions is pretty straightforward, and can be implemented in XForms like this:

<xf:submission id="sub-suggest"
 action="http://www.google.com/complete/search"
 method="get" separator="&amp;"
 ref="instance('inst-suggestions-rq')"
 replace="instance" instance="inst-suggestions-rs"
/>

<xf:instance id="inst-suggestions-rq">
  <query xmlns="">
    <qu />
    <hl>en</hl>
    <js>false</js>
  </query>
</xf:instance>

<xf:instance id="inst-suggestions-rs">
  <dummy xmlns="" />
</xf:instance>

If you've not dived into XForms yet, then this code creates two instances -- think of them as DOMs -- and a submission. The latter is configured to send instance rq to the Google server (http://www.google.com/complete/search) and place the returned instance into instance rs. You'll note that instance rq has an element qu that is empty -- that's where we'll place the item that we want suggestions for.

Note also that we're using XML to hold the values to be sent. XForms supports a number of 'serialisations', which determine exactly how the data is sent to the server. The idea is that we manipulate the data in our forms in only one way -- as XML. But when we send it to the server it could go out in all sorts of ways. In this example, the XML will be converted to an HTTP GET request. So if we want suggestions on "xf", our request will look like this:

  http://www.google.com/complete/search?qu=xf&hl=en&js=false

Google Suggest returns the following (click here if you want to see the results for yourself):

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
    <script>
      function bodyLoad()
      {
        if (parent == window) return;
        var frameElement = this.frameElement;
        parent.sendRPCDone(
          frameElement,
          "xf",
          new Array("xfire", "xfm", "xfx", "xfce", "xfactor", "xfree86", "xfl",
            "xfr.exe", "xforms", "xfiles"),
          new Array("504,000 results", "602,000 results", "1,980,000 results",
            "1,150,000 results", "53,300 results", "5,470,000 results",
            "270,000 results", "3,190 results", "836,000 results",
            "305,000 results"),
          new Array("")
        );
      }
    </script>
  </head>
  <body onload='bodyLoad();'>
  </body>
</html>

XForms expects the result to be XML, and as you can see, Google is returning the results as a pair of JavaScript arrays, which are themselves being passed into a JavaScript function. This is fine for their application, but pretty useless to some other web service consumer. Luckily, since the returned script is inside some well-formed XML, we can at least place the results into our XForms instance, and from there we can manipulate it.

So we'll pretend that the web service we made our query from really did provide us with a list of suggestions in XML, and we'll do that by adding a function that parses the JavaScript returned by Google Suggest, and converts it into a set of XML nodes.

Making our form consume XML rather than the particular format we see here gives us two new possibilities -- we can use suggestions from other sources:



and we can also use Google's suggestions to help search other locations.

Getting the Suggestions Into XML

You can skip over this bit if you like, since it's only about how to get the non-XML results that Google Suggest returns, into XML. By the end of the following process we'll have XML that looks like this, but of course, if we had a real 'suggestions' web service we wouldn't need this step:

<xf:instance id="inst-suggestions">
  <suggestions xmlns="">
    <suggestion query="xbox" comment="55,900,000 results" />
    <suggestion query="xanga" comment="1,190,000 results" />
    <suggestion query="xbox 2" comment="21,000,000 results" />
    <suggestion query="xvid" comment="3,090,000 results" />
    <suggestion query="xm radio" comment="2,030,000 results" />
    <suggestion query="xbox cheats" comment="12,000,000 results" />
    <suggestion query="xanga.com" comment="1 result" />
    <suggestion query="xvid codec" comment="630,000 result" />
    <suggestion query="xml" comment="120,000,000 results" />
    <suggestion query="xerox" comment="19,400,000 results" />
  </suggestions>
</xf:instance>

OK -- if you're still here, and you want to see how this is obtained, let's go. The first thing we need is another instance to put the results in:

<xf:instance id="inst-suggestions">
  <suggestions xmlns="" />
</xf:instance>

Next we need to register a handler for the event that will fire when we've finished getting suggestion data from Google. This is xforms-submit-done, which is a built-in event dispatched by all XForms processors when a submission completes successfully (if there is an error, xforms-submit-error is dispatched instead):

<xf:action ev:observer="sub-suggest" ev:event="xforms-submit-done">
  <xf:setvalue
   ref="instance('inst-dummy')/dummy"
   value="inline:texttoxml()"
  />
</xf:action>

This handler is set up to call a JavaScript function that converts the arrays to XML, and if this form is ever used to consume 'real' XML, this handler and the function can just be dropped.

The function itself is the only script we'll need in the form. The first thing it has to do is get a pointer to the XForms model:

<script type="text/javascript">
  // <![CDATA[
  function texttoxml()
  {

    var oModel = document.getElementById("mdl-main")

    if (oModel) {

Next get the instance that contains the results from Google:

      var oInst = oModel.getInstanceDocument("inst-suggestions-rs");

      if (oInst) {

As we saw before, the script element is where the real data is, in the parameters to the function call:

        var oScriptList = oInst.selectNodes("html/head/script/text()");

There is probably only one node in the returned nodeset, but we'll create a loop just in case:

        var oNode = oScriptList.nextNode();
        var s = "";

        while (oNode)
        {
          s += oNode.text;
          oNode = oScriptList.nextNode();
        }

s now contains the body of script.

Next we get the target node, the location where the suggestions are to be placed:

        oInst = oModel.getInstanceDocument("inst-suggestions");

        if (oInst)
        {
          oNode = oInst.documentElement;
          if (oNode) {

We remove all children of the target node -- i.e., the previous suggestions -- but they could be copied to another instance as a cache, so that if the user enters the same values again, we don't bother going to the server:

            while (oNode.hasChildNodes())
              oNode.removeChild(oNode.firstChild);

The next step is the most important -- we crack open the function call to find the two arrays that contain the list of suggestions:

            var oArrays = s.split("new Array(");
            var regExp = new RegExp("\"[^\"]*\"", "ig");
            var oNames = oArrays[1].match(regExp)
            var oPages = oArrays[2].match(regExp)

            if (oNames && oPages) {

Once we have the suggestions in an array, we are ready to create the new instance data:

              for (var i=0; i < oNames.length; i++)
              {
                var el = oInst.createElement("suggestion");

                el.setAttribute("query", oNames[i].replace(/\"/gi, ""));
                el.setAttribute("comment", oPages[i].replace(/\"/gi, ""));
                oNode.appendChild(el);
              }

Although XForms is designed to be used with declarative mark-up, a number of methods are provided to make it so that direct DOM manipulation can co-exist. We use the rebuild method to tell the processor that the instance has been changed, and this in turn causes the processor to rebuild it's dependency graphs, recalculate any calculations, and refresh the display:

              oModel.rebuild();
            }// if ( we created two arrays )
          }// if ( found the node to store the suggestions under )
        }//if ( found the instance where the suggestions will be stored )
      }//if ( found the instance holding the suggestions returned from Google )
    }// if ( found the model )
    return true;
  }// texttoxml()
  // ]]>
</script>

Triggering the Submission

We now have a submission that can request suggestions from Google, and at the end of the process we will have a list of XML suggestions. We could easily add this to a form, so that when a user presses enter in a control the submission is carried out. We'll create an input control and when we receive the DOMActivate event we'll trigger the submission:

<xf:input ref="instance('inst-suggestions-rq')/qu">
  <xf:label>Suggestions for:</xf:label>
  <xf:action ev:event="DOMActivate">
    <xf:send submission="sub-suggest" />
  </xf:action>
</xf:input>

Of course, we don't want our users to have to press [ENTER] every time they want some suggestions, which is where we find another of the patterns that XForms has captured, and turned into simple mark-up -- incremental.

XForms provides an attribute on form controls called incremental, which allows the author to provide a hint to the XForms processor that they would like to know when the value of the control has changed, even if the user hasn't finished changing the value. We can obtain a notification that the data our control is bound to has changed, by listening for the xforms-value-changed event, as follows:

<xf:input ref="instance('inst-suggestions-rq')/qu">
  <xf:label>Suggestions for:</xf:label>
  <xf:action ev:event="xforms-value-changed">
    <xf:send submission="sub-suggest" />
  </xf:action>
</xf:input>

The event will be dispatched whenever the user changes the value of the control, which on GUI-based XForms processors will be when they tab out of the control or press [ENTER]. However, if we set the incremental hint we are saying to the processor that we're happy to receive more xforms-value-changed events, if they become available. On most processors this will mean that if the user pauses in their typing (or pauses whilst dragging a slider) then we want to know:

<xf:input ref="instance('inst-suggestions-rq')/qu" incremental="true">
  <xf:label>Suggestions for:</xf:label>
  <xf:action ev:event="xforms-value-changed">
    <xf:send submission="sub-suggest" />
  </xf:action>
</xf:input>

So, with the addition of one attribute XForms has made this powerful feature available to any author, without needing to know script, or have the level of resources that Google has.

As a quick way to see the results, I'll put them into a select1 control:

<xf:select1 ref="q" appearance="full">
  <xf:label>Search for:
  <xf:itemset nodeset="instance('inst-suggestions')/suggestion">
    <xf:label style="display: block;" ref="@query" />
    <xf:value ref="@query" />
  </xf:itemset>
</xf:select1>

The options available to the select1 are automatically changed as the new results come in. The following real-time Flash movie shows the results (you will need to click on the right-hand mouse button and select 'play'):



Note that if you want to try this code in formsPlayer, you'll get a better experience with the latest preview release of formsPlayer 1.3. Previous versions of formsPlayer will work, but incremental causes xforms-value-changed to be dispatched on every character typed. See the end of this blog for details about downloading the software, and obtaining the source.

Searching Google

The second half of the application was the actual searching of Google, based on one of the suggestions. This is pretty easy. I won't go into all of the details here, although there are some great features that you can take advantage of when you are using Google's Web Services API.

The main thing we need to do is create some instance data and a submission that will search our desired web service (in the case of the example, it's Google). To achieve this we modify our input control so that when the user presses [ENTER] we perform the search. The search itself is performed by copying the current value from the control, into the search instance, ready for passing to the search web service:

<xf:input ref="instance('inst-suggestions-rq')/qu" incremental="true">
  <xf:label>Suggestions for:</xf:label>
  <xf:action ev:event="xforms-value-changed">
    <xf:send submission="sub-suggest" />
  </xf:action>
  <xf:action ev:event="DOMActivate">
    <xf:setvalue
     ref="instance('inst-search-rq')/SOAP-ENV:Body/gs:doGoogleSearch/q"
     value="instance('inst-suggestions-rq')/qu"
    />
    <xf:send submission="sub-search" />
  </xf:action>
</xf:input>

I may drill into the part of the sample code that searches Google more in the future, but for now I think it is commented enough to explain what it does. The following Flash movie shows the completed form in action (you will need to click on the right-hand mouse button and select 'play'):

Conclusion

We've shown how XForms can be used to interact with two web services from one form. In this particular illustration one web service is used to provide values that will help provide input into the second web service. The first service could be a list of tags from deli.cio.us or technorati, used to search the second web service, Google. Or the first web service could search a list of part numbers (a list too large to put into a simple drop-box), that is then used to populate an invoice that will be submitted.

In addition to communicating with two web services, we've also shown how the XForms incremental feature can be used to provide ongoing feedback to the user.

If you want to look at the full code for this sample it is available here. If you want to run it in the Sidewinder Viewer, you can copy it to your Sidewinder Webapps folder and double-click. formsPlayer 1.3 and the Sidewinder Viewer are available from the formsPlayer download page.

Tags: xforms | webapps | ajax | javascript | programming | patterns | google