Design by Committee: Mashups

WSO2 Mashup Server heap space fix

Mon, 25 Aug 2008 19:35:50 GMT

Thanks for Tyrell for finally locating the source of the elusive Java heap space error, which I came across sporadically during my daily Mashup Server stress testing. Tyrell describes the fix that you can add to your 1.5.1 installation here.

"Why SOA Architects Should Care" about Enterprise Mashups

Tue, 12 Aug 2008 21:25:21 GMT

Can't help but repeat a few choice bits from a recent article Enterprise Mashups Part II: Why SOA Architects Should Care by Chris Warner and John Crupi. Too bad I didn't see this before the Enterprise Capabilities of the WSO2 Mashup Server 1.5 webinar I gave this morning - I could have cribbed a few bits of wisdom! Like these:

Mashups bring a “user” into the SOA mix. ... By having the business build mashups upon the foundation of your service-oriented architecture, they essentially become SOA champions without knowing it.

This is a point we tried to hit strongly in the WSO2 Mashup Server, which can help bring individuals, teams, or enterprises together into a community around services, and strengthen the concept of a "user" and help build a diverse set of user experiences tied to services. Until the SOA platform gets connected to the business users, the promises of SOA (business agility) won't be fully realized.

In addition to SOA-friendly formats, such as RSS and Atom plus REST and SOAP, mashup creators can publish mashups to spreadsheets, as WSRP-compliant portlets, wiki- and blog-friendly widgets, or even into a mobile phone as a micro-application. Mashups can become the vehicle through which services become part of the everyday tools of the enterprise business user.

Which is why we've added on to the RSS, Atom, REST, and SOAP support in 1.0 the ability to interact with spreadsheets and databases (through Data Services), and are starting our foray into the world of portlets, widgets, and gadgets with support for Google gadgets.

Mashups can go well beyond leveraging an SOA by becoming part of that SOA, allowing developers to create customized “service skins” from core services. ...
Because mashups can be exposed as REST-, WSDL- and JSON-based services, they look and feel like a real SOA-based service to developers who want to consume them. ... The mashup, created by the developer, becomes the tailored service which is directly aligned with their particular need. Major enhancements to core services can be accommodated with a reformulated or updated mashup by the mashup creators themselves.

Yes, we've found our users (and ourselves) using the WSO2 Mashup Server for more than just combining services into a new one, but for rapid development of new services, or for customizing services to a particular scenario. From the outside, these don't just look like real services - they are real services - for example in the 1.5 release high levels of security can be applied to the services even though they are so easy to write. A good example of a "skinned" service - the digit2image sample service that comes with the mashup server tailors a very general purpose and complicated service (Flickr) for a particular narrow purpose - to find a random image, with appropriate copyright, for a single numeric digit. By providing a narrow API, this capability can be exposed more easily and efficiently (some caching is performed by the service to speed up returns). And it allows innovation without touching the original Flickr service which we don't have control over. For instance, I could search beyond Flickr for appropriate images without breaking the service contract or rewriting the Flickr service.

Toward the end, the authors write:

At this point any SOA architect worth his WSDL should see that mashups can greatly enhance an SOA and don’t necessarily ignore or break the principles that made your SOA great. But governance, granularity and scope for mashups do require subtle tweaks and adjustments to your enterprise toolset.

Indeed, governance is important as your SOA grows and chaos theory has a chance to get a toehold. The WSO2 Mashup Server is taking the first steps into governance by relying on the WSO2 Registry to store mashups in a versioned repository, manage ownership, users and roles, and facilitate communication and the building of trust in an Enterprise SOA community. But we still have to to a better job of working with the Registry's features for supporting multiple versions, product lifecycles, and dependency management. For instance, while the Mashup Server stores all it's mashups, comments, tags, ratings, etc. in the Registry, when you deploy a new mashup, it doesn't automatically populate the registry with the WSDL. We're looking at much better integration with the Registry as a primary feature of our next Mashup Server release.

Kudos to the authors for clearly presenting not just the case for service mashups in an SOA architecture, but why any SOA architecture that doesn't expand into the mashup space is missing out on huge opportunities.

WSO2 Mashup Server 1.5 released!

Mon, 21 Jul 2008 16:18:59 GMT

I'm proud to say that v1.5 of the WSO2 Mashup Server is now available for download! This release has been a few months in the making, and I'm thrilled to see a major update to the Mashup Server finally out. News release here.

Thanks to the efforts of Keith, Tyrell, Channa, Yumani, and many others at WSO2, we've been able to put some snazzy new features in this release:

Integrated award-winning Data Services: quickly and with zero-coding map queries on your databases, Excel spreadsheets, or CSV files into Web services.
Google Gadgets: power a Google gadget with a mashup service - we generate a try-it, a code template for customizing the gadget, and even a build in Gadget dashboard based on Apache Shindig.
Security, security, security: Limit access to your mashup service to only authorized users, encrypt and sign your messages, and access services which are secured. I think our security interface is the easiest way to engage advanced WS-Security features that's out there!
OpenID login support.

There's lots of little improvements too, and I hope to talk about those, as well as the major features above, in upcoming posts.

For now though, a hearty congratulations and a good night's sleep to the Mashup Server team!

A scraping incident

Tue, 27 May 2008 16:40:41 GMT

One of my mashups fell prey to the dreaded scrape rot - a complete overhaul of the target site that invalidated all of my scraping rules. The pages in question are from globalincidendmap.com, which previously powered my internationalincident mashup (see Sri Lanka Incident Mashup). The change was catastrophic - queryable content from the site is no longer free, but requires a paid membership and a login process. One option would be to pay for a membership, but besides the steep price I doubt that the license terms allow republishing of the data. I could support the mashup only for paid users of the service, collecting credentials and forwarding them on, but that again is both questionably secure (a user would have to trust that I didn't abuse the credentials temporarily in my possession), and unrealistic since few if any of my audience would spring for the cost of membership. In effect, my mashup has been totaled.

This illustrates one reason why scraping should be used only as a last resort, when no more stable forms of content are available - feeds or Web services. When you mix the content and presentation, changes in the presentation are easily confused with changes in the content. Although the scraping features of the WSO2 Mashup Server are popular, I like to think of them as a stop-gap while publishers find cost-effective ways of serving up presentation-free content, such as delivering simple services using the Mashup Server ;-). Ideally, more and more publishers will recognize the value of raw content, and the need for Web scraping will diminish. Gonna take a while though...

Even without scraping, there remains one of the deep problems with mashups and distributed programming, that of services that disappear, are altered, change usage terms, etc., breaking their dependent mashups in the process. There has been lots written on this, which can be generally summed up as "this is a hard problem."

One thing we plan to do in the future to make sure that service changes don't harm downstream dependents is use more of the advanced functionality of the WSO2 Registry upon which the WSO2 Mashup Server is built - namely, versioning. Today, each time a service changes, an old copy is retained in the database, but no longer is "alive" as a service. Some future version will have a simple interface for continuing to keep the old versions online, and help users to lock into one of these previous versions. Some cool dependency management features on the drawing board for the Registry will also help find and record dependencies and notify dependents of changes.

But would these help in the case of the internationalincident service? This is a case where there is a deliberate change which prevents "unauthorized" access. The solution in this case was to mark the service as obsolete, and go out and find a whole new source of data. The new srilankanincident service is a result - though the data is slightly different, perhaps a result of focusing narrowly on Colombo, it was a fairly short task to reprogram it, and even improve it, once I had found a new source of data. The speed of fixing catastrophic failures is my current best hope against scrape rot.

Mashup Server Webinar May 13th

Fri, 09 May 2008 14:20:11 GMT

I've got another free Webinar coming up - again an Introduction to the WSO2 Mashup Server and to mooshup.com on 13 May from 9-10AM PST. Join me if you:

Are curious about mashups, Mashup Server products, and want an overview of the capabilities of WSO2's offering.

Have services, web pages, or other information sources available but you want smarter ways to use those services.

Always wanted an application to do (x) on the Web but it was always too costly to develop.

Know Javascript and want to see what it can do outside the browser.

Register now!

Webinar - A New Approach to Web Service Composition

Tue, 08 Apr 2008 20:01:17 GMT

I'm giving a free Webinar on 15 April 2008 9-10AM PDT about the approach we took to service composition in the WSO2 SOA Platform. Instead of a declarative approach which my XSLT days showed can be powerful yet also limiting in many ways compared with a full programming language, the WSO2 Mashup Server uses a "scriptable Web Services" metaphor, and supports the ability to consume and produce Web Services using simple Javascript expressions. Add to that the ability to script non-Web Service materials like Web pages and feeds, and you've got a powerful yet accessible platform for creating Web Service mashups. Register now!

Sri Lankan Incident Mashup

Tue, 01 Apr 2008 00:37:06 GMT

I just posted a finished version of a Mashup designed to help answer the question "is Sri Lanka getting safer or not?" This is a question we on the global WSO2 team ask each time we arrange travel to that unfortunately troubled country. Despite a spate of violence early this year, designed to coincide with the formal dissolution of the cease fire that has done little to prevent violence, it seemed to me things were getting a little better. But I needed some facts to back that up.

The mashup plots bombings and other incidents as a bar chart, measuring the severity of the incident (how many killed & wounded) over the last 18 months. The idea was to see if there was a clear overall trend in the violence or not, something not readily apparent from a Google map (see right).

The mashup service itself consists of several items, each one a simple task accomplished in half a page of javascript:

Scrape the search results page at globalincidentmap.com to pull the essential data out of an HTML table for the country of interest, and put it into a simple XML structure.
Cache the page if it has already been accessed within 24 hours (scraping is expensive, the first access in any 24 hour period will be pretty slow.)
Parse the headline for patterns such as "eight killed" or "injures 7" and turn that into killed and wounded digits (this isn't perfect, but we can tolerate a few errors since we're trying to present an overall sense of the problem rather than perfectly accurate statistics.) Also filter out as much as possible the killing of LTTE, as that's more a measure of war than of terrorism.
Provide a helper method to look up details and get a link to a new story for any item.

Using this service (called internationalindicent) I created a custom HTML UI to format Sri Lanka-specific results into the bar chart and to make it interactive (click on a bar to see more info about the incident.) Then I used "share this mashup" to upload the service to mooshup.com so others could try it out (or copy the code.)

The rough version took a couple of hours, mostly figuring out the details of scraping the page and coming up with the headline patterns to look for, but then I spent a couple more polishing the HTML UI so I wouldn't be embarrassed to share it. In the process I demonstrated some of the powerful aspects of using mashup technologies in your development arsenal:

Instead of investigating the current situation using Google News for 30-60 minutes each time we plan a trip for the latest information, I can browse this site in a few minutes, see the trend, and get details of any recent incidents I'm interested in. This will pay for itself in terms of my own personal productivity before long.
There is also a small user group (really, just the handful of WSO2 employees based outside Sri Lanka) who can also benefit from this micro-application, increasing their productivity as well.
The WSO2 Mashup Server makes the data available as a service, so others can reuse it too, for alternate displays or to generate displays for other parts of the world.
And, it's just fun!

So ... is Sri Lanka getting safer? I'll have to let you be the judge of that. Go to http://mooshup.com/services/jonathan/internationalincident/ to see for yourself!

[Update 5/27: This service is no longer available. See http://auburnmarshes.spaces.live.com/blog/cns!F985A6952BC07C4!813.entry for more info.]

Mashup Camp 6

Tue, 25 Mar 2008 18:31:52 GMT

Just returned from some travel which included 2 days of Mashup University and a day of Mashup Camp. A few thoughts:

Maps are still well represented. This surprised me a bit, but I recognize I've been well immersed in the WSO2 Mashup Server which supports a wide variety of User Interactions (html pages including maps, but also feeds, email, instant messaging.) I predict non-map mashups will start to eat into the map-based mashup market share dominance over the next year.
There's lots of interest in consuming various Web APIs. Some vendors were promoting their APIs, others tools for consuming those APIs. I think the WSO2 Mashup Server can tap into an underserved market here, since it's the easiest way I know of to deliver a comprehensive Web API on top of a bit of Javascript logic, which in turn can front information sources as diverse as databases and scraped web pages. I think the Mashup Server can become a "design your own API" tool that can have great appeal to the mashup developer.
A lot of Javascript was in evidence, further validating our choice to use Javascript for mashup logic in the WSO2 Mashup Server.
For the first time I ran into a few people with serious interest in WADL. Has its time finally come? More on that in a subsequent post.
The "unconference" style was quite interesting and successful, especially if you're like me and are more interesting in connecting with interesting individuals than in hearing yet another vendor pitch (mine excepted of course!) ;-) One thing that is sorely lacking is any kind of organization to the conference Wiki, and a surprising lack of ability to record much of the activity there. I couldn't even find who won the Best Mashup contest...

I can't wait till next year!

Mashing up a National Geographic Photo of the Day Feed

Fri, 26 Oct 2007 16:10:49 GMT

(This article first appeared a few days ago on the WSO2 Oxygen Tank.)

I recently wrote a neat little mashup which demonstrates a little of the power of the WSO2 Mashup Server to flow information from one place to another, and from one format to another. I had a simple set of requirements:

I use the Google Photos Screensaver to show a slideshow of interesting photographs when the family-room computer isn’t being used. Since the family room computer includes the 37-inch LCD screen as a separate monitor, high quality photos come out really clear and make for a nice, constantly changing design element for the room. It works best if the set of photographs changes before it gets old.
I recently found the National Geographic site’s “photo of the day” section as an interesting source of high-quality photographs that updates on a daily basis. However, National Geographic doesn’t provide a feed for the photo of the day.

Essentially then the task was to scrape the URLs from the photo of the day, and package them into a feed. The complication comes from the fact that there doesn’t seem to be a list of photos of the day available on the National Geographic web site – just links from a particular photo to the one for the previous (or next) day. Because a feed of 30 photos requires 30 different pages to be scraped, some caching really becomes necessary to improve performance, especially since feed readers can be expected to bombard the service if it proves popular.

I initially broke down the task into three parts:

Scraping a photo of the day page to extract the useful metadata, including the date, title, photographer’s credit, and description of the photo, a set of links to the actual image in various sizes, and links to the page being scraped (so one can return there easily) and to the previous page in the photo stream. Since this metadata shouldn’t vary, cache it locally for faster retrieval.
Searching the cache or going to the web site (and thus populate the cache) to acquire the metadata for a particular date.
Formatting the metadata for a particular range of dates into a feed.

Here’s how I approached each of these tasks.

Scraping a photo of the day page

The first order of business for scraping a page like this is simply to fetch the page, tidy it into XML so we can navigate it using tools like XPath. The WSO2 Mashup Server provides a “Scraper” object that accepts an XML language describing the steps involved in scraping. This configuration language is defined by the Web Harvest component that we use for scraping. I usually start with a scraping mashup using a simple function that configures and performs the scrape, and returns the results:

 
function scrape_picture_page() {
  var config =
    <config>
      <var-def name='response'>
        <html-to-xml>
          <http method='get' 
              url="http://photography.nationalgeographic.com/photography/ ¬
                   photo-of-the-day" />
        </html-to-xml>
      </var-def>
    </config>;
 
  var scraper = new Scraper(config);
 
  var bodyWithoutXMLDecl = 
      scraper.response.substring(scraper.response.indexOf('?>') + 2);
  var result = new XML(bodyWithoutXMLDecl);
 
  return result;
}

The config language itself is pretty straightforward, once you learn to read it inside out – the <http> element fetches the requested URL, the <html-to-xml> does just what it sounds like and tidies the result, which is put into a variable named “response”. The scrape is performed by initializing a new “Scraper” object with the config, and the result is made available through the “response” property on the result – corresponding to the “response” variable we defined within the config file. One trick though – the result is a stream of XML text, including an XML declaration. The E4X extensions can parse this into XML (new XML()), but can’t handle the XML declaration. We have to strip off the declaration ourselves using string manipulation.

By placing the above function in a file named “nationalgeographic.js” in the “scripts” directory of the Mashup Server, a Web service with a scrape_picture_page operation will be deployed. We can get to it through the try-it page (http://localhost:7762/services/jonathan/nationalgeographic?tryit) and see what the tidied HTML looks like for the page.

Extracting the data from the page can be a tedious process, involving looking at HTTP request-response pairs and trolling through the HTML source of a page. Fortunately the National Geographic site’s HTML is simple and straightforwardly structured, with a number of well-placed identifiers to help us zero in on the interesting content. I usually end up using Firebug (Firefox debugging extension) to navigate the live HTML of the page and develop some XPath expressions that extract the desired metadata for the page. I’ve also found that, since Web Harvest communicates between components using strings rather than parsed XML, that defining a lot of XPath filters to extract information one element at a time during a scrape can perform poorly. Instead it seems much faster to wrap a series of XPath expressions into a simple XSLT stylesheet so the XML can be parsed once, queried as much as needed, and an XML structure containing the results returned in one action. To do that, I added an XSLT stylesheet to the above configuration:

  var config =
    <config>
      <var-def name='response'>
        <xslt>
          <xml>
            <html-to-xml>
              <http method='get' url={url} />
            </html-to-xml>
          </xml>
          <stylesheet><![CDATA[
           <xsl:stylesheet version="1.0"
                            xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
              <xsl:output method="xml" omit-xml-declaration="yes"/>
              <xsl:template match="/">
               <photo>
                  <xsl:for-each select="//*[@id='content-center-well']">
                    <date><xsl:value-of select="div[@class='date']"/></date>
                    <previous>http://photography.nationalgeographic.com ¬
                      <xsl:value-of 
                          select="div[@class='slide-navigation'][1]/p/a/@href"/>
                    </previous>
                    <xsl:for-each select="div[@class='image-viewer clearfix']">
                      <xsl:for-each select="table/tbody/tr[1]/td/a">
                        <page>http://photography.nationalgeographic.com/photography/¬
                          photo-of-the-day/<xsl:value-of
                          select="substring-before(substring-after(@href,'enlarge/'),
                          '_pod_image.html')"/>.html</page>
                        <xsl:variable name="href" 
                          select="concat('http://photography.nationalgeographic.com', 
                                substring-before(img/@src, '-ga.jpg'))"/>
                        <location type='small'>
                          <xsl:value-of select="$href"/>-ga.jpg</location>
                        <location type='medium'>
                          <xsl:value-of select="$href"/>-sw.jpg</location>
                        <location type='large'>
                          <xsl:value-of select="$href"/>-lw.jpg</location>
                        <location type='wide'>
                          <xsl:value-of select="$href"/>-xl.jpg</location>
                      </xsl:for-each>
                                               
                      <xsl:for-each select="div[@class='summary']">
                        <title><xsl:value-of select="h3"/></title>
                        <credit><xsl:value-of select="p[@class='credit']"/></credit>
                        <description>
                          <xsl:copy-of select="div[@class='description']/node()"/>
                        </description>
                      </xsl:for-each>
                    </xsl:for-each>
                  </xsl:for-each>
                </photo>
            </xsl:template>
            </xsl:stylesheet>
          ]]></stylesheet>
        </xslt>
      </var-def>
    </config>;

Again, fairly straightforward – the <xslt> task has two inputs, <xml> and <stylesheet>. The stylesheet unfortunately has to be enclosed in a CDATA section rather than as straight XML. One other nice trick though – when the output is an XSLT template, the “omit-xml-declaration” flag can be used to strip off the XML declaration so we don’t have to do it through text manipulation, simplifying and accelerating our Javascript code.

So we’re almost there with this capability. Some minor improvements and adding caching are all we need:

Add an optional “url” parameter to allow this page to work on any photo-of-the-day page URL. Using E4X’s curly braces we can substitute this value right into the config file.
If the result was successful (e.g. the <photo/> element has children), calculate the date in yyyy-mm-dd format and use the storexml service to cache it – choosing a path unlikely to conflict with other users of the storexml service. To make the storexml service easy to call, we import it’s stub, which I got from http://localhost:7762/services/system/storexml?stub&lang=e4x&localhost=true and saved into the nationalgeographic.resources folder which serves as the sandbox for this service. It’s important to save a copy because when the Mashup Server boots up the nationalgeographic service might be deployed before the storexml service – attempts to generate the stub at that time will fail and cause the nationalgeographic service to fail too. The Mashup Server doesn’t yet track these dependencies (and we’re still thinking about whether this is a tractable problem or not.)
Since this operation isn’t really supposed to be called by end-users of the feed, I could make it private using scrape_picture_page.visible = “false”, but instead I’ve just used the “operationName” property to rename it, indicating to users that it really is just for test purposes.
Add type annotations.
Add documentation annotations (not shown below.)

system.include("storexml.stub.js");
var cachePath = "nationalgeographic/cache/";
 
scrape_picture_page.operationName = "test_scrape_picture_page";
scrape_picture_page.inputTypes = {"url" : "xs:string?"};
scrape_picture_page.outputType = "xml";
function scrape_picture_page(url) {
  if (url == null)
    url = "http://photography.nationalgeographic.com/photography/photo-of-the-day";
  var config =
    <config>
      <var-def name='response'>
        <xslt>
          <xml>
            <html-to-xml>
            <http method='get' url={url} />
            </html-to-xml>
          </xml>
         <stylesheet>
            ...
          </stylesheet>
        </xslt>
      </var-def>
    </config>;
 
  var scraper = new Scraper(config);
 
  var result = new XML(scraper.response);
   
  if (result.hasComplexContent()) {
    var date = xsDate(new Date(result.date));
    storexml.store(cachePath + date, result);
  }
 
  return result;
}

xsDate.visible = false;
function xsDate(d)
{
  return d.getUTCFullYear() + "-" +
       (d.getUTCMonth() < 9 ? "0": "" ) + (d.getUTCMonth() + 1) + "-" +
         (d.getUTCDate() < 10 ? "0": "" ) + d.getUTCDate();
}

As an aside, this shows a couple of my wishes:

<xml> is a reserved tag name in XML, it’s unfortunate that Web Harvest doesn’t use something else.
Web Harvest’s requirement that the stylesheet be enclosed in a CDATA section is unfortunate – it means well-formedness errors can’t be caught at Javascript/E4X compile time, but at runtime. This slows down the development process. I could put the stylesheet in a separate file, but that just makes it harder to share the service and see what’s going on.
I’d prefer a way to get E4X XML back from Web Harvest directly so I wouldn’t have to parse it myself, worry about the XML Declaration, and so forth. Maybe we can do something about this in a future release.
Managing date formats becomes a bit of a chore. I prefer the operations and cache to work on xs:date format (yyyy-mm-dd), but the page metadata is in the form “Month day, year” (directly from the scraped page). And Javascript prefers to manipulate dates in its own Date object. Soon we’ll see that the RSS profile defines a subset of the Javascript serialization that means a fourth conversion.

Finding a picture for a particular date

Now that we have a function that can scrape a page given a URL, and given that the data returned and cached by that function contains a link to the page for the previous day’s page, we can do some walking around in the cache to find data for a particular date. That’s what this function does.

First, we look in the cache for a photo’s metadata. If it’s there, we can simply return it – we’re done. Otherwise we need to find the URL for the page representing that date and call the scrape_picture_page operation.

If I can’t find the requested date in the cache, I look for the next earlier date, and so on, until I do find a photo in the cache (or I reach today’s date). That’s the first while loop. Then, using the <previous> page url, I work backward again, incidentally populating the cache as I go, until I’m back to the date I was looking for. The couple of “if” statements look for exceptional conditions: the first one handles the case where I’ve looked all the way forward till today but still haven’t found anything in the cache, and the second makes sure that if a page can’t be scraped for some reason that we give up and return what little we have before we dig ourselves any deeper.

picture_for_date.inputTypes = {"date" : "xs:string"};
picture_for_date.outputType = "xml";
function picture_for_date(date) {
  try {
    return storexml.retrieve(cachePath + date);
  } catch (e) {
    print("failed to find cached photo for date " + date);
    var photo;
    var startDate = parseDate(date);
    var today = new Date();
    // work forwards in the cache until we find something (or hit today)
    while (startDate <= today) {
      try {
        photo = storexml.retrieve(cachePath + xsDate(startDate));
        break;
      } catch (e) {
        startDate.setUTCDate(startDate.getUTCDate() + 1);
      }
    }
    // start with the most current thing in the cache (if any) an work 
    // backwards to the requested date, filling in the cache as we go...
    var targetDate = parseDate(date);
    while (startDate > targetDate) {
      var previousPageUrl;
        if (photo == null) previousPageUrl = null;
        else previousPageUrl = photo.previous;
       
        print("fetching photo for " + startDate);
        photo = scrape_picture_page(previousPageUrl);
        if (!photo.hasComplexContent())
          break;
        startDate.setUTCDate(startDate.getUTCDate() - 1);
    }
       
    return photo;
  }
}

Generating the feed

Now we have all the pieces in place to aggregate the data and generate a list of some kind as output. The picture_of_the_day operation does that for us.

The function has some parameters controlling aspects of the feed – whether to link to the small, medium, large, or wide aspect ratio images, and how many items to include. If no number is specified, we generate a feed of the latest 30 photos – just long enough to enjoy the photo but not so long we get tired of it.

The WSO2 Mashup Server has a Feed object to help construct feeds, but because I’m targeting this feed at the Google Photos Screensaver I need to include some feed extensions that aren’t supported in the 0.2 release (though they’ve just been added to the nightly build!). It’s not hard to create an RSS by hand though, so that’s what I chose to do. First I prepopulate the channel with title, links, and description, and then loop through the photos adding an item for each of them. The first time through the loop, I also add in a <pubDate> reflecting the date of today’s photo.

Again, this isn’t rocket science – the hardest thing is simply to format the dates appropriately. During the loop I use Javascript Date objects to increment days and tick over at the end of the month. I convert that to an xs:date to access the cache, to an RSS Profile-conformant string for the <pubDate>, and to an xs:dateTime for use in the <atom:published/> element, which seems useful for the subscription page displayed in Internet Explorer 7.

picture_of_the_day.inputTypes = 
     {"size" : "small | medium | large | wide", "numPhotos" : "number?"};
picture_of_the_day.outputType = "#raw";
function picture_of_the_day(size, numPhotos) {
  if (numPhotos == null) numPhotos = 30;
 
  var feed =
    <rss version="2.0">
      <channel>
        <title>National Geographic Picture-of-the-day (from WSO2 Mashup Server)</title>
        <link>http://mashups.wso2.org/services/nationalgeographic/¬
              picture_of_the_day?size={size}</link>
        <description>WSO2 Mashup Server mashup acquiring and caching links to the ¬
          National Geographic Picture of the Day 
          (http://photography.nationalgeographic.com/photography/picture-of-the-day),
          and exposing them as a feed.  Sizes of "small", "medium", "large", and "wide"
          are available. A max number of photos can be specified with the "numPhotos" 
          parameter.</description>
      </channel>
    </rss>;
 
  var startDate = new Date();
  var photo, photoDate, url, urlsmall, entry;
  for (var i = 0; i < numPhotos; i++) {
    photo = picture_for_date(xsDate(startDate));
    if (photo.hasComplexContent()) {
      url = photo.location.(@type == size).toString();
      urlsmall = photo.location.(@type == 'small').toString();
      photoDate = new Date(photo.date.toString());
      if (i == 0) {
        feed.channel.appendChild(<pubDate>{rssDate(photoDate)}</pubDate>);
      }
      entry = <item xmlns:media="http://search.yahoo.com/mrss/">
          {photo.title}
                <description>
                  &lt;a href='{url}'>&lt;img src='{urlsmall}'/>&lt;/a>
                  {photo.description.*.toXMLString()}
                </description>
                <pubDate>{rssDate(photoDate)}</pubDate>
                <link>{photo.page.toString()}</link>
                <guid isPermaLink='false'>{photo.page.toString()}</guid>
                <media:content url={url} type="image/jpeg" />
                <media:thumbnail url={urlsmall} />
                <atom:published xmlns:atom="http://www.w3.org/2005/Atom"
                    >{xsDate(photoDate)}T00:00:00Z</atom:published>
              </item>;
      feed.channel.appendChild(entry);
    }
    startDate.setUTCDate(startDate.getUTCDate() - 1);
  }
  return feed;
}

You can access this operation through the try-it page at http://localhost:7762/services/jonathan/nationalgeographic?tryit and see that the operation returns a feed. However, the try-it uses SOAP by default under the covers, which isn’t terribly friendly to feed readers like the Google Photos Screensaver. No problem – the Mashup Server also exposes it’s operation through a REST interface. By accessing the URL http://localhost:7762/services/jonathan/nationalgeographic/photo_of_the_day?size=wide, you can see the feed directly in the browser, point the screen saver at it, subscribe to it, etc. By adjusting the “size” and “numPhotos” parameters you can generate variants of the feed that suit your purpose.

Publishing the feed

Once I had the service written, tried it for a day or two to ensure it was stable (and fixed a couple of edge cases as a result), I used the administrative UI in the Mashup Server to publish it to http://mooshup.com, which hosts the service live on the internet for others to use. The publishing process is simple – click the share button, confirm that http://mooshup.com is the destination, and click OK. While we have lots to do to make this site an attractive and useful place for members of the mashup community to hang out, it does give me a stable internet URL for the feed (for example http://mooshup.com/services/jonathan/nationalgeographic/picture_of_the_day?size=wide) so others can enjoy it. You can exercise the try-it page live from there, look at the metadata, or download the service to your local installation of the Mashup Server and run it there.

Last Word

Hopefully this helps you get a feel for the Mashup Server in action. We did some screen scraping, fairly sophisticated caching by invoking an external storexml Web service, formulated an RSS feed, and made it (and intermediate functions) available through a Web service including SOAP 1.2, SOAP 1.1, and HTTP bindings, including an HTTP GET binding amenable to RSS agents. Although we didn’t look at them in detail in this article, the Mashup Server generated a try-it page for debugging and exercising the service, WSDL, Schema, stubs for accessing the service simply from Javascript or E4X environments, even generated some human-readable documentation for the mashup. We ran the service locally, then published it live onto the internet. It also would not be hard to generate a custom HTML interface providing (for example) a slideshow of these photos, but in this case I wanted to show that user interfaces can go beyond just HTML pages by using Google Photos Screensaver as my ultimate user interface.

So what’s next for this service? The main improvement I can think of is rewriting the code to use the Feed object when it becomes capable of handling the images. It took me a while to figure out which RSS extensions were necessary and it would be nice not to worry about the representation of dates. Maybe I could even offer an Atom feed in parallel. Another idea related to performance would be to experiment with a different, perhaps additional, caching strategy – which is to cache the entire feed to disk and periodically refresh it using the recurrence capabilities of the mashup server. But those are perhaps good topics for future articles!

Until then, enjoy the great photos available from National Geographic!

[Updated 6 Feb 2008 - added "jonathan" user to endpoint urls as required by the Mashup Server 1.0 release, and changed the online links to point to http://mooshup.com.]