Design by Committee: Blog

Mooshup blazes

Mon, 25 Aug 2008 19:50:05 GMT

Keith upgraded mooshup.com to the 1.5.1 release. I used to find it pretty sluggish, but no longer! Try it out...

WSO2 Mashup Server heap space fix

Mon, 25 Aug 2008 19:35:50 GMT

Thanks for Tyrell for finally locating the source of the elusive Java heap space error, which I came across sporadically during my daily Mashup Server stress testing. Tyrell describes the fix that you can add to your 1.5.1 installation here.

The Open Source Social Contract

Fri, 22 Aug 2008 22:33:43 GMT

Now that I've had the opportunity to observe the Open Source industry from the inside for a couple of years, I find myself musing a fair bit about the economics - broadly speaking the exchange of value - involved. Much of the customer appeal for open source solutions comes from the sticker price, which is generally zero. But as the adage goes, if something sounds too good to be true, it generally is. If open source software is to be high quality and broadly applicable, the customer demand for low-cost software needs to be matched with incentives for vendors and individuals to continue to produce high-quality software. When the transaction isn't based on actual currency, what is the commodity that makes this a successful transaction for both sides?

For individual open source authors the incentives might include the joy of having a large and loyal user base. It might include fame and the development of skills that lead to greater personal satisfaction and a more impressive and marketable resume. But I'd like to focus here on the incentives for professional software development businesses to invest in producing more and better open source software.

The research, development and deployment of software is a complex and costly process. How are open source vendors able to accomplish these goals without getting licensing revenue in return? I believe there are many ways a vendor can receive value from a user besides an exchange of currency. Although the commodities exchanged aren't tangible the exchange is rarely a zero-sum game, and can enrich the supplier without depriving or depleting the customer. No matter how many smiles you give, you never run out.

Enforcement of non-tangible exchanges is impossible, and thus the exchange relies on the good will of the customer to give back. My intention in this post is to enumerate some of the ways this non-monetary economy works, and I hope to encourage users to participate more fully and consciously in holding up their end of the transaction, and thus to perpetuate a virtuous cycle of open source software development. If you use open source software, please consider one of the following ways you can remunerate the creator.

What the user gives		How the creator benefits
Tell the author whether you liked the product or not	=	Reduced cost of soliciting customer feedback, ability to target new features more cost-effectively
Tell a friend or blog about it	=	Reduced awareness-marketing costs
Rating the technology positively on sites Digg or Ohloh	=	Reduced awareness-marketing costs
Lend an eyeball to a promotion or advertisement	=	Makes marketing expenditures more productive
Become a registered user	=	Reduces costs of contacting users, helps accurately judge the popularity of the product and thus the level of continuing investment
Ask a question on the mailing list	=	Reduces costs of getting customer feedback
Answer someone else's question on the mailing list	=	Reduces general product support costs
Write an article or blog about creative uses of the product	=	Reduces documentation and marketing costs
File a bug	=	Reduces QA costs
Send a patch	=	Reduces development costs
Implement a new feature	=	Reduces development costs
Download additional products	=	Reduces marketing costs and strengthens the business
Consider purchasing other products or services from the author	=	Improves profitability and increases ongoing R&D
Be grateful for the software	=	Increases everyone's karma ;-)

The laws of economics state that the more rewards there are for a product or service, the more of that product and service will be produced. By increasing the rewards for vendors to create useful and high-quality open source software, you encourage more of that software in the future. Isn't that worth an investment of a little time? It doesn't even lighten your wallet!

New toy :-D

Fri, 22 Aug 2008 17:41:30 GMT

As I posted back in 2005:

The US automakers don't seem tapped into [the fuel economy] trend at all. They still seem to think circumventing mileage minimums by pumping out SUVs is the way to sustainable revenues. Last week Ford and GM were put on notice that they were wrong. At least the blue half of this country, and I suspect lots of export markets, are willing to invest their automobile acquisition budget in a choice that reduces pump costs, unsightly and unhealthy smog, and reduces our dependence on foreign oil, and maybe even get a bit of value appreciation while they're at it. They're even more motivated to vote with their dollars since their election votes haven't provided much of a visible return. Yet despite plenty of urging by the environmental community, Ford and GM seem to have ignored the inevitabilities of the long-term. More and more of those purchasing dollars will head straight to Japan. I suspect the next 15 years could be pretty rough as our automobile designers adapt.

High fuel prices have accelerated the timeline beyond what I had imagined, and Prius sales have accordingly boomed.

I'm proud to at last announce I've joined those voting with their wallets for fuel efficiency and low emissions (and unfortunately against our domestic automakers) with my new purchase!

Let's hope American competitiveness is up to the challenge, and hope that they do a much better job of recognizing and capitalizing on long-term trends in the future.

"Spontaneous Reflections" Podcast Launched!

Wed, 13 Aug 2008 13:51:15 GMT

Today I'm launching my new musical podcast of short piano or keyboard improvisations. I call it Spontaneous Reflections, and am planning to update it approximately weekly. It's available free in the iTunes store or you can subscribe directly here.

For those of you who don't know, I started my career in music school and playing keyboard in the showrooms of Reno, but found the vocational aspects much less exciting than I hoped, and switched over to a more intellectually and financially rewarding career in computer software.

After a long period of playing primarily pipe organ, a couple of years ago I returned to piano and keyboard and though I don't have my 20-year old chops (and pipe organ tends to ruin your sense of rhythm) I am exploring some new directions in my improvisation, really trying to dig into the instrument as a percussion instrument.

Live music is a funny thing though - each note goes out into the Ether and dissolves, gone forever except as ghosts in memory. And I find that playing is quite a different experience than listening, heavily colored with the subjective. To grow further I thought it valuable to be able to listen to some of my improvisations and see what works and what doesn't, and to help me practice more discipline about the overall shape of a piece. This podcast thus is an audio journal, not polished or refined, but genuinely spontaneous.

I hope you enjoy it!

"Why SOA Architects Should Care" about Enterprise Mashups

Tue, 12 Aug 2008 21:25:21 GMT

Can't help but repeat a few choice bits from a recent article Enterprise Mashups Part II: Why SOA Architects Should Care by Chris Warner and John Crupi. Too bad I didn't see this before the Enterprise Capabilities of the WSO2 Mashup Server 1.5 webinar I gave this morning - I could have cribbed a few bits of wisdom! Like these:

Mashups bring a “user” into the SOA mix. ... By having the business build mashups upon the foundation of your service-oriented architecture, they essentially become SOA champions without knowing it.

This is a point we tried to hit strongly in the WSO2 Mashup Server, which can help bring individuals, teams, or enterprises together into a community around services, and strengthen the concept of a "user" and help build a diverse set of user experiences tied to services. Until the SOA platform gets connected to the business users, the promises of SOA (business agility) won't be fully realized.

In addition to SOA-friendly formats, such as RSS and Atom plus REST and SOAP, mashup creators can publish mashups to spreadsheets, as WSRP-compliant portlets, wiki- and blog-friendly widgets, or even into a mobile phone as a micro-application. Mashups can become the vehicle through which services become part of the everyday tools of the enterprise business user.

Which is why we've added on to the RSS, Atom, REST, and SOAP support in 1.0 the ability to interact with spreadsheets and databases (through Data Services), and are starting our foray into the world of portlets, widgets, and gadgets with support for Google gadgets.

Mashups can go well beyond leveraging an SOA by becoming part of that SOA, allowing developers to create customized “service skins” from core services. ...
Because mashups can be exposed as REST-, WSDL- and JSON-based services, they look and feel like a real SOA-based service to developers who want to consume them. ... The mashup, created by the developer, becomes the tailored service which is directly aligned with their particular need. Major enhancements to core services can be accommodated with a reformulated or updated mashup by the mashup creators themselves.

Yes, we've found our users (and ourselves) using the WSO2 Mashup Server for more than just combining services into a new one, but for rapid development of new services, or for customizing services to a particular scenario. From the outside, these don't just look like real services - they are real services - for example in the 1.5 release high levels of security can be applied to the services even though they are so easy to write. A good example of a "skinned" service - the digit2image sample service that comes with the mashup server tailors a very general purpose and complicated service (Flickr) for a particular narrow purpose - to find a random image, with appropriate copyright, for a single numeric digit. By providing a narrow API, this capability can be exposed more easily and efficiently (some caching is performed by the service to speed up returns). And it allows innovation without touching the original Flickr service which we don't have control over. For instance, I could search beyond Flickr for appropriate images without breaking the service contract or rewriting the Flickr service.

Toward the end, the authors write:

At this point any SOA architect worth his WSDL should see that mashups can greatly enhance an SOA and don’t necessarily ignore or break the principles that made your SOA great. But governance, granularity and scope for mashups do require subtle tweaks and adjustments to your enterprise toolset.

Indeed, governance is important as your SOA grows and chaos theory has a chance to get a toehold. The WSO2 Mashup Server is taking the first steps into governance by relying on the WSO2 Registry to store mashups in a versioned repository, manage ownership, users and roles, and facilitate communication and the building of trust in an Enterprise SOA community. But we still have to to a better job of working with the Registry's features for supporting multiple versions, product lifecycles, and dependency management. For instance, while the Mashup Server stores all it's mashups, comments, tags, ratings, etc. in the Registry, when you deploy a new mashup, it doesn't automatically populate the registry with the WSDL. We're looking at much better integration with the Registry as a primary feature of our next Mashup Server release.

Kudos to the authors for clearly presenting not just the case for service mashups in an SOA architecture, but why any SOA architecture that doesn't expand into the mashup space is missing out on huge opportunities.

WSO2 Mashup Server 1.5.1 release

Fri, 08 Aug 2008 18:17:48 GMT

As announced on the mooshup blog, the WSO2 Mashup Server team completed a new release, which includes 50 bug fixes. Most of these are small, but a few are worthy of note:

Issues with Email host object in v1.5 fixed.
Private key management features.
Fixes in the SMTP and JMS transports (Keith is working on an article about how to enable them.)
WSRequest host object now supports a fire-and-forget message exchange pattern.
Supporting http proxies more consistently (e.g. in the Feed object).
Lots of little UI nits and polish.

I'm very pleased with this release, as it is often the case that some of the fit and finish work gets postponed in the last stages of a major release like 1.5. Open source is about rapid and continual improvement, and this release fits that bill nicely!

Download it today!

Upcoming Webinar: Enterprise capabilities in the WSO2 Mashup Server 1.5

Tue, 05 Aug 2008 16:29:46 GMT

As announced here:

Title: Enterprise Capabilities in the WSO2 Mashup Server 1.5
Date: August 12, 2008
Start Time: 9.00 am PST
Duration: 60 min
Presenter: Jonathan Marsh

The WSO2 Mashup Server provides a powerful platform for developing and deploying mashups. In this Webinar, we'll provide a brief introduction to the Mashup Server architecture and Javascript-based development model, as well as highlight the new features in the 1.5 release that support enterprise deployments. These include support for message level security (authentication, encryption, signing) on mashup services, and the ability to expose a data source such as a relational table, an Excel spreadsheet, or a simple CSV file as a zero-coding Data Service.
Attend this Webinar if you:

You have services, web pages, or other information sources available but you want smarter ways to use those services.
You want to expose or compose services in a simple yet powerful way, complete with enterprise-level security.
You are an existing WSO2 Mashup Server user and would like an overview of the new features in the 1.5 release.

Your presenter, Jonathan March, is the Director of Architecture for Mashup Technologies, and led the design and development of the WSO2 Mashup Server. Jonathan, a product designer and devoted script hacker, has spent over ten years developing and standardizing, XML and Web service technologies, most recently at Microsoft where he contributed to the standardization of XML, XSLT, XPath, WSDL, and other technologies.

Register now!

A Confluence of Fortuitous Circumstances

Tue, 05 Aug 2008 01:40:48 GMT

1) Project Auburn, a celebration of civic pride and elbow grease, in addition to renovating the classic State Theatre, has renovated the rear garden of OLAS, making it a great site for sculpture, relaxing, and entertainment. I helped put the fence up and did some of the repainting a few weeks ago when hundreds of volunteers turned out to make Auburn a better place. Kudos to the Rotary Club and other civic organizations for making our town a better place!

2) The Auburn Art Walk, a recurring evening of open studios and galleries, occurs every Second Thursday throughout the summer.

3) OLAS is celebrating the new landscaping, in conjunction with the Art Walk, with a show entitled From Earth To Sky, August 14th from 6:00 to 9:00 PM.

4) I acquired a new keyboard, a Yamaha CP300, at long last giving me some mobility to my music. (And making it easier to get quality recordings, more about which later.)

The outcome of all these fortuitous circumstances?

Jason and I will be playing our usual eclectic mix of improvisatory world/new age/jazz/undefinable music for the event.

If you're in the neighborhood, come on by!

Mashups that work despite cross-site scripting (XSS) browser restrictions

Sat, 02 Aug 2008 01:22:49 GMT

What is XSS and why is it restricted?

[Note: also published on the Mooshup.com blog.]

Disclaimer: I'm not a security guru, so what follows is my opinion, observation, and experience. Please feel free to comment and correct!

Modern browsers protect against release of private information to a third (possibly malicious) party by imposing “cross-site scripting” or “XSS” restrictions. The basic attack is that a browser is pointed to a web site that is trusted (to some degree) by the user. In using the web site the user may provide some confidential information to that web site, such as a password, bank account number, or some other information that could be exploited by a malicious entity. Naturally, the user would hesitate to provide this information to a site that he doesn’t trust (the user can be fooled through a set of techniques known as phishing, but that’s a different story.) Here we assume that the user has gone to a site that is authentic.

However, it is possible that even a trustworthy site, through poor construction or through compromised delivery mechanisms, could be “hacked” by a third party. For instance, accessing the site through an open (but malicious) wireless network may allow the page to be subtly changed during transmission. This change might be to insert a bit of script code that records the interactions the user has with the page, including information he enters such as a password, and also information that is provided by the website to the user. The inserted script could collect this private information, and then "phone home" to the attacker. HTTPS can mitigate such attacks by securing the communication channel, but interactions with plain HTTP sites may still disclose user secrets of various levels of sensitivity.

Attacks can also come, and generally do come, over a trusted internet connection, even possibly through HTTPS. Anytime user-generated content appears in a page (e.g. comments on a blog, etc.) there is a possibility that third party, and thus untrustworthy, content is piggybacked on a trusted site. Plain text third-party content is benign (what you see is what you get), but if the content can be submitted in html, it is possible that such html can include malicious scripts. For this reason, a trustworthy site that allows user-generated content must scrub any user-generated content provided to it, removing anything that could be executed as script.

From the perspective of browser vendors there are a lot of sites out there and not all of them consider the security implications of user-generated content adequately. To help protect the user, XSS counter-measures in the browser attempt to limit the ability of and scripts within the page to “phone home”. The is accomplished by preventing HTTP POST (the protocol used to submit forms and upload data) access to any web site domain called on by the page other than the one from which the main page originated. For instance, a page from http://wso2.org can access “safe” content from alternate domains like http://wso2.com, including images, stylesheets, even script libraries. The page however won’t be able to post a form containing user input to anywhere but http://wso2.org. The XMLHttp object provides a way to POST from script, and also prevents information from being posted to any domain other than the page domain.

So as a user, having a browser watch out for these types of attack and prevent them seems useful. But let's consider situations where they get in the way of useful, trustworthy work.

Why a developer might want to access alternate domains

For a web developer (especially a mashup developer) XSS can be quite a pain, as it limits your ability to write a page that spans domains. It limits your ability to host AJAX and Web Service interactions (powered by the XMLHttp object) anywhere other than your primary domain. For instance, you can't host a Web service on mooshup.com and use it within your own application (at least directly from the browser). Even though both sites may be trusted by you as the web site author, the browser enforces a blanket restriction on this access. (Each browser has mechanisms that may loosen this in some circumstances, but there isn’t anything with zero-config or cross-browser.)

This restriction limits applications such as gadget pages (e.g. iGoogle.com) that aggregate information from a large number of sources. The Google Gadget framework, for instance, provides a way to GET information through a proxy on the trusted server, but currently disallows similar capabilities for POSTing.

The Loophole - Script Injection

Don’t start feeling too secure as a user, or too disappointed as a developer trying to do legitimate work – there are some loopholes that can be exploited.

As described above, an HTTP GET operation is assumed to be safe across domains, while HTTP POST is not. If one could masquerade a POST as a GET, one could circumvent security restrictions. In particular, script can be fetched regardless of domain. This powers important functionality, such as third party libraries, an important feature supporting simplified development, analytics, and advertising. Basically one needs to translate the body of the POST into url parameters on the GET (recognizing there are length and encoding issues to deal with), insert a <SCRIPT> tag dynamically into the page which uses the GET, and the server on the external domain can access the "posted" information. It can even send a response back in the form of a block of script (essentially a callback). Of course, you need to insert script into the page initially to get the ball rolling - which can be pretty difficult over a secure connection or for sites that properly sanitize user-generated content. But if you're the owner of the original site, it's not terribly difficult once the technique has been mastered.

Naturally, a user can protect themselves reliably against these attacks by turning off Javascript in the browser. Or cutting the internet wires. Or burying the computer in the backyard and raising carrier pigeons. All quite practical alternatives, don't you think?

WSO2 Mashup Server 1.5

The new WSRequest.js in the new WSO2 Mashup Server 1.5 release has facilities to “exploit” this loophole when the developer wants to access a mashup from a page within another domain. If an "access denied" error is returned from the XMLHttp object, a script injection is attempted instead. This allows you to use the Mashup Server's convenient stubs within a page or a gadget without encountering the XSS pain. There are some restrictions in the fine print of course - only asynchronous calls are allowed, message size is limited, and the wire-level messages no longer are conformant to open standards for Web services, but those aren't unreasonable considering the alternatives.

Conclusion: time to drop XSS restrictions?

So, my question is, if XSS restrictions are so painful, yet circumvented with a modest bit of work (hey I'm no genius at this stuff and I did it) why are the XSS restrictions in place at all? Instead of trading off convenience for security, you're imposing convenience without actually making a meaningful contribution to the user's security. The additional level of security provided by making cross domain access simply obscure rather than truly prohibited doesn't seem worth it. Is it time to dump XSS restrictions? Or do we need to add a new (and further inconvenient) restriction against inserting <SCRIPT> tags into a page dynamically? As long as there is any cross-domain access I don't think I'll be completely secure. And that rules out advertisement insertions which I don't think is going to happen anytime soon!

WSO2 Mountain View - surrounded by history

Tue, 29 Jul 2008 23:25:09 GMT

Some of our Sri Lankan visitors wanted to see a bit more of the Silicon Valley than the inside of our conference rooms where we've been working all week. Follows is a little driving tour I took them on to see some of the local sights, both large and small.

WSO2 Mountain View office: Aspiring to someday be part of Silicon Valley lore, a fitting place to start when looking in the rear view mirror of history. http://wso2.com
Fairchild Semiconductor: After the invention of the IC at Shockley Laboratories, mass defections spurred the development of the integrated circuit by the eventual founders of Intel Corporation. http://en.wikipedia.org/wiki/Fairchild_Semiconductor
The GooglePlex: Formerly the site of Silicon Graphics campus, Google is the current symbol of Silicon Valley entrepreneurial success. http://en.wikipedia.org/wiki/Google_Campus
The Computer History Museum: Ironically located in the former Silicon Graphics Executive Offices building, vacated not long after occupation - a poster child for the valley's perpetual boom-bust cycle. http://en.wikipedia.org/wiki/Computer_History_Museum
Apple Computer: Famous for bringing computer technology to the general public through a continuing emphasis on usability. http://en.wikipedia.org/wiki/Apple_computer
Stanford University: the intellectual force that anchored Silicon Valley here rather than elsewhere. http://en.wikipedia.org/wiki/Stanford
Stanford Linear Accelerator: Not really central to Silicon Valley history, but it is big and impressive and highlights Stanford's role in attracting scientific talent to the area. http://en.wikipedia.org/wiki/SLAC
Sand Hill Road: the center of VC in the valley. http://en.wikipedia.org/wiki/Sand_Hill_Road
Hewlett-Packard Garage: considered the "birthplace of Silicon Valley" - the prototype of the garage entrepreneurial culture. http://en.wikipedia.org/wiki/HP_garage

The tour ends conveniently near to the best Ginger Gelato in the Whole Universe.

WSO2 Mashup Server 1.5 released!

Mon, 21 Jul 2008 16:18:59 GMT

I'm proud to say that v1.5 of the WSO2 Mashup Server is now available for download! This release has been a few months in the making, and I'm thrilled to see a major update to the Mashup Server finally out. News release here.

Thanks to the efforts of Keith, Tyrell, Channa, Yumani, and many others at WSO2, we've been able to put some snazzy new features in this release:

Integrated award-winning Data Services: quickly and with zero-coding map queries on your databases, Excel spreadsheets, or CSV files into Web services.
Google Gadgets: power a Google gadget with a mashup service - we generate a try-it, a code template for customizing the gadget, and even a build in Gadget dashboard based on Apache Shindig.
Security, security, security: Limit access to your mashup service to only authorized users, encrypt and sign your messages, and access services which are secured. I think our security interface is the easiest way to engage advanced WS-Security features that's out there!
OpenID login support.

There's lots of little improvements too, and I hope to talk about those, as well as the major features above, in upcoming posts.

For now though, a hearty congratulations and a good night's sleep to the Mashup Server team!

IM and Mashup Server

Wed, 16 Jul 2008 00:05:44 GMT

Yumani has published a nice tutorial on Sending Instant Messages via Mashup Server showing how to use the instant messaging capabilities built into the WSO2 Mashup Server. I think the ability to have instant alerts as part of your mashup is a cool feature, and as Yumani shows in this article, they are pretty trivial to do.

I'm looking forward to adding the next step in a future release - the ability to invoke a mashup using instant messaging. Then we'll have a scriptable platform for building agents that can communicate via IM. Imagine converting your favorite service to an IM conversational interface. Or making a data source available through an IM conversation. Or even imagine new games... Endless possibilities!

Loch Leven Lakes

Mon, 07 Jul 2008 04:36:55 GMT

Just posted a few snapshots from a Saturday day hike to Loch Leven Lakes, a wonderful and fairly accessible bit of the High Sierras. Haven't been there in a while, despite it being a fantastic place for some bouldering with the kids. It was good to get above California's persistent smoky haze for a few hours (though later in the afternoon even the high country took on a smoky hue.)

Happy Independence Day

Sat, 05 Jul 2008 06:16:03 GMT

Wishing everyone a last minute happy 4th of July, and for those of you who missed fireworks this year, here's a couple of photos to hold you till next year. Full Flickr set here.

Industry Interoperability Panel (TechEd Online)

Thu, 03 Jul 2008 00:57:57 GMT

Microsoft TechEd Online has posted a video taped in Orlando after our successful keynote appearance. This video captures a panel discussion that I participated in - I'll just quote the blurb:

This Panel discussion centers around the use of standards to facilitate interoperability between different technologies. The participants have all implemented their own version of a reference application and speak about any challenges they faced in implementing the application. The main discussion points of this Panel include: interoperability through SOAP; WS-* standards and how they facilitate interoperability; common problems with interoperability; identity and security in a distributed application; and future directions/remaining challenges in interoperability. With Raghu Thiagarajan, Gerald Beuchelt, Gregory Leake, Jonathan Marsh, and Chris Haddad.

Watch the video here, or go here, search for "Industry Interoperability" and choose your playback format.

Related news: The video of our demo within the keynote has been extracted from the entire keynote and made available on youtube. Here's the link!

Getting beyond passwords...

Wed, 25 Jun 2008 22:22:35 GMT

Microsoft, Google, and others have just launched the Information Card Foundation, to promote awareness and interoperability around InfoCards. This is great news - I think InfoCards have the potential to solve some real problems with managing identities and securing them against theft. Despite the client-side technology being available in Vista for a year plus, we're still slow to see installations emerge in the marketplace. The only use I make of InfoCard (and then still pretty rarely) is in the WSO2 Mashup Server, which has both InfoCard and OpenId support (though you can use an old fashioned username/password).

Managing identity and security without complicating the user experience is still a challenge. I'm hoping the Information Card Foundation can encourage broadly useable solutions as it gets rolling. And I'm hoping some large properties (PayPal is a member) may actually start putting stronger identity management technologies in place on a broad scale.

It should be no surprise with our products supporting InfoCards and OpenId that WSO2 is now a member of both the Information Card Foundation and the OpenID Foundation.

Tamales bay, sky views

Sun, 22 Jun 2008 23:18:16 GMT

Had a great though brief visit from Paul, during which we toodled around the north end of Tamales Bay, bordering the Point Reyes National Seashore, the site of one of our successful backpacking trip a couple of years ago. My primary goal, the Drake Estero, was closed for Harbor Seal pupping season, but we had good fun anyway. Flickr set here.

Also uploaded and organized some existing shots from plane windows. Looking out the window is by far the best part of flying! Flickr set here.

Microsoft features Open Source in TechEd keynote

Tue, 10 Jun 2008 14:03:56 GMT

Yes, you heard right. A few minutes ago in Orlando at TechEd IT 2008, Bob Muglia's keynote included a demo of StockTrader 2.0, an SOA sample application consisting of a client application, a business process service layer, and an order processing service in order to place sample stock trades. Gregory Leake of Microsoft showed the application, with each of the three components built in .NET 3.5, and then I came on stage, representing WSO2, and we swapped out the WPF smart client for a PHP application based on the WSO2 Web Services Framework for PHP. Then we swapped out the back end order processing service for a Java version hosted in the WSO2 Web Services Application Server. After each swap we placed a successful trade.

Watch the keynote here.

The demo featured the cross-platform interoperability between .NET, Java-based solutions, and unmanaged code solutions such as the PHP application. The Web Services used were completely secured with message-level security (WS-Security), and everything of course worked quite seamlessly.

You can download the WSO2 StockTrader 2.0 application as well, including PHP versions of the business service and the order processing service.

The good news in putting this demo together is that the wire-level interop worked pretty spectacularly out-of-the-box, just as the demo promotes. The actual interop between the three major development platforms in use today (CLR-based languages, JVM-based languages, and unmanaged code based at some level in C) is impressive, and while there is more work to do to complete and verify interop deeply across Security, Reliability, Transactions, and Policies, it really seems like the goal of making this stuff both universal and "just plumbing" is approaching pretty rapidly.

On another note - yesterday we were speculating backstage whether a keynote at a major Microsoft event had ever featured an Open Source partner on stage. None of us could think of any off the top of our heads. Can you? Were we the first?

Update: 10PM. Here's the press release. And a news article, with a nice (and accurate!) quote from me. ;-)

Microsoft Live Mesh and Vista - bummer together

Sat, 07 Jun 2008 20:07:34 GMT

I was pretty excited to try out Microsoft's new "personal cloud" computing initiative - Live Mesh. I signed up right away and it looks great and offers some very neat features. Yet I have one problem with it, and that's an unfortunate tie with Vista that makes one or the other virtually unusable.

Live Mesh, for some reason unknown to me, requires that User Account Control to be turned on. Vista users will know User Account Control (UAC) as an annoyance that comes turned on by default. It prevents unauthorized and potentially dangerous changes from being made without an Administrator's approval, which I concede might be beneficial in some circumstances (a classroom?) but is incredibly annoying if you are used to being the Administrator yourself. In that case all it does it pop up a regular stream of annoying modal warnings, to which you become accustomed to clicking through without reading in about 10 minutes, thereby rendering any protection useless.

But it's worse, these warnings actually prevent you from doing useful work when you really want to. Like renewing your IP address. Or adding or removing files in the Program Files folder - if you're editing them through an IDE, you don't even get the warnings, skipping directly to failure. You might have to reorganize your file system a bit to work around UAC. And some stuff simply doesn't work. I could not under any circumstances get Adobe Flash 9 to install for me under UAC.

You can see I'm an anti-fan of this non-feature of Vista, which promises security but provides no real benefit and quite a number of headaches.

I don't really understand why UAC would be required to run Live Mesh, but how important could it be if I can run Live Mesh on Windows XP, which has no such concept? Is this just a UAC marketing?

In any case the effect is that a serious Vista user (the kind who might be an early adopter of Mesh) is significantly disadvantaged. For me, the benefits of Live Mesh aren't worth the pain inflicted by UAC, and I'll generally restrict my Mesh use to non-Vista platforms, or extraordinary circumstances where it's worth doing a reboot to access the Mesh. What a shame.

Weirdest - but probably coolest - remix ever

Sat, 07 Jun 2008 19:39:52 GMT

Listen to this, and then check out this. Wow.

A scraping incident

Tue, 27 May 2008 16:40:41 GMT

One of my mashups fell prey to the dreaded scrape rot - a complete overhaul of the target site that invalidated all of my scraping rules. The pages in question are from globalincidendmap.com, which previously powered my internationalincident mashup (see Sri Lanka Incident Mashup). The change was catastrophic - queryable content from the site is no longer free, but requires a paid membership and a login process. One option would be to pay for a membership, but besides the steep price I doubt that the license terms allow republishing of the data. I could support the mashup only for paid users of the service, collecting credentials and forwarding them on, but that again is both questionably secure (a user would have to trust that I didn't abuse the credentials temporarily in my possession), and unrealistic since few if any of my audience would spring for the cost of membership. In effect, my mashup has been totaled.

This illustrates one reason why scraping should be used only as a last resort, when no more stable forms of content are available - feeds or Web services. When you mix the content and presentation, changes in the presentation are easily confused with changes in the content. Although the scraping features of the WSO2 Mashup Server are popular, I like to think of them as a stop-gap while publishers find cost-effective ways of serving up presentation-free content, such as delivering simple services using the Mashup Server ;-). Ideally, more and more publishers will recognize the value of raw content, and the need for Web scraping will diminish. Gonna take a while though...

Even without scraping, there remains one of the deep problems with mashups and distributed programming, that of services that disappear, are altered, change usage terms, etc., breaking their dependent mashups in the process. There has been lots written on this, which can be generally summed up as "this is a hard problem."

One thing we plan to do in the future to make sure that service changes don't harm downstream dependents is use more of the advanced functionality of the WSO2 Registry upon which the WSO2 Mashup Server is built - namely, versioning. Today, each time a service changes, an old copy is retained in the database, but no longer is "alive" as a service. Some future version will have a simple interface for continuing to keep the old versions online, and help users to lock into one of these previous versions. Some cool dependency management features on the drawing board for the Registry will also help find and record dependencies and notify dependents of changes.

But would these help in the case of the internationalincident service? This is a case where there is a deliberate change which prevents "unauthorized" access. The solution in this case was to mark the service as obsolete, and go out and find a whole new source of data. The new srilankanincident service is a result - though the data is slightly different, perhaps a result of focusing narrowly on Colombo, it was a fairly short task to reprogram it, and even improve it, once I had found a new source of data. The speed of fixing catastrophic failures is my current best hope against scrape rot.

Mashup Server Webinar May 13th

Fri, 09 May 2008 14:20:11 GMT

I've got another free Webinar coming up - again an Introduction to the WSO2 Mashup Server and to mooshup.com on 13 May from 9-10AM PST. Join me if you:

Are curious about mashups, Mashup Server products, and want an overview of the capabilities of WSO2's offering.

Have services, web pages, or other information sources available but you want smarter ways to use those services.

Always wanted an application to do (x) on the Web but it was always too costly to develop.

Know Javascript and want to see what it can do outside the browser.

Register now!

Am I the last to know we're cool?!

Mon, 05 May 2008 18:48:58 GMT

Seems the WSO2 crew has been blogging about WSO2 appearing on the "cool 5" companies in a recent Gartner report (paid subscribers only). What intrigued them about the WSO2 Mashup Server was support for the hitherto paradoxical "lightweight but enterprise-oriented" services.

And here I am a couple of days late. I guess for breaking news and the real skinny on "cool" you would do well to add the feeds of Paul, Sanjiva, Daniel, Glen, and Keith to your blogroll.

I'm on YouTube...

Mon, 21 Apr 2008 23:32:56 GMT

Just noticed this interview I gave at Mashup Camp posted on YouTube. (Does everyone hate hearing themselves talk as much as I do?)

Webinar - A New Approach to Web Service Composition

Tue, 08 Apr 2008 20:01:17 GMT

I'm giving a free Webinar on 15 April 2008 9-10AM PDT about the approach we took to service composition in the WSO2 SOA Platform. Instead of a declarative approach which my XSLT days showed can be powerful yet also limiting in many ways compared with a full programming language, the WSO2 Mashup Server uses a "scriptable Web Services" metaphor, and supports the ability to consume and produce Web Services using simple Javascript expressions. Add to that the ability to script non-Web Service materials like Web pages and feeds, and you've got a powerful yet accessible platform for creating Web Service mashups. Register now!

Sri Lankan Incident Mashup

Tue, 01 Apr 2008 00:37:06 GMT

I just posted a finished version of a Mashup designed to help answer the question "is Sri Lanka getting safer or not?" This is a question we on the global WSO2 team ask each time we arrange travel to that unfortunately troubled country. Despite a spate of violence early this year, designed to coincide with the formal dissolution of the cease fire that has done little to prevent violence, it seemed to me things were getting a little better. But I needed some facts to back that up.

The mashup plots bombings and other incidents as a bar chart, measuring the severity of the incident (how many killed & wounded) over the last 18 months. The idea was to see if there was a clear overall trend in the violence or not, something not readily apparent from a Google map (see right).

The mashup service itself consists of several items, each one a simple task accomplished in half a page of javascript:

Scrape the search results page at globalincidentmap.com to pull the essential data out of an HTML table for the country of interest, and put it into a simple XML structure.
Cache the page if it has already been accessed within 24 hours (scraping is expensive, the first access in any 24 hour period will be pretty slow.)
Parse the headline for patterns such as "eight killed" or "injures 7" and turn that into killed and wounded digits (this isn't perfect, but we can tolerate a few errors since we're trying to present an overall sense of the problem rather than perfectly accurate statistics.) Also filter out as much as possible the killing of LTTE, as that's more a measure of war than of terrorism.
Provide a helper method to look up details and get a link to a new story for any item.

Using this service (called internationalindicent) I created a custom HTML UI to format Sri Lanka-specific results into the bar chart and to make it interactive (click on a bar to see more info about the incident.) Then I used "share this mashup" to upload the service to mooshup.com so others could try it out (or copy the code.)

The rough version took a couple of hours, mostly figuring out the details of scraping the page and coming up with the headline patterns to look for, but then I spent a couple more polishing the HTML UI so I wouldn't be embarrassed to share it. In the process I demonstrated some of the powerful aspects of using mashup technologies in your development arsenal:

Instead of investigating the current situation using Google News for 30-60 minutes each time we plan a trip for the latest information, I can browse this site in a few minutes, see the trend, and get details of any recent incidents I'm interested in. This will pay for itself in terms of my own personal productivity before long.
There is also a small user group (really, just the handful of WSO2 employees based outside Sri Lanka) who can also benefit from this micro-application, increasing their productivity as well.
The WSO2 Mashup Server makes the data available as a service, so others can reuse it too, for alternate displays or to generate displays for other parts of the world.
And, it's just fun!

So ... is Sri Lanka getting safer? I'll have to let you be the judge of that. Go to http://mooshup.com/services/jonathan/internationalincident/ to see for yourself!

[Update 5/27: This service is no longer available. See http://auburnmarshes.spaces.live.com/blog/cns!F985A6952BC07C4!813.entry for more info.]

Mashup Camp 6

Tue, 25 Mar 2008 18:31:52 GMT

Just returned from some travel which included 2 days of Mashup University and a day of Mashup Camp. A few thoughts:

Maps are still well represented. This surprised me a bit, but I recognize I've been well immersed in the WSO2 Mashup Server which supports a wide variety of User Interactions (html pages including maps, but also feeds, email, instant messaging.) I predict non-map mashups will start to eat into the map-based mashup market share dominance over the next year.
There's lots of interest in consuming various Web APIs. Some vendors were promoting their APIs, others tools for consuming those APIs. I think the WSO2 Mashup Server can tap into an underserved market here, since it's the easiest way I know of to deliver a comprehensive Web API on top of a bit of Javascript logic, which in turn can front information sources as diverse as databases and scraped web pages. I think the Mashup Server can become a "design your own API" tool that can have great appeal to the mashup developer.
A lot of Javascript was in evidence, further validating our choice to use Javascript for mashup logic in the WSO2 Mashup Server.
For the first time I ran into a few people with serious interest in WADL. Has its time finally come? More on that in a subsequent post.
The "unconference" style was quite interesting and successful, especially if you're like me and are more interesting in connecting with interesting individuals than in hearing yet another vendor pitch (mine excepted of course!) ;-) One thing that is sorely lacking is any kind of organization to the conference Wiki, and a surprising lack of ability to record much of the activity there. I couldn't even find who won the Best Mashup contest...

I can't wait till next year!

Paul asks "Mashup or Integration?"

Sat, 23 Feb 2008 01:24:55 GMT

Paul Downey poses some interesting considerations for what a mashup consists of. I think I'd list a pretty different set of criteria, but for now I'll just start by comparing a mashup hosted by the WSO2 Mashup Server against his list to see how we stack up:

Zero touch: +1. Each mashup is accessible, documented, try-able, and so forth right out of the box. The admin console makes them discoverable, and if you have a user account (email-validated guests are supported too) you can tag, rate, comment a mashup to build a community around them to make any touch beyond "zero" positive and reciprocal.
Safe: +.5. The Mashup Server supports and encourages safe interactions, but it does not prevent users from hanging themselves if they so choose. That is, I can have an operation exposed through GET, explicitly marked as safe by the author even, that has seriously detrimental side effects.
Cool URIs: +1. Each mashup has neat URIs for the endpoints, for the metadata and tooling associated with it, for the admin capabilities associated with it, and even for accessing the operations over HTTP.
Open data formats: +1. We're partial to XML and use Web Services under the hood, but also provide bridges to HTML, JSON, Feeds, etc.
Eschew RIA: +1. By default we don't provide any rich interface.
HTML form access: +1. By default we do provide an HTML try-it form, and stubs to quickly build your own HTML pages.
Accessible URIs: +1. All endpoints are available through both http and https by default. Mashups can be easily migrated outside local contexts so their visibility is greater. E.g. mooshup.com.
SOAP/WSDL: -1. Everything we do has rich metadata and SOAP 1.1 and SOAP 1.2 bindings along with the REST/POX binding, and we also make it easy to consume web services in these formats. However, you don't have to know anything about these formats to write a successful mashup - they're just valuable parts of the plumbing. So maybe a -1 is too stingy.
Authentication: +0. We support username/password and Infocard to access the Mashup admin site, but there isn't a drop-dead simple way to restrict access to the service based on these controls. Not too hard to provision higher levels of security using WS-Security, but I'm not sure Paul would agree that's sufficient.
Scratch your itch: +1. Subjective, but the whole product is designed to serve the needs of individual developers to hack up services as easy as they can hack up a simple web page.
Fun: +2!! As long as you're willing to hack a bit of Javascript.

So I think the Mashup Server stacks up pretty well using Paul's criteria, which I think can be summed up as "has a nice web interface" and "fun and easy". But some of Paul's criteria don't fit with that summary and those are unsurprisingly the ones I take some issue with:

RIA: I don't see any reason why a rich interface to a mashup is always bad. Only when it locks up the data in a way that's impractical to reuse. In our model, we support the separation of content and presentation in order to lift the limits on the kinds of presentation environments the user prefers. A single mashup can (and ideally should) support as many presentation media as are appropriate, including simple web pages, RIAs, feeds, notifications and messages, portlets, widgets, whatever. If an RIA "scratches my itch" then what's wrong with it?

For example, my iPod Touch comes with a YouTube app, which is an RIA for the YouTube site optimized to the screen limitations of the product. I can also point Safari right at YouTube, but the optimized version actually is easier for simple access. Is this bad? It really depends on what the consumer wants. I agree dead ending in an RIA may be inappropriate for some users of that service, but providing an RIA front end to a clean and well-documented interface that can be repurposed in other ways seems like a good user-centered feature that even Paul would support. I think he probably meant this item to mean "Does the site rely solely on so-called 'rich user experience' technologies in a way that precludes data from being accessed though a nice web interface?"

SOAP/WSDL: I understand why Paul would add this to the list. Up until the Mashup Server Web Services were just too hard to consume to allow them to stand on a list with "scratch your itch" and "fun". However I think we've turned the corner on that. A Mashup Server author rarely has to even consider whether SOAP is involved in delivering a Web Service - we can take care of that for them. Right now it's simply an alternative to the REST interface.

As for WSDL, in the Mashup Server it strongly supports the other goals of zero touch, cool URIs, and open data formats. I'm pretty tired of doing one-off hand coding to access REST sites or sucking up their hand-crafted and varying-quality stubs. WSDL has a big role to play in simplifying these interactions for the developer, in a way that I think Paul would appreciate. It's a primary artifact in providing the "nice web interface" that we all agree is invaluable.

Anyway, thanks Paul for a provocative post!

Music at Pachamama's, take 2

Thu, 14 Feb 2008 17:20:12 GMT

Bro Jason and I again will improvise around a few sets at PachaMama's Organic Cafe (map) this Friday, February 15 7:00 – 9:00 PM. Drop in for a listen!

Upcoming Webinar, conference

Thu, 07 Feb 2008 20:36:49 GMT

I'm giving the first of a regular series of free Webinars to introduce people to the WSO2 Mashup Server and to mooshup.com on 12 Feb from 9-10AM PST. Register now!

I'll also be talking about mashups at next Monday's Web Services on Wall Street conference, starting with the opening panel: "Enterprise Mashups For Wall Street – Leveraging SOA and Web 2.0". If you're there, come say hi!

WSO2 Mashup Server 1.0

Mon, 28 Jan 2008 18:34:55 GMT

At last - we've shipped the 1.0 release of the WSO2 Mashup Server! This project has been in the making since I joined WSO2 over a year ago. A lot of hard work goes into a project like this, but it's amazing to me how much we've accomplished in so little time with so modest resources.

I'll be talking more about the Mashup Server from here on out - how it provides a powerful platform for consuming and exposing Web information of many formats, but focused on Web services (REST and WS-*). But for now you can get a flavor of it from the press release.

At the same time we've launched an on-line site for hosting and sharing mashups, at mooshup.com. I think it will be a pretty fun site, and shows some of the cool community features we've built in from the ground up. Also you might want to subscribe to the Mooshup.com blog, where we'll have some great discussions about how to use the Mashup Server and mooshup.com, and what's next for the product (we'll need your input here!)

Kudos to the team, esp. Keith, Channa, Tyrell, Thilina, and Yumani, with the help of many others.

But now, I think I've earned a little nap ;-).

Christmas in Singapore

Thu, 27 Dec 2007 16:44:18 GMT

See the photos here.

We had a great time exploring Singapore over the last few days. Here are just a few of the highlights:

Arriving at the hotel well after midnight and reuniting with our German friends as we were checking in.
The Shangri-La hotel breakfast buffet. Each morning we power up for the day with a buffet the includes a build-your-own noodle soup section, Indian fare, dim sum, Japanese and Korean fare, fruits and pastries in abundance in addition to the full western buffet. Three plates each usually fattens us up enough to skip lunch, not to mention discourage an early dinner, but not enough to taste everything that looks delicious.
Double-decker hop-on-hop-off bus tour of central Singapore, introducing us to the city, a mix of colonial buildings, traditional two-story storefronts with upper-story shutters in a rainbow of colors, and glass and steel skyscrapers. Laced through with impeccably maintained greenery. I didn't expect Singapore to feel so spacious and gracious. It doesn't have that intensely urban feel like Hong Kong or New York.
Wandering through Little India, a maze of shops and eateries heady with burning incense.
Dinner at the home of a colleague of our friends, getting a picture of home life and how east meets west.
Trawling the many malls on Orchard Road. A little of that goes a long way for me, unless followed by...
A daily afternoon dip in the pool, before, during or after the brief but sometimes heavy daily rainstorm. An hour with a good book under a beach umbrella, listening to children in the pool or the patter of rain just beyond the shelter.
The Night Safari - a cross between a zoo and a wild animal park, but in a dark and rain-glistening tropical jungle lit by a full moon and artistically placed lighting not much brighter than the moon itself. Never though have I seen such an active and alert a collection of animals - Malaysian tigers, jackals, tapirs, capybaras, elephants, barbarosas (a lumpy wild pig with upturned tusks sticking right through the roof of it's snout), giant anteaters, sloth bears, bat-eared foxes, giraffes, hippos, water buffalo, antelope...
A boat loop on the Singapore river, starting at Boat Quay, a quaint strip of eateries tucked up next to the polished spires of the financial district.
Chinatown - a section of quaint and colorful storefronts and street vendors - though very few hawking cheap Chinese merchandise. For some reason Indian trinkets dominate. The Hindu temple is festooned with a layer cake of blue characters and a menagerie of animals, and the courtyard walls are topped with images of lounging sacred white cows.
Meeting with our friends in the vaulted lobby for Christmas eve present sharing, at the foot of a 30 foot tree, as a live choir performs intricately harmonized Christmas songs nearby.
A new iPod touch ;-).
Indochine, a trendy southeast Asian eatery housed in a wing of the Singapore Asian Civilization Museum on the waterfront looking towards Boat Quay and the financial district, provided the perfect Christmas Eve dinner venue. My top choices - a beef and prawn salad, steamed cod in lemon sauce, green mussels in coconut curry, lemongrass creme brule, mango and sticky rice. Wow!
The Singapore Art Museum, housed in a colonial former school, boasts an interesting collection of Asian contemporary art, some of which seems rather primitive to me, others quite sophisticated.
Perfectly manicured botanical gardens brimming with more kinds of palm tree than you had ever imagined existed. The fantastic orchid garden. My favorite specimen is the spindly and bizarrely twisting brownish-purple "Margaret Thatcher".
Club Chinois on Orchard Road for Christmas dinner - upscale, trendy, light oriental. Featured scallops on a slab of silken tofu, fois gras on crispy duck skin, chicken drumstick on a Chinese sweet radish salsa, minced 5-spice chicken on a disk of silken tofu, cod braised in a clay pot with baby bok choy, and cubes of tenderloin stir-fired with ginger and green onion in a crispy noodle basket. Chased down with apple pie, chocolate lava, peanut-encrusted rice balls filled with bean paste with a honey-ginseng tea, and a warm creamy almond "soup" in a new coconut shell. One of the best dinners ever!
Strolling through Christmas street party on Orchard Drive, with elaborate lights and decorations, bizarre floats (my favorite - the jumbo sliced pannatone loaf with fern-like shoots springing from its top and dotted with bread loaves in case you didn't get the "Jesus is the bread of life" theme), a concert. But mostly filled with shutterbugs milling around. At any one time, 1/3 were taking photos, 1/3 were posing for photos, and 1/3 were waiting to take a photo or pose in one. I'm not exaggerating!
Taking a cable car to Mount Haber (only a 100 meters high or so), and back down to Sentosa Island, which is gradually turning from a beach resort and golf course into an island-sized theme park. We caught a computer-rendered 3D chair-hurtling "log ride" down a mountain. Then strolled through Undersea World which held a number of interesting specimens such as the tiny red hearted but otherwise translucent sea angels, giant Japanese Spider crabs, and a long underwater tunnel where we could watch scuba divers feeding the manta rays and hordes of other fish. Even a dugong.
The Pink Dolphin show in the lagoon - standard fare with tricks and petting from volunteers. With quite a jostling crowd and corny tourist patter, it was remarkable only in that we actually saw the pink dolphins.
Dinner at The Banana Leaf Apolo in Little India, where dinner is served on a banana leaf mat that acts both as a placemat and plate. Gen was the only one who ate the whole meal of samosas, Tandoori chicken, chicken tikka masala, paneer in a creamy tumeric sauce, and a paneer/potato kofta in saag. Chased down with mugs of limeade.