Here are the slides from my talk/demo on ql.io at NodePDX.

Here are the slides from the workshop on ql.io at the Node Summit titled “ql.io: Consuming HTTP at Scale”.

This is a beta release of ql.io on node.js 0.6.x.

• Use native cluster module to start the app
• Upgrade all dependencies to the latest
• Limit response size to 10000000 bytes from upstream sources. You can change this with maxResponseLength in the config.
• Limit outgoing requests per statement to 50. You can change this with maxRequests in the config.
• Chain events for logging done with log-emitter.
• Add a new JSON based interface to browse tables and routes. Try /routes to start browsing.

If you are interested in running ql.io on node.js 0.4.x, use the 0.3 branch.

To create an app using ql.io 0.4 modules, follow the usual steps:

 mkdir myapp
 cd myapp
 curl https://raw.github.com/ql-io/ql.io/master/modules/template/init.sh | bash
 bin/start.sh

One of the key visible benefits of ql.io is that it eliminates the code noise that is common in writing HTTP client apps. As a DSL for writing HTTP client code, it focuses on automating the task of making multiple HTTP requests and processing responses in the best order possible taking care of paralleization, orchestration, projections and normalizations behind the scenes.

In this post, I would like to present the baseline performance benchmarks of ql.io running on node.js 0.4.12. Though I have done some ad hoc tests in the last 2-3 months for hardware sizing purposes, this is my first systematic attempt.

If you’re interested in setting a ql.io app instance on a Joyent’s no.de, here is how.

• Clone the template app

git clone git@github.com:ql-io/ql-io.no.de.git

This template includes a server.js, some sample ql.io scripts.

• Create a SmartMachine instance on no.de.

• Push the app to no.de

# Assuming your smart machine name is "foo"
cd ql-io.no.de
git remote add foo.no.de foo.no.de:repo
git push foo.no.de master

This will push the app to your SmartMachine, and bring it up.

• Try a sample route.

curl http://foo.no.de/myapi

Use http://foo.no.de/console to view the console.

See ql.no.de/myapi to see this example in action.

• Clients can occasionally get socket hangup errors when origin servers close connections without sending a Connection: close header. See https://github.com/joyent/node/issues/1135 for some background. To avoid such errors, http.request.js now automatically retries the request once provided the statement that caused the HTTP request is a select.

• The engine can now consume CSV response in addition to XML and JSON.

• Fixed request body processing for routes (see issue 161).

• OAuth example - OAuth2 is trivial as ql.io proxies headers from clients to servers. OAuth1 requires glue code to compute the Authorization header. See http://ql.io/docs/oauth for an example.

• Use npm installed modules for ql.io-site (see issue 116).

• Handle empty response bodies gracefully (see issue 98).

• Recover from partial failures in case of scatter-gather calls (see issue 90) - some statements can result in multiple HTTP requests. When this happens, the engine used to fail the entire statement if any of those requests fail. The engine now looks for success responses and aggregates them.

• Update CodeMirror to support line-wrapping (See issue 11) - no need to split lines manually anymore.

See Announcing ql.io.