Should backend servers send XML or Protobuf responses to ql.io? That is the question this post addresses.

I setup a ql.io application hitting a mock server and ran JMeter to generate load and collect results.

Source code for the app is on Github. Mock server is here and JMeter script is here.

ql.io tables for eBay’s Marketplaces APIs are here.

The table used for this test is findItemsByKeywords.ql.

Test Setup

Server: Dev workstation ca. 2011 (Dell Precision T5500 with Intel Xeon E5630 2.53GHz 24Gb RAM), Linux 3.0.0-20-generic #34-Ubuntu SMP Tue May 1 17:24:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Client: Dev workstation ca. 2007 (Dell Precision 690 Intel Xeon DP 5060 3.2GHz, 8GB RAM), SunOS 5.11 joyent_20120517T192048Z i86pc i386 i86pc

The server ran node v0.6.18a and ql.io app 0.7.4. On the same box another node process was serving canned xml and protobuf responses from eBay’s FindingService on port 6000 (it was on the same box in order to take the network out of the equation).

The test app was running on port 3000, with two route/table/patch combinations (one for XML, one for Protobuf) pointing to the above response server on localhost:6000.

The client was JMeter 2.7, running in server mode (jmeter-server), with HEAP=”-Xms1024m -Xmx1024m”.

See Appendix for implementation details.

Test 1: 200 users, running for 1 hour, hitting the XML path

Rate: 203.43 trans/sec. Average response time: 72 ms.

Number of Samples:           734,339
Average Response Time:            72 ms
Minimim Response Time:            25 ms
Maximum Response Time:         1,002 ms
Standard Deviation:               49 ms
Error Percentage:               0.00 %
Transaction rate:             203.43 trans/sec
Throughput:                   12,248 KB/sec

Test 2: 200 users, running for 1 hour, hitting the Protobuf path

Rate: 213.91 trans/sec. Average response time: 24 ms.

Number of Samples:           772,200
Average Response Time:            24 ms
Minimim Response Time:             8 ms
Maximum Response Time:           198 ms
Standard Deviation:               26 ms
Error Percentage:               0.00 %
Transaction rate:             213.91 trans/sec
Throughput:                   12,388 KB/sec

Discussion

The results show that Protobuf is three times faster than XML!

Is it surprising? Yes. Even with the optimize_for = SPEED option turned on in the .proto file, this is a bit extreme.

The exercise of comparing Protobuf to JSON is left to the reader. It is reasonable to assume that they should be on par, as any conversion step is going to be slower than the native format.

This test doesn’t take into account network latency. Payload sizes play an important role when network is involved; for the resultset being used in the test (50 items) the sizes were as follows:

mock.xml - 83 KB
mock.protobuf - 27 KB

Conclusion

According to the results of the test, it certainly makes sense to use Protobuf instead of XML.

Further testing (involving the network), needs to be done in order to determine whether protobuf is a better solution than json, but given that there is a significant reduction in payload size, Protobuf is likely to come out a winner in that test as well.

Appendix

URLs hit by JMeter (client)

http://10.xx.xx.xx:3000/ebay/finding/keywords/xml/ipad
http://10.xx.xx.xx:3000/ebay/finding/keywords/protobuf/ipad

Routes (server)

return select searchResult.item, errorMessage
from ebay.finding.findItemsByKeywordsXML where keywords = '{keywords}'
via route '/ebay/finding/keywords/xml/{keywords}' using method get;

return select searchResult.item, errorMessage
from ebay.finding.findItemsByKeywordsProtobuf where keywords = '{keywords}'
via route '/ebay/finding/keywords/protobuf/{keywords}' using method get;

Tables (server)

create table ebay.finding.findItemsByKeywordsXML
on select post to 'http://localhost:6000/mock.xml'
     using headers 'X-EBAY-SOA-SECURITY-APPNAME'='{config.tables.ebay.finding.appname}',
                   'X-EBAY-SOA-OPERATION-NAME'='findItemsByKeywords'
     using defaults format = "JSON", limit = 5, offset = 0
     using patch 'findItemsByKeywordsXML.js'
     using bodyTemplate "findItemsByKeywords.ejs" type 'application/xml'
     resultset 'soapenv:Envelope.soapenv:Body.findItemsByKeywordsResponse'

create table ebay.finding.findItemsByKeywordsProtobuf
on select post to 'http://localhost:6000/mock.protobuf'
     using headers 'X-EBAY-SOA-SECURITY-APPNAME'='{config.tables.ebay.finding.appname}',
                   'X-EBAY-SOA-OPERATION-NAME'='findItemsByKeywords'
     using defaults format = "JSON", limit = 5, offset = 0
     using patch 'findItemsByKeywordsProtobuf.js'
     using bodyTemplate "findItemsByKeywords.ejs" type 'application/xml'
     resultset 'findItemsByKeywordsResponse'    

Patches (server)

The following patch was used in the XML path: findItemsByKeywordsXML.js.

This additional code was inserted into the above patch to decode Protobuf responses:

var fs = require('fs'),
    _ = require('underscore'),
    Schema = require('protobuf').Schema,
    fis_schema = new Schema(fs.readFileSync(__dirname + '/util/FindItemsByKeywords.desc')),
    FindItemsByKeywordsResponse = fis_schema['com.ebay.marketplace.search.v1.services.finditemservice.FindItemsByKeywordsResponse'];

exports['parse response'] = function(args) {
    var length = 0, idx = 0;
    _.each(args.body, function(b) {
        length += b.length;
    });

    var buf = new Buffer(length);
        _.each(args.body, function(b) {
            idx = idx + b.copy(buf, idx);
    });

    var fir = { 'findItemsByKeywordsResponse' : FindItemsByKeywordsResponse.parse(buf) };

    return {
        type: 'application/json',
        content: JSON.stringify(fir)
    };
}

Note: two optimizations can be made to the above code: a) allow the patch to return the json structure instead of a string that will need to be parsed again and b) receive buffer length as an argument in order to avoid looping through the data buffers twice.

Mock Server

var _ = require('underscore'),
fs = require('fs'),
url = require('url'),
util = require('util'),
http = require('http');

var port = 6000;

function endsWith(str, suffix) {
    return str.indexOf(suffix, str.length - suffix.length) !== -1;
}

var server = http.createServer(function(req, res) {
    var file = __dirname + '/data/' + req.url

    var cType;

    if (endsWith(req.url, '.xml')) {
        cType = 'text/xml;charset=UTF-8';
    } else if (endsWith(req.url, '.json')) {
        cType = 'application/json;charset=UTF-8';
    } else if (endsWith(req.url, '.protobuf')) {
        cType = 'application/octet-stream;charset=UTF-8';
    }

    var stat = fs.statSync(file);
    res.writeHead(200, {
        'Content-Type' : cType,
        'Content-Length' : stat.size
    });

    var readStream = fs.createReadStream(file);
    util.pump(readStream, res, function(e) {
        if (e) {
            console.log(e.stack || e);
        }
        res.end();
    });
});

server.listen(port, function() {
    console.log('\nmock server listening on ' + port);
});

Please send comments/suggestions to ql.io Google Group.

Today’s release of ql.io 0.7 includes the following changes:

Features

• Fallback syntax to the language - see https://github.com/ql-io/ql.io/wiki/%5BProposal%5D-Optional-Inputs-and-Errors
• Compiler rewritten to output the DAG with dependencies
• Explicit depedencies between modules
• Support for pre-requisite params - see https://github.com/ql-io/ql.io/wiki/%5BProposal%5D-Optional-Inputs-and-Errors
• Retry once for idempotent requests on timeouts
• Update the context with the udf filtered data
• Support C style block comments
• No Compression if the CPU load is > 50%

Bug Fixes

• Http client Agent maxSockets increased to 1000 to avoid request backlog on any given socket - https://github.com/ql-io/ql.io/issues/512
• Fix https://github.com/ql-io/ql.io/issues/478
• Fix https://github.com/ql-io/ql.io/issues/13 Disable autorun in the console.
• Add file path/name to comiple errors

One of the core strengths of ql.io is evented orchestration of reads and writes to HTTP APIs using a declarative language. In recent weeks, the core processing algorithm used to process q.l.io scripts went through an overhaul to easily infer what goes on when you submit a script for execution. The outcome of this exercise is a rewrite of the compiler which now takes a given script and outputs an execution plan. This helped us achieve two things - further simplification of the orchestration algorithm (which is now just about 80 lines long), and visualization to identify potential latecy bottlenecks.

Read on to find out how to generate and visualize execution plans.

For instance, consider the script

prodid = select ProductID[0].Value from eBay.FindProducts where
    QueryKeywords = 'macbook pro';
details = select * from eBay.ProductDetails where
    ProductID in ('{prodid}') and ProductType = 'Reference';
reviews = select * from eBay.ProductReviews where
    ProductID in ('{prodid}') and ProductType = 'Reference';

return select d.ProductID[0].Value as id, d.Title as title,
    d.ReviewCount as reviewCount, r.ReviewDetails.AverageRating as rating
    from details as d, reviews as r
    where d.ProductID[0].Value = r.ProductID.Value
    via route '/myapi' using method get;

A visualization of the execution plan of this script is below.

By looking at this execution plan we can infer the following:

• The select statemet on line 8 depends on the statements on lines 3 and 5.
• The overall latency of this script depends on the slowest of the statements on lines 3 and 5.

Here is the execution plan of another script. This script takes two inputs - a user’s identity and a set of IDs of some items, and gets some details from two different APIs (the bottom two nodes). The responses from those APIs trigger some in-process data extractions and transformations which join on the node below the node at the top.

Again, the overall latency depends on the bottom two nodes.

Here is the execution plan of another script which shows one node ([5]) blocking on another ([1]).

Generating Excecution Plan

Generating the execution plan is easy. Here is a node.js script.

You can use the compiler in the browser too. Here is script that works in any modern browser.

Visualization

I wrote a small tool to compile a script and generate a .dot file, and feed the output to Graphviz.

Dot file generator - to generate .dot files for ql.io scripts.

Today’s release of ql.io 0.6 includes the following changes:

• Support array style reference in columns clause, such as select 'b-1', 'b-3'['c-1'] from a.
• Disable ability to enable/disable ecv checks by default. You can turn it on by adding arg --ecvControl true to the start script.
• Add optional parameters in route. Including “with optional params” in route would make params without ^ prefix optional. When this clause is present, only required tokens are used for matching a request to a route.
• Be able to start the server on multiple ports
• Added support for multiple attachments. See docs on insert http://ql.io/docs/insert
• End pending connections on close after responses are written.
• Support cache events (hit, miss, new, error, info, heartbeat)
• Switch to new cluster2
• Added new syntaxes “with part” and opaque insert param.
• Fix expression parsing in string template so that a token like "{obj.prop[?(@.price > 2)]}" is valid
• Add support for escaped quotes in string values
• Update PEG.js to 0.7.
• Remove duplicates from in clause.
• Use hasOwnProperty in place of prop lookup while joining
• Deal with non UTF-8 encodings from upstream resources
• When joining, use ‘==’ to maintain backwards compat
• Refactor logging to error, access, proxy and default logs. The proxy log file contains outgoing req/resp, access log contains incoming requests, error log contains all errors and warnings, and the rest go to ql.io.log. All these files are rotated.
• Include a payload with begin events
• Support local offset and limit
• Fix the case of alias names with joins and UDFs.
• Add UDFs in where clause to post process rows. You can either tweak or remove a row. See https://gist.github.com/2334012 for semantics of UDFs. UDF support for the where clause is coming in version 0.7.

cluster2 is a node.js (>= 0.6.x) compatible multi-process management module. This module grew out of our needs in operationalizing node.js for ql.io at eBay. Built on node’s cluster, cluster2 adds several safeguards and utility functions to help support real-world production scenarios:

• Scriptable start, shutdown and stop flows
• Worker monitoring for process deaths
• Worker recycling
• Graceful shutdown
• Idle timeouts
• Validation hooks (for other tools to monitor cluster2 apps)
• Events for logging cluster activities

See cluster2 for more info.

Today’s release of ql.io 0.5 includes the following changes:

Here are the steps to deploy ql.io on Cloud Foundry with node 0.6.x. This setup automatically spawns a cluster of node processes using node’s native cluster. A sample app to demonstrate these steps is on github - this includes an app.js and a package.json with all the dependencies.

A route in ql.io is a new consumer-optimized HTTP interface. Routes superimpose a simple and familiar HTTP interface on ql.io scripts without needing to specify an elaborate script in the request. In other words, routes make ql.io a platform to “build your own APIs”.

Having built this capability, in this post I want to highlight some potential ways to take advantage of routes.

Once in a while you come across an HTTP API that uses HTTP in complicated and incorrect ways. There are many examples of this on the Web today including those from eBay, Amazon, Google, Microsoft and many many others. These can be hard to use as they require you to follow proprietary styles for constructing requests and parsing responses. Some of those also don’t work well with common HTTP infrastructure like caches.

In this post, I would like to show how you can, in four simple steps, use ql.io to hide the complexity of such APIs.

Verson 0.4 of ql.io is out to npm today. Here is a quick summary of changes.

• Use native cluster module to start the app
• Upgrade all dependencies to the latest
• Limit response size to 10000000 bytes from upstream sources. You can change this with maxResponseLength in the config.
• Limit outgoing requests per statement to 50. You can change this with maxRequests in the config.
• Chain events for logging done with log-emitter.
• Add a new JSON based interface to browse tables and routes. Try /routes to start browsing.
• Integrate har-view