One of the core strengths of ql.io is evented orchestration of reads and writes to HTTP APIs using a declarative language. In recent weeks, the core processing algorithm used to process q.l.io scripts went through an overhaul to easily infer what goes on when you submit a script for execution. The outcome of this exercise is a rewrite of the compiler which now takes a given script and outputs an execution plan. This helped us achieve two things - further simplification of the orchestration algorithm (which is now just about 80 lines long), and visualization to identify potential latecy bottlenecks.

Read on to find out how to generate and visualize execution plans.

For instance, consider the script

prodid = select ProductID[0].Value from eBay.FindProducts where
    QueryKeywords = 'macbook pro';
details = select * from eBay.ProductDetails where
    ProductID in ('{prodid}') and ProductType = 'Reference';
reviews = select * from eBay.ProductReviews where
    ProductID in ('{prodid}') and ProductType = 'Reference';

return select d.ProductID[0].Value as id, d.Title as title,
    d.ReviewCount as reviewCount, r.ReviewDetails.AverageRating as rating
    from details as d, reviews as r
    where d.ProductID[0].Value = r.ProductID.Value
    via route '/myapi' using method get;

A visualization of the execution plan of this script is below.

By looking at this execution plan we can infer the following:

• The select statemet on line 8 depends on the statements on lines 3 and 5.
• The overall latency of this script depends on the slowest of the statements on lines 3 and 5.

Here is the execution plan of another script. This script takes two inputs - a user’s identity and a set of IDs of some items, and gets some details from two different APIs (the bottom two nodes). The responses from those APIs trigger some in-process data extractions and transformations which join on the node below the node at the top.

Again, the overall latency depends on the bottom two nodes.

Here is the execution plan of another script which shows one node ([5]) blocking on another ([1]).

Generating Excecution Plan

Generating the execution plan is easy. Here is a node.js script.

You can use the compiler in the browser too. Here is script that works in any modern browser.

Visualization

I wrote a small tool to compile a script and generate a .dot file, and feed the output to Graphviz.

Dot file generator - to generate .dot files for ql.io scripts.