[ts-gen] Downstream architecture (1st) [Was: Hooking into shim directly ...]
pippin at owlriver.net
Fri Aug 28 20:53:10 EDT 2009
About downstream architectural approaches to the IB tws api, this will
be the first of three posts. This one tries to provide direct answers
to your questions, although I suspect you'll find them unsatifying.
The next will focus on general architectural issues, while the last of
these will consider a more specific approach, and for that matter, the
one I recommend. First, though, you mention code duplication:
> ... I began to realize that .. code I ... maintain would require
> duplication ... which is not a good idea ...
Yes. Very, very true. Code duplication is *always* a headache, the
best known route to self-inflicted defeat by entropy. There are a
number of interface projects to the IB tws api that are now moribund and
seem to have foundered on the issue of code maintenance given IB's version
changes. I've been determined to avoid that sinkhole, and believe I've
After some consideration, I believe any effective solution to the
problem of wire-level request and message format specification, once
given IB's frequent version changes, requires either higher-level
translation of their sample client code, or well-designed tables
of what I call type tags. Although IANAL, this first solution to
the best of my knowledge produces code derived from IB's copyright
code, and shares that copyright. There are theoretical problems
as well with such source-to-source translation --- my dissertation
work was in this area, and I know the issues well --- and as far as
I'm concerned, starting from IB's sample client sources is *out*
of the question. That leaves attribute tables.
In an earlier post in reply to one of yours, subj: "src, tag, ver ->
table_name", there occurs the following exchange:
> Now, about table lookup of request and message event indices:
>> ...it would be great if there were a meta-table ... so we can just
>> look [them] up in the database.
> This is on our roadmap as part of the next major functional area,
> versioning. The shim will use the ApiLevel variable to determine
> request and message formats, check the event indices against the
> database, and complain/exit if there are differences. Downstream
> users would then be free to read from the same tables to learn
> about the supported events for any given level of the api.
Here is the key issue for avoiding code duplication. Once given
such type tag tables to define token level attributes, much of the
problem of code duplication solves itself.
Now, a relatively brief reply to the detailed questions you raise:
> ... I looked up popen() to replace ruby's IO.popen(). It appears
> that the call is at such a low level that only one direction (read
> or write) can be opened at a time. To do bi-directional piping,
> I have to fork() and do all sorts of not-so-fun-stuff.
I'll deal with a better alternative, popen3(), in my last post of
this set. I agree that fork/join code in ruby, or C, or anything else
for that matter, is prone to trouble. It's in effect do-it-yourself,
manual thread management. I'll use such when I need to, but I'd like
to avoid inflicting it on those developing downstream clients.
> I began to wonder ... is there a way to ... hook into the Shim
> directly? I just want to write commands into it and read
> responses, just like we normally do in ruby. In particular, I
> need to implement my own risk.rb equivalent in C/C++. In
> licensing terms, I have no intention to use this code for anything
> but my personal use.
Last first, there is no problem with licensing; anything you derive
from the shim is GPL V3 code, and freely redistributable, and you
can use it for whatever you like as long as you follow the license.
And, if it breaks you get to keep both pieces.
Still in reverse order, again, I'll be discussing equivalents to
risk.rb in my last post of this series. And, if all you want to do
is communicate over a duplex IO channel, formally it doesn't matter
what language you use, although I'll argue later that scripting
languages like ruby have significant advantages. If you think more
about this issue, however, I believe you'll see that there is more
than channel communication going on here. Shared state matters too.
> So what I would like to know is, is there a clear API in the Shim
> for me [to] write send commands in and read responses back?
As for the first question you asked in the para several back, and
ask again above, linking to the shim, I could say: use the source,
Luke, except that trying for a single monolithic executable is almost
certainly a mistake. Tempting, but still, I believe, a bad idea.
Yes, you can do so. The GPL license explicitly supports just such,
and we want people to work on and add to the code, but you'll need
to explore the code yourself, and it's not the right way to do what
you say you want to do. So, I'm not going to devote time now to
leading readers of this mailing list through the nooks and crannies
of the code, although I intend to write more formal documentation
later, and of course anyone who wishes is free to engage in such
source code exploration on their own.
As for how, in brief, besides IO via pipes or sockets, where again,
I believe ruby is the better approach, in C++ you can also build
command objects by copying text into the command listener stream's
input buffer, apply the command language start symbol to that stream
to extract the command object, generate a request from that command
object, add it to the Feeder to be enqueued, and let the (brief)
passage of time lead to the request send upstream to the IB tws.
This is, after all, what the shim itself does, except that, outside
of initialization, the text comes from stdin rather than internally
generated character sequences. For an example of building commands
from strings, see late.c, and again, fair warning, you *don't* want
to take this approach.
As for why, this brings me to the question of what an api program
brings to the table, of what downstream programs should concentrate
on, and how to avoid code duplication between the two. That's
the topic of my next post.
More information about the ts-general