[ts-gen] order states [Was: Experiments with Java API]

Bill Pippin pippin at owlriver.net
Wed Oct 7 13:43:08 EDT 2009


Ken,

About order states, I disagree with your claim that order states:

> "... are ignored/filtered-out by the Shim ..."

Not so!  Every order state defined in IB's APIguide.pdf, and so
every status attribute value that is permitted to occur in an order
status message, is accepted by the shim on input, echoed to the log,
and written as is to the database.  I challenge you to identify an
IB-defined tws api status attribute value for order status messages
that is "ignored" or "filtered out".

If you see new state strings in order status messages that are not
mentioned in IB's APIguide.pdf file, please let us know.  By their
nature, we are unable to predict unobserved and newly defined
states unless they have been foreshadowed by the APIguide.pdf
document.  If you have seen state strings in order status messages
that are *not* in that document, and that is what you are speaking
of here, please say so. 

Also, as for:

> I don't know the design decision that went into filtering these
> other states out ...

You imply that the shim's design involves such a choice, and as I
note above, such claim is false.

Again, the shim's design goes to some effort to exactly reflect
the universe of possible values via the OrderStatus table's status
attribute declaraction, even going so far as to include start states
that, according to IB's own docs, *can't* occur in order status
messages.

I've duplicated the set of possible order states as defined by IB for
the tws api in the create table statement for the OrderStatus table:

>   status  enum('PendingSubmit',
>                'PendingCancel',
>                'PreSubmitted', 'Submitted',
>                'Cancelled', 'Filled', 'Inactive') not null,

In the list above, the first two states are start states, and should
not occur in status messages, as noted in IB's docs; I quote their
explanation from APIguide.pdf below:

>   PendingSubmit ... Note: This order status is not sent by the
>   TWS and should be explicitly set by the API developer when an
>   order is submitted.

The injunction to the developer to use PendingSubmit presumably
reflects IB's belief that programmers using the sample client sources
are unable to originate their own start states, or would be confused
if such were not predefined.  The key point above is IB's claim that
the PendingSubmit value is not sent by the IB tws.

Older versions of the shim did not include these first two states,
and presumably they could be elided from the enum declaration, but
I included them just to avoid the kind of objection you make above. 
If you are using this older version of the database, you need to
upgrade.  This issue, and the fix thereof, is dealt with in detail
in a post from back in August:

"Order states and db version ++"
http://www.trading-shim.org/pipermail/ts-general/2009-August/000548.html

You note having seen PreSubmitted and PendingSubmit:

> ... I have definitely seen PreSubmitted and PendingSubmit.
> In fact, I previously had to look for these in the ... logs to
> capture them ...

Interesting.  Of course PreSubmitted occurs frequently, but I did
not expect that PendingSubmit would occur as part of order status
messages, since according to IB's docs it does not, as noted above.
Of course it isn't unheard of for the docs to conflict with reality;
and when theory and observed fact disagree, the facts win out.
For that matter, I believe I remember Nils noting that he'd seen
PendingSubmit, so it's very possible that you're correct here.

No matter from the view point of the shim; all IB-defined tws api
states are included in the enum declaration for the status attribute,
so the values *are* captured to the journal, and in any case have
always been echoed to the log.

If you see log order status messages that occur in the log, but
do not appear in the database, perhaps you are using an obsolete
version of the shim, and should upgrade.  The problem whereby order
status messages might have duplicate keys one session to the next,
if and when the IB tws resets the order id counter back to one, has
also been fixed; see:

"Debugging intermittent lost posts:
http://www.trading-shim.org/pipermail/ts-general/2009-October/000664.html 

Back to the various order states:

> ... I actually prefer to interpret PendingSubmit as Submitted ...

You may of course interpret the states as you wish; but the actual
values as observed in the order status messages are determined by
the IB tws, and the shim follows the policy of posting those values
exactly as received.

> ... I actually find it useful to have all the states, and then
> leave the interpretation to the Shim user ...

You do have all the states, as noted above; and their interpretation
is up to the downstream client, as you wish.  The current version
of the shim reads order status messages from the IB tws, and writes
each one to the journal.  If your app is reading the log-formatted
event stream, it has all the data; the shim does not filter order
status messages outside of syntax errors, which are reported via
copious output to the stderr, and which you should report to the
list, e.g., if IB adds a new order state without telling anyone.

> ... after browsing the IB documentation [discovered that] ...
> [order status] seems to have a whole bunch of states ...
> Cancelled, Inactive, Pending, Filled, Partial, PendingCancel,
> PendingSubmit, PreSubmitted, Submitted.

Mostly true.  A subset of your list occurs in IB's docs, as well as
the create table statement as noted above.  Matching the finite
domain set from that source with your list gives:

    OrderStatus:        Ken Feng's list
    ----------------    ----------------
                      * Pending
    PendingSubmit       PendingSubmit
    PendingCancel       PendingCancel
    PreSubmitted        PreSubmitted
    Submitted           Submitted
    Cancelled           Cancelled
                      * Partial
    Filled              Filled
    Inactive            Inactive

I claim that the list on the left is, at least according to the
newest version I know of for IB's APIguide.pdf, the complete list of
order states that IB has defined.  Do you have convincing evidence
to the contrary, in particular for your inclusion of the two
extra states?

In the older version of the database I also included other
states conceivably of interest for future use by the shim, and
perhaps you are now in the process of recapitulating this design
progression.  You may also be looking at some dead code in sets.c,
where by "dead" is meant a table (OrsI::instances()) not used
elsewhere by the shim.  The comments there list the IB order states,
while the table itself lists the hypothetical state set:

    Pending
    Submitted
    Filled
    Partial
    Cancelled

This finite domain set is an abortive attempt to merge the two
view points of the shim and IB tws with respect to order state,
for use by the shim in tracking order state.  It may or may not
be of use in the future, but is not now used to filter input or
produce output.  Ignore it.  The shim itself tracks the create
and successive order modification commands to track order state,
e.g., to refuse multiple creates, or a modify without a previous
create.

This matter of two state values, one determined by the shim and one
by the IB tws, is reflected in the notion of the two "sides" to the
journal, the cmd and msg tables, where CreateEvent and ChangeOrder
are in the first category, and OrderStatus, ActiveOrder, OrderResult,
and OrderReport are on the msg side.  Cmd-side order state is defined
by the following six events reflected in the ChangeOrder table:

>   act     enum('create', 'modify',
>                'submit', 'cancel',
>                'modsub', 'urgent') not null,

If you want to learn about order state strings as known to the
shim, see the create table statements in sql/xact.sql.  That's
where the enum declaration quotes I've used come from.

Also, in a previous post I claimed that once an order state transition
had occurred on the IB tws side, and been reported --- or not, as the
case may be --- it was gone; I said the "... train has left the
station."  I'd like to note here that, although this is true from the
perspective of receiving order status messages, which are actually
events, not state query answers, there is potentially, though not yet
for the shim, another source for this information.

The open order message for newer levels of the api includes an
"OrderState" subsequence; from the 960 version of the EReader.java
file of the IB sample client sources:

>   case OPEN_ORDER: {
    ...
>       OrderState orderState = new OrderState();

>       if (version >= 16) {

>           order.m_whatIf = readBoolFromInt();

>           orderState.m_status = readStr();
>           orderState.m_initMargin = readStr();
>           orderState.m_maintMargin = readStr();
>           orderState.m_equityWithLoan = readStr();
>           orderState.m_commission = readDoubleMax();
>           orderState.m_minCommission = readDoubleMax();
>           orderState.m_maxCommission = readDoubleMax();
>           orderState.m_commissionCurrency = readStr();
>           orderState.m_warningText = readStr();
>       }
    ...
>   }

Presumably the m_status attribute above is the same as the order
status state transition next value as reported via the order status
message; since the shim is still at an earlier version of the
open order message, I haven't yet checked this.  Note also that
this newest version of the open order message includes commission
information, which may be of interest to some users.

Thanks,

Bill


More information about the ts-general mailing list