[ts-gen] Speculation about lost order status messages

Ken Feng kfmfe04 at gmail.com
Wed Oct 7 15:06:12 EDT 2009


Hi Bill,

Thank you for you comment/observation.

I suspected that I might be doing something wrong in the downstream  
client - as you mentioned, no one else seems to be observing these  
issues; otoh, I don't know if many users are hitting the system as  
frequently as I am.  Lost order status still happens to me several  
times a day.

As I have been stuck too long at this last piece of getting the whole  
system working, and I am not confident that I have the expertise to  
manipulate low-level Ruby or C correctly, especially in the presence  
of threading code, I will take a short hiatus from that part of the  
Shim and see if I can get this last piece (Risk) working through the  
Java API.

As an aside, the Data Shim process has been running continuously and  
very reliably for me.

- Ken

On Oct 8, 2009, at 3:35 AM, Bill Pippin wrote:

> Ken,
>
> About your speculation as to the cause of the lost order status  
> messages
> that Nils was dealing with:
>
>> Could this buffering/LiveLock problem potentially cause OrderStatus
>> messages from the upstream not to make it through the Shim and down  
>> to
>> the Ruby client?
>
> No; the problem is observed as a hang, not intermittent lost messages.
> And no one has yet noted a missing order status *message* to the  
> shim's
> log --- Nils' problem was with silent, lost posts, which issue seems  
> to
> have been pinned down to a problem, since fixed, with duplicate keys
> interacting with the mysql "ignore" attribute in insert statements;  
> see:
>
> "Debugging intermittent lost posts:
> http://www.trading-shim.org/pipermail/ts-general/2009-October/000664.html
>
> Now, back to your speculations about order status messages reaching  
> the
> client.  All upstream IO uses low-level IO via reads and writes to a
> socket.  As I noted in your quote from my original post below, the key
> issue is lost IO events; a busybody stdlib IO layer that insists on
> buffering multiple IO events is *discarding* the initial subsequence  
> of
> those events from the viewpoint of upper layers in the IO stack, and
> non-trivial concurrent systems will not tolerate such interference.
>
>>> The livelock would occur where commands to the shim were  
>>> accumulating
>>> in a stdlib buffer, the shim did not know of them and was loop  
>>> waiting
>>> using select for commands, and the client was loop waiting for  
>>> events
>>> that would not occur until the commands were processed.
>
> Buffering is legitimate for a layered API such as TCP/IP, where pieces
> for one layer, on one side, are buffered, or aggregated, by the *same*
> layer on the other side of the stack.  E.g., network splitting and
> coalescence of packets occurs all over the Internet.
>
> Problems arise when some busybody library component throws away
> event notifications for a client layer higher in the stack.
>
> Since C stdlib IO and ruby do not do TCP/IP style encapsulation to
> distinguish the multiple IO layers --- C file descriptor, C file
> pointer, and ruby IO object --- it is up to the application developer
> to expose low-level IO state, and the related events, by eliminating
> buffering entirely for input, and limiting it to line-buffering for
> output.
>
> Thanks,
>
> Bill
> _______________________________________________
> ts-general mailing list
> ts-general at trading-shim.org
> http://www.trading-shim.org/mailman/listinfo/ts-general



More information about the ts-general mailing list