[ts-gen] Downstream architecture (3rd)

Ken Feng kfmfe04 at gmail.com
Wed Oct 7 01:35:07 EDT 2009


Hi Bill,

This is really interesting.

Could this buffering/LiveLock problem potentially cause OrderStatus  
messages from the upstream not to make it through the Shim and down to  
the Ruby client?  This problem sounds insidious, not unlike threading  
bugs I've seen in the past.

Thank you for instituting test coverage; it will certainly help ensure  
functional stability over future revisions.

- Ken

On Oct 7, 2009, at 8:46 AM, Bill Pippin wrote:

> Today's release includes a new directory, rsc, for ruby sample client,
> with example code for popen3 and select.  The code is preliminary and
> leaves much room for improvement, yet includes the minimum  
> architectural
> features of a call to Open3.popen3 to open the connection to a shim
> process, and use of IO.select to read both the shim stdout and stderr.
>
> Users may want to use the ruby class definitions in the rsc directory
> as the starting point for downstream script development.
>
> Since these sample client sources are also used by the exs/test.rb
> regression script, they are a starting point for me in improving
> test coverage, in particular to avoid problems such as the one noted
> earlier this week by Ken, where execution report requests had silently
> stoped working.
>
> In an earlier post back in Aug,
>
> http://www.trading-shim.org/pipermail/ts-general/2009-August/000570.html
>
> I wrote:
>
>> ... users should already be using popen3() or its equivalent
>> in their favorite scripting language, and select() ditto ...
>
>> It would be nice to provide example code via a ruby sample
>> client, and that's just one of many things ... [not] completed
>> yet.
>
> Users attempting to use Open3.popen3 as suggested above have probably
> had problems with livelock, and may well have fallen back on using
> psuedo ttys.  [Note that Pty.spawn(...) is nearly a drop in replacment
> for Open3.popen3(...), with the difference that the ptty merges stdout
> and stderr, so that the client must disambiguate them.]
>
> The current release of the shim changes the buffering for the shim
> stdin to unbuffered, which is the last step needed to overcome the
> problems with livelock.  Previous changes included the choice of
> line buffering for the shim's stdout, with the cout option; and
> advice to the list to eliminate script buffering to the pipes, e.g.,
> via the assignments in the popen3 block below:
>
>>  Open3.popen3(text.cmd_line) do | cmd_send, msg_recv, err_text |
>
>>    cmd_send.sync =
>>    msg_recv.sync =
>>    err_text.sync = true
>
>>    test.prog(cmds, read, proc, cmd_send, msg_recv, err_text)
>>  end
>
> The livelock would occur where commands to the shim were accumulating
> in a stdlib buffer, the shim did not know of them and was loop waiting
> using select for commands, and the client was loop waiting for events
> that would not occur until the commands were processed.
>
> The text of the new ruby sample client follows my sig, and, as noted
> earlier, is also included in the rsc directory in the today's release.
>
> Also, the problem with history queries for ECBOT:YM is new, known, and
> I'll be getting to that next.  It seems to be data (symbol) dependent.
>
> Thanks,
>
> Bill
>
> ::::::::::::::
> test.rb
> ::::::::::::::
> #!/usr/bin/ruby
>
> #  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to  
> the list
> #  copyright (c) 2008-2009 Trading-shim.com, LLC  Columbus, OH
> #  GPL version 3 or later, see COPYING for details
>
> =begin
>
>  Given the command line defined by CmdArgText; the predefined  
> commands,
>  by CannedCmds; the select/read loop, by SelectRead; and the  
> processing
>  filter, by ProcFilter: for the class Popen3Shim, exec the command  
> as a
>  child using popen3, and use the SampleTest instance to send cmds to  
> it,
>  read what it writes back, and dump results.  See the ProcFilter class
>  for further details of processing.
>
> =end
>
> require "rsc/CmdArgText"
> require "rsc/CannedCmds"
> require "rsc/SelectRead"
> require "rsc/ProcFilter"
> require "rsc/SampleTest"
> require "rsc/Popen3Shim"
>
> text = CmdArgText.new
> cmds = CannedCmds.new
> read = SelectRead.new
> proc = ProcFilter.new
> test = SampleTest.new
>       Popen3Shim.new.prog(text, cmds, read, proc, test)
> exit
> ::::::::::::::
> SampleTest.rb
> ::::::::::::::
> #  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to  
> the list
> #  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
> #  GPL version 3 or later, see COPYING for details
>
> =begin
>
>  Given both: objects to send commands, perform processing, and read
>  results; and files for output, input, and stderr for the related IO;
>  then glue the object processing together via a simple test harness.
>
> =end
>
> class SampleTest
>
>  def prog(cmds, read, proc, cmd_send, msg_recv, err_text)
>
>       cmds.test(cmd_send)
>    if read.loop(msg_recv, err_text, proc)
>       cmds.exit(cmd_send)
>       proc.dump
>    end
>
>  end
> end
>
> ::::::::::::::
> CmdArgText.rb
> ::::::::::::::
> #  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to  
> the list
> #  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
> #  GPL version 3 or later, see COPYING for details
>
> =begin
>
>  Define a reasonable command line with which to start a data mode  
> shim,
>  given the goal of regression testing; feel free to modify as needed.
>
> =end
>
> class CmdArgText
>
>  def cmd_line
>    "./shim --data cout file save join diff"
>  end
> end
>
> ::::::::::::::
> Popen3Shim.rb
> ::::::::::::::
> #  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to  
> the list
> #  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
> #  GPL version 3 or later, see COPYING for details
>
> =begin
>
>  Open the command line Text using popen3, and apply the prog method of
>  the Test object to each of the object parameters Cmds, Read, and Proc
>  as well as the resulting pipes.
>
>  Note that pseudo ttys can be used as nearly a drop in replacement for
>  popen3.  I.e. Pty.spawn(text.cmd_line) do | cmd_send, msg_recv, pid |
>  There's the drawback that the process filter must distinguish tws api
>  events from shim writes to the stderr, but given the log format, this
>  is not difficult.
>
> =end
>
> require 'open3'
>
> class Popen3Shim
>
>  def prog(text, cmds, read, proc, test)
>
>    Open3.popen3(text.cmd_line) do | cmd_send, msg_recv, err_text |
>
>      cmd_send.sync =
>      msg_recv.sync =
>      err_text.sync = true
>
>      test.prog(cmds, read, proc, cmd_send, msg_recv, err_text)
>    end
>  end
> end
>
> ::::::::::::::
> CannedCmds.rb
> ::::::::::::::
> #  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to  
> the list
> #  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
> #  GPL version 3 or later, see COPYING for details
>
> =begin
>
>  Define as text various commands in the trading-shim input language,  
> and
>  provide operations to print those commands as part of regression  
> tests.
>
>  Currently this class includes only non-transactional tests, those  
> that do
>  not perform any trades.  For now, see the test script exs/risk.rb  
> for an
>  example of simple trades.
>
>  Although this script is meant in part to illustrate correct syntax  
> for
>  the shim commands included herein, there are a number of cases  
> where the
>  alternatives in the plain shim scripts --- executable text files  
> with the
>  shim itself as the shebang interpreter, found in the directory ../ 
> exs ---
>  are more flexible and upto date.
>
>  In particular, the comments in the tick, info, and bars scripts may  
> be of
>  interest to new users.
>
>  test_cmds(shim_cmd)
>
>      Once contructed, and given an IO object connected to the  
> unbuffered
>      stdin of a trading-shim process, send session level, market data,
>      marked depth, and history commands to the shim.  This top-level  
> cmd
>      will trigger all the other methods of this class, and so is the  
> only
>      method the client need use.
>
>      By the way, the contracts are meant to be for Apple and the  
> current
>      front month of ECBOT:YM .
>
> =end
>
> class CannedCmds
>
>  def test shim
>      data_test(shim)
>      tick_test(shim)
>      book_test(shim)
>      past_test(shim)
>      tick_test(shim)
>  end
>
>  def data_test(shim); shim.write data_text(); end
>  def tick_test(shim); shim.write tick_text(); end
>  def book_test(shim); shim.write book_text(); end
>  def past_test(shim); shim.write past_text(); end
>  def exit     (shim);
>                     STDERR.print "exit \n"
>                       shim.write "exit;\n";   end
>
>  def data_text()
>    return <<-"EOT"
> set loglevel Detail;
> select next;
> select exec AAPL 9:00:00;
> select news all;
> cancel news;
> select info ibc:  266093 at SMART new;
> select info ibc:56578477 at ECBOT all;
>
>    EOT
>  end
>
>  def tick_text()
>    return <<-"EOT"
> select tick ibc:  266093 at SMART 1;               wait  2;
> cancel tick ibc:  266093 at SMART;                 wait  0;
>
> select tick ibc:56578477 at ECBOT 1;               wait  2;
> cancel tick ibc:56578477 at ECBOT;                 wait  0;
>
>    EOT
>  end
>
>  def book_text()
>    return <<-"EOT"
> select book ibc:56578477 at ECBOT 5;               wait  4;
> cancel book ibc:56578477 at ECBOT;                 wait  0;
>
>    EOT
>  end
>
>  def past_text()
>    return <<-"EOT"
> select past ibc:56578477 at ECBOT h1  11  1d now;  wait  6;
>
>    EOT
>  end
> end
>
> ::::::::::::::
> SelectRead.rb
> ::::::::::::::
> #  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to  
> the list
> #  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
> #  GPL version 3 or later, see COPYING for details
>
> =begin
>
>  Given two inputs, the first for a shim format event log, and the  
> second
>  from the shim stderr, loop to multiplex reads for each via select,  
> send
>  each log input line to proc.filter, each error text line to  
> proc.stderr,
>  and terminate if:
>
>      eof or errors occur with either the event or error input files;  
> or
>      twelve or more one-second timeouts occur during the call to  
> select.
>
> =end
>
> class SelectRead
>
>  def loop(msg_recv, err_text, proc)
>
>    limit = 12
>    count = 0
>    child = true
>    input = [msg_recv, err_text]
>
>    while count < limit && child
>      result = IO.select(input, nil, input, 1)
>      if result != nil
>        if result[2] == []
>          result[0].each do |f|
>            case f
>              when msg_recv then child &= proc.filter msg_recv.gets
>              when err_text then child &= proc.stderr err_text.gets
>            end
>          end
>        else STDERR.print "IO error in select loop\n"; child = false  
> end
>      else count += 1 end
>    end
>    return child
>  end
> end
>
> ::::::::::::::
> ProcFilter.rb
> ::::::::::::::
> #  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to  
> the list
> #  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
> #  GPL version 3 or later, see COPYING for details
>
> =begin
>
>  Given shim format event lines or stderr text, count events for the  
> former,
>  and echo the latter; and otherwise, if the input line is nil, again  
> echo
>  it, and otherwise do nothing.  The dump method lists event totals.
>
>  For debugging and other purposes there is also a commented-out  
> statement
>  to list log format events.  That statement will, given the split on  
> bars,
>  slice the array to drop the first three fields, reconstitute via  
> join,
>  and slice the resulting string to fit the terminal display.
>
>  The event counting is crude, and will in the future be modified to  
> use
>  database information about the various event types.
>
> =end
>
> class ProcFilter
>
>  attr_reader :cmd_totals, :req_totals, :msg_totals
>  def initialize
>    @cmd_totals = []
>    @req_totals = []
>    @msg_totals = []
>    (0..100).each do |i|
>      @cmd_totals[i] = 0
>      @req_totals[i] = 0
>      @msg_totals[i] = 0
>    end
>  end
>
>  def filter line
>    if line != nil
>      fields = line.chop.split('|')
>      length = fields.size
>
>      if length >= 5 then
>        src = fields[3].to_i
>        tag = fields[4].to_i
>        ver = fields[5].to_i
>        if src == 3
> #         STDERR.print fields[3,length].join('|')[0..78], "|\n"
>        end
>        case src
>          when 1 then cmd_totals[tag] += 1
>          when 2 then req_totals[tag] += 1
>          when 3 then msg_totals[tag] += 1
>        end
>      end
>    end
>    return line
>  end
>
>  def stderr line
>    STDERR.print line
>    return line
>  end
>
>  def dump
>    1.upto(20) { |i| printf("%4u", cmd_totals[i]) }
>    1.upto(20) { |i| printf("%4u", req_totals[i]) }
>    1.upto(20) { |i| printf("%4u", msg_totals[i]) }
>  end
>
> end
>
> _______________________________________________
> ts-general mailing list
> ts-general at trading-shim.org
> http://www.trading-shim.org/mailman/listinfo/ts-general



More information about the ts-general mailing list