Friday, 20 March 2015

Things I wish interactive brokers would do with their API software


As regular readers know I use interactive brokers (IB) to run an automated futures trading system. Now in many ways IB are great. They have several huge plus points compared to most other brokers:

  • Uniquely (?) they offer an API that allows you to build your own automated trading system. What's more the trading server software is available for Unix (and Mac, if that's your thing), not just Windows.
  • They have extremely low commissions.
  • They have a flexible and straightforward portfolio margin and multi-currency system.
  • They have a very wide range of markets.
However now I have been running my system for around a year with real money (with several months of test trading before that) there are a few things that are really annoying.

There are a few more things that other people might find annoying, but which I don't. For example IB don't offer UK spread betting, but then spread betting is the work of Satan so that's fine with me. I don't mind the lack of futures execution algos, despite the fact there are loads for stocks, because I'm perfectly happy with my own. The API documentation is... sketchy... but you can read my blog instead!

No python, matlab, or R API


(In 2017, a couple of years after I wrote this post, IB finally offered a native python API. I wrote a series of posts on this, starting here. I've edited this section to reflect that change). 

IB offers their API in five flavors:

  • Excel DDE 
  • Active X
  • C++
  • Java
  • C#
  • Python
Now there has some very big omissions from that list. The five languages used most in the systematic trading world are (in no particular order): C++, Java, Python, R and Matlab.

I've never really understood why Active X needs to be included as well as DDE; surely one spreadsheet based solution is enough? I've never met anyone who uses C#, though I'm sure they exist, but isn't having C++ and C# gilding that particular lily?

So why no Python, R or Matlab? An entire middle ground of people between Excel using 'F9 monkeys' and hairy chested C++ programmers has been missed out.

Fortunately there are third party API's available to support the likes of Python, R and Matlab; such as the swigibpy API that I use which wraps around the C++ code. However it makes me slightly nervous to use a third party product which has been written and is only supported by someone on a part time basis (though I hasten to add this hasn't given me any problems yet). It particularly worries me when combined with the possibility of an IB API upgrade breaking everything (see below). And it would absolutely terrify me if I was using this platform to trade other peoples money.

Why can't IB create proper API's for these three two high level languages? I am not an expert on Matlab and R, but it would be trivial for IB to do the wrapping of their C++ (as they appear to have done for POSIX). At least then we have some comfort that we are using an official product.


API and server updates aren't always backwardly compatible


There is nothing more annoying than people changing their software so that the changes aren't backwardly compatible. Add new functions? By all means, I don't have to use them do I? Fix bugs? Feel free, as long as it's not a 'bug' whose behaviour I have been relying on (so that is an undocumented feature, not a bug).

Change the name of an existing function just to annoy me? Force me to download the latest version of your API server or stop me trading altogether? No thank you. Especially not while I am relying on a third party to update their python API so it still works. Also why should I have to rewrite my code, retest it and generally make sure its still stable?


No free delayed data via API


When I use the IB front ends (of which more in a second), and I try and look at FTSE futures prices I get the nice friendly message 'Would you like to use 15 minute delayed data, you cheapskate?' (I'm paraphrasing). Click yes, and I can. I can even trade the dammn things, once I've ticked another disclaimer 'Would you really put your families future at stake by submitting a blind market order, just because you refuse to pay the very reasonable fee that Euronext LIFFE want to charge you for live data?'. Oh yes I would.

However when you use the API to get a FTSE futures price you get a very unfriendly message 'IB error id 23 errorcode 200: No security definition has been found for the request'.

Why oh why oh why is it so hard to have a tickbox in the IB API server that says 'Yes I want delayed data where I don't have a subscription. Yes I don't mind trading blind. Yes I am a cheapskate'.

Please IB - give us delayed API data.


No multiple trading login


You can only have be logged into IB in one place at a time.

This means:
  • It means you can't have multiple IB servers running for one account. No problem since you can connect over a network to your server from multiple machines. 
  • It means you can't have multiple IB servers for two linked accounts. Cue lots of messy code specifying which account you are trading from and hoping the server doesn't get it wrong. So a small problem, but with a solution.
  • It means you can't have a manual front end (see next section) open, perhaps to monitor your trades or do a little manual trading in another account. Unsolvable problem.
  • It means you can't have an IB server running a test account, and another for your live account, so running proper tests is impossible. Unsolvable, and very annoying, problem.
  • It means, it sucks.

No lightweight stable front end for manual trading


Apart from the API server, which is a very stable product that you can leave running for weeks on end, IB offer a number of ways to trade manually. At the time of writing these are:
  • Client side TWS. You download this thing and run it on your machine. It is an obese piece of software with far too many bells and whistles. When I run it it freezes eithier immediately, or when I try and do something complicated, like search for a symbol.
  • Server side TWS. Half the time this doesn't download the thin client properly and the other half the time it freezes.
  • Web trader. The vanilla version of this just freezes. The 'beta' version of this runs brilliantly, at least compared to the other front ends. Which is to say that I've managed to use it to submit one order at a time. After that, when I try and find the next symbol, it freezes. And I have to close the session. And login again (using my stupid security card, see below). Half a dozen end of tax year housekeeping trades took me the best part of an hour.
  • Mobile. I haven't tried this. But if something doesn't run on my 4GB laptop it probably won't run on my little Samsung...
I don't know if it's my machine, operating system or browser that is at fault (though I've tried it on every machine I have which covers both windows and linux, though not Mac). But all the UK retail stockbrokers I use seem to manage to create simple lightweight web front ends that are platform independent, and just work. Yet IB cannot.


1990's style security access protocol


Darn, a power cut. I'll just reset my fuse box. Now all I have to do is use vncviewer to log in to my headless trading box*, run the IB gateway server, pull out a stupid plastic card thing from my wallet (assuming I haven't temporarily mislaid it) which I need to log in and restart my own client processes.

It's like digital security was never invented. If I have to login to something like webtrader then fair enough, let me use a cruddy plastic card. But why can't I just have a secure ID sitting on a file on my machine? I can then setup my crontab so that the IB API, and all my server side stuff, restarts when I switch the machine on. After each power cut all I would need to do is reset the fuse box and then turn my server back on again.

* This now reflects my current setup - in earlier versions of this post I wasn't using vnc. Thanks to all those who commented below so that I could address this issue - the security is still rubbish but at least I don't have to plug a keyboard and monitor into my headless trading box.

To conclude...


It does seem as an outsider that the IB development team spend most of their time adding new features to the already bloated TWS front end, and designing things that they think are cool and pretty. It might be that the API community are responsible for only a tiny fraction of IB's income. However I seriously doubt that since algorithimic traders will trade a lot more, and probably use margin more aggresively, than the average Joe.

So please guys, I love the product, lets make it better.

27 comments:

  1. Rob, I think you can still ssh to your server and use GNU screen to run the gateway. You can then detach from the session and close your terminal window. The session will be alive on the server.

    ReplyDelete
    Replies
    1. Hi Alex, good to hear from you.

      I can't remember if screen was one of the remote window programs I tried to get running before, but I'll have a look at it and see if it works.

      Delete
    2. I checked out screen and unfortunately it doesn't do the trick.

      ssh to headless server - works
      run screen on headless server - fine
      run IB gateway within screen - fine
      detach screen session - fine

      The problem then begins if I try and close the terminal window where I did the original ssh from. It hangs, and if I force close it or turn off the machine then the detached screen session dies, taking the running server with it. I guess there is something about the server that doesn't like being detached.

      Anyway thanks for the idea, but back to the drawing board....

      Delete
    3. Hi Rob,

      I recommend using tmux instead of screen. I can't remember the exact reasons anymore but I looked into both a couple of years ago and decided tmux was the way to go (I think tmux has a more modern client/server architecture and a more liberal BSD licence).

      Your screen problem sounds very odd though. screen's raison d'etre is to allow the persistence of long-running processes after closing the login shell so what you described should work.

      Maybe give tmux a try?

      Delete
    4. tmux doesn't work eithier but slightly different issue. When I detach the session I get the same problem that I can't shut the client side terminal window that the ssh was run from. If I then force close it I can reconnect to the orphan tmux window, but before I've done that the IB server terminates itself.

      The IB server is a java program, which doesn't seem to like being detached from the screen.

      Incidentally I have some sporadic success using x11vnc / vncviewer which is a remote display server / client rather than detachable window solution. It doesn't seem to consistently work however, which is a bit frustrating. It did work today however.

      Delete
    5. Ok, if IB Server is X based then I'm out of my depth.

      Maybe this post is of some help?
      http://stackoverflow.com/questions/12309389/keeping-a-x11-application-alive-which-i-can-disconnect-from-and-reconnect-to-af

      On a somewhat different note, have you looked at this ib-controller package?

      https://github.com/ib-controller/ib-controller

      By the sound of it, it may allow you to sort out your automated login issue.

      Delete
    6. Re: being out of your depth, that makes two of us :-)

      IB controller is something I looked at a while ago, unfortunately it won't work unless you disable the physical security on your account (displays codes in CAPTCHA form, for which you have to enter the corresponding matching codes on a plastic card). Right now the minor inconvenience of having to plug a bunch of stuff in every few weeks doesn't outweigh the concerns I have about dropping back to just username/password security.

      Thanks for all your thoughts on this issue though.

      Delete
    7. You're welcome. I enjoy reading your blog so glad to try and help out.

      Delete
    8. if you want a process to persist, install vncserver on the server (apt-get install vncserver), run a vncserver session via ssh (vnc4server :1). "vnc" into the server (vncviewer 192.168.1.100:1), run xterm in the vnc session, run whatever command-line process you'd like in xterm, then just exit out of the vnc session. the vnc server will persist, together with the open xterm window and attached running processes. you don't need to keep the ssh session open during any of this. the vncserver session stays alive after ssh is terminated.

      Delete
    9. Another tip, in case you want to switch back to command line:
      https://mosh.mit.edu/

      Delete
  2. Hello Rob. I use ActiveX with Visual Basic 6.0, so this is not necessarily a second spreadsheet based solution. But of course this happens because I don't code in C++ or Java, so I haven't had much choice...

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. To help you solve Interactive Brokers TWS performance issue (freezing) - you may want to increase amount of memory your Java Virtual Machine is allowed to use when its running. You can find instructions here:

    http://www.tradinggeeks.net/2014/05/interactive-brokers-performance-optmization/

    ReplyDelete
  5. I think its really hard to find a Broker that has worse manual trading interfaces. It seems that they don't care that people are only trading 1/5 of what they would because of their rubbish interfaces. TWS is a piece of junk and webtrader keeps logging me quicker than I can enter an order... IB could quadruple their income from trade commissions if they employed some decent designers and coders for 12 month. Whoever wrote/design the rubbish they have now should be take out and shot in the testicles and that would only be 1/2 as painful as using one their interfaces!
    Anyone have any better solutions I would love to know... I'm not going to start writing my own interface and fighting with any of their API servers.
    Rgds
    Tony

    ReplyDelete
  6. Hi Rob does this mean that you can run multiple instances of the API Gateway on multiple machines on the same network simultaneously?

    ReplyDelete
    Replies
    1. No, that's exactly what you *can't do*. Unless you have multiple accounts (with different logins).

      You can run multiple clients though. Just need to add the client local IP address to the whitelist on the gateway software.

      Delete
  7. Hi Rob
    You should be able to run two instances of the IB Gateway on the same machine, if you log in as different users on the two instances.

    Your test/paper account should have a separate login for it.

    For example, I have a server set up to run both a Gateway client with my live account under one username and another client with the paper account under a separate username.

    Also have you tried the IB Customer Activity Monitor? It allows you to view (read-only) orders and executions on your account, even when you have IB Gateway/TWS open. I've found this to be very useful sometimes :)

    Regards
    Craig

    ReplyDelete
    Replies
    1. Hi Craig
      My paper account has the same username as my live one (though a different account number) - I thought that was the norm?

      I haven't tried the activity monitor. Unfortunately a quick google reveals it only runs on windows machines...

      Rob

      Delete
    2. Yeah I run on Windows :)
      That's quite unusual about your paper account. When you log into TWS under your username, do you see only your live account details? Or do you see the paper details as well?

      I think when you set up a paper account, via account management, you have enter a username. I always entered a separate name but possibly you can enter the same name as live?

      Delete
  8. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. As far as I know, yes you need a GUI.

      Delete
  9. Hello! Anybody use PTMC platform which recenly has connected to Interactive Brokers? http://www.financemagnates.com/forex/technology/pfsofts-multibroker-platform-protrader-now-connects-to-interactive-brokers/

    ReplyDelete
  10. The delayed market data is available after you call the reqMarketDataType(3) function and the field IDs will be changed in the msg from the server. But interestingly, you cannot get historical data from API without a live data feed subscription.

    ReplyDelete
  11. You can actually get access to delayed market data if you call reqMarketDataType(3) before you request your market data. But I have been told you cannot get access to historical data through API without a live data feed subscription.

    ReplyDelete