Wednesday 8 March 2017

Historic data from native IB python API

This is the second in a series of posts on how to use the native python API for interactive brokers. This post is an update of the post I wrote here, which used the 3rd party API swigibpy.

Okay so you have managed to run the time telling code in my last post.

Now we will do something a bit more interesting, get some market prices. Arguably it is still not that interesting, and this stuff will never be as interesting as a decent book or a good bottle of red wine, but we've all got to get the money to buy those books and bottles and this is better than many other options for achieving that.

Note: This post has been updated to use a more robust method for dealing with concurrency.


Let us peruse the IB ABI instructions...

Whenever you are trying something new out with the API you need to identify the EClient (command to ask the server for stuff) and EWrapper (command run by the wrapper client when stuff is presented unto it) functions. So Reading The Frigging Manual is worth doing.

And indeed there is not just one but four places in the weird and mysterious world of EClient where it looks like prices could be found (ignoring whacky methods for things like options):



  • reqMktData
  • reqHistoricalData
  • reqRealTimeBars
  • reqMarketDepth


Market Data: Returns a stream of price ticks (updated quotes and trades). Because it is a stream you have to ask for it, and then ask for it to stop lest you be buried under a huge pile of ticks. It is level 1, eg 'top of the order book' data.

Historical Data: Returns a one off chunk of data, both prices and volumes, with a given look back history and frequency.

Real Time Bars: Returns a stream of averages of the trades (or bid, ask, midpoint) over an interval, currently only 5 seconds. So its kind of like pre-clumped tick data arriving at regular intervals. Its a stream, so you have to ask for it... you get the idea.

Market Depth: Returns a stream of updates to the order book, both prices and size. This requires a requires a level 2 subscription. Its a stream...


Because real time barsmarket data and market depth deliver very similar information I won't discuss them all (and since I am a tightwad I don't have any Level 2 data subscriptions!). In this post we'll go through the 'one shot' historical data; and in the next one we'll look at one kind of streaming data (market data).


Historical Data AKA Those Who Cannot Remember the Past Are Doomed...

First get the code from this gist somewhere on a server far far away...

Also the relevant manual pages are here.



 Getting some price data 

 Just run the file (assuming you still have an IB connection available as per the previous post). You should get something like this:

Getting full contract details from the server... 
Getting historical data from the server... could take 10 seconds to complete 
historic_data
[('20170312', 98.145, 98.16, 98.135, 98.14, 53208), ('20170313', 98.14, 98.16, 98.135, 98.145, 53600), ('20170314', 98.135, 98.16, 98.135, 98.14, 21673)]


"Obviously" these are tuples containing daily prices (date, open, high, low, close, volume) for the September 2018 Eurodollar futures contract. If you get an error you might need to change the expiry date for the contract. I'll show you how to do this later in the post.

(Note before mixing these with any kind of other prices or doing proper backtesting I would personally append a 'fixed' end of date timestamp to them eg 23:59:59 to ensure there is no slight look forward bias. This is discussed more here.)


  The IB contract object or How IB Knows What We Want

Some code:

ibcontract = IBcontract()
ibcontract.secType = "FUT"
ibcontract.lastTradeDateOrContractMonth="201809"
ibcontract.symbol="GE"
ibcontract.exchange="GLOBEX"

(Note if you got an error above here is your chance to change the expiry to the current one)

Here is a real 'gotcha' - the IB contract object (when I say 'gotcha' it can be stuff I've literally spent days banging my metaphorical head against and in some cases my actual head). This is a complete description which identifies a particular thing you can trade. Most of the data you need is available from the https://www.interactivebrokers.com/ product listing pages. Some problems I have (there may be more associated with other asset classes) mainly in not giving enough information to uniquely identify a contract:

  • Getting the exchange name right - it needs to be exactly the same as on the product page. 
  • Having to specify a currency. This seems to be a problem for CAC40 and AEX futures (specify ibcontract.currency="EUR") and silver ("USD").
  • Having to specify a multiplier effectively contract movement in cash terms per price point (For CAC ibcontract.multiplier="10", AEX is 200 and SILVER is 5000)
  • Be careful, some contracts actually have an expiry date in a different
  • You have to specify the exact expiry YYYYMMDD of a VIX contract rather than just YYYYMM because there are now weekly expiries.
(By the way I don't actually use this contract object in the rest of my code. Instead I have another class which contains more useful information, and then a convert_to_ibcontract function. This function handles the horrible special cases above and also translates between meaningless names like  and better names like JohnBoy - my own pet name for the Eurodollar future *).

 (* This is a joke)

The next thing we need to do is get the contract details. This may sound a bit weird, we already have the contract details since we've just defined it? In practice the definition of a contract is very light and the details we get back are much richer.

There is one specific case where you must do this. To see why suppose it is a few months time and you are trading hundreds of futures contracts. A fill comes in for a particular contract instrument code and month/year. However as we shall see the identification for the contract isn't the 'yyyymm' code we use to define the contract, but the full expiry 'yyyymmdd'. You can't just take the first 6 characters of that either since some futures actually expire the month before they should. So to be on the safe side I usually get the expiry from IB before trading to make sure I am applying fills against the correct thingy.

Actually I don't do this immediately before trading but in a separate process that populates a database of contract information...


So we now need to "resolve" this contract into a fully featured contract which properly specifies the entire expiry date:

resolved_ibcontract=app.resolve_ib_contract(ibcontract)

Getting full contract details from the server... 

By the way if you haven't specified the initial contract object precisely enough, so there are multiple possible contracts, then you'll get a warning. You may also get a warning if the IB server doesn't deliver a "I am done with sending you contracts now sir" message by activating the wrapper method contractDetailsEnd()

All this is Fascinating Rob But How do we Actually get the Prices?!


historic_data = app.get_IB_historical_data(resolved_ibcontract)

This calls the following code in the TestClient object:


def get_IB_historical_data(self, ibcontract, durationStr="1 Y", barSizeSetting="1 day",
                           tickerid=DEFAULT_HISTORIC_DATA_ID):

    """    Returns historical prices for a contract, up to today

    ibcontract is a Contract
    :returns list of prices in 4 tuples: Open high low close volume    """

    self.wrapper.init_error()
    print("Getting historical data from the server... ")
    
    ## Make a place to store the data we're going to return    historic_data_queue = finishableQueue(self.init_historicprices(tickerid))
    today = datetime.datetime.now()

    # Request some historical data. Native method in EClient    
    self.reqHistoricalData(
       tickerid,  # tickerId,       ibcontract,  # contract,       datetime.datetime.today().strftime("%Y%m%d %H:%M:%S %Z"),  # endDateTime,       durationStr,  # durationStr,       barSizeSetting,  # barSizeSetting,       "TRADES",  # whatToShow,       1,  # useRTH,       1,  # formatDate       False,  # KeepUpToDate <<==== added for api 9.73.2       [] ## chartoptions not used     )

    historic_data_queue = finishableQueue(self.init_historicprices(tickerid))
    ## Wait until we get a completed data, an error, or get bored waiting    MAX_WAIT_SECONDS = 10    historic_data = historic_data_queue.get(timeout = MAX_WAIT_SECONDS)

    while self.wrapper.is_error():
       print(self.get_error())

    if historic_data_queue.timed_out():
       print("Exceeded maximum wait for wrapper to confirm finished - seems to be normal behaviour")

   self.cancelHistoricalData(tickerid)


   return historic_data
The general structure of this function should be obvious to anyone who read the last post. We:

  • call some init functions in the wrapper instance (self.wrapper) so there is somewhere for error messages and our price data to go
  • Ask the IB API server for something with the inherited EClient method self.reqHistoricalData()
  • Wait patiently until the data is received, or until we get bored 
  • Complain about any errors, or return the results as appropriate 

The self.reqHistoricalData() asks the IB server for historical price data. We can set length of time (a year in this case, although because we are using a demo account there isn't that much data available) durationStr, the end time, the bar size (in this case days; not actually the size of the kind of bar you get drinks from) and some other things you can look at the documentation yourself to find out about. Just to note not all the combinations of time period and bar length are permitted - you can't get second by second prices for a year (see this link.) even if your hard drive could store them.

About the only thing that might not be obvious is tickerid. You might be the kind of wild crazy person who requests loads of historical data for different contracts. In which case you would get data coming back randomly and you wouldn't know which contract it was for. So you have to provide some kind of label for the request if you request loads of things at once. Note you can't actually do that because if you request more than 1 set of data about every 10 seconds you get a 'pacing violation'  (basically, don't take the mickey).

This is only one side of the story, as....

Oh Yeah I remember now, we have to have an EWrapper function on the server side... 

From the TestWrapper class, as usual we have a storage method where we put stuff in a Queue:


## Historic data code

def init_historicprices(self, tickerid):
    historic_data_queue = self._my_historic_data_dict[tickerid] = queue.Queue()

    return historic_data_queue

And some methods in EWrapper which have been overriden:


def historicalData(self, tickerid , bar):

    ## Overriden method    ## Note I'm choosing to ignore barCount, WAP and hasGaps but you could use them if you like    bardata=(bar.date, bar.open, bar.high, bar.low, bar.close, bar.volume)

    historic_data_dict=self._my_historic_data_dict

    ## Add on to the current data    if tickerid not in historic_data_dict.keys():
        self.init_historicprices(tickerid)

    historic_data_dict[tickerid].put(bardata)


def historicalDataEnd(self, tickerid, start:str, end:str):
    ## overriden method
    if tickerid not in self._my_historic_data_dict.keys():
        self.init_historicprices(tickerid)

    self._my_historic_data_dict[tickerid].put(FINISHED)


There is no magic in creating these things; to write this stuff you look in the API reference to see what the wrapper function that requires overriding (in this case historicalData) returns. In this case the wrapper function beastie is called repeatedly with the same arguments for each row of data. In theory historicalDataEnd would also be called, however as with getting contract details this might never happen (I've had mixed success with it). 


<snip>

Back in the TestClient object:


    historic_data_queue = finishableQueue(self.init_historicprices(tickerid))
    ## Wait until we get a completed data, an error, or get bored waiting    MAX_WAIT_SECONDS = 10    historic_data = historic_data_queue.get(timeout = MAX_WAIT_SECONDS)


It's now worth understanding what a finishableQueue is; 

## marker for when queue is finishedFINISHED = object()
STARTED = object()
TIME_OUT = object()

class finishableQueue(object):

    def __init__(self, queue_to_finish):

        self._queue = queue_to_finish
        self.status = STARTED

    def get(self, timeout):
        """
        :param timeout: how long to wait before giving up waiting for a FINISHED flag        :return: list of Queue elements        """        contents_of_queue=[]
        finished=False
        while not finished:
            try:
                current_element = self._queue.get(timeout=timeout)
                if current_element is FINISHED:
                    finished = True                    self.status = FINISHED
                else:
                    contents_of_queue.append(current_element)
                    ## keep going and try and get more data
            except queue.Empty:
                ## If we hit a time out it's most probable we're not getting a finished element                ## give up and return what we have                finished = True                self.status = TIME_OUT


        return contents_of_queue

    def timed_out(self):
        return self.status is TIME_OUT


This is just a simple class for keeping track of Queues of multiple items in the wrapper thread where the results have some 'finish' point, which may not come.

And once again, back in the TestClient object:

    historic_data = historic_data_queue.get(timeout = MAX_WAIT_SECONDS)


    while self.wrapper.is_error():
       print(self.get_error())
    if historic_data_queue.timed_out():
       print("Exceeded maximum wait for wrapper to confirm finished - seems to be normal behaviour")
   self.cancelHistoricalData(tickerid)

   return historic_data

So we get this output:

Getting historical data from the server... could take 10 seconds to complete 
historic_data
[('20170312', 98.145, 98.16, 98.135, 98.14, 53208), ('20170313', 98.14, 98.16, 98.135, 98.145, 53600), ('20170314', 98.135, 98.16, 98.135, 98.14, 21673)]

(By the way exactly the same finishableQueue idea is used when we resolve the contract details)

That then is everything you need to know about historical data. I will cover market data, i.e. the wonderful world of streaming data, in the next post.

This is the second in a series of posts. 
The previous post was: qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html

The next post is qoppac.blogspot.co.uk/2017/03/streaming-market-data-from-native.html


70 comments:

  1. Rob,

    the definition of ibcontract's parameters is a bit difficult to read due to this web page's layout. Can you change this layout please?

    Although I'm not using Python, I do get good results from historicalDataEnd. I'm not sure why that callback doesn't work for you.

    ReplyDelete
    Replies
    1. Fixed (something weird is going on when I copy and paste code from pycharm into blogger and all the \n vanish).

      Yes it's a problem I have with my current setup (swigibpy) as well. Not sure if it's a 'feature' or I'm doing something wrong.

      Delete
    2. Thank you. Now are the specified parameters good readable. I agree with you that it is rather ambiguous which parameters you need to provide for a contract. To avoid this am I using the IB conid. Disadvantage is that I need to (manually) investigate which conid I need.

      I have to retract my statement about historicalDataEnd. I looked once again in my code and notice that I am actually not using it. I have in my program a boolean called HistoricalPricesReceived. in the overriden method historicalData am I using a statement "if string date contains the string 'finish' then HistoricalDataReceived is true".

      Delete
  2. Rob, this is great. I took the code off the gist and it just about worked right away. I had increase the timeout on the login to give me a chance to hit the OK button twice on TWS. That might work different on the gateway.

    If you want to look at how to specify a contract, look at the ContractSamples.py in the IB sample code.

    I'm also seeing the historicalDataEnd() get called before the timeout, so probably an easy fix there.

    Another great thing is it all works inside a notebook, which makes it easy to test.

    ReplyDelete
  3. this looks like daily data for a ticker. is it possible to get hourly data for a defined period?

    ReplyDelete
    Replies
    1. Yes by changing: barSizeSetting="1 day"
      You will probably want to change durationStr="1 Y" as well. Have a look at the official documentation.

      Delete
  4. On line 239, you need to add now (for API 9.73.2):
    "TRADES", # whatToShow,
    1, # useRTH,
    1, # formatDate
    False, # KeepUpToDate <<==== add this
    [] ## chartoptions not used
    )

    Else you get the error: TypeError: reqHistoricalData() missing 1 required positional argument: 'chartOptions'

    ReplyDelete
  5. Hi everyone,

    I am running the code verbatum in this tutorial, including the edemo account rather than my own IB account. I am getting an error.

    I am supposed to be getting:

    Getting historical data from the server... could take 10 seconds to complete
    historic_data
    [('20170312', 98.145, 98.16, 98.135, 98.14, 53208), ('20170313', 98.14, 98.16, 98.135, 98.145, 53600), ('20170314', 98.135, 98.16, 98.135, 98.14, 21673)]

    What I'm actually getting:

    Getting full contract details from the server...
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:demohmds
    Getting historical data from the server... could take 10 seconds to complete
    Exception in thread Thread-11:
    Traceback (most recent call last):
    File "/Users/kevin/anaconda/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
    File "/Users/kevin/anaconda/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
    File "/Users/kevin/Desktop/twsapi_macunix/IBJts/source/pythonclient/ibapi/client.py", line 235, in run
    self.decoder.interpret(fields)
    File "/Users/kevin/Desktop/twsapi_macunix/IBJts/source/pythonclient/ibapi/decoder.py", line 1154, in interpret
    handleInfo.processMeth(self, iter(fields))
    File "/Users/kevin/Desktop/twsapi_macunix/IBJts/source/pythonclient/ibapi/decoder.py", line 703, in processHistoricalDataMsg
    self.wrapper.historicalData(reqId, bar)
    TypeError: historicalData() missing 8 required positional arguments: 'open', 'high', 'low', 'close', 'volume', 'barCount', 'WAP', and 'hasGaps'
    Exceeded maximum wait for wrapper to confirm finished - seems to be normal behaviour
    []

    I am unsure if it's a personal path error, or why it's telling me that the historicalData() method is missing the arguments. Thanks for the help.

    ReplyDelete
    Replies
    1. If you look at previous comments for this post you will see this issue should have been fixed. Make sure you are using the latest GIST.

      Delete
    2. Hello Rob,

      are you referring to Wessel's fix?

      '
      On line 239, you need to add now (for API 9.73.2):
      "TRADES", # whatToShow,
      1, # useRTH,
      1, # formatDate
      False, # KeepUpToDate <<==== add this
      [] ## chartoptions not used
      )

      Else you get the error: TypeError: reqHistoricalData() missing 1 required positional argument: 'chartOptions'
      '

      I'm using the latest gist and still getting the same error as the commenter.

      Delete
    3. Hello Rob,

      are you referring to Wessel's fix above? I am using the latesst GIST (updated July 6) and getting the same error as him.

      Delete
    4. Sorry I didn't read the trace properly. It says ALL the arguments are missing. That's very strange. I can't seem to reproduce the error which suggests you have a path, or similar, problem.

      Delete
    5. I got the same error "historicalData() missing 8 required positional arguments..."

      Delete
    6. I've updated my API software and now I'm getting the same error. It looks like IB have changed the historicalData function arguments (why they keep ****** doing this I don't know...). I'll have to modify the code to cope with this...

      Delete
    7. Ok - if you download the new GIST link and you're using both the latest gateway and API then this should work now.

      Delete
  6. Hey Rob Can I ask you I want to get a set of a Historical data. I have post it on stack overflow do you have any suggestion?

    https://stackoverflow.com/questions/45793114/how-to-get-series-of-requests-in-ibapi-on-python

    ReplyDelete
  7. Hello:
    I tried
    try:
    app.disconnect()
    except:
    print("bug in IB API to disconnect")
    to avoid the annoying bug error message. But sometimes the exception is not caught and with a [AttributeError: 'NoneType' object has no attribute 'close'] , I just copied the codes above.

    ReplyDelete
  8. Hi Rob,Thanks for your extremely useful posting.
    I am testing your code without change, but can't get the same result. It just keeps saying "No market data permissions for GLOBEX FUT". I have already tried changing the contract details, but the same type of error for all other security markets, and types of securities.
    Could you please guide me if something seems wrong, or could IB have possibly changed their data access policy on demo accounts recently..?
    Thanks in advance!

    Here is the full error message:
    Getting full contract details from the server...
    IB error id -1 errorcode 2104 string Market data farm connection is OK:usfuture
    IB error id -1 errorcode 2104 string Market data farm connection is OK:usfarm
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:ushmds.us
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:ilhmds
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:hkhmds
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:ushmds
    Getting historical data from the server... could take 10 seconds to complete
    IB error id 50 errorcode 162 string Historical Market Data Service error message:No market data permissions for GLOBEX FUT
    Exceeded maximum wait for wrapper to confirm finished - seems to be normal behaviour

    ReplyDelete
    Replies
    1. Are you using the edemo account, or your own paper trading account? Paper trading accounts have the same permissions as the live account they mirror.

      Delete
    2. Does your account have permission to trade futures? Have you subscribed to (and paid for) live market data for futures?
      The error description you provide hints that one of these requirements is not fulfilled.

      Delete
    3. I am trying this with e-demo account without registration, so yes, no permissions I have for now. Just testing if API works with Python, before making a regular account.
      So my understanding was that you (Rob) also tried your code with demo account, because you mentioned demo account in your post as follows:

      (quote)
      The self.reqHistoricalData() asks the IB server for historical price data. We can set length of time (a year in this case, although because we are using a [[[demo account]]] there isn't that much data available) durationStr, the end time, the bar size (in this case days; not actually the size of the kind of bar you get drinks from) and some other things you can look at the documentation yourself to find out about. Just to note not all the combinations of time period and bar length are permitted - you can't get second by second prices for a year (see this link.) even if your hard drive could store them.
      (end quote)

      Your comments are suggesting that IB does not provide with any kind of data to e-demo account, am I right?
      Anyway many thanks to your quick reply, for both of you guys!

      Delete
    4. Yes I used an edemo account to test the code. You can usually get some data from the edemo account and it's unusual to see the 'no permission' message. The restrictions about dates and bar lengths apply to all accounts.

      Delete
    5. OK, then I think I should ask IB about it.
      Thanks so much anyway!

      Delete
    6. Problem solved, Just for sharing.
      I was trying the code with my free-trial demo account (represented by my email address), not with the sample demo account. (edemo / demouser)
      Other users, just don't make "Try the demo" words in TWS login window deceive you.

      Delete
  9. Hi Rob,
    Thank you a lot for your series on Python + IB API.

    While running your code, there are times I encounter this error. Would you mind taking a look at it? It is really weird since this error just appeared recently (11/20/17 to be exact) . Thank you a lot.

    File "C:/Users/Tho.thodo-pc/Documents/test ibpy.py", line 297, in
    historic_data = app.get_IB_historical_data(resolved_ibcontract)

    File "C:/Users/Tho.thodo-pc/Documents/test ibpy.py", line 243, in get_IB_historical_data
    [])

    File "C:\Users\Tho.thodo-pc\Anaconda3\lib\site-packages\ibapi\client.py", line 2135, in reqHistoricalData
    self.sendMsg(msg)

    File "C:\Users\Tho.thodo-pc\Anaconda3\lib\site-packages\ibapi\client.py", line 76, in sendMsg
    self.conn.sendMsg(full_msg)

    File "C:\Users\Tho.thodo-pc\Anaconda3\lib\site-packages\ibapi\connection.py", line 79, in sendMsg
    nSent = self.socket.send(msg)

    ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine

    ReplyDelete
    Replies
    1. Sorry I haven't seen that one before.

      Delete
    2. Hi Rob,

      Thank you for your quick reply.

      I went through the code that I copied from your Git and found out that while I removed the last line (app.disconnect()), the problem would not happen again. I guess there must be something wrong when I leave API Client connecting to IB Gateway all the time.

      Is it possible if I use your code to pull historical data of other asset classes (i.e stocks, options, etc) by changing parameters in "ibcontract"?

      Thank you a lot!

      Delete
  10. Hi Rob, thank for your great posts! I am playing with your historical data code with the edemo account. I am getting price quotes that do not match reality. For example, according to what I get, the closing price of AAPL on March 14 was $178.13. The true closing price was $178.44. Is this due to the use of the edemo account?

    ReplyDelete
    Replies
    1. Yes prices and positions from edemo are essentially garbage and shouldn't be trusted.

      Delete
  11. Hello Rob,

    I've a subscription for L1 data and experimenting with the API. For the historical data, prices (High/Low/Open/Close) are correct but the volume is not the same as the data I get from Quandl or the CME website.
    For example Eurodollar Jun18 contract on 2018/03/23:
    - IB: 260.10K (Extended Trading Hours) or 108K (Regular Trading Hours)
    - Quandl: 394K
    - CME website: 399K

    Do you know where the difference come from? Is it then save to use IB volume to make roll decisions?

    Kris

    ReplyDelete
    Replies
    1. Sorry no idea. I don't use volume as a signal so it doesn't bother me.

      Delete
    2. I also don't use volume as a signal, only use it for roll decisions...

      Delete
  12. Hi Rob,
    This is very useful introduction, thank you.
    I've bumped into the following issue, wondering if you seen this before:

    {code}
    File "/Users/Uriel/Documents/Python/Ibis/get_history.py", line 268, in
    resolved_ibcontract=app.resolve_ib_contract(ibcontract)
    File "/Users/Uriel/Documents/Python/Ibis/get_history.py", line 189, in resolve_ib_contract
    resolved_ibcontract=new_contract_details.summary
    AttributeError: 'ContractDetails' object has no attribute 'summary'
    {\code}

    I dont see the 'summary' in the official manual, and if I remove it then
    I get:
    {code}
    AttributeError: 'ContractDetails' object has no attribute 'conId'
    {\code}

    what am I doing wrong ?

    Attila

    ReplyDelete
    Replies
    1. IB has changed the code for the callback method related contract details. The actual contract information used to be stored in a field called "summary". This has changed to a different name.

      Delete
    2. summary is now contract in the new API.

      resolved_ibcontract=new_contract_details.summary should be
      resolved_ibcontract=new_contract_details.contract

      Delete
  13. Hi Rob, I was wondering if you have dealt with an issue I am having. For some reason, the "End" wrapper functions, such as "historicalDataEnd" and "contractDetailsEnd", when they place FINISHED into the queue, instead of telling the queue it's "Finished" and breaking, the "FINISHED" object is actually making its way all the way through the queue and into my data object, but I don't see any reason why. So, I'm getting errors when I work on my data because I have this empty object at the end of my data object. For now, I've been leaving out the End statements, but I'm trying to diagnose the cause of the problem. Is this something you have seen before? Thanks!!

    ReplyDelete
    Replies
    1. That is weird, but then anything to do with asynchrous operations can always cause weird stuff to happen (sorry that isn't helpful...)

      Delete
  14. hi Rob Thank you very much for this post!. I am new with Interactive Api. I copied your code but i am facing a problem while running it .. it tells me " 'ContractDetails' object has no attribute 'summary' ".

    Do you have an idea what am I missing please ?

    (my purpose is just to pull out historical data as price, volumes and the book..)

    ReplyDelete
    Replies
    1. See above where the same question was asked.

      Delete
    2. Sorry for the delay in repling.

      API change -> gist now updated.

      Delete
  15. Hi - thanks as always for your educational posts.
    I am trying to run the latest gist with the latest TWS API (v9.72) and it is barfing at the line
    resolved_ibcontract=new_contract_details.summary
    and complaining there is no such attribute. I am successfully connecting using the demo login, there are plenty of other reasonable looking attributes in new_contract_details, but no summary.
    Does this mean the API has changed since this gist was put together, or am I missing something.

    ReplyDelete
    Replies
    1. See earlier questions. IB has changed names of fields in the callback method for getting contract details.

      Delete
    2. API change, gist now updated.

      Delete
  16. Hello Rob,

    Could you please explain the function in class finishableQueue ?

    def timed_out(self):
    return self.status is TIME_OUT

    Best,
    Thanasarn

    ReplyDelete
    Replies
    1. It's just to avoid exposing the innards of the class. What it's telling you is hopefully self documenting.

      Delete
  17. Hi Rob, thanks for all the teaching!
    It really helped me greatly!

    As of 28/08/2018, the sample contract requires market subscription.

    And "resolved_ibcontract=new_contract_details.summary
    contract" goes to error.

    I have to change this to make the code work.
    resolved_ibcontract=new_contract_details.contract

    Hope this helps the others. Thanks!!

    ReplyDelete
  18. As far as I can make out, when there is an error, the code hangs until timeout, and then continues processing. E.g. when calling contract_details_queue.get(timeout = MAX_WAIT_SECONDS) in your gist, if there is no match found, nothing happens for seconds, and then the error is thrown. Which would make sense since nothing is being placed into the queue by IB (Does it indeed behave this way or am I doing something wrong?) If so, is there a way to handle the error as soon as it occurs. Thanks as always...

    ReplyDelete
  19. Hi Rob,

    Thanks for this post, really awesome that you shared this with the community.

    I have 2 questions about your code, I have re-read your post a million times and tried some stuff, but i can't seem to figure it out. (I am also quite new to Python, did a 6 hour course and made some easy programs with it).

    Question 1:
    When the program is not able to make a connection with the api, your code seems to re-try it 1 more time, and than it quits trying. (i think this related to "get bored waiting"). For some reason, I deal with this a lot (also when I am using a different code, seems to be either API related or related to my own internet connection, not sure)

    Basically, I would like this program to keep trying untill it retreives the data (basically, "I never get bored waiting") Could you tell me how to achieve this?

    Question 2:
    If the program than returns the data, it returns a string that goes sideways for eternity (or untill we have retreived all the data for the contract) for example:

    'AAPL', ';', '20190517; ; 15:30:00', ';', 187.06, ';', 187.59, ';', 186.76, ';', 187.14, ';', 11626, '\n'), ('AAPL', ';', '20190517; ; 15:31:00', ';', 187.13, ';', 187.68, ';', 187.09, ';', 187.52, ';', 1794, '\n')

    As you can see with my '\n' command, i would like every output to be on a new row, but (I think) because your code writes a str, it writes the whole thing and does not uses the '\n' command.

    Could you tell me what I should change in order for your code to return the data every single row? I can't seem to find the related str command that does this.

    Again thanks a million times for posting this and it would be great if you could help me out with my issues.

    Regards

    ReplyDelete
    Replies
    1. Jack,
      As a general point I think you need to learn a *lot* more python before it would be safe to try writing a trading system using the language, because these are quite basic questions. I'm worried that you will be a danger to your own bank balance if you trade with real money at this stage.

      As a kindness I will answer your questions, but this blog is not meant to be a 'help you learn python' blog. There are better and more well qualified experts out there for that, than me:

      Q1. This is handled in the API not my code; to deal with this you could wrap the line 266 in the gist with a try: except: inside an infinite while loop


      Q2: That isn't a string it's returning, but the string representation of historic_data which is produced by the print() command. So historic_data is a list of tuples. You wouldn't normally want to turn those into strings for a real trading system, but if you wanted to print them out as individual rows for diagnostic purposes then you'd do something like this:

      for one_tuple in list_of_tuples:
      print(",".join(one_tuple))

      Delete
  20. Thank you for you answer Rob!

    Luckily I am not trying to make an automated trading system, I am just trying to download historical data. But thank you for your concerns.

    As for question 1, and I am sorry to ask another question, I am not quite sure how to achieve that, since I can't find the value that is supposed to be true (which should be connection). So I have tried a couple things:
    while False
    try:
    self.connect(etc.)
    except:
    True

    This does not loop forever, and also gives me the ib 504 error: not connected.

    If I use while True the program does nothing

    And I have tried calling all the functions (while function: False/True try self.connect(etc.)

    So basically, since I do not know what's supposed to be True I can't specify what to do if it's false (If that makes sence)

    Would you be so kind to tell me what's supposed to be true or false and what to call when except happens? (or write that piece of code for me (: )

    As for question 2, I have been able to fix that by deleting everything that you wrote in the "def historicalData" function and replacing it with the following line:

    print(symbol, ";", bar.date.replace(' ', '; '), ";", bar.open, ";", bar.high, ";", bar.low, ";", bar.close, ";", bar.volume)

    First I would like to thank you again for trying to help me with my issue and for posting this code!

    Second I would like to ask you if you would be so kind to help me out with question 1, since I am unsure what the conditions are for the while loop that should be true or false.

    Really the only thing that is holding me back now for downloading bulk data is that connection issue, so it would be amazing if you could help me out with that!

    Have a nice day and thank you again!

    ReplyDelete
    Replies
    1. This is what I mean:

      working=False
      while not working:
      try:
      self.connect (etc)
      worked=True
      except:
      pass

      However I haven't tested it and I don't know if the self.connect() will throw an exception or like it if you try to repeat that command multiple times.

      Delete
    2. The tabs have vanished from that post so let's try again with spaces:

      working=False
      while not working:
      try:
      self.connect (etc)
      worked=True
      except:
      pass

      Delete


    3. working=False
      while not working:
      try:
      self.connect (etc)
      worked=True
      except:
      pass

      Delete
    4. Okay - replace * with tab:

      working=False
      while not working:
      *try:
      **self.connect (etc)
      **worked=True
      *except:
      **pass

      Delete
    5. Thank you again for your answer.

      Unfortunately If i try to use that line of code, the program does nothing. I think that it should be specified somewhere in the TestClient class, but im not sure.

      Delete
    6. UPDATE:

      Sorry, i forgot the
      **worked = True

      So, after adding that line AND deleting self.cancelHistoricalData, the program does not lose it's connection, so that is good. but when it fails to retreive data, it now skips whatever it failed to retreive and continues on with the next one (progress! :P)

      EDIT: (deleted comment and copied that to this reply so clean your blogpost a bit, sorry for the missclicks).

      Alright, so I have tried a couple new things so far. Basically I don't think that this works because self.connect(etc) does not return anything itself, it just tries to connect and because it tries it will automatically turn to true (I think).

      changing "pass" with the following things will not work aswell, for the same reason.

      continue, TestApp, working=False, finishableque, EClient, EWrapper.

      I think that it would work if we could add something like:

      if we get an output, working = true
      else
      working = false
      retry

      the problem is that I do not know how to define that output, could you help me with that? (again, sorry for asking so many questions)

      Thanks!

      Delete
    7. Its not possible to modify self.connect as this is a method provided by the IB API objects. However we can check for a connection:

      working=False
      while not working:
      *self.connect (etc)
      *if self.IsConnected():
      **worked=True

      Delete
    8. Unfortunately this does not seem to work.

      first, apparently it is
      self.isConnected(): (manual page says "IsConnected", but client.py says "isConnected"

      Then, trying the following:

      ...
      *if self.isConnected():
      **working = True
      *if not self.isConnected():
      **working = False

      Does not work aswell, for some reason.

      Changing "if not..."by "except" gives me an error and says "statement expected"

      This is verry strange to me, as now we are calling for isConnected, which is False because it's not connected, but still it skips the loop and goes on with the next symbol

      Delete
    9. UPDATE:

      another weird thing, when I say:
      self.isConnected():
      print("Connection established " + symbol)

      it says that a connection is established, eventhough it's not retreiving the data and gives me a timeout after 10 sec.

      Delete
    10. Alright So, I have added:

      *if self.isConnected():
      **working = True
      **print("Connection Established")
      *if not self.isConnected():
      **working = False
      **print("Connection not Established")

      eventhough the ib terminal does not return:
      "IB error id -1 errorcode 2104 string Market data farm connection is OK:usfarm"

      It does say:
      "Connection Established"
      but this is mentioned before the terminal returns:
      "Getting full contract details from server"

      So after digging through the manual pages, I found that I should be able to call:
      "EWrapper.error(self, reqId, errorCode, errorString)"
      in order to verify that my connection is good and that I can now receive data.

      When I replace:
      self.isConnected():

      with that code, it just returns me the code as a string, so I am not able to use that for some reason.

      But since "isConnected" does not seem to work, I think we should find a way to make python do something like this:

      If we do not get error-1 within 5 seconds, working = False & retry code.

      What I really still think is strange, is that it does not attempt to receive the data after the first connection is failed.

      Could you help me with that? :)

      Delete
    11. Alright, I would like to thank you for your help Rob.

      I have been able to fix my connection issues and now my program does not seem to have any issues anymore.

      Again, thank you a lot for all your help and your code.

      Regards

      Delete
  21. This comment has been removed by the author.

    ReplyDelete
  22. Hello.

    Will someone please tell me if a market data subscription is required to download historical end-of-day data via the ibapi?

    I suspect so, but just wanted to make sure before subscribing.

    ReplyDelete
    Replies
    1. Yes, market data subscription is required.

      Delete
  23. Rob, thanks for writing this and the other related posts.

    I've been accumulating data using ib_insync for a while now, but this and the previous post have made OOP coding just that little bit clearer, from the consistency of thick mud, to that of a good pea and ham soup.

    Thanks again, and back to the reading :)

    ReplyDelete
  24. Does this method allow you to collect data for long ago expired contracts, or is it limited to contracts expiring less than 2 years ago?

    ReplyDelete
    Replies
    1. Whatever IB has available.

      Delete
    2. Right, 2 years of expired contracts then. :(

      Anyone got ideas of the best place for price data for backtesting for a non-coder? I've been using Barchart for Excel but the data is riddled with errors and gaps.

      Delete

Comments are moderated. So there will be a delay before they are published. Don't bother with spam, it wastes your time and mine.