Error when fetch the history data with minute timeframe.

[tonylegend]

Hi,

I encounter this error when I upgrade to version alpaca-backtrader-api-0.14.1. It can be reproduced using the code of strategy_readme_sample.py . The only change is backtest and the timeframe of minute bars.

File "/Users/xxx/lib/python3.8/site-packages/alpaca_backtrader_api/alpacastore.py", line 397, in _t_candles
cdl = self.get_aggs_from_alpaca(dataname,
File "/Users/xxx/lib/python3.8/site-packages/alpaca_backtrader_api/alpacastore.py", line 586, in get_aggs_from_alpaca
response = _iterate_api_calls()
File "/Users/xxx/lib/python3.8/site-packages/alpaca_backtrader_api/alpacastore.py", line 522, in _iterate_api_calls
r = self.oapi.get_bars(dataname,
File "/Users/xxx/lib/python3.8/site-packages/alpaca_trade_api/rest.py", line 651, in get_bars
bars = list(self.get_bars_iter(symbol,
File "/Users/xxx/lib/python3.8/site-packages/alpaca_trade_api/rest.py", line 637, in get_bars_iter
for bar in bars:
File "/Users/xxx/lib/python3.8/site-packages/alpaca_trade_api/rest.py", line 564, in _data_get_v2
for item in items:
TypeError: 'NoneType' object is not iterable

The code:

import alpaca_backtrader_api
import backtrader as bt
from datetime import datetime

ALPACA_API_KEY = "<key_id>"
ALPACA_SECRET_KEY = "<secret_key>"

IS_BACKTEST = True
IS_LIVE = False
symbol = "AAPL"


class SmaCross(bt.SignalStrategy):
    def __init__(self):
        sma1, sma2 = bt.ind.SMA(period=10), bt.ind.SMA(period=30)
        crossover = bt.ind.CrossOver(sma1, sma2)
        self.signal_add(bt.SIGNAL_LONG, crossover)


if __name__ == '__main__':
    cerebro = bt.Cerebro()
    cerebro.addstrategy(SmaCross)

    store = alpaca_backtrader_api.AlpacaStore(
        key_id=ALPACA_API_KEY,
        secret_key=ALPACA_SECRET_KEY,
        paper=not IS_LIVE,
    )

    DataFactory = store.getdata  # or use alpaca_backtrader_api.AlpacaData
    if IS_BACKTEST:
        data0 = DataFactory(dataname=symbol, historical=True,
                            fromdate=datetime(2021, 8, 12),
                            timeframe=bt.TimeFrame.Minutes,
                            data_feed='iex')
    else:
        data0 = DataFactory(dataname=symbol,
                            historical=False,
                            timeframe=bt.TimeFrame.Days,
                            data_feed='iex')
        # or just alpaca_backtrader_api.AlpacaBroker()
        broker = store.getbroker()
        cerebro.setbroker(broker)
    cerebro.adddata(data0)

    print('Starting Portfolio Value: {}'.format(cerebro.broker.getvalue()))
    cerebro.run()
    print('Final Portfolio Value: {}'.format(cerebro.broker.getvalue()))
    cerebro.plot()

I found that the earliest_sample in _iterate_api_calls() is some time in the day of fromdate(start) like 2021-08-12 04:03:00 which is never earlier than the start date time 2021-08-12 00:00:00. So the got_all is till false when the data fetch is done. It works for the days timeframe but failed on minutes timeframe. I am not sure if that is the root cause, and just post my finding.

[jdeighton]

If you set the fromdate parameter to also include the hour and minute for the start of the market, it should work ok. At least that sorted it for me. Good luck.

...
data0 = DataFactory(dataname=symbol, historical=True,
                            fromdate=datetime(2021, 8, 12, 9, 30),
                            timeframe=bt.TimeFrame.Minutes,
                            data_feed='iex')
...

[tonylegend]

Thanks. I also used this workaround by setting the market open minutes. Hopefully alpaca API team could handle this timeframe difference internally or just update the API spec.

…

On Sun, Sep 5, 2021, 11:41 AM jdeighton ***@***.***> wrote: If you set the fromdate parameter to also include the hour and minute for the start of the market, it should work ok. At least that sorted it for me. Good luck. ... data0 = DataFactory(dataname=symbol, historical=True, fromdate=datetime(2021, 8, 12, 9, 30), timeframe=bt.TimeFrame.Minutes, data_feed='iex') ... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#182 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZRFOTHJZ7WEG32IVQREL3UAO2UBANCNFSM5CENQW2Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[cspipaon]

Hi, I am getting started with Backtrader and Alpaca and seem to be getting the same error. I have run the code in the first post by @tonylegend with the edit in DataFactory recommended by @jdeighton but still get the same error.

Traceback (most recent call last):
  File "/home/xxx/anaconda3/envs/algotrading_env/lib/python3.8/site-packages/alpaca_backtrader_api/alpacastore.py", line 397, in _t_candles
    cdl = self.get_aggs_from_alpaca(dataname,
  File "/home/xxx/anaconda3/envs/algotrading_env/lib/python3.8/site-packages/alpaca_backtrader_api/alpacastore.py", line 586, in get_aggs_from_alpaca
    response = _iterate_api_calls()
  File "/home/xxx/anaconda3/envs/algotrading_env/lib/python3.8/site-packages/alpaca_backtrader_api/alpacastore.py", line 522, in _iterate_api_calls
    r = self.oapi.get_bars(dataname,
  File "/home/xxx/anaconda3/envs/algotrading_env/lib/python3.8/site-packages/alpaca_trade_api/rest.py", line 651, in get_bars
    bars = list(self.get_bars_iter(symbol,
  File "/home/xxx/anaconda3/envs/algotrading_env/lib/python3.8/site-packages/alpaca_trade_api/rest.py", line 637, in get_bars_iter
    for bar in bars:
  File "/home/xxx/anaconda3/envs/algotrading_env/lib/python3.8/site-packages/alpaca_trade_api/rest.py", line 564, in _data_get_v2
    for item in items:
TypeError: 'NoneType' object is not iterable

[jvb2358]

I'm running into this issue too. I seem to be able to make the data calls for fromdate -> todate ranges less than 30 days. But I'm still debugging the reason.

My best guess so far is that the call is hitting a limit on the free alpaca api subscription service. At least I've gotten a similar NoneType error when I trying to access data I wasn't subscribed to.

More specifically in alpacastore.py, _iterate_api_calls(),
There's logic to work around calls for data larger than 1000 samples, by breaking up the request into multiple api calls. I'm wondering if in doing so, the system is hitting a limit for the free subscription. For instance, Alpaca free api is limited to 30 symbols, which is obviously different than 30 days, but i'm wondering if calling the same symbol 30 times might also trigger this limit. Bit of a wild guess. I'm just getting started with alpaca / backtrader tbh.

I've seen a number of posts of people running into this type of issue when using the free alpaca api subscription. Does anyone know a way of printing out the api response from store.getdata(...) calls?