update docs

docs/_sources/deployment.rst.txt

@@ -414,6 +414,9 @@ In addition to broker-specific secrets, the following environment variables are
    * - **Secret**
      - **Description**
      - **Example**
+   * - LUMIWEALTH_API_KEY
+     - Your API key from the BotSpot.trade website so that you can track your bot's performance. To get this API key, visit the `BotSpot.trade <https://botspot.trade/>`_ website and add/create a bot. After creating the bot, you will receive an API key.
+     - 694rr2c8d9234b43a40fab494a79f5634ghd4f39d44ccf2e
    * - LIVE_CONFIG
      - Your live config file, only needed for strategies that have multiple configurations (there will be a folder named "configurations" in the src/ folder) and if you are running the strategy live.
      - paper_1

docs/deployment.html

@@ -855,35 +855,39 @@ <h1>General Environment Variables<a class="headerlink" href="#general-environmen
 </tr>
 </thead>
 <tbody>
-<tr class="row-even"><td><p>LIVE_CONFIG</p></td>
+<tr class="row-even"><td><p>LUMIWEALTH_API_KEY</p></td>
+<td><p>Your API key from the BotSpot.trade website so that you can track your bot’s performance. To get this API key, visit the <a class="reference external" href="https://botspot.trade/">BotSpot.trade</a> website and add/create a bot. After creating the bot, you will receive an API key.</p></td>
+<td><p>694rr2c8d9234b43a40fab494a79f5634ghd4f39d44ccf2e</p></td>
+</tr>
+<tr class="row-odd"><td><p>LIVE_CONFIG</p></td>
 <td><p>Your live config file, only needed for strategies that have multiple configurations (there will be a folder named “configurations” in the src/ folder) and if you are running the strategy live.</p></td>
 <td><p>paper_1</p></td>
 </tr>
-<tr class="row-odd"><td><p>IS_BACKTESTING</p></td>
+<tr class="row-even"><td><p>IS_BACKTESTING</p></td>
 <td><p><strong>(Optional)</strong> Set to <strong>“True”</strong> to run the strategy in backtesting mode, set to <strong>“False”</strong> to run the strategy live (defaults to False).</p></td>
 <td><p>False</p></td>
 </tr>
-<tr class="row-even"><td><p>POLYGON_API_KEY</p></td>
+<tr class="row-odd"><td><p>POLYGON_API_KEY</p></td>
 <td><p><strong>(Optional)</strong> Your API key from your Polygon account, only needed if you are backtesting.</p></td>
 <td><p>a7py0zIdhxde6QkX8OjjKNp7cD87hwKU</p></td>
 </tr>
-<tr class="row-odd"><td><p>DISCORD_WEBHOOK_URL</p></td>
+<tr class="row-even"><td><p>DISCORD_WEBHOOK_URL</p></td>
 <td><p><strong>(Optional)</strong> Your Discord webhook URL, only needed if you want to send notifications to Discord. Learn how to get a Discord webhook URL here: <a class="reference external" href="https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks">Discord Webhooks</a></p></td>
 <td><p><a class="reference external" href="https://discord.com/api/webhooks/123456789/">https://discord.com/api/webhooks/123456789/</a></p></td>
 </tr>
-<tr class="row-even"><td><p>DB_CONNECTION_STR</p></td>
+<tr class="row-odd"><td><p>DB_CONNECTION_STR</p></td>
 <td><p><strong>(Optional)</strong> Your connection string to your account history database, only needed if you want to save your account history to a database.</p></td>
 <td><p>sqlite:///account_history.db</p></td>
 </tr>
-<tr class="row-odd"><td><p>STRATEGY_NAME</p></td>
+<tr class="row-even"><td><p>STRATEGY_NAME</p></td>
 <td><p><strong>(Optional)</strong> The name of the strategy. This will change the strategy_id in the database and in the Discord messages.</p></td>
 <td><p>My Strategy</p></td>
 </tr>
-<tr class="row-even"><td><p>MARKET</p></td>
+<tr class="row-odd"><td><p>MARKET</p></td>
 <td><p><strong>(Optional)</strong> The market you want the bot to think it is. Eg. “24/7” will make the bot think it is trading in a 24/7 market.</p></td>
 <td><p>24/7</p></td>
 </tr>
-<tr class="row-odd"><td><p>POLYGON_MAX_MEMORY_BYTES</p></td>
+<tr class="row-even"><td><p>POLYGON_MAX_MEMORY_BYTES</p></td>
 <td><p><strong>(Optional)</strong> The maximum memory in bytes that the Polygon API can use. This is useful for limiting memory usage during backtesting.</p></td>
 <td><p>512000000</p></td>
 </tr>

docsrc/_build/html/_sources/deployment.rst.txt

@@ -414,6 +414,9 @@ In addition to broker-specific secrets, the following environment variables are
    * - **Secret**
      - **Description**
      - **Example**
+   * - LUMIWEALTH_API_KEY
+     - Your API key from the BotSpot.trade website so that you can track your bot's performance. To get this API key, visit the `BotSpot.trade <https://botspot.trade/>`_ website and add/create a bot. After creating the bot, you will receive an API key.
+     - 694rr2c8d9234b43a40fab494a79f5634ghd4f39d44ccf2e
    * - LIVE_CONFIG
      - Your live config file, only needed for strategies that have multiple configurations (there will be a folder named "configurations" in the src/ folder) and if you are running the strategy live.
      - paper_1

docsrc/_build/html/deployment.html

@@ -855,35 +855,39 @@ <h1>General Environment Variables<a class="headerlink" href="#general-environmen
 </tr>
 </thead>
 <tbody>
-<tr class="row-even"><td><p>LIVE_CONFIG</p></td>
+<tr class="row-even"><td><p>LUMIWEALTH_API_KEY</p></td>
+<td><p>Your API key from the BotSpot.trade website so that you can track your bot’s performance. To get this API key, visit the <a class="reference external" href="https://botspot.trade/">BotSpot.trade</a> website and add/create a bot. After creating the bot, you will receive an API key.</p></td>
+<td><p>694rr2c8d9234b43a40fab494a79f5634ghd4f39d44ccf2e</p></td>
+</tr>
+<tr class="row-odd"><td><p>LIVE_CONFIG</p></td>
 <td><p>Your live config file, only needed for strategies that have multiple configurations (there will be a folder named “configurations” in the src/ folder) and if you are running the strategy live.</p></td>
 <td><p>paper_1</p></td>
 </tr>
-<tr class="row-odd"><td><p>IS_BACKTESTING</p></td>
+<tr class="row-even"><td><p>IS_BACKTESTING</p></td>
 <td><p><strong>(Optional)</strong> Set to <strong>“True”</strong> to run the strategy in backtesting mode, set to <strong>“False”</strong> to run the strategy live (defaults to False).</p></td>
 <td><p>False</p></td>
 </tr>
-<tr class="row-even"><td><p>POLYGON_API_KEY</p></td>
+<tr class="row-odd"><td><p>POLYGON_API_KEY</p></td>
 <td><p><strong>(Optional)</strong> Your API key from your Polygon account, only needed if you are backtesting.</p></td>
 <td><p>a7py0zIdhxde6QkX8OjjKNp7cD87hwKU</p></td>
 </tr>
-<tr class="row-odd"><td><p>DISCORD_WEBHOOK_URL</p></td>
+<tr class="row-even"><td><p>DISCORD_WEBHOOK_URL</p></td>
 <td><p><strong>(Optional)</strong> Your Discord webhook URL, only needed if you want to send notifications to Discord. Learn how to get a Discord webhook URL here: <a class="reference external" href="https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks">Discord Webhooks</a></p></td>
 <td><p><a class="reference external" href="https://discord.com/api/webhooks/123456789/">https://discord.com/api/webhooks/123456789/</a></p></td>
 </tr>
-<tr class="row-even"><td><p>DB_CONNECTION_STR</p></td>
+<tr class="row-odd"><td><p>DB_CONNECTION_STR</p></td>
 <td><p><strong>(Optional)</strong> Your connection string to your account history database, only needed if you want to save your account history to a database.</p></td>
 <td><p>sqlite:///account_history.db</p></td>
 </tr>
-<tr class="row-odd"><td><p>STRATEGY_NAME</p></td>
+<tr class="row-even"><td><p>STRATEGY_NAME</p></td>
 <td><p><strong>(Optional)</strong> The name of the strategy. This will change the strategy_id in the database and in the Discord messages.</p></td>
 <td><p>My Strategy</p></td>
 </tr>
-<tr class="row-even"><td><p>MARKET</p></td>
+<tr class="row-odd"><td><p>MARKET</p></td>
 <td><p><strong>(Optional)</strong> The market you want the bot to think it is. Eg. “24/7” will make the bot think it is trading in a 24/7 market.</p></td>
 <td><p>24/7</p></td>
 </tr>
-<tr class="row-odd"><td><p>POLYGON_MAX_MEMORY_BYTES</p></td>
+<tr class="row-even"><td><p>POLYGON_MAX_MEMORY_BYTES</p></td>
 <td><p><strong>(Optional)</strong> The maximum memory in bytes that the Polygon API can use. This is useful for limiting memory usage during backtesting.</p></td>
 <td><p>512000000</p></td>
 </tr>

docsrc/deployment.rst

@@ -414,6 +414,9 @@ In addition to broker-specific secrets, the following environment variables are
    * - **Secret**
      - **Description**
      - **Example**
+   * - LUMIWEALTH_API_KEY
+     - Your API key from the BotSpot.trade website so that you can track your bot's performance. To get this API key, visit the `BotSpot.trade <https://botspot.trade/>`_ website and add/create a bot. After creating the bot, you will receive an API key.
+     - 694rr2c8d9234b43a40fab494a79f5634ghd4f39d44ccf2e
    * - LIVE_CONFIG
      - Your live config file, only needed for strategies that have multiple configurations (there will be a folder named "configurations" in the src/ folder) and if you are running the strategy live.
      - paper_1

lumibot/data_sources/data_source.py

@@ -2,6 +2,7 @@
 from abc import ABC, abstractmethod
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from datetime import datetime, timedelta
+import traceback
 import time
 
 import pandas as pd
@@ -307,42 +308,47 @@ def get_bars(
         length,
         timestep="minute",
         timeshift=None,
-        chunk_size=10,
-        max_workers=200,
+        chunk_size=2,
+        max_workers=2,
         quote=None,
         exchange=None,
         include_after_hours=True,
     ):
         """Get bars for the list of assets"""
 
         def process_chunk(chunk):
-            """Process a chunk of assets."""
             chunk_result = {}
             for asset in chunk:
-                chunk_result[asset] = self.get_historical_prices(
-                    asset,
-                    length,
-                    timestep=timestep,
-                    timeshift=timeshift,
-                    quote=quote,
-                    exchange=exchange,
-                    include_after_hours=include_after_hours,
-                )
+                try:
+                    chunk_result[asset] = self.get_historical_prices(
+                        asset,
+                        length,
+                        timestep=timestep,
+                        timeshift=timeshift,
+                        quote=quote,
+                        exchange=exchange,
+                        include_after_hours=include_after_hours,
+                    )
+
+                    # Sleep to prevent rate limiting
+                    time.sleep(0.1)
+                except Exception as e:
+                    # Log once per asset to avoid spamming with a huge traceback
+                    logging.warning(f"Error retrieving data for {asset.symbol}: {e}")
+                    tb = traceback.format_exc()
+                    logging.warning(tb)  # This prints the traceback
+                    chunk_result[asset] = None
             return chunk_result
 
         # Convert strings to Asset objects
         assets = [Asset(symbol=a) if isinstance(a, str) else a for a in assets]
 
-        # Chunking the assets
+        # Chunk the assets
         chunks = [assets[i : i + chunk_size] for i in range(0, len(assets), chunk_size)]
 
-        # Initialize ThreadPoolExecutor
         results = {}
         with ThreadPoolExecutor(max_workers=max_workers) as executor:
-            # Submit tasks
             futures = [executor.submit(process_chunk, chunk) for chunk in chunks]
-
-            # Collect results as they complete
             for future in as_completed(futures):
                 results.update(future.result())
 

lumibot/entities/bars.py

@@ -112,11 +112,11 @@ def __init__(self, df, source, asset, quote=None, raw=None):
         self._raw = raw
 
         if "dividend" in df.columns:
-            df["price_change"] = df["close"].pct_change()
-            df["dividend_yield"] = df["dividend"] / df["close"]
-            df["return"] = df["dividend_yield"] + df["price_change"]
+            df.loc[:, "price_change"] = df["close"].pct_change()
+            df.loc[:, "dividend_yield"] = df["dividend"] / df["close"]
+            df.loc[:, "return"] = df["dividend_yield"] + df["price_change"]
         else:
-            df["return"] = df["close"].pct_change()
+            df.loc[:, "return"] = df["close"].pct_change()
 
         self.df = df
 

lumibot/tools/yahoo_helper.py

@@ -1,6 +1,7 @@
 import logging
 import os
 import pickle
+import time
 from datetime import datetime, timedelta
 
 import pandas as pd
@@ -11,6 +12,7 @@
 from .helpers import get_lumibot_datetime
 
 INFO_DATA = "info"
+INVALID_SYMBOLS = set()
 
 
 class _YahooData:
@@ -91,6 +93,10 @@ def dump_pickle_file(symbol, type, data):
 
     @staticmethod
     def format_df(df, auto_adjust):
+        # Check if df is empty
+        if df is None or df.empty:
+            return df
+
         if auto_adjust:
             del df["Adj Ratio"]
             del df["Close"]
@@ -169,41 +175,104 @@ def get_symbol_last_price(symbol):
 
         return df["Close"].iloc[-1]
 
-    @staticmethod
     def download_symbol_data(symbol, interval="1d"):
-        ticker = yf.Ticker(symbol)
-        try:
-            if interval == "1m":
-                # Yahoo only supports 1 minute interval for past 7 days
-                df = ticker.history(interval=interval, start=get_lumibot_datetime() - timedelta(days=7), auto_adjust=False)
-            elif interval == "15m":
-                # Yahoo only supports 15 minute interval for past 60 days
-                df = ticker.history(interval=interval, start=get_lumibot_datetime() - timedelta(days=60), auto_adjust=False)
-            else:
-                df = ticker.history(interval=interval, period="max", auto_adjust=False)
-        except Exception as e:
-            logging.debug(f"Error while downloading symbol day data for {symbol}, returning empty dataframe for now.")
-            logging.debug(e)
+        """
+        Attempts to download historical data from yfinance for the specified symbol and interval.
+        Retries on empty/None data in case of transient rate limits.
+        If all attempts fail, marks the symbol as invalid (added to INVALID_SYMBOLS) to skip it in future.
+        If symbol info is unavailable, we just skip timezone adjustments (do not return None).
+        """
+
+        # If we've already marked this symbol invalid, skip further calls
+        if symbol in INVALID_SYMBOLS:
+            logging.debug(f"{symbol} is already marked invalid. Skipping yfinance calls.")
             return None
 
-        # Adjust the time when we are getting daily stock data to the beginning of the day
-        # This way the times line up when backtesting daily data
-        info = YahooHelper.get_symbol_info(symbol)
-        if info.get("info") and info.get("info").get("market") == "us_market":
-            # Check if the timezone is already set, if not set it to the default timezone
-            if df.index.tzinfo is None:
-                df.index = df.index.tz_localize(info.get("info").get("exchangeTimezoneName"))
-            else:
-                df.index = df.index.tz_convert(info.get("info").get("exchangeTimezoneName"))
-            df.index = df.index.map(lambda t: t.replace(hour=16, minute=0))
-        elif info.get("info") and info.get("info").get("market") == "ccc_market":
-            # Check if the timezone is already set, if not set it to the default timezone
-            if df.index.tzinfo is None:
-                df.index = df.index.tz_localize(info.get("info").get("exchangeTimezoneName"))
+        ticker = yf.Ticker(symbol)
+
+        # --- HISTORICAL DATA RETRY LOGIC ---
+        max_retries = 3
+        sleep_sec = 1
+        df = None
+
+        for attempt in range(1, max_retries + 1):
+            try:
+                if interval == "1m":
+                    df = ticker.history(
+                        interval=interval,
+                        start=get_lumibot_datetime() - timedelta(days=7),
+                        auto_adjust=False
+                    )
+                elif interval == "15m":
+                    df = ticker.history(
+                        interval=interval,
+                        start=get_lumibot_datetime() - timedelta(days=60),
+                        auto_adjust=False
+                    )
+                else:
+                    df = ticker.history(
+                        interval=interval,
+                        period="max",
+                        auto_adjust=False
+                    )
+            except Exception as e:
+                logging.debug(f"{symbol}: Exception from ticker.history(): {e}")
+                if attempt < max_retries:
+                    logging.debug(f"{symbol}: Attempt {attempt} failed. Sleeping {sleep_sec}s, then retry.")
+                    time.sleep(sleep_sec)
+                    sleep_sec *= 2
+                    continue
+                else:
+                    logging.debug(f"{symbol}: All {max_retries} attempts failed. Marking invalid.")
+                    INVALID_SYMBOLS.add(symbol)
+                    return None
+
+            if df is None or df.empty:
+                logging.debug(f"{symbol}: Attempt {attempt} returned empty or None data.")
+                if attempt < max_retries:
+                    logging.debug(f"{symbol}: Sleeping {sleep_sec}s, then retry.")
+                    time.sleep(sleep_sec)
+                    sleep_sec *= 2
+                else:
+                    logging.debug(f"{symbol}: Data still empty after {max_retries} attempts. Marking invalid.")
+                    INVALID_SYMBOLS.add(symbol)
+                    return None
             else:
-                df.index = df.index.tz_convert(info.get("info").get("exchangeTimezoneName"))
-            df.index = df.index.map(lambda t: t.replace(hour=23, minute=59))
+                # Successfully got data, so break out of the loop
+                break
 
+        # --- SYMBOL INFO (OPTIONAL) ---
+        info = None
+        try:
+            info = YahooHelper.get_symbol_info(symbol)
+        except Exception as e:
+            logging.debug(f"{symbol}: Exception from get_symbol_info(): {e}")
+
+        # If we have valid info, handle timezone adjustments.
+        # Using sub_info to avoid accessing .get() on None.
+        if info and isinstance(info, dict):
+            sub_info = info.get("info", {})
+            if isinstance(sub_info, dict):
+                market = sub_info.get("market", "")
+                tz_name = sub_info.get("exchangeTimezoneName", None)
+
+                # US market
+                if market == "us_market" and tz_name:
+                    if df.index.tzinfo is None:
+                        df.index = df.index.tz_localize(tz_name)
+                    else:
+                        df.index = df.index.tz_convert(tz_name)
+                    df.index = df.index.map(lambda t: t.replace(hour=16, minute=0))
+
+                # Crypto/CCC market
+                elif market == "ccc_market" and tz_name:
+                    if df.index.tzinfo is None:
+                        df.index = df.index.tz_localize(tz_name)
+                    else:
+                        df.index = df.index.tz_convert(tz_name)
+                    df.index = df.index.map(lambda t: t.replace(hour=23, minute=59))
+
+        # Finally, run any custom DataFrame processing
         df = YahooHelper.process_df(df, asset_info=info)
         return df
 

setup.py

@@ -5,7 +5,7 @@
 
 setuptools.setup(
     name="lumibot",
-    version="3.8.24",
+    version="3.8.26",
     author="Robert Grzesik",
     author_email="[email protected]",
     description="Backtesting and Trading Library, Made by Lumiwealth",
@@ -21,7 +21,8 @@
         "yfinance>=0.2.46",
         "matplotlib>=3.3.3",
         "quandl",
-        "numpy>=1.20.0",
+        # Numpy greater than 1.20.0 and less than 2 because v2 has compatibility issues with a few libraries
+        "numpy>=1.20.0, <2",
         "pandas>=2.2.0",
         "pandas_market_calendars>=4.3.1",
         "plotly>=5.18.0",