Building Blocks of Algorithmic Trading: Collecting End-of-Day Data

Algorithmic trading begins not with signals or models, but with data — specifically, clean, consistent, and reliable End-of-Day (EOD) market data. Whether you're building a trend-following system or a multi-factor screener, your models are only as good as the data they run on. In this first article of our algorithmic trading series, we explore the essential sources of EOD data and demonstrate how to store and access it efficiently using Redis, a high-performance in-memory database.

Why EOD Data?

EOD data — which includes open, high, low, close prices and volume (OHLCV) at the end of each trading day — is the bedrock for many algorithmic trading strategies, especially those that don’t require high-frequency execution. It's cleaner, more accessible, and less volatile than intraday data, making it ideal for:

• Backtesting strategies
• Screening assets
• Generating entry/exit signals
• Risk and portfolio management

Where Can You Get EOD Data?

Here are several common and reliable ways to collect EOD data:

1. Professional Market Data Providers

If data cost is not an issue, consider subscribing to a professional provider:

• Quandl (now part of Nasdaq Data Link) – Offers premium and free datasets.
• EOD Historical Data – Affordable, easy-to-integrate REST API with global coverage.
• Tiingo – Clean, high-quality financial data with generous free API tiers.

These providers often include dividend and split adjustments, which are essential for accurate backtesting.

2. Investment-Oriented Websites

Web scraping from investment portals is possible, but not always recommended for production systems:

• Investing.com – Offers EOD data for equities, indices, forex, and commodities.
• Yahoo Finance – Widely used; its unofficial API can be accessed using libraries like yfinance.
• Market Index – Australian oriented investment website with free ASX EOD data provided.

When using Investing.com, you might come across the Python library investpy. While it offers a convenient interface, it hasn’t worked reliably in my experience:

Copy Error retrieving data for TSLA: ERR#0015: error 403, try again later.

Luckly "investing.com" still provides a function on their website to export the historial data for many instruments. So the worst case scenario is to download the data manualy from it.

3. Broker APIs

Many brokers provide EOD data via REST or WebSocket APIs:

• Interactive Brokers (IBKR) – Offers EOD historical data via their TWS API.
• Alpaca – Provides free stock EOD data through their REST API.
• IG, TD Ameritrade, OANDA, Polygon.io, and others also offer options.

Always check for:

• Historical limits
• Frequency caps
• Adjustments for splits/dividends

The problem with broker APIs is that each broker has their own unique interface and data structure. Therefore, you may need to write custom code to handle the data retrieval and storage.

4. Trading Platforms

Platforms like MetaTrader 5 (MT5) offer built-in historical data:

• MT5 can export EOD data directly from chart history.
• Supported by most brokers
• You can use Python with MetaTrader5 API (via MetaTrader5 Python package) to automate downloads.

Copy from MetaTrader5 import *
initialize()
rates = copy_rates_range("EURUSD", TIMEFRAME_D1, start, end)

Using MetadataTrader5 is a great way to get EOD data, but be aware of the limitations:

• On available to Windows users: The python library for the MetaTrader5 platform is only available on Windows, so you need to run your code on a Windows machine or use a virtual machine.
• Limited to Forex and CFDs: Not all assets are available.
• Data Quality: Ensure the data is clean and adjusted for splits/dividends.

Personally, I prefer using the MetaTrader5 API for retrieving EOD trading data. While its coverage of tradable assets is somewhat limited, it offers more than enough instruments to get started. Unlike some broker-specific APIs, MetaTrader 5 is widely adopted by traders, supported by many brokers, and backed by a large community.

A Python script will be provided to help you download EOD data directly from the MetaTrader5 platform. Setting up a demo account is also straightforward, making it easy to begin testing right away.

Storing EOD Data with Redis

Once you've collected your data, you need fast, structured access for backtesting and model development. Redis, an in-memory key-value store, is an excellent choice for its speed and simplicity.

Why Redis?

• Ultra-fast read/write
• Easy to structure time-series data
• Built-in expiration and memory management
• Great for small-to-medium-sized datasets or caching layer

Redis Data Schema

For EOD data, we can use a key, subkey, value structure:

• Key: [PREFIX]:symbol (e.g., nasdaq:AAPL)
• Subkey: date (e.g., 20250510)
• Value: JSON string of OHLCV data
• Example: nasdaq:AAPL:20250510 -> {"open": 180.12, "high": 182.55, "low": 179.70, "close": 181.30, "volume": 25300000}

Since instruments traded on different markets or exchanges can share the same symbol, it's important to define a unique key for each asset. To achieve this, we introduce a prefix for each symbol, separated by a colon.

The format is:
<EXCHANGE_PREFIX>:<SYMBOL>

Examples of common prefixes:

• nasdaq:AAPL — Apple Inc. on the NASDAQ
• nyse:IBM — IBM on the New York Stock Exchange
• asx:BHP — BHP Group on the Australian Securities Exchange
• crypto:BTCUSD — Bitcoin as a cryptocurrency
• forex:EURUSD — Euro to US Dollar exchange rate
• metals:XAUUSD — Gold to US Dollar exchange rate
• energies:XBRUSD — Brent Crude Oil to US Dollar exchange rate
• indices:SPX500 — S&P 500 Index

You can define the prefix based on the exchange or asset class you're working with. This convention ensures consistency and avoids symbol collisions across your dataset.

Run Redis Locally

To run Redis locally, you can use Docker Compose. Here’s a simple command to get started:

If you are on Linux, you can use the following command to run Redis locally:

Copy cd redis
docker-compose up -d

If you are on Windows, you can contact me if you need help to run Redis locally.

From MetaTrader to Redis: Storing EOD Data Efficiently

Downloading EOD Data from MetaTrader5

Now it is time to get our hands dirty and start collecting EOD data. I hope you are comfortable with command line (Command Prompt) and Python.

As discussed, we prefer to use the MetaTrader5 API to download EOD data. As Python is the only language supported by MetaTrader5, we have to use the Python with MetaTrader5 Python package to download EOD data and store it in Redis.

I have made a simple script to download EOD data from MetaTrader5 and save them into files.

You need to have the both Python and MetaTrader5 platform installed and running on your machine. You can download the MetaTrader5 platform from the official website.

Then, download the code repository from GitHub:

Copy git clone https://github.com/tyolab/tyosis-import.git
cd tyosis-import
pip install -r requirements.txt

To run the data download script, ensure you have MetaTrader 5 installed and running, and that you are logged into your trading account. Then for example, to download EOD data for XAUUSD (Gold) from 2025-01-01 to today:

Copy python utils\mt5.py XAUUSD 2025

The content of downloaded looks like this:

Copy symbol,date,open,high,low,close,volume
XAUUSD,02/01/2025,2624.78,2660.34,2621.7,2657.66,198834
XAUUSD,03/01/2025,2657.49,2665.26,2636.63,2639.46,169658
XAUUSD,06/01/2025,2641.42,2649.44,2614.37,2635.92,229184
XAUUSD,07/01/2025,2636.07,2664.16,2632.88,2648.79,205976
XAUUSD,08/01/2025,2648.42,2669.91,2645.29,2661.36,194517
XAUUSD,09/01/2025,2661.5,2678.11,2655.45,2669.91,151068
XAUUSD,10/01/2025,2670.79,2697.79,2663.96,2688.75,244597
XAUUSD,13/01/2025,2689.1,2693.39,2656.55,2662.87,289738
XAUUSD,14/01/2025,2663.47,2678.0,2659.89,2676.74,189931
XAUUSD,15/01/2025,2678.15,2697.32,2669.31,2696.63,176113
XAUUSD,16/01/2025,2694.86,2724.67,2689.89,2713.7,194921
XAUUSD,17/01/2025,2713.75,2717.32,2699.23,2702.2,172786
XAUUSD,20/01/2025,2702.94,2713.68,2689.31,2707.06,169458
XAUUSD,21/01/2025,2706.95,2745.93,2702.88,2744.69,224338

A detailed explanation of how to use the script is provided below:

Copy python utils\mt5.py <symbol> <year> 

Replace <symbol> with the trading symbol you want to download data for (e.g., EURUSD) and <year> with the year you are interested in (e.g., 2023). The script will then connect to your MetaTrader 5 platform, download the end-of-day historical data for the specified symbol and year, and save it to a text file named YYYY.txt within a directory structure based on the market and symbol (e.g., data/Forex/EURUSD/2023.txt). If the file already exists, the script will append new data to it, starting from the day after the last recorded date.

If you want to download data for the last 20 years of Gold (XAUUSD), you can run the following command:

Copy for /L %i in (2005,1,2025) do python utils\mt5.py XAUUSD %i

Now we have the EOD data stored in text files. The next step is to load this data into Redis.

Loading EOD Data into Redis

To load the EOD data into Redis, we can use the utils/import.py script provided in the repository. This script will read the text files and store the data in Redis using the key structure we defined earlier.

For example,

Copy python utils/import.py --key-prefix "metals:" /data/Metals/XAUUSD/2025.txt 

If you would like to import the last 20 years of Gold (XAUUSD) data, you can run the following command:

Copy for /L %i in (2005,1,2025) do python utils/import.py --key-prefix "metals:" /data/Metals/XAUUSD/%i.txt 

Checking the Data in Redis

Copy redis-cli
>hgetall metals:XAUUSD
  1) "20250307"
  2) "{\"O\": 2911.85, \"H\": 2930.43, \"L\": 2896.95, \"C\": 2910.82, \"V\": 164598}"
  3) "20250127"
  4) "{\"O\": 2768.22, \"H\": 2772.58, \"L\": 2730.49, \"C\": 2740.11, \"V\": 264527}"
  5) "20250315"
  6) "{\"O\": 2983.94, \"H\": 2988.8, \"L\": 2981.73, \"C\": 2986.03, \"V\": 2279}"
  7) "20250211"
  8) "{\"O\": 2908.07, \"H\": 2942.65, \"L\": 2881.57, \"C\": 2897.44, \"V\": 252658}"
  9) "20250206"
 10) "{\"O\": 2867.64, \"H\": 2873.29, \"L\": 2834.22, \"C\": 2855.5, \"V\": 198609}"
 11) "20250116"
 12) "{\"O\": 2694.86, \"H\": 2724.67, \"L\": 2689.89, \"C\": 2713.7, \"V\": 194921}"
 13) "20250321"
 14) "{\"O\": 3044.99, \"H\": 3047.41, \"L\": 2999.37, \"C\": 3023.0, \"V\": 119857}"

Best Practices

• Normalize your data: Adjust for splits/dividends if applicable.
• Consistency: Store all assets using the same structure.
• Backup: Regularly export Redis snapshots (RDB) or enable AOF for persistence.
• Metadata: Store additional info like timezone (if you need data with smaller timeframe like h4, h1, or m15 etc), data source, and update timestamp.

Final Thoughts

Before building signals, portfolios, or trading bots, build your data foundation. Collecting clean EOD data and storing it efficiently gives you the confidence that your backtests are meaningful and your signals trustworthy.

Now that you've gathered EOD data for your chosen instruments, you have a solid foundation for backtesting your trading strategies. You can start exploring different indicators, rules, and risk management techniques to see how they would have performed historically. Remember to thoroughly evaluate your backtests and consider factors like transaction costs and slippage to get a realistic assessment of your strategy's potential.