
{/* banner prompt: Cinematic realistic photograph, 16:9 wide banner composition: a laptop on a wooden desk showing blurred lines of code in a dark editor, beside it a second screen with an abstract heatmap of green and red blocks, a notebook with hand-drawn diagrams, warm desk lamp light at night. Muted teal and warm amber colour grade, soft natural light, shallow depth of field, high detail, photorealistic, no readable text, no numbers, no logos, no visible faces. */}

This is the final post in the order book craft track of the [ASX order book series](/blog/2026/10/12-reading-the-asx-order-book-how-intraday-traders-actually-find-edge/). The previous seven posts described what experienced intraday traders look at: [depth over charts](/blog/2026/10/13-why-the-best-intraday-traders-barely-look-at-charts/), [protection](/blog/2026/10/14-protection-the-liquidity-you-lean-on/), the [opening auction](/blog/2026/10/15-reading-the-asx-opening-auction/), [gap cause](/blog/2026/10/16-why-did-it-gap-mechanical-gaps-vs-information-gaps/), [cross-market leads](/blog/2026/10/17-cross-market-leads-and-the-indicative-open/), [fake and hidden size](/blog/2026/10/18-spoofs-icebergs-and-reloading-orders/) and [time-based exits](/blog/2026/10/19-when-the-tape-stops-confirming-time-stops-and-liquidity-exits/).

All of that is discretionary. A good trader does it by eye in a few seconds. But almost every piece of it can be expressed as a number, and once it's a number we can test whether it actually predicts anything. That's the job of this post.

As I've written before, [an indicator is a measurement, not a decision](/blog/2026/09/13-indicators-measure-strategies-decide-alpha-is-what-survives/). Everything below is measurement. Whether any of it becomes a strategy depends on what the testing says.

<KeyPoints>

- Python features for imbalance, microprice, protection, resistance, wall persistence and auction mispricing.
- Label with short forward horizons and evaluate per event or per day, after costs and latency.
- The likely edge is in narrow event setups, not in a continuous imbalance signal.

</KeyPoints>

## Getting depth data

You need order book snapshots: at a given timestamp, the price and volume at each of the top N levels on each side. For ASX stocks, options include your broker's API if it provides market depth (Interactive Brokers, for example, offers a market depth request, subject to data subscriptions), or a commercial data vendor. Depth data is bulky, so most people capture snapshots at a fixed interval or on every change for a short watchlist.

I'll assume a simple schema:

```python
# one row per snapshot
# ts, code, bid_px_1..N, bid_sz_1..N, ask_px_1..N, ask_sz_1..N
import numpy as np
import pandas as pd

N = 5
BID_PX = [f"bid_px_{i}" for i in range(1, N + 1)]
BID_SZ = [f"bid_sz_{i}" for i in range(1, N + 1)]
ASK_PX = [f"ask_px_{i}" for i in range(1, N + 1)]
ASK_SZ = [f"ask_sz_{i}" for i in range(1, N + 1)]
```

## Feature 1: order book imbalance

The classic measure: how lopsided is the visible depth near the touch?

```python
def imbalance(df: pd.DataFrame, levels: int = 3, decay: float = 0.5) -> pd.Series:
    """Weighted depth imbalance in [-1, 1]. Positive = more bid depth."""
    w = np.array([decay ** i for i in range(levels)])
    bids = df[BID_SZ[:levels]].to_numpy() @ w
    asks = df[ASK_SZ[:levels]].to_numpy() @ w
    return pd.Series((bids - asks) / (bids + asks), index=df.index)
```

The decay weights nearer levels more heavily, because size one tick away matters more than size five ticks away.

## Feature 2: microprice

The midpoint ignores size. The microprice tilts the mid toward the side with *less* depth, because that side is more likely to be consumed next.

```python
def microprice(df: pd.DataFrame) -> pd.Series:
    b, a = df["bid_px_1"], df["ask_px_1"]
    qb, qa = df["bid_sz_1"], df["ask_sz_1"]
    return (b * qa + a * qb) / (qa + qb)
```

Microprice minus mid, expressed in ticks, is a neat one-number summary of top-of-book pressure.

## Feature 3: protection ratio

This is the discretionary idea from the protection post, made explicit: depth behind an entry, relative to how much volume normally trades.

```python
def protection_ratio(df: pd.DataFrame, side: int, recent_vol_per_min: pd.Series,
                     levels: int = 3) -> pd.Series:
    """side=+1 for a long (protection = bids), -1 for a short (protection = asks).
    recent_vol_per_min: rolling average traded volume per minute, aligned to df.
    """
    cols = BID_SZ[:levels] if side == 1 else ASK_SZ[:levels]
    depth = df[cols].sum(axis=1)
    return depth / recent_vol_per_min.replace(0, np.nan)
```

A ratio of 10 means it would take roughly ten minutes of normal trading to consume the protection. A ratio of 0.5 means it could be gone in thirty seconds.

## Feature 4: resistance ahead

The twin of protection: how much stands in the way of the move you want?

```python
def resistance_ratio(df: pd.DataFrame, side: int, levels: int = 3) -> pd.Series:
    """Depth ahead of the trade divided by depth behind it. Lower is better."""
    ahead = df[(ASK_SZ if side == 1 else BID_SZ)[:levels]].sum(axis=1)
    behind = df[(BID_SZ if side == 1 else ASK_SZ)[:levels]].sum(axis=1)
    return ahead / behind.replace(0, np.nan)
```

## Feature 5: wall persistence

To deal with spoofs, measure whether large orders survive. One simple approach: track the maximum single-level size near the touch, and how many consecutive snapshots a level above some size threshold has existed at the same price.

```python
def wall_persistence(df: pd.DataFrame, side: int, threshold: float) -> pd.Series:
    """Consecutive snapshots that a level >= threshold has existed at the same price."""
    px_cols, sz_cols = (BID_PX, BID_SZ) if side == 1 else (ASK_PX, ASK_SZ)
    wall_px = []
    for _, row in df.iterrows():
        big = [row[p] for p, s in zip(px_cols, sz_cols) if row[s] >= threshold]
        wall_px.append(big[0] if big else np.nan)
    wall_px = pd.Series(wall_px, index=df.index)
    same = wall_px.eq(wall_px.shift()) & wall_px.notna()
    return same.groupby((~same).cumsum()).cumsum()
```

A wall that has persisted, and especially one that has persisted while volume traded at its price, is more trustworthy than one that just appeared.

## Feature 6: auction mispricing

At the open, the most useful single feature from the earlier posts is how far the indicative gap deviates from what the overnight lead and sector peers imply.

```python
def auction_mispricing(indicative_gap: pd.Series, lead_move: pd.Series,
                       beta: pd.Series, peer_median_gap: pd.Series,
                       w_lead: float = 0.5) -> pd.Series:
    """Positive = indicated too high versus references (fade candidate)."""
    expected = w_lead * beta * lead_move + (1 - w_lead) * peer_median_gap
    return indicative_gap - expected
```

Combined with the auction surplus sign and a protection ratio from the pre-open book, this is a reasonable first attempt at encoding the "is the open mispriced?" judgement.

## Labelling: what are we predicting?

Features need a target. For scalping-style questions the natural targets are short horizons:

```python
def forward_return_ticks(mid: pd.Series, tick: float, horizon: str) -> pd.Series:
    """Mid-price change over a forward time horizon, in ticks."""
    fwd = mid.shift(freq=f"-{horizon}").reindex(mid.index, method="ffill")
    return (fwd - mid) / tick
```

Use several horizons (5 seconds, 30 seconds, 2 minutes). Different features tend to work at different horizons, and the time-stop post suggests that most of the edge in scalps shows up very early.

## Evaluating without fooling yourself

This is the part people skip. A few rules I apply:

<Table data={
    {
        headers: ['Pitfall', 'Guard'],
        rows: [
            ['Lookahead', 'Only use information available at the snapshot timestamp. Rolling volumes must be trailing.'],
            ['Mid-price fantasy', 'Mid moves are not tradeable. Re-test with entry at the touch you would actually pay.'],
            ['Costs', 'Subtract brokerage and the spread. Many imbalance signals vanish after costs.'],
            ['Latency', 'Add a realistic delay between signal and order. A one second delay changes a lot.'],
            ['Overlapping samples', 'Snapshots seconds apart are not independent. Evaluate per event or per day.'],
            ['Regime', 'Split by period and by market condition. A feature that only worked in one month is noise.'],
            ['Universe', 'Test on stocks you would actually trade, with similar liquidity and tick size.'],
        ]
    }
} />

A simple first evaluation is to bucket a feature into quantiles and look at the average forward return in each bucket, after costs, with confidence intervals computed per day rather than per snapshot.

```python
def bucket_report(feature: pd.Series, fwd: pd.Series, q: int = 5) -> pd.DataFrame:
    df = pd.concat({"f": feature, "y": fwd}, axis=1).dropna()
    df["bucket"] = pd.qcut(df["f"], q, labels=False, duplicates="drop")
    return df.groupby("bucket")["y"].agg(["count", "mean", "std"])
```

If the top and bottom buckets don't separate clearly after costs, the feature isn't useful at that horizon, however intuitive it looked.

## What I expect to find

I'll be honest about priors. Raw top-of-book imbalance is one of the most studied microstructure signals there is, and most of its edge goes to participants who are much faster than any human or retail system. I don't expect it to survive costs on its own.

What I'm more interested in is the *combination* the discretionary traders used, at the specific moments they used it: a mispriced auction, plus protection that is large relative to volume, plus little resistance ahead, plus a fast time stop. That's a narrow, event-driven setup, not a continuous signal, and it's where human depth readers seem to have found their edge. Whether it holds up in data is exactly the kind of question I'll be testing in our own systems.

That wraps up the order book craft track. Next, we move to the business side of trading, starting with the most misunderstood pair of numbers in any trading journal.

<SeriesNav series="asx-order-book" />

<Disclaimer />


49212afc165b75b602da1f5391f84e71862eff13bc3db9741fba959ae94bb105

Python examples for computing order book imbalance, microprice, protection ratio and auction mispricing from market depth snapshots, and how to backtest them.

Turning discretionary depth reading into numbers you can test. Python code for order book imbalance, microprice, a protection ratio, a resistance score and an auction mispricing feature, plus how to label and evaluate them honestly without fooling yourself.

Measuring Order Book Imbalance in Code


{/* banner prompt: A playful but polished 3D clay-render, matte materials in sage green, cream and terracotta, of a stylized detective magnifying glass hovering over a small padlock icon that is cracking open, with a tangled network cable looping around a proxy-server box in the background, soft studio shadows, simple background, 16:9. */}

`yt-dlp` gave us the least helpful error message it has: `HTTP Error 410: Gone`. The standard fix is `yt-dlp -U`. We ran it. Already on the latest stable release. The video was not, in fact, gone, but proving that took three separate bugs stacked on top of each other, and the last one had nothing to do with `yt-dlp` at all.

We were debugging a small local downloader tool: a browser extension that hands a page URL and cookies to `yt-dlp` on the user's own machine, optionally routed through a local HTTP proxy for geo-unblocking. One specific site started failing every time, on a URL that worked fine a week earlier.

## The first clue was buried in `-v`

The plain error told us nothing. Adding `-v` for verbose output surfaced a line that's easy to scroll past:

```
WARNING: [SiteExtractor] The extractor is attempting impersonation, but no
impersonate target is available. If you encounter errors, then see
https://github.com/yt-dlp/yt-dlp#impersonation for information on
installing the required dependencies
```

That's the real signal. A growing number of sites fingerprint the TLS handshake and HTTP/2 frame ordering of incoming requests to tell a real browser apart from a scripted client, even when every header looks identical. `yt-dlp` can spoof a browser's TLS fingerprint via an optional dependency, `curl_cffi`, which wraps `libcurl` with impersonation profiles for Chrome, Safari, Firefox and so on. Without it, some extractors still try the request, get rejected at the TLS layer, and the site's edge returns a generic `410` instead of anything that explains what actually happened. The video was fine. `yt-dlp` just didn't look enough like a browser to be allowed to ask.

## The version window nobody tells you about

The fix looked trivial:

```sh
python3 -m pip install curl_cffi
```

It installed cleanly, at `curl_cffi-0.16.0`, the latest release. `yt-dlp` still refused to use it, and `--list-impersonate-targets` came back with every target marked `(unavailable)`. The debug line was the giveaway:

```
[debug] Optional libraries: ... curl_cffi-0.16.0 (unsupported) ...
```

`yt-dlp` doesn't just check that `curl_cffi` is importable. It pins a narrow set of versions it actually trusts, because `curl_cffi`'s impersonation API has changed shape across releases. Digging through the installed `yt-dlp` build (it ships as a zip-app; `python -m zipfile` opens it like any other zip) turned up the exact check:

```python
curl_cffi_version = tuple(map(int, re.split(r'[^\d]+', curl_cffi.__version__)[:3]))

if curl_cffi_version != (0, 5, 10) and not (0, 10) <= curl_cffi_version < (0, 16):
    curl_cffi._yt_dlp__version = f'{curl_cffi.__version__} (unsupported)'
    raise ImportError('Only curl_cffi versions 0.5.10 and 0.10.x through 0.15.x are supported')
```

Latest-and-greatest was one minor version too new. The fix was to pin below the ceiling:

```sh
python3 -m pip install "curl_cffi>=0.10,<0.16"
```

**Lesson:** when a library depends on another native-extension package for an optional feature, don't assume "latest" is "supported." Check for a version gate specifically on the feature you're trying to turn on, not just on whether the import succeeds.

## Collateral damage from `--force-reinstall`

The first install attempt used `--force-reinstall` to make sure the correct version actually took. It did, but it also silently upgraded packages that were already correctly pinned for something else entirely: `certifi`, `cffi`, `markdown-it-py`, and `pygments` all jumped to newer versions than another tool in the same shared Python environment required. `pip` said so plainly, right at the end of the install:

```
ERROR: pip's dependency resolver does not currently take into account all
the packages that are installed. This behaviour is the source of the
following dependency conflicts.
some-other-tool 2.15.0 requires certifi==2026.4.22, but you have
certifi 2026.7.22 which is incompatible.
some-other-tool 2.15.0 requires cffi==2.0.0, but you have cffi 2.1.1 ...
```

None of those four packages actually needed upgrading; `curl_cffi`'s own requirement (`cffi>=2.0.0`, `certifi>=2024.2.2`) was already satisfied by the versions in place. `--force-reinstall` doesn't just reinstall the package you named; it re-resolves and reinstalls its entire dependency subtree, and when it does that it grabs the latest version satisfying each constraint rather than leaving an already-satisfied pin alone. In a shared base environment with several unrelated tools installed, that's a quiet way to break something you weren't even looking at.

The fix was to pin those four straight back:

```sh
python3 -m pip install "certifi==2026.4.22" "cffi==2.0.0" \
  "markdown-it-py==4.0.0" "pygments==2.20.0"
```

**Lesson:** avoid `--force-reinstall` in a shared or base Python environment. If you need a specific version of one package, ask for exactly that (`pip install "pkg==x.y.z"`) rather than forcing a full dependency re-resolve you don't control.

## A new error, and a wrong theory

With the right `curl_cffi` version in place, the impersonation warning was gone, replaced by a new one, appearing on every single attempt:

```
SSLError: Failed to perform, curl: (35) TLS connect error:
error:00000000:invalid library (0):OPENSSL_internal:invalid library (0)
```

This looks exactly like a known class of `curl_cffi` bug: its bundled BoringSSL build can clash with another OpenSSL-family library already loaded in the same Python process, commonly `cryptography`, pulled in transitively by something unrelated like browser-cookie decryption support. The natural fix is isolation: a clean virtual environment with only `yt-dlp` and `curl_cffi`, nothing that could drag in a conflicting crypto library.

We built that environment. Same error, 100% reproducible, in complete isolation: no `cryptography`, no `secretstorage`, nothing else installed at all.

That result was more useful than a fix would have been. A clean-environment reproduction rules out an entire category of theories in one shot. It wasn't a Python packaging conflict. It had to be something outside Python's import graph.

## Stripping back to the dumbest possible test

The next step was to remove every layer of abstraction and test the actual thing that was failing: a TLS handshake over the proxy. Not `curl_cffi`. Not even `yt-dlp`. Plain, unimpersonated `requests`:

```python
import requests
r = requests.get("https://www.example.com",
                  proxies={"https": "http://127.0.0.1:8082"})
```

Same SSL error. Against a domain with nothing to do with the original site. That was the moment the whole investigation flipped: this was never a `curl_cffi` bug, or even a Python bug. Something about the proxy itself was breaking HTTPS.

Raw `curl` confirmed it in the plainest terms available:

```
* CONNECT tunnel established, response 200
> Client hello (1)
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.com:443
```

The proxy's `CONNECT` tunnel came up cleanly, with a `200 Connection Established`, and then the TLS handshake died immediately afterward, for every client, on every site. Nothing about `curl_cffi`'s TLS fingerprint mattered; the tunnel itself wasn't relaying bytes correctly once real TLS traffic started flowing through it.

The proxy client had several selectable exit gateways by region. Switching from one region to another and rerunning the exact same `curl` command fixed it instantly: clean `200`, valid response, every time. The original gateway node was in a broken state for HTTPS relaying specifically; nothing in our code, `yt-dlp`, or `curl_cffi` was ever going to fix that from the client side.

## What actually shipped

Three real problems, three real fixes, none of them where the first error message pointed:

<Table data={
    {
        headers: ['Symptom', 'Looked like', 'Actually was'],
        rows: [
            ['HTTP 410: Gone', 'Stale yt-dlp / dead video', 'Site requires TLS impersonation'],
            ['curl_cffi "(unsupported)"', 'Broken install', "Version above yt-dlp's pinned ceiling"],
            ['Unrelated tool broke after install', 'Unrelated bug', '--force-reinstall re-resolved shared deps'],
            ['SSL "invalid library" error', 'curl_cffi vs OpenSSL clash', 'Proxy gateway failing the TLS relay'],
        ]
    }
} />

The lasting fix was to isolate `curl_cffi`-enabled `yt-dlp` into its own virtual environment, so a future dependency bump anywhere else on the machine can never touch it again, and to symlink the `PATH`-resolved `yt-dlp` at that environment's build, so every caller picks it up without extra configuration.

## The general shape of this kind of bug

None of the individual fixes here are exotic. What made this one take longer than it should have was trusting each error message's own framing for too long:

- **"410 Gone" told us the video was gone.** It wasn't. It was an anti-bot response wearing the shape of a different error.
- **"unsupported" told us the install was broken.** It wasn't broken, it was one version too new for a pin we didn't know existed.
- **"invalid library" pointed straight at `curl_cffi` and OpenSSL.** It was neither. It was a proxy gateway with a bad TLS relay.

The pattern that actually broke each false lead was the same one every time: reproduce with the fewest possible moving parts. A clean virtual environment ruled out the packaging theory. Plain `requests` with no impersonation ruled out `curl_cffi` entirely. Raw `curl` against an unrelated domain ruled out the target site. Each step removed one whole category of suspects instead of guessing at the next fix, and the actual root cause, when it finally showed up, wasn't in any of the code we'd been staring at.


baa180f9f2841f2725dcc099270d855fd31fabe10b91b90a1deb3bcfc6f49f60

How we debugged a yt-dlp 410 Gone error that wasn't staleness: curl_cffi TLS impersonation, a strict version-pinning gotcha, a --force-reinstall dependency mishap in a shared Python environment, and a proxy gateway failure that mimicked a library bug.

A yt-dlp download kept failing with 'HTTP Error 410: Gone' even on the latest version. The real cause was three layered problems stacked on top of each other: missing TLS impersonation, a narrow curl_cffi version-compatibility window, and an unrelated proxy gateway fault that looked exactly like a library bug.

Debugging a Fake 410: yt-dlp, curl_cffi, and a Proxy That Lied


# Building Blocks of Algorithmic Trading: Collecting End-of-Day Data

{/* # Algorithmic Trading Series #1: Collecting End-of-Day (EOD) Data — The Foundation of a Trading System */}

Algorithmic trading begins not with signals or models, but with **data**: specifically, clean, consistent, and reliable **End-of-Day (EOD)** market data. Whether you're building a trend-following system or a multi-factor screener, your models are only as good as the data they run on. In this first article of our algorithmic trading series, we explore the essential sources of EOD data and demonstrate how to store and access it efficiently using **Redis**, a high-performance in-memory database.

## Why EOD Data?

EOD data, which includes open, high, low, close prices and volume (OHLCV) at the end of each trading day, is the **bedrock** for many algorithmic trading strategies, especially those that don’t require high-frequency execution. It's cleaner, more accessible, and less volatile than intraday data, making it ideal for:
- Backtesting strategies
- Screening assets
- Generating entry/exit signals
- Risk and portfolio management

## Where Can You Get EOD Data?

Here are several common and reliable ways to collect EOD data:

### 1. **Professional Market Data Providers**
If data cost is not an issue, consider subscribing to a professional provider:
- [**Quandl (now part of Nasdaq Data Link)**](https://data.nasdaq.com/publishers/QDL) – Offers premium and free datasets.
- [**EOD Historical Data**](https://eodhd.com/) – Affordable, easy-to-integrate REST API with global coverage.
- [**Tiingo**](https://www.tiingo.com/) – Clean, high-quality financial data with generous free API tiers.

These providers often include dividend and split adjustments, which are essential for accurate backtesting.

### 2. **Investment-Oriented Websites**
Web scraping from investment portals is possible, but not always recommended for production systems:
- **Investing.com** – Offers EOD data for equities, indices, forex, and commodities. 
- **Yahoo Finance** – Widely used; its unofficial API can be accessed using libraries like `yfinance`.
- [**Market Index**](https://www.marketindex.com.au/) – Australian oriented investment website with free ASX EOD data provided.

When using Investing.com, you might come across the Python library `investpy`. While it offers a convenient interface, it hasn’t worked reliably in my experience:

```shell
Error retrieving data for TSLA: ERR#0015: error 403, try again later.
```

Luckly "investing.com" still provides a function on their website to export the historial data for many instruments. So the worst case scenario is to download the data manualy from it. 

### 3. **Broker APIs**
Many brokers provide EOD data via REST or WebSocket APIs:
- **Interactive Brokers (IBKR)** – Offers EOD historical data via their TWS API.
- **Alpaca** – Provides free stock EOD data through their REST API.
- **IG**, **TD Ameritrade**, **OANDA**, **Polygon.io**, and others also offer options.

Always check for:
- Historical limits
- Frequency caps
- Adjustments for splits/dividends

The problem with broker APIs is that each broker has their own unique interface and data structure. Therefore, you may need to write custom code to handle the data retrieval and storage. 

### 4. **Trading Platforms**
Platforms like **MetaTrader 5 (MT5)** offer built-in historical data:
- MT5 can export EOD data directly from chart history.
- Supported by most brokers
- You can use Python with MetaTrader5 API (via `MetaTrader5` Python package) to automate downloads.

```python
from MetaTrader5 import *
initialize()
rates = copy_rates_range("EURUSD", TIMEFRAME_D1, start, end)
```
Using MetadataTrader5 is a great way to get EOD data, but be aware of the limitations:
- **On available to Windows users**: The python library for the MetaTrader5 platform is only available on Windows, so you need to run your code on a Windows machine or use a virtual machine.
- **Limited to Forex and CFDs**: Not all assets are available.
- **Data Quality**: Ensure the data is clean and adjusted for splits/dividends.

Personally, I prefer using the `MetaTrader5` API for retrieving EOD trading data. While its coverage of tradable assets is somewhat limited, it offers more than enough instruments to get started. Unlike some broker-specific APIs, MetaTrader 5 is widely adopted by traders, supported by many brokers, and backed by a large community.

A Python script will be provided to help you download EOD data directly from the MetaTrader5 platform. Setting up a demo account is also straightforward, making it easy to begin testing right away.


## Storing EOD Data with Redis

Once you've collected your data, you need fast, structured access for backtesting and model development. **Redis**, an in-memory key-value store, is an excellent choice for its speed and simplicity.

### Why Redis?
- Ultra-fast read/write
- Easy to structure time-series data
- Built-in expiration and memory management
- Great for small-to-medium-sized datasets or caching layer

### Redis Data Schema 

For EOD data, we can use a key, subkey, value structure:
- **Key**: `[PREFIX]:symbol` (e.g., `nasdaq:AAPL`)
- **Subkey**: `date` (e.g., `20250510`)
- **Value**: JSON string of OHLCV data
- **Example**: `nasdaq:AAPL:20250510` -> `{"open": 180.12, "high": 182.55, "low": 179.70, "close": 181.30, "volume": 25300000}`

Since instruments traded on different markets or exchanges can share the same symbol, it's important to define a **unique key** for each asset. To achieve this, we introduce a **prefix** for each symbol, separated by a colon.

The format is:  
`<EXCHANGE_PREFIX>:<SYMBOL>`

Examples of common prefixes:
- `nasdaq:AAPL`: Apple Inc. on the NASDAQ
- `nyse:IBM`: IBM on the New York Stock Exchange
- `asx:BHP`: BHP Group on the Australian Securities Exchange
- `crypto:BTCUSD`: Bitcoin as a cryptocurrency
- `forex:EURUSD`: Euro to US Dollar exchange rate
- `metals:XAUUSD`: Gold to US Dollar exchange rate
- `energies:XBRUSD`: Brent Crude Oil to US Dollar exchange rate
- `indices:SPX500`: S&P 500 Index

You can define the prefix based on the exchange or asset class you're working with. This convention ensures consistency and avoids symbol collisions across your dataset.


{/* You can use symbol-date as a Redis key and store OHLCV as a JSON string: */}

{/* ```python
import redis
import json

r = redis.Redis(host='localhost', port=6379, db=0)

symbol = "AAPL"
date = "2025-05-10"
key = f"{symbol}:{date}"

data = {
    "open": 180.12,
    "high": 182.55,
    "low": 179.70,
    "close": 181.30,
    "volume": 25300000
}

r.set(key, json.dumps(data))
```

To retrieve:

```python
raw = r.get("AAPL:2025-05-10")
record = json.loads(raw)
print(record["close"]) */}

### Run Redis Locally
To run Redis locally, you can use Docker Compose. Here’s a simple command to get started:

If you are on Linux, you can use the following command to run Redis locally:
```
cd redis
docker-compose up -d
```
If you are on Windows, you can contact me if you need help to run Redis locally.

## From MetaTrader to Redis: Storing EOD Data Efficiently

### Downloading EOD Data from MetaTrader5
Now it is time to get our hands dirty and start collecting EOD data. I hope you are comfortable with command line (`Command Prompt`) and Python. 

As discussed, we prefer to use the MetaTrader5 API to download EOD data. As Python is the only language supported by MetaTrader5, we have to use the Python with `MetaTrader5` Python package to download EOD data and store it in Redis.

I have made a simple script to download EOD data from MetaTrader5 and save them into files.

You need to have the both Python and MetaTrader5 platform installed and running on your machine. You can download the MetaTrader5 platform from the official website.

Then, download the code [repository](https://github.com/tyolab/tyosis-import) from GitHub:
```
git clone https://github.com/tyolab/tyosis-import.git
cd tyosis-import
pip install -r requirements.txt
```

To run the data download script, ensure you have MetaTrader 5 installed and running, and that you are logged into your trading account. Then for example, to download EOD data for XAUUSD (Gold) from 2025-01-01 to today:
```
python utils\mt5.py XAUUSD 2025
```

The content of downloaded looks like this:
```csv
symbol,date,open,high,low,close,volume
XAUUSD,02/01/2025,2624.78,2660.34,2621.7,2657.66,198834
XAUUSD,03/01/2025,2657.49,2665.26,2636.63,2639.46,169658
XAUUSD,06/01/2025,2641.42,2649.44,2614.37,2635.92,229184
XAUUSD,07/01/2025,2636.07,2664.16,2632.88,2648.79,205976
XAUUSD,08/01/2025,2648.42,2669.91,2645.29,2661.36,194517
XAUUSD,09/01/2025,2661.5,2678.11,2655.45,2669.91,151068
XAUUSD,10/01/2025,2670.79,2697.79,2663.96,2688.75,244597
XAUUSD,13/01/2025,2689.1,2693.39,2656.55,2662.87,289738
XAUUSD,14/01/2025,2663.47,2678.0,2659.89,2676.74,189931
XAUUSD,15/01/2025,2678.15,2697.32,2669.31,2696.63,176113
XAUUSD,16/01/2025,2694.86,2724.67,2689.89,2713.7,194921
XAUUSD,17/01/2025,2713.75,2717.32,2699.23,2702.2,172786
XAUUSD,20/01/2025,2702.94,2713.68,2689.31,2707.06,169458
XAUUSD,21/01/2025,2706.95,2745.93,2702.88,2744.69,224338
```

A detailed explanation of how to use the script is provided below:
```
python utils\mt5.py <symbol> <year> 
```

Replace `<symbol>` with the trading symbol you want to download data for (e.g., EURUSD) and `<year>` with the year you are interested in (e.g., 2023). The script will then connect to your MetaTrader 5 platform, download the end-of-day historical data for the specified symbol and year, and save it to a text file named YYYY.txt within a directory structure based on the market and symbol (e.g., `data/Forex/EURUSD/2023.txt`). If the file already exists, the script will append new data to it, starting from the day after the last recorded date.

If you want to download data for the last 20 years of Gold (XAUUSD), you can run the following command:
```
for /L %i in (2005,1,2025) do python utils\mt5.py XAUUSD %i
```

Now we have the EOD data stored in text files. The next step is to load this data into Redis.

### Loading EOD Data into Redis

To load the EOD data into Redis, we can use the `utils/import.py` script provided in the repository. This script will read the text files and store the data in Redis using the key structure we defined earlier.

For example,

```
python utils/import.py --key-prefix "metals:" /data/Metals/XAUUSD/2025.txt 
```

If you would like to import the last 20 years of Gold (XAUUSD) data, you can run the following command:
```
for /L %i in (2005,1,2025) do python utils/import.py --key-prefix "metals:" /data/Metals/XAUUSD/%i.txt 
```

### Checking the Data in Redis
```shell
redis-cli
>hgetall metals:XAUUSD
  1) "20250307"
  2) "{\"O\": 2911.85, \"H\": 2930.43, \"L\": 2896.95, \"C\": 2910.82, \"V\": 164598}"
  3) "20250127"
  4) "{\"O\": 2768.22, \"H\": 2772.58, \"L\": 2730.49, \"C\": 2740.11, \"V\": 264527}"
  5) "20250315"
  6) "{\"O\": 2983.94, \"H\": 2988.8, \"L\": 2981.73, \"C\": 2986.03, \"V\": 2279}"
  7) "20250211"
  8) "{\"O\": 2908.07, \"H\": 2942.65, \"L\": 2881.57, \"C\": 2897.44, \"V\": 252658}"
  9) "20250206"
 10) "{\"O\": 2867.64, \"H\": 2873.29, \"L\": 2834.22, \"C\": 2855.5, \"V\": 198609}"
 11) "20250116"
 12) "{\"O\": 2694.86, \"H\": 2724.67, \"L\": 2689.89, \"C\": 2713.7, \"V\": 194921}"
 13) "20250321"
 14) "{\"O\": 3044.99, \"H\": 3047.41, \"L\": 2999.37, \"C\": 3023.0, \"V\": 119857}"
```

## Best Practices

- **Normalize your data**: Adjust for splits/dividends if applicable.
- **Consistency**: Store all assets using the same structure.
- **Backup**: Regularly export Redis snapshots (RDB) or enable AOF for persistence.
- **Metadata**: Store additional info like timezone (if you need data with smaller timeframe like h4, h1, or m15 etc), data source, and update timestamp.

## Final Thoughts

Before building signals, portfolios, or trading bots, **build your data foundation**. Collecting clean EOD data and storing it efficiently gives you the confidence that your backtests are meaningful and your signals trustworthy.

Now that you've gathered EOD data for your chosen instruments, you have a solid foundation for backtesting your trading strategies. You can start exploring different indicators, rules, and risk management techniques to see how they would have performed historically. Remember to thoroughly evaluate your backtests and consider factors like transaction costs and slippage to get a realistic assessment of your strategy's potential.

{/* In the next part of this series, we’ll explore how to **backtest a simple moving average crossover strategy** using the data we've just stored in Redis. This will involve retrieving the data, calculating moving averages, and simulating trades based on crossover signals.

Stay tuned for a step-by-step guide on implementing and evaluating this strategy! */}

 {/* In the next part of this series, we’ll explore how to **backtest a simple moving average crossover strategy** using the data we've just stored. */}


<ContactFooter />

ce4db250086366b9be1bf79b6e065520de7e584d7395c860ef50a389215463c8

In this article, we explore the essential sources of EOD data and demonstrate how to store and access it efficiently using Redis, a high-performance in-memory database.

Building Blocks of Algorithmic Trading: Collecting End-of-Day Data


## Introduction
MetaTrader doesn't have strong support for algorithmic trading if you want to develop a trading bot that doesn't heavily rely on this particular trading platform. Typically, people doing automated trading on MetaTrader platforms have to code it completely in MQL, and a terminal must be run actively in the foreground.

To reduce the dependency solely on the MQL programming language or to increase the flexibility of a trading system (bot) that can be coded in a different language or run on different operating systems using any preferred technical analysis libraries, some MetaTrader libraries using a message queue (MQ) service, such as ZeroMQ, have been developed. In this way, the part of the code that is coupled with MetaTrader platforms acts only as a relay (proxy agent).

Some people want to completely overcome the limitations of what MetaTrader can provide, so they hack the underlying communication protocol between the MetaTrader terminal and the server and develop libraries in a programming language of their choice, such as C# or C++. With such MetaTrader libraries, an active terminal running in the foreground is no longer necessary, and the trading bot can run headless on Windows, MacOS, or Linux. However, there is a catch with this method: since such a hack is not officially supported by the trading platform developer, it may need constant updates due to requirements for latest terminal version or may simply stop working due to a change in the underlying protocol.

In this article, we will try to develop a trading gateway (or proxy agent) that is intended to reduce the dependency on MetaTrader platform, but also maintain the stability and reliability of such a program in the long term. When the gateway is running or live trading is in action, we don't want the unpredicted interrupt the whole workflow: our trading system (bot) could stop receving market data updates all of a sudden, or it could not place order or receive order status information anymore without an update to the trading program.

## The Design of the Trading System (Bot)

In order to achieve the following goals:
- You do not have to use MQL and do extra compilations, and needs as little operations as possible on MetaTrader, other than leaving it running in the foreground
- The gateway should be stable and reliable. The gateway should continue working even with that the updates of MetaTrader is required constantly.

Fortunately, MetaQuote, the developer of MetaTrader, has provided an official Python package for integration since MetaTrader 5. With this package, you can create a trading bot entirely coded in Python, or you can build a trading gateway that acts as a bridge between the trading platform and an existing trading system (bot) developed in any language (e.g., NodeJS, C++, C#, Java). In the latter case, you can leverage the ecosystem of each language and enjoy the freedom from limitations in both programming language and operating system.

In this article we will focus on developing such a trading gateway using the MetaTrader 5 Python package for the itegration. To ensure seamless communication within the trading system, a complete setup should consist of the following main components:
- a message queue server for messaging and message relaying (price, orders, etc.) 
- trading gateway(s) for getting price updates and managing orders
- indicator updater after receiving new price data  
- a trading decision maker (strategy) for generating buy or sell signals

A gateway is a small but essential part of the entire traiding system. It obtains the latest price updates and manages the placement of new orders or the closure of existing orders when buy or sell signals are triggered.

The following figure illustrates the workflow of a complete trading system:

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/mt5-python-gateway/Trading System Flowchart.png)

## The Implementation of Core Funtinalities of the MetaTrader Gateway

In this section we will discuss the core functionalities of a trading gateway. We will expect at least the following for the MetaTrader Gateway:
- pull the price updates
- submit an order when a signal is triggered
- close the order when it is time to 

A complete implementation of a proper gateway could be complicated, more functionalities might be also needed, for examples, sush as acquisition of account's balance, current P&L or historial data. All depend on what is required for the trading system. Being able to get the account balance could be benificial but is not an absolute requirement.

The actual implementation of this gateway could be straight-forward, as we can just call the coresponding functions provided by the API. However, how to use those functions to achieve the goals of auto trading may be not as that simple as you may think. The following sections will discuss the each aspect of the functionalities in details.

### Initialzation of MetaTrader Terminal
As discussed previously, the MetaTrader python library requires MetaTrader terminal running in the foreground, so it can communicate with the terminal in order to retrieve the price updates and place orders.

There are two parts in this process:
- start the terminal if it has not been running already
- loging into your account

#### Start The terminal
```python
    path = "C:\\Program Files\\MetaTrader 5\\terminal64.exe"
    timeout = 1000000
    portable = False 
    if not mt5.initialize(path=path, portable=portable, timeout=timeout):
        print("initialization failed, error code =", mt5.last_error())
        quit()
```
To start a terminal, the library needs to know where the terminal executable (terminal64.exe) is located, and that is the variable "path" for. In some cases the library might have difficulties in building communciation with the terminal, "timeout" is the time limit in millionseconds when it will stop trying.  

Also, different terminal executables provided by the brokers are actually all the same just with different brandings and a different installation path. You can copy and paste the terminal program to a different directory and run it from there. In this case, you can specify the "portable" mode to "True", so the data from the terminal will be installed in that directory.

#### Account Access
Once the terminal is up and running. You will need to authenciate yourself with your broker's MetaTrader trading server.
```python
    account = 888888
    password = "Your Password Here"
    server = "MetaQuotes-Demo"
    timeout = 60000
    authorized = mt5.login(account, password=password, server=server, timeout=timeout)
    if authorized:
        print("Login successful")
    else
        print("Failed to connect to the trade account with error code =", mt5.last_error())
```

### Getting Price Updates
One major limitation of this package is that you cannot get the price updates through "pumping", which is a machanism that you can get notification whenever new data arrives. In order to get the latest price data, you have to make query to the terminal to see if there is any new data. A chart window for the symbol you are watching should be opened as well, as this active window would make sure the terminal to receive the price updates from the server. 

So we can set a timer, or create a infinit loop inside a child thread and then wake up after being asleep for a certain amount of time. In the example below, we will use a timer to get latest bitcoid price:
```python
last_updated = None
timezone = pytz.timezone("Etc/UTC")
def check_updates():
    # print ("checking for updates") 
    # for different brokers, different tickers for BITCOIN could be used
    # such as 'BITCOIN' or 'BTCUSD'
    time_from = None
    time_to = datetime.now(timezone)
    if last_updated is None:
        time_from = datetime.now(timezone)
        # this is the first time we get the price we will try get all ticks in the last two mintues
        time_from = time_from  - timedelta(minutes=2)
    else:
        time_from = last_updated
        time_from = time_from + timedelta(seconds=1)

    ticks = mt5.copy_ticks_range('BTCUSD', time_from, time_to, mt5.COPY_TICKS_ALL)
    if ticks is None or len(ticks) == 0:
        # print("No ticks received from ", utc_from, "to", server_now, "for ", symbol)
        return
    
    # print("Ticks received:", len(ticks))
    time = None
    for tick in ticks:
        # print the time of the tick
        # named time, bid, ask, last and flags 
        # convert ecpoch to datetime
        time = datetime.fromtimestamp(tick[0])
        # conver time to native with timezone
        time = time.astimezone(timezone)

        # quote = quote()
        # quote.bid = tick[1]
        # quote.ask = tick[2]
        # quote.time = time
        # quote.last = tick[3]
        # quote.volume = tick[4]
        #
        # DO WHATEVER YOU WANT WITH THE DATA
        # For example we can send the quote data to the trading system
        # where indicators kept updated and strategies look for signals 
        # to send quotes to the trading system you can use a message queue system
        # such as ZeroMQ, RabbitMQ, Kafka, etc.
        # or you can use a simple REST API to send the data to the trading system
        # or you can my implementation of a message queue server ('TYO-MQ')
    last_updated = time
    set_timer()

def set_timer():
    # Will the bitcoin price again in 5 seconds
    # If you want to do high-frequency trading obviously this is not good for you
    threading.Timer(5.0, check_updates).start()  

# Don't forget to start the timer
set_timer()
```

### Placing a New order
Once we get a signal from the strategy we can place an order.

```python
# ORDER_TYPE_BUY
# Market Buy order
# ORDER_TYPE_SELL
# Market Sell order
# ORDER_TYPE_BUY_LIMIT
# Buy Limit pending order
# ORDER_TYPE_SELL_LIMIT
# Sell Limit pending order
# ORDER_TYPE_BUY_STOP
# Buy Stop pending order
# ORDER_TYPE_SELL_STOP
# Sell Stop pending order
# ORDER_TYPE_BUY_STOP_LIMIT
# Upon reaching the order price, a pending Buy Limit order is placed at the StopLimit price
# ORDER_TYPE_SELL_STOP_LIMIT
# Upon reaching the order price, a pending Sell Limit order is placed at the StopLimit price
# ORDER_TYPE_CLOSE_BY
# 
# depends on what the signal indicates for buying or sell
# the order type can be one of the above
# if we want buy from the market we can use: mt5.ORDER_TYPE_BUY
# if we want to sell to the market we can use: mt5.ORDER_TYPE_SELL
# if we want to place a buy limit order we can use: mt5.ORDER_TYPE_BUY_LIMIT

def place_order(symbol, market, type, price, volume, stop_loss, take_profit, comment, stop_price, expire_time):
    # Order to close a position by an opposite one
    trade_request = {
        "action": mt5.TRADE_ACTION_DEAL,
        "symbol": symbol,
        "volume": float(volum),
        "type": type,
        "price": price,
        "sl": float(stop_loss),
        "tp": float(take_profit),
        "comment": comment, # a comment for the order, max 31 characters
        "type_time": mt5.ORDER_TIME_GTC,
        "type_filling": mt5.ORDER_FILLING_IOC, # mt5.ORDER_FILLING_RETURN
        "deviation": 20,
        "magic": 0,
        # "expiration": datetime.now() + timedelta(minutes=5), # 5 minutes from now
    }
    trade_result = mt5.order_send(trade_request)
    # you will get a None if the trade_request is not constructed properly
    # for example, the volume is not a float
    # or the price is not a float
    # or the stop_loss is not a float
    # or the take_profit is not a float
    # or the stop_price is not a float
    # or the expire_time is not a datetime object
    # or the comment is not a string or longer than 31 characters
    if trade_result.retcode != mt5.TRADE_RETCODE_DONE:
        print("Error placing order: ", symbol, ", type: ", type, ", volume: ", volume, ", price: ", price, ", stop_loss: ", stop_loss, ", take_profit: ", take_profit, ", comment: ", comment, ", stop_price: ", stop_price, ", expire_time: ", expire_time)
        return None

    print("Order placed: ", trade_result.order, ", for ", symbol, ", type: ", type, ", volume: ", volume, ", price: ", price, ", stop_loss: ", stop_loss, ", take_profit: ", take_profit, ", comment: ", comment, ", stop_price: ", stop_price, ", expire_time: ", expire_time)

    return trade_result.order
```

If the order is successfully placed, the "place_order" function will return a ticket for the trade.

### Closing an order
Once we have an order in place, we may expect two outcomes:
- The signal is valid, and the price moves in the same direction as the signal indicates. In this case, we could exit: 1) if it reaches the price target; 2) when the trend reverses.

- The signal is not valid, and the price moves in the opposite direction than we expect. In this case, we should exit if the price goes beyond the stop-loss.

The stop-loss can be pre-calculated by the trading strategy or a pre-set threshold if the loss reach x percentage of the total value.

To close an existing order, what we have do is just to create a reverse order and include the ticket number as well. So if the order is a long position, we will create a sell order.

```python
def reverse_order_type(type):
    # if the order is a buy order, we will create a sell order
    # adjust the function based on your need, you may want to create a limit order instead of a market order
    if type == mt5.ORDER_TYPE_BUY:
        return mt5.ORDER_TYPE_SELL
    else
        return mt5.ORDER_TYPE_BUY

def close_order(symbol, type, price, volume, ticket, comment):
    trade_request = {
        "action": mt5.TRADE_ACTION_DEAL,
        "symbol": symbol,
        "volume": float(volume),
        "type": reverse_order_type(type),
        "type_time": mt5.ORDER_TIME_GTC,
        "type_filling": mt5.ORDER_FILLING_IOC, # mt5.ORDER_FILLING_RETURN,
        "deviation": 20,
        "magic": 0,
        "position": int(ticket),
    }
    trade_result = mt5.order_send(trade_request)

    # if the trade_request is not constructed properly, the trade_result will be None
    # for example, the ticket is not valid (e.g. it is a string not an integer)
    # and vloume is not a float
    if trade_result is None or trade_result.retcode != mt5.TRADE_RETCODE_DONE:
        print ("Failed to close order: ", ticket)
        return False
    return True
```

Please note there are at least three types of filling for opening or closing order: 1) 'ORDER_FILLING_IOC' means that after whatever maximum availabe volume get filled in the market, the remaining volume will be cancelled; 2) 'ORDER_FILLING_FOK' means an order has to be executed as a whole with the specified volume, so it is basically "deal or no deal"; 3) 'ORDER_FILLING_RETURN' can do other types can't do, for example, if the order is partially executed and the rest of the order won't be cancelled. However, not all brokers support this type of filling. It is understandable that the broker may not need the 'ORDER_FILLING_RETURN' filling type because for forex, metals, and other instruments, the volume is usually not a problem and for the share CFD products, basically it is you against the broker so as long as within the maximum volume you can trade, it can be garanteed that your order will be filled when the order price matches. 


### Conclusion

In this article, we have discussed the core functionalities of a trading gateway using the MetaTrader's python library. The core idea is that it is intended to reduce the dependency on MetaTrader platform, but also maintain the stability and reliability of such a program (the gateway) in the long term. We have discussed the initialzation of the MetaTrader terminal, getting price updates, placing a new order, and closing an existing order. After the implementation of these core functionalities, the gateway can be used as a bridge between the MetaTrader 5 trading platform and an existing trading system (bot) developed in any language (e.g., NodeJS, C++, C#, Java) once a messaging mechanism is in place.

### Further Reading

We will continue this topic in part II. In next part we will present a complete implementation of such a gateway for trading bitcoin with a messaging server for the communciation between the trading system and the gateway. Setting up a development environment for futher implementation of building a complete trading system will be also discussed.

## Contact

<ContactFooter />

{/* For further information, I can be reach via:

- X (formally twitter): [Eric Tang](https://x.com/_e_tang)
- email: [eric@tyo.com.au](mailto:eric@tyo.com.au) */}

3c973ca0827cf0b78701e312d02e2d77dec4b900451c18fda0fe7311de026ffc

In this article, we will try to develope a trading gateway (or proxy agent) that is intended to reduce the dependency on MetaTrader platform, but also maintain the stability and reliability of such a program in the long term. When the gateway is running or live trading is in action, we don't want the unpredicted interrupt the whole workflow: our trading system (bot) could stop receving quote updates all of a sudden, or it could not place order or receive order status information anymore without an update to the trading program.

A tutorial about implementing a trading gateway with MetaTrader 5 Python library. 

Tag: Python

Tag: Python

Measuring Order Book Imbalance in Code

Debugging a Fake 410: yt-dlp, curl_cffi, and a Proxy That Lied

Building Blocks of Algorithmic Trading: Collecting End-of-Day Data

Building a Trading Gateway (Bot) with MetaTrader 5 Python Library - Part 1