Complete Project Guide

1. What the Bot Does

Think of it like a weather forecaster for currency prices. It looks at recent data (RSI, moving averages, volatility) and predicts: Will the price go up or down in the next X candles? If it's confident enough (prob_up ≥ 60% for BUY, ≤ 40% for SELL), it tells MT5 to execute. The goal: make more winning trades than losing ones.

2. Quick Start

Prerequisites: Python 3.10+, pip install -r requirements.txt

1. Set TWELVE_DATA_API_KEY in Admin → App Settings (for market data)
2. Set TWELVE_DATA_SYMBOLS (e.g. EUR/USD,GBP/USD,USD/JPY)
3. python manage.py runserver — scheduler starts automatically
4. Use Dashboard → Actions → Run All (Manual) for Backfill → Aggregate → Train → Backtest, or run:
   • python manage.py backfill_historical_data --outputsize 500
   • Aggregate (Dashboard button or wait for hourly job)
   • python manage.py train_ml_model --target momentum
   • python manage.py backtest_ml --bars 500 --stress

3. Dashboard & URLs

Page	URL
Dashboard	/dashboard/
Actions	/dashboard/actions/
Trades	/dashboard/trades/
Signals	/dashboard/signals/
Model & Learning	/dashboard/model/
Guide (this page)	/dashboard/guide/
Admin	/admin/

4. Dashboard Actions (Manual & Auto)

Manual — choose target yourself. Auto — try all targets, pick best by Sharpe.

Action	In Plain Language
Backfill	Download past price data from Twelve Data so the bot has something to learn from. Need at least 90 bars per symbol.
Aggregate	Resample 1min bars into 1H bars (used for training).
Train	Teach the bot using that data. Uses target from dropdown (momentum, mean_reversion, triple_barrier).
Auto-tune	Try momentum, triple_barrier, mean_reversion; train and backtest each; pick best by Sharpe. Updates ML_TARGET_TYPE.
Run Backtest	Simulate trades on past data (Sharpe, drawdown, Litmus).
Stress test	Same backtest with 2× costs — does the edge survive?
Diagnose	Check model & risk status.
Run All (Manual)	Backfill → Aggregate → Train (your target) → Backtest
Run All (Auto)	Backfill → Aggregate → Auto-tune → Backtest
Target: momentum	Bot learns "price moved significantly up or down" — bigger moves, not tiny wiggles.
Target: triple_barrier	Bot learns "price hit +X or -Y ATR before the other" — event-based.
Target: mean_reversion	Bot learns "price reverted toward SMA" — ranging markets.
Walk-forward	Split data 60/20/20 to check if the bot works on unseen data.
Disabled regimes	Don't trade when market is in certain states (e.g. ranging_mid_vol, trending_low_vol).

Insufficient data (need 90+ bars per symbol):

• 1. Set TWELVE_DATA_API_KEY and TWELVE_DATA_SYMBOLS in Admin → App Settings (see below)
• 2. Run Backfill (e.g. 500 bars) — fetches 1min data from Twelve Data
• 3. Run Aggregate — resamples 1min → 1H for training
• 4. Then Run All (Manual) or Run All (Auto). Active symbols must match TWELVE_DATA_SYMBOLS or they'll have 0 bars.

5. MT5 EA Setup

1. Allow WebRequest: MT5 → Tools → Options → Expert Advisors → Allow WebRequest for http://127.0.0.1 or http://127.0.0.1:8000
2. Compile EA: MetaEditor → open mt5_ea/SmartTradingBot_ML.mq5 → F7 (0 errors)
3. Attach: Drag EA onto chart (e.g. EURUSD H1). Set InpBaseURL = http://127.0.0.1:8000, InpTradeAllowed = true
4. Algo Trading: Toolbar green, smiley face on chart. Close Strategy Tester if running.

EA reports trades: When EA opens/closes, it POSTs to /api/bot/trade/report/. Dashboard Trades and P&L update from this. Learning-from-trades requires EA to report on close (exit_price, pnl).

6. Metrics & Litmus Test

Metric	Plain Language
Sharpe	Risk-adjusted return. > 1.2 = strong. Negative = losing.
Brier score	How reliable probabilities are. < 0.25 = better than random.
Monotonic	Does higher confidence mean better results? "No" = misleading.
Bootstrap CI	Does edge hold when we shuffle trades? 0 inside = shaky.
p-value	Is profit/loss real or luck? < 0.05 = statistically meaningful.

Litmus Test (before real money): Sharpe > 1.2 (after --stress), max drawdown < 150 pips, monotonic yes, Brier < 0.25, bootstrap CI excludes 0.

7. Weekly Automation

Job	Schedule	What it does
periodic_retrain	Sunday 4:00	Retrain on latest data + closed trades. With LightGBM + INCREMENTAL_TRAINING=true: adds trees on top of previous model (does not start from scratch).
weekly_ml_pipeline	Sunday 5:00	Backfill → Aggregate → Train (or Auto-tune if enabled) → Backtest
auto_train_if_ready	Every 30 min	First-time train when 90+ bars per symbol, no model yet
fetch	Every 6 min	Fetch 1min bars from Twelve Data
aggregate	Hourly (minute 0)	1min → 1H
purge	Daily 00:00	Delete old market data (DATA_RETENTION_DAYS)

Incremental training (LightGBM): When ML_ENGINE=lightgbm and INCREMENTAL_TRAINING=true, each retrain adds ~30 trees on top of the existing model instead of retraining from scratch. RF always retrains from scratch.

AUTO_TUNE_TARGETS=true: Weekly pipeline tries momentum, triple_barrier, mean_reversion; picks best by Sharpe. No manual target choice.

8. Config Reference (Admin → App Settings)

Required for data (Backfill / first run): Set TWELVE_DATA_API_KEY (your Twelve Data API key) and TWELVE_DATA_SYMBOLS (e.g. EUR/USD,GBP/USD,USD/JPY). Without these, Backfill fails and Auto-tune reports "0 bars".

Key	Default	Description
TWELVE_DATA_API_KEY	—	Required for backfill/fetch
TWELVE_DATA_SYMBOLS	EUR/USD,GBP/USD,USD/JPY	Symbols to fetch
ML_TARGET_TYPE	momentum	direction | momentum | mean_reversion | triple_barrier
ML_ATR_THRESHOLD	0.5	ATR multiplier for momentum target
ML_TRIPLE_BARRIER_UPPER_ATR	1.0	Upper ATR for triple_barrier target
ML_TRIPLE_BARRIER_LOWER_ATR	1.0	Lower ATR for triple_barrier target
ML_FORWARD_CANDLES	10	Horizon for target (candles)
ML_ENGINE	rf	rf | lightgbm
ML_CONFIDENCE_THRESHOLD	0.6	Min prob for BUY (≥0.6) or SELL (≤0.4)
INCREMENTAL_TRAINING	true	Each retrain adds trees on top of previous (LightGBM only)
INCREMENTAL_NEW_TREES	30	Trees to add per incremental retrain
RECENCY_WEIGHT_DECAY	0	Weight recent data higher (0=off)
USE_REGIME_MODELS	false	Train separate model per regime
USE_META_MODEL	false	Meta-model gates trades (HOLD when failure prob high)
META_FAILURE_THRESHOLD	0.6	Above this → HOLD
DISABLED_REGIMES	(empty)	Comma list: ranging_mid_vol, trending_low_vol, etc.
AUTO_TUNE_TARGETS	false	Auto-try targets, pick best by Sharpe
BACKTEST_SPREAD_PIPS	1.0	Spread in backtest
BACKTEST_SLIPPAGE_PIPS	0.5	Slippage per side
RISK_PERCENT	0.01	1% risk per trade
MAX_DAILY_LOSS_PERCENT	0.03	Stop trading when daily loss ≥ this
STOP_LOSS_PERCENTAGE	2.0	Stop loss %
TAKE_PROFIT_PERCENTAGE	4.0	Take profit %
STARTUP_CATCHUP_BARS	90	Bars to fetch on startup (≥90 for auto-tune)
DATA_RETENTION_DAYS	30	Days of market data to keep

9. Improving the Model

Target types: momentum (trending), mean_reversion (ranging), triple_barrier (breakout). Use Auto-tune to try all.

ML_ENGINE: rf (RandomForest) or lightgbm (often better calibration). Requires pip install lightgbm.

Incremental (LightGBM only): INCREMENTAL_TRAINING=true (default for LightGBM). Each retrain adds trees on top of the previous model (INCREMENTAL_NEW_TREES=30). Continuous improvement without throwing away learned state. RF always retrains from scratch.

Regime-specific (USE_REGIME_MODELS=true): Separate model per regime (trending_high_vol, ranging_low_vol, etc.).

Meta-model (USE_META_MODEL=true): Predicts when primary model is likely wrong; HOLD when failure prob > META_FAILURE_THRESHOLD.

RECENCY_WEIGHT_DECAY: Weight recent data higher (e.g. 2) for market adaptation.

DISABLED_REGIMES: Skip bad regimes, e.g. ranging_mid_vol,trending_low_vol.

10. Troubleshooting

Problem	Solution
Backfill fails	Set TWELVE_DATA_API_KEY in Admin
Auto-tune: insufficient data	Run Backfill (500+ bars), then Aggregate. Need 90+ bars per symbol.
No model / always HOLD	Run train_ml_model or wait for auto_train_if_ready. Need 90+ bars.
Feature count mismatch	Retrain (train_ml_model)
LightGBM not found	pip install lightgbm
Negative Sharpe / Brier > 0.25	Try different target (triple_barrier), more data, DISABLED_REGIMES
EA not opening trades	WebRequest allowed for http://127.0.0.1:8000; Algo Trading on; Strategy Tester closed; backend running
No requests from EA in backend log	Check WebRequest URL, InpBaseURL, firewall; restart MT5 after URL change

11. Manual Commands

python manage.py backfill_historical_data --outputsize 500
python manage.py train_ml_model --target momentum --walk-forward
python manage.py backtest_ml --symbol "EUR/USD" --bars 500 --stress
python manage.py diagnose_trading

Production (multi-worker): Set RUN_SCHEDULER=false on web workers; run python manage.py run_scheduler separately.

12. Project Structure

smartTradingBot/
├── ml/                    # ML package
│   ├── features.py       # RSI, ATR, SMA, regime, etc.
│   ├── targets.py        # momentum, mean_reversion, triple_barrier
│   ├── trainer.py        # RF, LightGBM, regime-specific, meta-model
│   ├── predictor.py      # BUY/SELL/HOLD
│   └── storage.py        # Model paths
├── marketdata/            # Data, config, facade
│   ├── ml_service.py     # Facade → ml/
│   ├── tasks.py          # Fetch, aggregate, auto_tune, weekly pipeline
│   └── ...
├── models/                # rf_model.joblib, lightgbm_model.joblib
├── mt5_ea/                # SmartTradingBot_ML.mq5
└── dashboard/             # Views, templates

13. Learning from Trades

ModelVersion: Each training run stored (data range, n_samples, metrics). Retrain can build on last state.

SignalSnapshot: When backend sends BUY/SELL, stores features + signal. Links "what model saw" to the trade.

TradeOutcome: EA reports open (entry, mt5_ticket) and close (exit_price, pnl). Closed outcomes used as (features, win/loss) in training. Trades with clear setups (oversold/overbought in reason) get 1.5× weight.

Requirement: EA must POST to /api/bot/trade/report/ on both open and close. Without close report, learning-from-mistakes does not work.

14. Storage & Retention

What	Limit
Market data	Last DATA_RETENTION_DAYS (default 30). Purge daily at 00:00.
Logs	Weekly rotation: files in logs/ older than 14 days removed.
Model	models/rf_model.joblib or lightgbm_model.joblib. RF: overwritten each train. LightGBM+incremental: same file, trees added on top.

15. Will It Auto-Improve Every Sunday?

Short answer: It retrains every Sunday — but that does NOT guarantee profitability.

What happens Sunday 4:00 (periodic_retrain): Loads latest price data + closed trades, then trains. With LightGBM + INCREMENTAL_TRAINING=true, it adds ~30 trees on top of the existing model (does not start from scratch). With RF, it retrains from scratch.

Sunday 5:00 (weekly_ml_pipeline): Backfill → Aggregate → Train (or Auto-tune) → Backtest. Same incremental behavior when LightGBM.

It will: incorporate new data, recent wins/losses (TradeOutcomes), and with incremental — build on previous trees.

It will NOT: automatically become profitable if the edge does not exist. More data or more trees do not turn a loser into a winner.

Bottom line: Sunday jobs keep the model up to date and growing (incremental). Use backtest and Litmus Test to check performance.

Quick checklist: (1) Backfill + Aggregate + Train. (2) Run backtest with --stress. (3) If Litmus mostly FAIL, not ready for real money. (4) EA: WebRequest, Algo Trading, report on open+close.