bnair/CryptoTrading
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add core infrastructure: config, Binance adapter, docs, and auto-setup

Browse Source 
- Add Pydantic settings with trading mode validation (paper/testnet/live)
- Implement Binance USDⓈ-M Futures adapter with hedge mode, isolated margin
- Add type definitions for orders, positions, and market data
- Create documentation (PLAN.md, ARCHITECTURE.md, SECURITY.md)
- Add setup.sh with uv/pip auto-detection for consistent dev environments
- Configure Docker multi-stage build and docker-compose services
- Add pyproject.toml with all dependencies and tool configs
This commit is contained in:
bnair123
2025-12-27 13:28:08 +04:00
parent f1f9888f6b
commit 6f602c0d19
22 changed files with 3396 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

370
docs/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,370 @@
# TradeFinder Architecture
## System Overview
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ TradeFinder │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌───────────┐ │
│ │ Streamlit │ │ Engine │ │ Optimizer │ │ Backtester│ │
│ │ UI │◄───►│ (Core) │◄───►│ (Optuna) │ │ │ │
│ └─────────────┘ └──────┬──────┘ └─────────────┘ └───────────┘ │
│ │ │
│ ┌───────────────────┼───────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Regime │ │ Risk │ │ Order │ │
│ │ Classifier │ │ Engine │ │ Manager │ │
│ └─────────────┘ └─────────────┘ └──────┬──────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Strategies │ │ Allocator │ │ Exchange │ │
│ │ Suite │ │ │ │ Adapter │ │
│ └─────────────┘ └─────────────┘ └──────┬──────┘ │
│ │ │
├──────────────────────────────────────────────────┼──────────────────────────┤
│ Data Layer │ │
│ ┌─────────────┐ ┌─────────────┐ │ │
│ │ DuckDB │ │ Redis │ │ │
│ │ (Analytics) │ │ (State) │ │ │
│ └─────────────┘ └─────────────┘ │ │
└──────────────────────────────────────────────────┼──────────────────────────┘
│
▼
┌───────────────┐
│ Binance │
│ Futures API │
│ (Testnet) │
└───────────────┘
```
---
## Component Details
### 1. Core Engine (`src/tradefinder/core/`)
The main orchestrator that coordinates all components.
```
core/
├── __init__.py
├── config.py # Pydantic settings
├── main.py # Entry point, event loop
├── engine.py # Trading engine orchestrator
├── backtest.py # Backtesting entry point
└── optimize.py # Optimization entry point
```
**Responsibilities**:
- Load configuration from environment
- Initialize all components
- Run main trading loop (signal → risk → order)
- Handle graceful shutdown
### 2. Exchange Adapters (`src/tradefinder/adapters/`)
Abstract interface for exchange connectivity with Binance implementation.
```
adapters/
├── __init__.py
├── base.py # Abstract ExchangeAdapter interface
├── binance_usdm.py # Binance USDⓈ-M Futures implementation
└── types.py # Shared types (Order, Position, etc.)
```
**Key Features**:
- Hedge mode support (dual position side)
- Isolated margin per symbol
- Order types: limit, stop-market, market
- WebSocket streams for real-time data
**Binance USDⓈ-M Specifics**:
```python
# Required API calls on initialization
POST /fapi/v1/positionSide/dual # Enable hedge mode
POST /fapi/v1/marginType # Set ISOLATED margin
POST /fapi/v1/leverage # Set leverage per symbol
```
### 3. Data Layer (`src/tradefinder/data/`)
Market data ingestion, storage, and retrieval.
```
data/
├── __init__.py
├── fetcher.py # REST historical data fetcher
├── streamer.py # WebSocket real-time streams
├── storage.py # DuckDB operations
└── schemas.py # Database schemas
```
**Data Sources**:
| Data Type | Source | Frequency |
|-----------|--------|-----------|
| OHLCV | REST (backfill) + WS (live) | 1m, 5m, 4h |
| Mark Price | WebSocket | Real-time |
| Funding Rate | REST + WS | 8h intervals |
| Order Book | WebSocket (optional) | Real-time |
**Storage Schema** (DuckDB):
```sql
CREATE TABLE candles (
symbol VARCHAR,
timeframe VARCHAR,
timestamp TIMESTAMP,
open DOUBLE,
high DOUBLE,
low DOUBLE,
close DOUBLE,
volume DOUBLE,
PRIMARY KEY (symbol, timeframe, timestamp)
);
CREATE TABLE trades (
id VARCHAR PRIMARY KEY,
symbol VARCHAR,
side VARCHAR,
entry_price DOUBLE,
exit_price DOUBLE,
quantity DOUBLE,
pnl_usdt DOUBLE,
pnl_pct DOUBLE,
strategy VARCHAR,
entry_time TIMESTAMP,
exit_time TIMESTAMP
);
```
### 4. Strategies (`src/tradefinder/strategies/`)
Trading strategy implementations with common interface.
```
strategies/
├── __init__.py
├── base.py # Abstract Strategy interface
├── supertrend.py # Trend-following
├── squeeze.py # Volatility breakout
├── mean_reversion.py # Range-bound
└── signals.py # Signal types
```
**Strategy Interface**:
```python
class Strategy(ABC):
@abstractmethod
def generate_signal(self, data: pd.DataFrame) -> Signal | None:
"""Analyze data and return entry/exit signal."""
pass
@abstractmethod
def get_stop_loss(self, entry_price: float, side: Side) -> float:
"""Calculate stop-loss price for risk sizing."""
pass
@property
@abstractmethod
def parameters(self) -> dict[str, Any]:
"""Current strategy parameters (for UI display)."""
pass
```
**Regime-Strategy Mapping**:
| Regime | Primary Strategy | Secondary |
|--------|------------------|-----------|
| Trending | Supertrend | - |
| Ranging | Mean Reversion | - |
| High Volatility | Squeeze Breakout | - |
| Uncertain | Reduce exposure | - |
### 5. Risk Engine (`src/tradefinder/core/risk.py`)
Position sizing and risk management.
**Position Sizing Formula**:
```
Position Size = (Account Equity × Risk %) / |Entry Price - Stop Loss|
Example:
- Equity: $3000
- Risk: 2% = $60
- Entry: $100,000 (BTC)
- Stop: $98,000
- Stop Distance: $2,000
Position = $60 / $2,000 = 0.03 BTC
Notional = 0.03 × $100,000 = $3,000
```
**Risk Limits**:
| Parameter | Value |
|-----------|-------|
| Risk per trade | 1-3% |
| Max per strategy | 25% equity |
| Max total exposure | 100% equity |
| Max leverage | 2x (initial) |
### 6. Allocator (`src/tradefinder/core/allocator.py`)
Multi-strategy capital allocation.
**Allocation Rules**:
1. Each strategy gets max 25% of equity
2. Hedge mode allows simultaneous long/short
3. Reduce allocation during uncertain regimes
4. Track "portfolio heat" (total risk exposure)
### 7. Order Manager (`src/tradefinder/core/orders.py`)
Order lifecycle management.
**Order Flow**:
```
Signal → Risk Check → Size Calculation → Order Submission
│
┌─────────────────────────┼─────────────────────────┐
│ │ │
▼ ▼ ▼
[PENDING] [FILLED] [REJECTED]
│ │
▼ ▼
[CANCELLED] [STOP PLACED]
│
▼
[STOP TRIGGERED]
```
**Order Types**:
| Purpose | Order Type | Rationale |
|---------|------------|-----------|
| Entry | Limit | Better fills, avoid slippage |
| Stop-Loss | Stop-Market | Guaranteed execution |
| Take-Profit | Limit | Optional, better exit price |
### 8. Regime Classifier (`src/tradefinder/core/regime.py`)
Market regime detection.
**Indicators Used**:
| Indicator | Purpose | Threshold |
|-----------|---------|-----------|
| ADX | Trend strength | >25 = trending |
| ATR% | Volatility | Relative to historical |
| BB Width | Squeeze detection | Narrow = breakout pending |
**Regime States**:
```python
class Regime(Enum):
TRENDING_UP = "trending_up"
TRENDING_DOWN = "trending_down"
RANGING = "ranging"
HIGH_VOLATILITY = "high_volatility"
UNCERTAIN = "uncertain"
```
### 9. UI Layer (`src/tradefinder/ui/`)
Streamlit dashboard for monitoring.
```
ui/
├── __init__.py
├── app.py # Main Streamlit app
├── pages/
│ ├── dashboard.py
│ ├── trades.py
│ ├── strategies.py
│ └── optimizer.py
└── components/
├── charts.py
└── tables.py
```
**Dashboard Sections**:
1. **Overview**: Equity, PnL (USDT/CHF/EUR), positions
2. **Regime**: Current market state, indicator values
3. **Strategies**: Active strategies, parameters, allocation
4. **Trades**: Recent trades, performance metrics
5. **Optimizer**: Last run results, parameter history
---
## Data Flow
### Trading Loop (Every Signal Interval)
```
1. Fetch latest candles (4h)
2. Update regime classification
3. Select active strategy based on regime
4. Generate signal from strategy
5. If signal:
a. Calculate position size (risk engine)
b. Check allocation limits
c. Submit order (limit entry + stop-market)
6. Log state to DuckDB
7. Update Redis with current positions
```
### Order Execution (1m/5m Monitoring)
```
1. Monitor open orders
2. If limit not filled within timeout:
a. Cancel and retry at new price, OR
b. Abandon if price moved too far
3. Track fills and update position
4. Ensure stop-loss is active
```
---
## Deployment Architecture
```yaml
# docker-compose services
services:
engine: # Main trading loop
ui: # Streamlit dashboard
redis: # Real-time state
optimizer: # Weekly Optuna runs
backtester: # On-demand backtesting
```
**Volume Mounts**:
```
/opt/trading/crypto/
├── engine/ # Engine logs, state
├── optimizer/ # Optimization results
├── redis/ # Redis persistence
└── shared/ # DuckDB, shared data
```
---
## Error Handling
| Error Type | Response |
|------------|----------|
| Exchange API error | Retry with backoff, log, alert if persistent |
| Order rejected | Log reason, notify, do not retry blindly |
| WebSocket disconnect | Auto-reconnect with backoff |
| Invalid signal | Log and skip, continue loop |
| Risk limit exceeded | Block order, log warning |
---
## Security Considerations
See [SECURITY.md](./SECURITY.md) for detailed security guidelines.
**Key Points**:
- API keys in environment variables only
- No withdrawal permissions on API keys
- Testnet keys separate from production
- Secrets never logged

195
docs/PLAN.md Normal file
View File

@@ -0,0 +1,195 @@
# TradeFinder Development Plan
## Vision
Automated crypto trading system for BTC/ETH perpetual futures with regime-adaptive strategy selection, risk management, and full transparency via web UI.
---
## Phase 1: Foundation (Current)
**Goal**: Core infrastructure, exchange connectivity, paper trading capability
### Milestone 1.1: Project Setup ✅
- [x] Repository structure
- [x] pyproject.toml with dependencies
- [x] Docker configuration
- [x] Environment template
### Milestone 1.2: Configuration & Core
- [ ] Pydantic settings with validation
- [ ] Structured logging (structlog)
- [ ] Error handling patterns
### Milestone 1.3: Exchange Adapter
- [ ] Abstract exchange interface
- [ ] Binance USDⓈ-M Futures adapter
- [ ] Hedge mode (dual position side)
- [ ] Isolated margin per symbol
- [ ] Leverage configuration
- [ ] Order types: limit, stop-market
- [ ] Testnet connectivity verification
### Milestone 1.4: Data Ingestion
- [ ] REST OHLCV fetcher (historical backfill)
- [ ] WebSocket streams (real-time)
- [ ] Kline/candlestick
- [ ] Mark price
- [ ] Funding rate
- [ ] DuckDB storage schema
- [ ] Data validation & gap detection
---
## Phase 2: Trading Logic
**Goal**: Regime detection, strategies, risk management
### Milestone 2.1: Regime Classifier
- [ ] ADX-based trend detection
- [ ] ATR% volatility measurement
- [ ] Bollinger Band width for squeeze detection
- [ ] Regime states: trending, ranging, high_volatility
### Milestone 2.2: Strategy Suite
- [ ] Base strategy interface
- [ ] Supertrend (trend-following)
- [ ] Squeeze Breakout (volatility expansion)
- [ ] Mean Reversion (range-bound)
- [ ] Grid logic (optional, later)
### Milestone 2.3: Risk Engine
- [ ] Position sizing from stop distance + risk%
- [ ] Per-trade risk limits (1-3%)
- [ ] Strategy allocation caps (25% max)
- [ ] Portfolio heat tracking
### Milestone 2.4: Order Manager
- [ ] Limit order entries with retry logic
- [ ] Stop-market circuit breakers
- [ ] Order state machine
- [ ] Fill tracking & slippage logging
---
## Phase 3: Optimization & Backtesting
**Goal**: Walk-forward optimization, performance measurement
### Milestone 3.1: Backtester
- [ ] Event-driven backtesting engine
- [ ] Realistic fill simulation (slippage, fees)
- [ ] Funding rate impact modeling
- [ ] Multi-symbol support
### Milestone 3.2: Optimizer
- [ ] Optuna integration
- [ ] Walk-forward validation
- [ ] Objective functions: Calmar, Sortino
- [ ] Weekly scheduled optimization runs
### Milestone 3.3: Scorecard
- [ ] Performance metrics calculation
- [ ] Alpha, Beta
- [ ] Sharpe, Sortino, Calmar ratios
- [ ] Max drawdown, CAGR
- [ ] Win rate, profit factor
- [ ] CHF/EUR currency conversion
- [ ] Historical comparison
---
## Phase 4: User Interface
**Goal**: Full transparency via Streamlit dashboard
### Milestone 4.1: Core Dashboard
- [ ] Account overview (equity, PnL)
- [ ] Current positions display
- [ ] Recent trades table
### Milestone 4.2: Regime & Strategy View
- [ ] Real-time regime indicator
- [ ] Active strategy display
- [ ] Parameter visibility
- [ ] Allocation breakdown
### Milestone 4.3: Analytics
- [ ] Performance charts (equity curve)
- [ ] Drawdown visualization
- [ ] Trade distribution analysis
- [ ] Optimizer results viewer
---
## Phase 5: Production Readiness
**Goal**: Reliable deployment, monitoring, alerting
### Milestone 5.1: CI/CD
- [ ] Gitea Actions pipeline
- [ ] Container builds & registry push
- [ ] Automated testing on PR
- [ ] Version tagging
### Milestone 5.2: Monitoring
- [ ] Health check endpoints
- [ ] Error alerting (email/webhook)
- [ ] Performance logging
- [ ] Resource monitoring
### Milestone 5.3: Documentation
- [x] PLAN.md (this file)
- [ ] ARCHITECTURE.md
- [ ] SECURITY.md
- [ ] RUNBOOK.md (operations)
- [ ] API reference
---
## Phase 6: Enhancements (Future)
### Email Reports
- [ ] Daily/weekly PDF reports
- [ ] Embedded charts
- [ ] Performance summary
### Additional Exchanges
- [ ] Bybit USDT perpetuals
- [ ] Alpaca crypto
- [ ] Abstract multi-exchange support
### Advanced Strategies
- [ ] Dynamic leverage based on regime
- [ ] Cross-margin mode (optional)
- [ ] Funding rate arbitrage
---
## Constraints & Decisions
| Decision | Rationale |
|----------|-----------|
| Custom engine (not Freqtrade) | Freqtrade futures requires one-way mode; we need hedge mode |
| Binance USDⓈ-M only (initial) | Best testnet support, most liquidity |
| Hedge mode ON | Long+short simultaneously for strategy isolation |
| Isolated margin | Contain risk per position |
| 4h signal timeframe | Longer swings, less noise, lower fees |
| Limit orders for entry | Better fills, avoids slippage |
| Stop-market for exits | Guaranteed execution on stop-loss |
| DuckDB for analytics | Fast OLAP queries, single file |
| Redis for state | Real-time position/order tracking |
---
## Timeline (Estimated)
| Phase | Duration | Status |
|-------|----------|--------|
| Phase 1: Foundation | 2 weeks | In Progress |
| Phase 2: Trading Logic | 3 weeks | Not Started |
| Phase 3: Optimization | 2 weeks | Not Started |
| Phase 4: UI | 1 week | Not Started |
| Phase 5: Production | 1 week | Not Started |
**Target**: Paper trading operational in ~8 weeks

270
docs/SECURITY.md Normal file
View File

@@ -0,0 +1,270 @@
# TradeFinder Security Guidelines
## API Key Management
### Key Permissions
**Binance Futures API Keys**:
| Permission | Required | Notes |
|------------|----------|-------|
| Enable Futures | Yes | Required for USDⓈ-M trading |
| Enable Reading | Yes | Account info, positions, orders |
| Enable Spot & Margin Trading | No | Not needed for futures |
| Enable Withdrawals | **NEVER** | Do not enable under any circumstances |
| Enable Internal Transfer | No | Not needed |
### Key Separation
Maintain separate API keys for:
1. **Testnet** (paper trading): Lower security concern, can be more permissive
2. **Production** (live trading): Maximum security, IP whitelist required
```bash
# .env structure
BINANCE_TESTNET_API_KEY=xxx # Testnet keys
BINANCE_TESTNET_SECRET=xxx
BINANCE_API_KEY=xxx # Production keys (empty until ready)
BINANCE_SECRET=xxx
```
### IP Whitelisting
For production keys:
1. Enable IP restriction in Binance API settings
2. Whitelist only the server IP running the bot
3. If using Docker, whitelist the host machine's public IP
---
## Secrets Storage
### Environment Variables (Required)
All secrets must be stored in environment variables:
```bash
# Local development
cp .env.example .env
chmod 600 .env # Restrict file permissions
# Docker deployment
docker run --env-file .env tradefinder
```
### What NOT to Do
| Violation | Risk |
|-----------|------|
| Commit `.env` to git | Keys exposed in repo history forever |
| Hardcode keys in source | Keys exposed to anyone with code access |
| Log API keys | Keys visible in log files |
| Share testnet keys | Still bad practice, builds bad habits |
### .gitignore
Ensure these patterns are in `.gitignore`:
```gitignore
.env
.env.*
!.env.example
*.pem
*.key
secrets/
```
---
## Logging Security
### Sensitive Data Handling
```python
# BAD - Never log secrets
logger.info(f"API Key: {api_key}")
# GOOD - Mask sensitive data
logger.info(f"API Key: {api_key[:4]}...{api_key[-4:]}")
# BEST - Don't log at all
logger.info("API credentials loaded successfully")
```
### Structlog Configuration
```python
# Filter sensitive fields from logs
def mask_sensitive(_, __, event_dict):
sensitive_keys = ['api_key', 'secret', 'password', 'token']
for key in sensitive_keys:
if key in event_dict:
event_dict[key] = "***MASKED***"
return event_dict
```
---
## Network Security
### API Endpoints
| Environment | Base URL | Notes |
|-------------|----------|-------|
| Testnet | `https://testnet.binancefuture.com` | Safe for testing |
| Production | `https://fapi.binance.com` | Real money |
### Request Signing
All authenticated requests must be signed:
```python
import hmac
import hashlib
def sign_request(params: dict, secret: str) -> str:
query_string = urlencode(params)
signature = hmac.new(
secret.encode('utf-8'),
query_string.encode('utf-8'),
hashlib.sha256
).hexdigest()
return signature
```
### Rate Limiting
Respect Binance rate limits to avoid IP bans:
| Limit Type | Value |
|------------|-------|
| Request weight | 2400/minute |
| Order rate | 300/minute |
| WebSocket connections | 5/second |
---
## Docker Security
### Container Isolation
```yaml
# docker-compose.yml security settings
services:
engine:
security_opt:
- no-new-privileges:true
read_only: true
tmpfs:
- /tmp
user: "1000:1000" # Non-root user
```
### Secrets in Docker
```yaml
# Use Docker secrets (production)
secrets:
binance_api_key:
file: ./secrets/binance_api_key.txt
binance_secret:
file: ./secrets/binance_secret.txt
services:
engine:
secrets:
- binance_api_key
- binance_secret
```
---
## Operational Security
### Pre-Deployment Checklist
- [ ] API keys have NO withdrawal permission
- [ ] Production keys are IP-whitelisted
- [ ] `.env` is not in git history
- [ ] Logs do not contain secrets
- [ ] Container runs as non-root
- [ ] Redis is not exposed to public network
- [ ] DuckDB file has restricted permissions
### Monitoring for Compromise
Watch for:
- Unexpected positions or orders
- API calls from unknown IPs (check Binance API logs)
- Unusual account balance changes
- Failed authentication attempts
### Incident Response
If keys are compromised:
1. **Immediately** delete API key in Binance console
2. Check for unauthorized trades/withdrawals
3. Generate new API key with fresh permissions
4. Review how compromise occurred
5. Update security practices
---
## Testing Security
### Testnet Isolation
- Always start with testnet credentials
- Verify `TRADING_MODE=testnet` before any trade logic
- Production code should fail-safe if mode is unclear
```python
def validate_mode(config: Config) -> None:
if config.trading_mode == TradingMode.LIVE:
if not config.binance_api_key:
raise ValueError("Production API key required for live trading")
if config.binance_testnet_api_key:
logger.warning("Testnet keys present in live mode - ignoring")
```
### Paper Trading
For in-house simulation (no API needed):
- Use simulated order fills
- No actual API calls
- Safe for strategy development
---
## Dependency Security
### Regular Updates
```bash
# Check for security vulnerabilities
pip audit
# Update dependencies
pip install --upgrade -r requirements.txt
```
### Pinned Versions
Use exact versions in production:
```toml
# pyproject.toml
dependencies = [
"ccxt>=4.2.0,<5.0.0", # Pin major version
"pydantic>=2.5.0,<3.0.0",
]
```
---
## Summary
| Category | Requirement |
|----------|-------------|
| API Keys | No withdrawal permission, IP whitelisted |
| Secrets | Environment variables only, never committed |
| Logging | Mask all sensitive data |
| Network | HTTPS only, signed requests |
| Container | Non-root, read-only where possible |
| Monitoring | Watch for unauthorized activity |