Reinforcement Learning for Dynamic Position Sizing in Volatile Markets

At Venym Labs, we operate on a core quantitative premise: alpha generation is only half the battle; alpha extraction is the other. In liquid, highly volatile markets—particularly digital assets and dispersed macro futures—the difference between a world-class strategy and a blown-out account rarely lies in the entry signal. It lies in position sizing.

Most quantitative teams treat position sizing as an afterthought, relying on static heuristics like the Kelly Criterion or fixed fractional risk. We view this as mathematically bankrupt in non-stationary environments. When realized volatility can swing from 15% to 150% annualized in a matter of hours, a static sizing model is a liability.

To solve this, we have spent the last two years building a proprietary Reinforcement Learning (RL) framework dedicated entirely to dynamic position sizing. This post details our architecture, the mathematical pitfalls of traditional approaches, and the empirical edge we have observed when handing the sizing decision over to a well-conditioned Soft Actor-Critic (SAC) agent.

The Failure of Static Heuristics#

The Kelly Criterion, defined as $f^* = \frac{\mu - r}{\sigma^2}$, is the gold standard for theoretical optimal sizing. It maximizes the expected logarithmic growth of wealth. However, Kelly makes two catastrophic assumptions: it assumes returns are independently and identically distributed (i.i.d.), and that the estimated mean ($\mu$) and variance ($\sigma^2$) are known constants.

In volatile equity and crypto markets, these assumptions fail spectacularly.

Consider a mean-reversion strategy on BTC perpetual futures. During a low-volatility regime, a 30-day rolling window might yield $\mu = 0.002$ and $\sigma = 0.015$. Standard Kelly might suggest a leverage of $8.7x$. When a sudden regime shift occurs—driven by a liquidity vacuum or macro shock—volatility expands to $\sigma = 0.06$ while the mean reversion signal temporarily inverts. The static Kelly model is still holding $8.7x$ leverage into a blizzard, typically resulting in a drawdown exceeding 40% before the lookback window adjusts.

To quantify this, we backtested a standard z-score mean reversion strategy on BTC from 2022 to 2025 using three sizing methods:

1. Fixed Fractional: Risking 1% of portfolio equity per trade.
2. Static Kelly: Recalculated at the end of every 24-hour period.
3. Volatility-Scaled Kelly: Scaling static Kelly by the ratio of short-term realized vol to long-term realized vol.

While Volatility-Scaled Kelly improved the maximum drawdown from -38.2% (Static Kelly) to -24.5%, it still suffered from severe lag. The scaling factor is inherently backward-looking. By the time the short-term volatility estimate spikes, the damage to the portfolio's convexity has already been done. We need a sizing model that is forward-looking and adaptive.

Formulating Position Sizing as an MDP#

To move beyond heuristics, we formulate dynamic position sizing as a Markov Decision Process (MDP). The RL agent does not generate the alpha signal; it sits downstream of our existing alpha engines and acts purely as a risk-adjusted capital allocator.

The MDP is defined by the tuple $(S, A, P, R, \gamma)$:

State Space ($S$)#

The state represents the current market regime and portfolio context. We encode this as a 128-dimensional continuous vector updated at 1-minute intervals. Key features include:

• Microstructure: Top-of-book bid/ask imbalance, rolling 5-minute order flow toxicity (VPIN).
• Volatility Surface: Term structure of implied volatility (e.g., 1-day vs 7-day IV), realized vs. implied volatility spread.
• Portfolio Context: Current net leverage, unrealized PnL, portfolio heat (sum of active position variances).
• Signal State: The raw alpha score from the predictive model, and its rolling Sharpe ratio over the last 4 hours.

Action Space ($A$)#

We use a continuous action space $a_t \in [-1, 1]$, which is mapped to a target portfolio weight. We explicitly avoid discrete action spaces (e.g., "Buy 100%, Hold 0%, Sell 100%") because granular sizing is where edge is preserved in illiquid or highly impacted environments.

Reward Function ($R$)#

This is where most RL implementations fail. Training an agent on raw PnL leads to catastrophic risk-seeking behavior. We utilize a risk-adjusted reward formulation based on the differential Sharpe ratio, combined with a strict drawdown penalty.

def calculate_reward(portfolio_returns: np.ndarray, 
                    risk_free_rate: float = 0.0,
                    dd_penalty: float = 5.0) -> float:
    # Differential Sharpe Ratio component
    excess_returns = portfolio_returns - risk_free_rate
    sharpe = np.mean(excess_returns) / (np.std(excess_returns) + 1e-8)
    
    # Maximum Drawdown component
    cumulative = np.cumprod(1 + portfolio_returns)
    running_max = np.maximum.accumulate(cumulative)
    drawdowns = (running_max - cumulative) / running_max
    max_dd = np.max(drawdowns)
    
    # Asymmetric penalty: reward Sharpe, heavily punish drawdowns
    reward = sharpe - (dd_penalty * max_dd ** 2)
    return reward

Architecture: Why Soft Actor-Critic (SAC)#

At Venym Labs, our primary sizing agent is built on Soft Actor-Critic (SAC). While Proximal Policy Optimization (PPO) dominates the RL literature for game-playing and language model alignment, we strongly prefer SAC for financial execution due to its maximum entropy framework.

In high-frequency volatile markets, deterministic policies are fragile. If a PPO agent learns that 0.5x leverage is optimal in a specific regime, it will output exactly 0.5x every time. This creates a fingerprint that can be adversarially exploited by other market participants (or simply broken by minor distribution shifts).

SAC maximizes both expected return and entropy:

The entropy term $\mathcal{H}$ forces the agent to maintain stochasticity. In practice, this means our sizing agent might output a distribution of target weights centered around 0.5x, sometimes choosing 0.45x and sometimes 0.55x. This stochasticity smooths out our market impact and makes our aggregate order flow significantly harder to detect.

import torch
import torch.nn as nn

class SACPolicyNetwork(nn.Module):
    def __init__(self, state_dim: int, action_dim: int, hidden_dim: int = 256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU()
        )
        self.mean_head = nn.Linear(hidden_dim, action_dim)
        self.log_std_head = nn.Linear(hidden_dim, action_dim)
        
        # Bound the standard deviation to prevent numerical instability
        self.log_std_bound = nn.Parameter(torch.tensor([-5, 2]), requires_grad=False)

    def forward(self, state: torch.Tensor) -> torch.distributions.Normal:
        features = self.net(state)
        mean = self.mean_head(features)
        log_std = self.log_std_head(features)
        log_std = torch.clamp(log_std, self.log_std_bound[0], self.log_std_bound[1])
        
        return torch.distributions.Normal(mean, torch.exp(log_std))

System Architecture and Execution Pipeline#

Integrating an RL agent into a live trading loop requires extreme engineering rigor. Latency is the enemy of execution. We do not run the neural network inference in the hot path of our matching engine. Instead, we utilize an asynchronous, event-driven architecture.

The Risk Constraint Layer is a non-differentiable, hard-coded safety net. The RL agent outputs a desired weight, but the risk layer enforces absolute limits: maximum gross leverage, maximum single-asset concentration, and real-time portfolio Value-at-Risk (VaR) thresholds. If the agent suggests a $3x$ leverage but the portfolio VaR is already at 95% of its limit, the risk layer clips the action to $1.2x$. This ensures that even if the agent experiences a catastrophic hallucination, the fund's capital is protected by deterministic bounds.

Handling Non-Stationarity and Regime Shifts#

The biggest hurdle in financial RL is covariate shift. The data distribution in a bull market is fundamentally different from a bear market. An agent trained solely on 2023-2024 data will fail miserably in a 2022-style crash.

We address this through two mechanisms:

1. Adaptive State Normalization via Online Statistics#

We do not use standard z-score normalization (subtracting the training mean and dividing by training std). Instead, we use an exponential moving average (EMA) of the mean and variance, updated continuously in production. This ensures the state vector presented to the SAC agent remains statistically consistent regardless of the market regime.

2. Curriculum Learning with Explicit Regime Tagging#

During training, we tag our historical data with a hidden Markov model (HMM) that identifies distinct volatility regimes (e.g., Regime 0: Low Vol/High Correlation; Regime 1: High Vol/Low Correlation). We initialize the agent's

training on the most statistically common regime (typically low-volatility trending) to establish a stable baseline policy. As the agent's validation loss plateaus, we progressively inject trajectories from more chaotic, high-volatility regimes into the replay buffer. This curriculum forces the policy network to learn robust, generalized sizing behaviors rather than overfitting to a single market archetype. By the end of training, the agent has learned to dynamically adjust its entropy coefficient—becoming highly stochastic in unfamiliar regimes (erring on the side of de-risking) and more deterministic in high-conviction, familiar states.

Key Takeaways#

• Static heuristics are fundamentally broken in non-stationary markets: Traditional models like the Kelly Criterion rely on backward-looking, i.i.d. assumptions that fail catastrophically during sudden volatility regime shifts.
• Sizing should be decoupled from alpha generation: By framing position sizing as a downstream Markov Decision Process, quantitative teams can optimize capital allocation independently from signal prediction.
• Reward engineering dictates survival: Training RL agents on raw PnL incentivizes dangerous risk-seeking behavior; utilizing a differential Sharpe ratio combined with an asymmetric drawdown penalty is critical for robust policy convergence.
• Stochastic policies prevent adversarial exploitation: Soft Actor-Critic (SAC) is superior to deterministic algorithms (like PPO) for execution because its maximum entropy framework smooths market impact and eliminates predictable order flow fingerprints.
• Neural networks require deterministic guardrails: No matter how well-trained an RL agent is, live deployment requires a non-differentiable risk constraint layer to enforce absolute leverage, concentration, and Value-at-Risk (VaR) limits.

Conclusion#

At Venym Labs, replacing heuristic sizing with a reinforcement learning agent has fundamentally shifted our portfolio construction philosophy. We no longer view risk management as a static shield, but as a dynamic, adaptive weapon that actively contributes to our overall portfolio Sharpe ratio. As we continue to expand this framework across additional asset classes and incorporate cross-asset correlation dynamics into the state space, the gap between theoretical alpha and realized returns will only narrow. In the modern quantitative landscape, the edge no longer belongs solely to those who predict the future best, but to those who size their convictions most intelligently.