Optimization of Stock Trading Strategies Using a Hybrid Reinforcement Learning and Forecasting Model

Abstract

Stock price prediction is an interesting challenge in machine learning due to the non-linear nature of the market. Although forecasting models can predict prices, they often do not provide optimal trading strategies. Reinforcement learning (RL) has the potential to optimize strategies, but it is highly dependent on the input states. This study integrates two methods—a CNN-LSTM forecasting model and RL (A3C)—to develop an algorithmic trading strategy. The model is evaluated using historical INDF stock data (2016–2024) with a data-split validation protocol of 80% training and 20% testing. Backtesting simulations on the period (Feb 2023–Dec 2024) show that the hybrid model achieves a cumulative total return of 121.44%. This result was obtained using an all-in trading strategy (one full position at a time) and includes transaction costs: a trading fee of 0.01% per transaction and a borrow interest rate of 0.0003% per day for short positions. This performance significantly outperforms traditional strategies: Buy and Hold (23.45%), MA Crossover (51.13%), RSI (9.09%), and MACD (−29.08%). The hybrid model also achieves a Sharpe Ratio of 2.381 (annualized, assuming a 0% risk-free rate).