Reinforcement Learning in Financial Trading

Our Methodology - Reinforcement Learning (RL)

Trade Like A Machine utilises a branch of artificial intelligence (AI) known as 'Reinforcement Learning'. This is a machine learning technique that enables an agent to learn from its actions and feedback from its environment. The learning process behind RL is inspired by how humans and animals learn from trial and error, by rewarding good behaviors and punishing bad ones. RL has been successfully applied to various domains, such as gaming, robotics, self-driving cars, healthcare, and finance.

The key difference between the supervised learning branch of AI, and RL is key to understand. RL does not merely make predictions. It actually ‘learns’ what actions are required (and when) to maximise reward. In short, in our context, it means that it ‘learns to trade’.

RL Success Stories

One of the most famous examples of RL is DeepMind’s AlphaGo, which is a computer program that learned how to master the ancient board game Go by playing millions of games against itself. In 2016, AlphaGo defeated Lee Sedol, one of the world’s best Go players, in a historic match that demonstrated the power and potential of RL. AlphaGo was able to discover new strategies and moves that no human player had ever seen or used before, and to outsmart its human opponent with its creativity and intuition.

Another example of RL is DeepMind’s AlphaFold, which is an algorithm that learns how to predict the three-dimensional structure of proteins by using data from previous experiments. In 2020, AlphaFold achieved a breakthrough in solving one of the most challenging problems in medical science, which could have huge implications for drug discovery and disease treatment. AlphaFold was able to accurately predict the shape of proteins that are essential for life and to surpass the previous performance of human experts and other methods.

RL in Financial Trading

In finance, RL has also been gaining popularity and attention as a way to design adaptive and intelligent trading algorithms. However, due to the proprietary and commercially sensitive nature in this sector, research and development is much less often made public. There have however been several studies in the academic space that have shown RL to be significantly more effective than traditional algorithmic trading techniques.

At Trade Like A Machine we have long-standing experience of trading the financial markets using 'classical' automated strategies (typically termed 'Algorithmic Trading'), but as of the beginning of December 2023 have completely migrated all of our trading strategies to models developed using RL. This followed an intensive 24 month period of research and development. So far, the new RL-based strategies are generating significantly more alpha from the markets than our classical agorithmic trading strategies did.

Deep Q-Learning

One of the main challenges of RL is how to deal with complex and high-dimensional state spaces, which are the sets of possible situations that an agent can encounter. For example, in financial markets, an agent needs to consider many factors, such as prices, volumes, indicators, etc., when making trading decisions. To cope with this challenge, we decided to use deep Q-learning (DQL), which is a combination of RL and deep neural networks (DNNs).

DNNs are a type of artificial neural network (ANNs) that consist of multiple layers of interconnected nodes that can learn complex patterns from data. DNNs have been widely used for various tasks, such as image recognition, natural language processing, and speech synthesis. DQL uses DNNs to approximate the Q-function, which is a function that estimates the expected future reward for each action given a state. By using DNNs, DQL can handle large state spaces and learn nonlinear relationships between states and actions.