Computers for predicting economic moves

Researchers are trying to find models that will allow computers to predict economic moves. Is this possible, and what are the difficulties in predicting economic developments?

Israel Benjamin Galileo Magazine

"It's the economy, stupid" was one of the slogans that accompanied the election campaign of Bill Clinton for the presidency of the United States in 1992. Clinton used this slogan to convince the electorate to focus on the candidates' ability to solve the economic problems, because the US economy was going through a period of slowdown. As you may recall, in the end Clinton was able to defeat George W. Bush Sr., even though only a year earlier Bush had won an unprecedented level of public support of 89%.

Clinton's slogan drew its strength from the enormous impact that the economic situation has on every person, rich and poor. It is also a strong motivation to find ways to predict economic developments. The economic policy makers want not only to identify the economic trends and know whether they are likely to reverse their direction, but also to understand how the actions they are considering taking will affect the development of these trends: is it worth changing the interest rate? Will supporting manufacturing plants and financial institutions protect jobs or just waste public money?

The difficulties in predicting economic processes

Since economic forecasts and analyzes process a very large amount of data, it is reasonable to ask how well computers can help with economic predictions. The answer today is rather disappointing: although economists and financiers now make sophisticated use of computers and software, reliable predictive ability is still far from their reach.

Part of the reason for this lies in the general difficulty of predicting the process of the development of chaotic systems: even the weather, whose behavior equations are known, cannot be predicted more than a few days into the future. The computerized forecasting process places in these equations the current measurements (temperature, humidity, air pressure, wind direction and speed) for as many places as possible and calculates from this the expected measurement values ​​after a short period of time. When you put the result of the calculation in the same equations, you can find the expected values ​​after more time has passed, and so on until you reach a prediction for the desired time in the future. The difficulty in forecasting is due to the fact that small errors in the data measurement and in the calculations themselves accumulate during repetitions of the calculation, so that the calculated result may be very far from the real result.


Economic processes are even more difficult to predict. One of the factors of difficulty is the effect of the forecasting process itself on the development that the process is trying to predict: for example, if only one person received a reliable forecast showing that copper prices will rise next week, he would be able to purchase a significant amount of copper without raising its price, and sell it at a profit the following week. Unfortunately for him, he will find that once there is a method for reliable forecasting, there will be more traders who will use it. The result will be an immediate increase in the price of copper - perhaps even above the expected price for it next week. Is it possible to predict the effect of the prediction itself on the price of copper? It is clear that such a prediction will also encounter the same problem once more traders use it.

The result is similar to the mental magic circle of "Do they know that I know that they know that I know?", and as in this circle there is no sure way to know when it is right to stop the spinning. A process that refers to itself (self reference) is obtained, similar to a microphone that transmits to the speaker the sounds that the microphone picks up from that speaker, so that a loud and harsh sound is created.

A different assessment of the risk

The most basic difficulty is in understanding the laws governing the economic processes. Compared to meteorology, where it is known that air will flow from an area of ​​high pressure to an area of ​​lower pressure (this is an abstraction, but it is not too far from reality), we expect to see that money will flow to places where the interest rate is higher. This comparison is far from complete: apart from the difference between the interest rates, there are many other factors that affect the amount of money that will flow to the high interest rate, and even when these factors are known it is difficult to estimate their effect.

For example, one of the factors is risk, but different people assess risk differently. Even if the actors operating in the market assess the same level of risk for some action, some may perform the action while others will avoid it, as a result of different propensity to accept risks, from different compositions of their other investments, their short-term and long-term goals, etc.
The most basic difficulty is in understanding the laws governing the economic processes. One of the factors is risk, but different people assess risk differently

Is it possible to conclude from this that economics cannot be predicted and that economic models have no value? Of course not, but even the best models are just tools whose operation is more art than science. Sometimes the prediction succeeds, but sometimes it leads to severe failure, as the world has seen in recent months.

The problems of chaotic systems or self-relating processes are difficult to deal with, but it is possible to discover more about the laws by which the market operates, so that economic models can be used more intelligently. Much research has been done in this field, and naturally most of the achievements have been contributed by economists. This article describes additional modest contributions, which have come from the field of computer science and artificial intelligence.

Hidden links in the web of influence

Everyone who is involved in the stock market knows that there are connections between the changes in the value of different stocks. Sometimes the changes come together, for example when a general slowdown in a certain field causes the shares of the companies operating in that field to fall together. In other cases, a decline in the value of shares in some sector often foreshadows a future change, in the same direction or in the opposite direction, in shares in another sector. These cases are particularly interesting, of course, because knowing the future reaction allows prediction, at least in the form of an informed assessment of probabilities.

Some connections are known and obvious, but other connections are difficult to discover. It is easy to use statistical tools that look for a correlation in the change between any pair of stocks, and find out whether the correlation is positive or negative (the correlation is positive when an increase in one stock predicts an increase in another stock, and vice versa), what is the strength of the correlation, and how long does it take to manifest itself (that is, do the changes in both stocks occur at the same time or with some time difference).

The problem with using these tools is the large amount of correlations that are revealed: when we hear on the radio that "the New York Stock Exchange is currently showing a trend of rising prices", we understand that most stocks have risen, and therefore we will discover correlations between almost every pair of stocks. Therefore, the great majority of the connections between stock behavior are not interesting, but the large amount of connections and the strength of the "uninteresting" connections hide the interesting connections, like hiding thin and pale threads rolled in a large ball of wool.

Finding economic models that make sense

In mid-December 2008, a group of researchers from Dartmouth College (Dartmouth College, in the state of New Hampshire, USA) published a new way to discover the hidden connections within that cocoon of threads. The mathematical methods developed for this purpose are quite complex, but they can be described as a series of operations in each of which different clusters are identified. To explain this, we will look at the daily values ​​of 2547 different stocks over 1251 trading days by applying the method: after the correlations between each pair of stocks were calculated, the basic groups of stocks that behave in a similar way were identified. It is not surprising that these clusters generally corresponded to well-known sectors (sectors) - they have a similar type of occupation or a similar geographic territory.

Clusters were also found that combined stocks from several sectors: for example, in one of these clusters stocks from the fields of technology, services and finance appeared, when they were all related to healthcare companies. The matching of the clusters to sectors (or to cross-sector groups that tend to trade with each other) confirms that the mathematical method indeed reveals logical economic models.

For stocks that are in the same cluster, the correlations are strong, so it is impossible to identify other effects without subtracting the first correlations found. Subtracting the correlations produces new series of stock values, which represent how the stock values ​​would have behaved had they not been affected by those correlations already found. A hypothetical example: if most stocks in the biotech sector fell in a certain period relative to stocks in other fields, we would not be able to find out that biotech stocks related to green energy rose a little less than the others, as a result of the drop in crude oil prices. In this case, the high positive correlation between biotech stocks hides a lower positive correlation between some of this group and oil prices, and the second correlation will be revealed only after we subtract the effect of the first correlation.

The operation can be repeated for these new series, and then new correlations are revealed that define additional groups of stocks (note: the idea of ​​identifying main effects and subtracting their influence to identify additional effects is not new to this study; the innovation here is in the mathematical method suitable for a large amount of correlations between series of values).

When the researchers did this, a new pattern of connections between different business sectors was discovered: a pattern of a circle in which changes in one sector predict a change in another sector, and it predicts a change in another sector, etc., until the circle is closed. This pattern is also well known, and is known in economic analyzes as "sector rotation". The ability of the new method to confirm the theory of sector rotation and to calculate its numerical indices strengthens the hope that using such methods will help to understand the laws of economic behavior and create more accurate economic models than the ones we have today.

When the rules are broken

To study economic behavior, computer programs are used that simulate the behavior of an independent person, and respond to their environment according to the behavior model defined for them

The researchers from Dratmat point out in their article that they chose to identify the normal behavior and not to investigate the extraordinary behavior - those events in which the economic world turns sharply to a new path. In such events, the old rules of prediction lose their validity, and the market changes its behavior with a speed and power much higher than the behavior in normal times. After a while the market returns to a more "calm" behavior, which may obey the old prediction rules or start acting according to new rules.

The rules of prediction are discovered by analyzing the behavior of the market, but if we can explain what makes these rules true, we may also understand the point at which these rules "break".

One step towards this ambitious goal was presented in November 2008 by Charles Macal, a systems scientist at the US Department of Energy's Argonne Laboratory. As economists have known for a long time, the traditional economic models assume that economic systems reach an ideal balance between the interests of all participants in that system - producers, consumers, financiers, etc. This assumption is essentially an assumption of unlimited rationality, where all participants have access to the same knowledge, and they make the best decisions based on that knowledge.

One of the main contributors to the understanding that this assumption is flawed was the American psychologist Herbert Simon, who won the Nobel Prize in Economics in 1978 for his pioneering research in economic decision-making.

To investigate more realistic economic behavior, Makal and his partners used a model of "agents" - computer programs that simulate the behavior of an independent person, and respond to their environment and the behavior of other agents according to the behavior model defined for them. In this context, the agent is a person or organization active in the stock market, and the simulation defines economic action patterns for each "agent" according to a behavior model drawn from studies and surveys - for example, how willing the agent is to bear risks, how much value he places on future profit versus current profit, And how much effort he puts into making his decisions.

The software system mediates between the software parts that simulate each agent, and monitors the development of the market as a result of the interaction between the agents. In one of the studies, Argon's laboratory was asked to create a model of the electricity market in the state of Illinois in the USA. When Illinois prepared for the release of the market from regulation, there was a need to predict the consequences of this process, among them the effect of the different prices in different regions of the country on the behavior of the market.

Among other things, the researchers examined the hypothesis that in such a market, where there are a large number of producers and consumers, no party will be able to manipulate the market by excessively raising prices or artificially lowering the scope of electricity production. Running the software showed that this hypothesis is not true in situations of heavy load, and these situations can lead to situations of "breaking the rules" where the market seems to behave in an irrational and unpredictable way. These findings led to the recommendation that a certain level of control over pricing and energy production quantities be implemented.

Computing power is growing

As stated at the beginning of the article, there is still no method in sight that can prevent, or at least predict, economic crises like the one we are experiencing at this time. However, modest contributions such as those presented here may help the understanding of economic processes, if only somewhat.

Although some of the methods described here were already used a decade or more ago, the computing power required to run these methods became available only in recent years: already in the first stage of the cluster identification method, correlations between several million pairs of stocks were required, and the agent method used for the electricity market In Illinois, it requires simulating the behavior of thousands of manufacturers, factories and consumers. Technological progress was among the causes of economic growth, and the results of that progress, in the form of enormous computing power, may be able to help somewhat to restart growth.

Israel Binyamini works at ClickSoftware developing advanced optimization methods.

From: Galileo Magazine, February 2009