About the fattest cow that saved the honor of democracy, and the tweets that predicted the future of the stock market

The median yielded the closest result to the weight of the bull, even closer than that of all the cattle experts who participated in the competition. The average, by the way, was even closer.

The farmers examined the herd carefully, and chose the fattest and most attractive cow. The horse was tied around his neck and put on display on the stage, where he was exposed to all the audience in the weight-jumping competition at the annual fair in Plymouth, England. Over eight hundred competitors purchased tickets, at the price of sixpence each, on which they wrote their names and guesses for the bull's weight. The guesses closest to the bull's weight earned their owners valuable prizes. And this is where the whole episode would have ended, had it not been for the presence in the audience of one of the most prominent scientists of the 19th century, Francis Galton.

"If we take the median of the people's votes, what will be the result?" Galton wondered. The median is a statistical tool that is similar in nature to the average, but ignores the most extreme votes up and down. When Galton summed up all the votes and calculated the median, he was pleasantly surprised: the median yielded the closest result to the bull's weight, even closer than that of all the cattle experts who participated in the competition. The average, by the way, was even closer.

Galton published the surprising phenomenon in the journal Nature in 1907, under the name 'vox populi' - the voice of the masses. "It is likely that the average participant in the contest was competent in estimating the weight of the bull, just as the average voter is competent in understanding the value of most of the political issues on which he votes." Wrote, with a large degree of justice. The individual voter may not be smarter or wiser than the average, but when the votes of all the citizens are weighed, it can be expected that they will reach a decision close to the right one. "This result, I think, gives more validity to our confidence in the democratic judgment than could have been expected."

There is of course a big difference between determining the weight of a bull and choosing the most suitable politician for the job, but Galton's idea was preserved and established as the 'wisdom of the masses'. It seems that in some cases the crowd is able to reach decisions and insights that are deeper and more accurate than those of each individual on his own.

These days, a new study shows that the feelings of the crowd are also capable of predicting more bizarre trends than voting for the Labor Party, as hard as it is to believe. A summary of nearly 10 million tweets ('tweets' - a nickname for a short message on the social network Twitter) showed that the level of calm in the tweet-count can predict whether the Dow Jones index will rise or fall in the coming days with an accuracy of 87.6 percent.

The researchers examined 9.8 million tweets, and classified some of them that expressed emotions according to the type of emotion: calmness, vigilance, confidence, liveliness, generosity and joy. They divided the tweets by day, and created a graph that showed the level of each public emotion according to the date. The next step was to check how each public sentiment changes according to specific dates. The researchers initially focused on the three days surrounding Election Day in the United States. The day before election day, the level of calmness drops, but it rises again on election day itself, along with a significant jump in the ranking of liveliness, joy and generosity. Another test was conducted on Christmas Day in November 2008, where a significant increase in the happiness rating can be found only.
These results confirm the algorithm's ability to analyze the feelings of the Tweet-count, but the big surprise came when the researchers tried to compare the national mood with the Dow Jones Industrial Average. They found to their surprise that the calm index correlates well with the ups and downs of the stock market, but leads them by a few days at a time.

This is not a strong proof in itself, so the researchers decided to try and check whether Twitter can be used to predict the future of the stock market. They created an algorithm that predicted whether the Dow Jones index would rise or fall based on the Dow Jones averages of the past three days. This algorithm was able to predict the future of the market with an accuracy of 73.3 percent. It's not bad, but when the calm data from Twitter was also entered into the algorithm, the accuracy percentages increased to 86.7 percent.

Although the results look impressive at first glance, it is recommended to remain skeptical. There is always the fear that the researchers determined the expected research results in advance, and adapted the methods in order to reach them. The fear increases due to the fact that the authors of the study chose to publish it in an online journal - arXiv - which is not officially reviewed by other scientists. This journal usually publishes mathematical and physical articles, so it is particularly difficult to understand why the researchers chose it as the place to publish the research. All of these raise question marks regarding the reliability of the results. As always, it is advisable to wait for further studies that will confirm the algorithm from the current study. And if other researchers come and refute the given results?

Well, that's the power of the wisdom of crowds.