Technical overview of the ratings system

Let's take a very simple situation. You want to know whether it is going to rain tomorrow. You don't know so you ask two knowledgeable sources this question. One of them believes it will rain with a probability of 60%. The other believes it will rain with a probability of 80%. We can sketch this situation as follows:

You combine these probabilities using some aggregation technique. One such technique is Bayes' equation:

$$P_{comb}={{0.6(0.8)} \over {0.6(0.8)+0.4(0.2)}}=0.857$$

Another technique is a straight average of the two answers:

$$P_{comb}={{0.6+0.8} \over {2}}=0.7$$

There are many possible aggregation techniques and which one we pick depends on the nature of the data. Here, for Bayes, we assume that the input values are the result of some probabilistic test backed up by data. In this case, if the two sources are independent, the answers will tend to reinforce each other. Bayes tends to be very rigorous about data and is most useful in situations where we have scientifically controlled tests (eg efficacy of a new medicine for curing a disease). For the averaging technique the data is not as rigorous. Maybe they represent two friends who are providing an off-the-cuff opinion on the chance of rain tomorrow.

This example assumes that we trust our two sources. But what if we only partially trust them? In that case we would expect to reduce the weight of the answers we get from them. Let's suppose we have a 70% trust in our first source and a 90% trust in our second source:

We can now modify our probabilities using the following equation:

$$P_{mod}=P_{nom}+T(P-P_{nom})$$

where

$P_{nom}$ is the nominal probability, ie 50%

$P_{mod}$ is the modified probability

$T$ is Trust

$P$ is the Probability assuming complete Trust

Note that for zero trust this equation reduces the probability to 50%, which is the same as a random answer and provides no meaningful information.

With this equation in mind we calculate our new probability.

$P_{mod1}=$