Technical overview of the ratings system

Let's take a very simple situation. You want to know whether it is going to rain tomorrow. You don't know so you ask two knowledgeable sources this question. One of them believes it will rain with a probability of 60%. The other believes it will rain with a probability of 80%. We can sketch this situation as follows:

You combine these probabilities using some aggregation technique. One such technique is Bayes' equation which, when applied to this case gives us:

$$P_{comb}={{0.6(0.8)} \over {0.6(0.8)+0.4(0.2)}}=0.857$$

Another technique is a straight average of the two answers:

$$P_{comb}={{0.6+0.8} \over {2}}=0.7$$

There are other aggregation techniques and which one we pick depends on the nature of the data. Here, for Bayes, we assume that our two weather predictors are independent and have a test to arrive at their probabilities. Since both tests are independent, they tend to reinforce each other. Bayes is very rigorous about data and is most useful in situations where we have scientifically controlled tests (eg efficacy of a new medicine for curing a disease). For the averaging technique the data is not as rigorous. Maybe they represent two friends who are providing an off-the-cuff opinion on the chance of rain tomorrow.

This example assumes that we trust our two sources. But what if we only partially trust them? In that case we would expect to reduce the weight of the answers we get from them. Let's suppose we have a 70% trust in our first source and a 90% trust in our second source:

We can now modify our probabilities using the following equation:

$$P_{mod}=P_{nom}+T(P-P_{nom})$$

where

$P_{nom}$ is the nominal probability, ie 50%

$P_{mod}$ is the modified probability

$T$ is Trust

$P$ is the Probability assuming complete Trust

Note that for zero trust this equation reduces the probability to 50%, which is the same as a random answer and provides no meaningful information.

With this equation in mind we calculate our new probability.

$P_{mod1}=0.5+0.7(0.6-0.5)=0.57$

$P_{mod2}=0.5+0.9(0.8-0.5)=0.77$

Notice how the Trust values pushed both probabilities closer to 0.5.

The network in this example consists of one level, but we can have more. If our two sources in turn rely on their sources we might have a situation like this:

Here the questioner (0) asks source 1 for their opinion on whether it will rain tomorrow. 1 has a personal opinion on this subject but asks two other contacts for their opinion and rolls all the numbers up into an aggregate which is transmitted to 0. If we use Bayes, the aggregate is calculated as follows:

First we modify the probabilities by the trust 1 has in 3 and 4:

$P_{mod3}=0.5+0.8(0.55-0.5)=0.54$

$P_{mod4}=0.5+0.6(0.95-0.5)=0.77$

These two values are combined via the Bayes eqn:

$$P_{comb34}={{0.54(0.77)} \over {0.54(0.77)+0.46(0.23)}}=0.797$$

We can then combine this with 1's own opinion. Since 1 trusts himself, presumably, we don't have to modify his 60% probability with a trust factor. We simply apply Bayes' equation again:

$$P_{comb134}={{0.797(0.6)} \over {0.797(0.6)+0.203(0.4)}}=0.855$$

We do the same thing for Source 2 with respect to its sources (5 and 6):

$P_{mod5}=0.5+0.75(0.65-0.5)=0.613$

$P_{mod6}=0.5+1.0(0.9-0.5)=0.9$

$$P_{comb56}={{0.613(0.9)} \over {0.613(0.9)+0.387(0.1)}}=0.934$$

$$P_{comb256}={{0.8(0.934)} \over {0.8(0.934)+0.2(0.066)}}=0.983$$

Now we have two aggregated opinions for 1 and 2, 0.855 and 0.983. These are now modified by the trust 0 has for 1 and 2:

$P_{mod1}=0.5+0.7(0.855-0.5)=0.748$

$P_{mod2}=0.5+0.9(0.983-0.5)=0.935$

These two values are then aggregated via Bayes to obtain the overall probability, ie the final answer:

$$P_{comb,overall}={{0.748(0.935)} \over {0.748(0.935)+0.252(0.065)}}=0.977$$