A trust weighted averaging technique to supplement straight averaging and Bayes

Main article: Aggregation techniques

Introduction

We have put forth a straightforward averaging technique for situations where Bayes was not appropriate. Probabilities were trust-modified (per Sapienza or an augmented method) and passed up the tree until the top-most node was reached where they would be averaged. The Bayesian combined probability was also found and a “weighting toward Bayes” could be used to achieve a result in between the average and the Bayesian combined probability.

The trust-modification for both the average and Bayesian combined probability was the same (ie Sapienza or other). This means that Trust = 0 reduces any probability to 50% where, in Bayes, it stops contributing to the calculation. However, for averaging, it does contribute to the calculation. 100 50% opinions and 1 100% opinion will average to produce an opinion pretty close 50%. In Bayes, the combined probability in such a situation would be 100%.

Indeed, this phenomenon is one of the reasons we advocated the averaging approach in the first place. If 100 people are uncertain about something and a 101st person expresses total confidence, you would probably be a little skeptical of the 101st person’s opinion.

Why does a 50% answer lead to such different results?

Still, these results are so starkly different that they demand some fundamental explanation. What’s going on here? Why does this dichotomy exist? And is there anything can we do about it in terms of trust? Can we employ trust consistently to get comparable results for the averaging and Bayesian technique?

So, to answer this question, let’s note that there is a difference between someone who replies “I don’t know” because they don’t know anything about the subject (simple ignorance) and someone who says the same because the answer really is unknown. The first person should be removed from the calculation. They simply shouldn’t count. The second person should count because they are contributing a valuable piece of information, that uncertainty is our current state of knowledge on some particular question.

If 100 people in the first category answer the question and then a 101st person, an expert, answers we should ignore the first 100 and go with the expert. If 100 people in the second category answer and a 101st person contradicts their opinion, we should take that contradiction with a grain of salt because the uncertainty of the 100 is probably closer to the truth.

In Bayes, the 50% answer doesn’t count, leading us to believe that Bayesian answers are in the first category. Does this make sense? A Bayesian answer is really just a test that has a probability associated with it. If the test gives you a 50-50 answer it’s as good as a coin toss. The test doesn’t really know anything about the limits of knowledge in a particular area. So it does seem like the Bayesian 50% answer is just like a dumb human answer, one that should be removed from the calculation.

A human being answering using their own judgement is in the second category, provided their judgement is based on real knowledge. This is something we can presumably assign a trust value to.

The Trust-weighted Method

In the simple averaging scheme we were trust-modifying the probabilities using the Sapienza eqn. Therefore Trust = 0% leads to a probability of 50% which counts in the straight averaging approach (although it doesn’t count in Bayes). Here we propose a scheme in which the Trust is used as a weighting in the averaging scheme itself. Trust = 0% causes a source to not count at all, same as in Bayes. Trust = 100% causes the source to be counted as a full source. One advantage of this approach is that user’s could assign Trust > 100% if they have an especially high regard for their source.

For a single parent at Node 0 and n children nodes all on the same level, the equation looks like this:

$$P_{ave,w}={\sum _{i=0}^{n}P_{i}T_{i} \over {\sum _{i=0}^{n}T_{i}}}$$

where

${\textstyle P_{i}}$ is the probability for node ${\textstyle i}$

${\textstyle T_{i}}$ is the trust between the parent node and node ${\textstyle i}$

This allows Trust to have a similar effect on averaging that it has for Bayes: it removes a source from the calculation when that source’s Trust is zero.

Example 1

First we will reproduce the example from the simple averaging case and outline the calculation procedure for the weighted average case.

This calculation works by starting at the bottom and collecting all the probability and trust numbers into a list that we can then apply the averaging calculations to. The leaf nodes, at the very bottom, require no calculation and are ignored.

The first significant Node is Node 1 which consists of a list of its own information and that of its children. Each sublist has probability (P,1-P) followed by the trust.

${\textstyle PT_{1}=[[0.5,0.5,1],[0.6,0.4,0.9],[0.7,0.3,0.9]]}$

The weighted average for this case is (just doing P, not 1-P):

${\textstyle P_{ave,w1}={.5(1)+0.6(0.9)+(0.7)(0.9) \over {1+0.9+0.9}}=0.596}$

Node 2 is similar:

${\textstyle PT_{2}=[[0.5,0.5,1],[0.8,0.2,0.9],[0.9,0.1,0.9]]}$

${\textstyle P_{ave,w2}={.5(1)+0.8(0.9)+(0.9)(0.9) \over {1+0.9+0.9}}=0.725}$

Node 0 now consists of its own node and the two previous results for Node’s 1 and 2 with an additional trust of 0.9 to represent the link from 0-1 and 0-2:

${\textstyle PT_{0}=[[0.5,0.5,1.0],[0.5,0.5,1.0,0.9],[0.6,0.4,0.9,0.9],[0.7,0.3,0.9,0.9],[0.5,0.5,1.0,0.9],[0.8,0.2,0.9,0.9],[0.9,0.1,0.9,0.9]]}$

With this information the calculation can proceed for the weighted average of the top-node:

${\textstyle P_{ave,w0}={0.5(1)+0.5(1)(0.9)+0.6(0.9)(0.9)+0.7(0.9)(0.9)+0.5(1)(0.9)+(0.8)(0.9)(0.9)+0.9(0.9)(0.9) \over {1+1(0.9)+(0.9)(0.9)+0.9(0.9)+1(0.9)+0.9(0.9)+0.9(0.9)}}=0.634}$

You will recall that this example yielded the following results for the straight averaging technique and Bayes:

${\textstyle P_{ave}=0.616}$

${\textstyle P_{bay}=0.964}$

The weighted average and straight average yield similar results because Trust = 0.9 at worst so all nodes have a relatively high weight. Indeed, if the Trust were set equal to 1, both the straight and weighted average results would be exactly the same:

$$P_{ave0}=P_{ave,w0}=0.643$$

The following example will show how different the results of these two calculations can be and will illustrate some of the properties mentioned above.

Example 2

${\textstyle P_{ave,w0}=0.7}$

${\textstyle P_{ave,0}=0.58}$

${\textstyle P_{bay}=0.9}$

In this example, the only trustworthy nodes are 0 and 4. So the weighted average technique correctly drops nodes 1-3 from the calculation, yielding an average of 0.7, which can be confirmed by inspection. The straight average is 0.58, however, because nodes 1-3 are counted at 50% in this case (after adjusting them for Trust per Sapienza). The Bayesian combined probability is 0.9 because 50% probabilities simply don’t count in the Bayes equation.

The 0.7 is a reasonable answer given that we trust ourselves (Node 0) and an expert (Node 4) who has a more certain answer. Note that the 0.58 unduly influences us toward Nodes that we do not trust because we believe them to be ignorant of the subject matter. The Bayesian result is, for reasons discussed above, also a bit too optimistic since we have another knowledgeable node with a valid, if uncertain, opinion.

This snippet will reproduce the above calculations and similar examples of your choosing.