View source for Other possible algorithms for calculating binary predicates

{{Main|Aggregation techniques}}

This is a work in progress, for now just look at it as a collection of notes.

This is the original graph from example 1, just modified to show opinions as true/false with confidence values to avoid confusion.

<kroki lang="graphviz">
digraph G {
    fontname="Helvetica,Arial,sans-serif"
    node [fontname="Helvetica,Arial,sans-serif"]
    edge [fontname="Helvetica,Arial,sans-serif"]
    layout=dot
    0 [label="0, no opinion"]
    1 [label="1, no opinion"]
    2 [label="2, no opinion"]
    3 [label="3, true, confidence 20%"]
    4 [label="4, true, confidence 40%"]
    5 [label="5, true, confidence 60%"]
    6 [label="6, true, confidence 80%"]
    0 -> 1 [label="T=0.9"];
    0 -> 2 [label="T=0.9"];
    1 -> 3 [label="T=0.9"];
    1 -> 4 [label="T=0.9"];
    2 -> 5 [label="T=0.9"];
    2 -> 6 [label="T=0.9"];
    }
</kroki>
<span id="previous-results"></span>
== Previous results ==

In [[A_trust_weighted_averaging_technique_to_supplement_straight_averaging_and_Bayes|this analysis]], these values are given:

Pure Baysean <math display="inline">P_{bay} = 0.964</math> (<code>true</code>, with 92.8% confidence)

Simple average: <math display="inline">P_{ave} = 0.616</math> (<code>true</code>, with 23.2% confidence)

Weighted Average: <math display="inline">P_{ave,w0} = 0.634</math> (<code>true</code>, with 26.8% confidence)

<span id="derating"></span>
== Derating ==

At each step, we scale the confidence by our [[trust]] in the source. Working our way up, node 1 derates the answers from 3 and 4 using the trust factors, then averages the results: <math display="block">PT_1 = \frac{0.9 \cdot 0.2 + 0.9 \cdot 0.4}{2} = \frac{0.18 + 0.36}{2} = 0.27</math> <math display="block">PT_2 = \frac{0.9 \cdot 0.6 + 0.9 \cdot 0.8}{2} = \frac{0.54 + 0.72}{2} = 0.63</math> <math display="block">PT_0 = \frac{0.9 \cdot 0.27 + 0.9 \cdot 0.63}{2} = \frac{0.243 + 0.567}{2} = 0.405</math> Which is to say node 0’s computed [[opinion]] is <code>true</code>, with a 40.5% level of confidence (normally we’d express this as 0.7025). This makes sense, because if you look at all the nodes with an opinion, they all think it’s <code>true</code> with an average of 50% confidence, but that confidence is watered down by two layers of middlemen.

<span id="cons"></span>
===== Cons: =====

With this system, if I’m node number two and I only have node 6 as a source (pretend 5 doesn’t exist), my computed opinion will be: true, with 72% confidence (6’s 80% derated by their .9 trust). Fine. But when I add node 5 back in, my computed opinion falls to 63% confidence. This seems wrong. Adding a new node with the same opinion but a smaller <math display="inline">confidence \times trust</math> value seems like it shouldn’t decrease my computed opinion, regardless of whether it’s low because we have low trust in the source or if they have low trust in themselves. If I ask a medical question, and a doctor I have high trust in answers <code>true</code>, I shouldn’t reduce my confidence just because an accupuncturist I have low trust in also answers <code>true</code>. We’re assuming 0 trust means we think the party doesn’t know, not that they’re always wrong.

<span id="possible-fixes"></span>
===== Possible Fixes: =====

We could simply choose the max of the <math display="inline">confidence \times trust</math> values, instead of trying to average them. We haven’t discussed the case where we have conflicting answers, but my current mental model is look at the <math display="inline">max(confidence\times trust)</math> of the <code>true</code> responders and the <math display="inline">max(confidence\times trust)</math> of the <code>false</code> responders separately. Our resulting answer is whichever is larger, and our confidence is the difference.

Pros: Additional answers don’t pull the most trusted/confident answer down, which seems good. Additional answers also don’t boost your confidence, which could be good if the answers are all just reporting the same source of information. When you have a lot of conflicting trusted answers, your confidence will be low.

Cons: If you have ten trusted sources that say <code>true</code> and one trusted source that says <code>false</code>, your confidence will be low. This could be ok if the <code>true</code> responders were all just echoing the same source, or bad if they were providing unique data.

<span id="simple-weighted-averaging"></span>
== Simple weighted averaging ==

An alternate algorithm (different behavior) would be to take average of 3 and 4’s answers, weighting using the trust factor. <math display="block">PT_{1_{alt}} = \frac{0.9 \cdot 0.2 + 0.9 \cdot 0.4}{0.9 + 0.9} = \frac{0.18+0.36}{1.8} = 0.3</math> <math display="block">PT_{2_{alt}} = \frac{0.9 \cdot 0.6 + 0.9 \cdot 0.8}{0.9 + 0.9} = \frac{0.54 + 0.72}{1.8} = 0.7</math> <math display="block">PT_{0_{alt}} = \frac{0.9 \cdot 0.3 + 0.9 \cdot 0.7}{0.9 + 0.9} = \frac{0.27+0.63}{1.8} = 0.5</math> So with this method, node 0’s computed opinion would be <code>true</code> with a 50% confidence, which is what you’d get from a straight average of all the nodes with an opinion. In other words, in this algorithm, nodes with no opinion just pass on the opinion/confidence of their sources without derating them. The trust factor’s only effect is when a node has multiple sources and needs to value one opinion more highly than the other. I think during discussion, we decided that we actually prefer to have opinions become less potent the further away they are, so that argues against this algorithm.