View source for Straight average algorithm with continuous input distributions, complex trust, and intermediate results

{{Main|[[Aggregation techniques|Aggregation techniques]]}}

In [[Exercising the algorithm interface with more complex data types|this previous post]] we discussed an algorithm for modifying probabilities using a more complex [[trust]] factor, one involving random [[Deception|lying]], [[Bias|bias]], and biased lying. We also discussed a simple case of combining [[Continuous distribution|continuous distribution]]s using [[Bayes' theorem|Bayes]], straight averaging and [[Trust-weighted averaging|trust-weighted averaging]]. However, the continuous distributions in this case did not use the complex [[Trust factor|trust factor]].

In [[Notes on setting up and using the sandbox|this post from last week]] we discussed a way to modify probabilities using the complex trust factor with continuous distributions. Doing so requires that we break up the continuous distribution into the set of choices being offered in the question (eg rainy, cloudy, sunny would be 3 choices) and apply the lying and biased lying portion of the [[Trust|trust]] modification to each choice accordingly.

None of this previous work, however, allowed us to calculate a full multi-level tree of nodes where we use <code>intermediate_results</code> to transfer calculations up to the parent node so the calculation can continue. This post is about an algorithm that rectifies this shortcoming and yields the ability to do straight averaging with complex trust, continuous input distributions, and multi-level calculations.

The algorithm, at its essence, keeps track of each node’s a) [[Personal opinion|personal opinion]] and b) the sum of the [[Opinion|opinion]]s of all its children (after they have been modified by trust). These two pieces of information become the <code>intermediate_results</code> which are used to calculate the average for that node and are passed up to the parent node. Each parent, in turn, sums all its children’s <code>intermediate_results</code> together to form, along with its personal [[opinion]], its own <code>intermediate_results</code>.

First, let’s start by reminding ourselves of how <code>intermediate_results</code> is constructed in this context:

<code>[personal opinion, sum of child opinions, population of children]</code>

Since we are dealing with continuous distributions, we include the x-values of the points in this list:

<code>[[x points of personal opinion, pdf points of personal opinion], [xpoints of child opinions, sum of pdf points of child opinions], population of children]]</code>

The following is an example of this:

<syntaxhighlight lang="python">inter_results012 =  [[[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], [2.00051038563241, 2.00051038563241, 2.00051038563241, 2.00051038563241, 2.00051038563241, 2.00051038563241, 1.5350918071000005e-06, 1.5350918071000005e-06, 1.5350918071000005e-06, 1.5350918071000005e-06, 1.5350918071000005e-06]], [[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], [5.821628442333012, 5.821628442333012, 5.821628442333012, 5.821628442333012, 5.821628442333012, 5.821628442333012, 6.144719133017774, 6.144719133017774, 6.144719133017774, 6.144719133017774, 6.144719133017774]], 6]
</syntaxhighlight>
Let’s also remind ourselves of the complex <code>trust_factor</code> which is broken up into a judgement trust and a communication trust:

<syntaxhighlight lang="python">trust_factor = [ [0.8, 0.1, 0.0, 0.02, 0.03, 0.02, 0.03],    #Tj, judgement trust
                 [0.9, 0.1, 0.0, 0.0, 0.0, 0.0, 0.0] ]  #Tc, communication trust</syntaxhighlight>
Here there are two choices (say rainy and sunny) so each <code>Tj</code> and <code>Tc</code> could be described as:

<code>[Ttruth, Trandom, Tlierandom, Tliebias_rainy, Tliebias_sunny, Tbias_rainy, Tbias_sunny]</code>

This is discussed at some length in [[Exercising the algorithm interface with more complex data types|Exercising the algorithm interface with more complex data types]].

The new algorithm is encapsulated in the function <code>straight_ave_points_continuous_complex_trust_intermediate</code> which looks like the following and is available in the following [https://gitlab.syncad.com/peerverity/trust-model-playground/-/snippets/151 snippet]:

<syntaxhighlight lang="python">@register_algorithm()
def straight_ave_points_continuous_complex_trust_intermediate(data: AlgorithmInput):
    convert_functions_to_points_continuous(data)
    set_initial_intermediate_results_sapccti(data)
    modify_prob_data_for_continuous_complex_trust_intermediate(data)
    sum_kids_intermediate_results_and_putinto_parent(data)
    x_list, Pdave_list = ave_points_continuous_complex_trust_intermediate4(data)
    opinion_output = OpinionData([x_list, Pdave_list], 1)
    intermediate_results = data.components[0].intermediate_results #not really necessary
    algout = AlgorithmOutput(opinion_output, intermediate_results)
    return algout</syntaxhighlight>
At this outermost level the algorithm is composed of 5 basic steps, the first five function calls in the script above:

# Converting input continuous distributions (provided by a [[wikipedia:Weibull distribution|Weibull distribution]]) to discrete points. This is governed by the <code>misc_input</code> parameter, particularly the number of points, <code>n</code>, desired. The function <code>convert_functions_to_points_continuous</code> handles this.
# Setting initial <code>intermediate_results</code> to zero for leaf nodes. These nodes only have personal opinions. However, by forcing them to have <code>intermediate_results</code> of the same form as all other nodes, the calculations can be applied consistently to all nodes (with no need for special casing the leaf nodes). The function <code>set_initial_intermediate_results_sapccti</code> handles this.
# Modifying the probability data using complex trust as discussed in [[Exercising the algorithm interface with more complex data types|this previous post]]. <code>modify_prob_data_for_continuous_complex_trust_intermediate</code> handles this.
# Take the sum of all the children <code>intermediate_results</code> and place this result as part of the parent <code>intermediate_results</code> (along with the parent personal opinion). This is done in <code>sum_kids_intermediate_results_and_putinto_parent</code>.
# Use the result of 4 to take the straight average for the node, which consists of its personal opinion summed with the <code>intermediate_results</code> of all the children. The function <code>ave_points_continuous_complex_trust_intermediate4</code> handles this. Incidentally, the 1-3 variants of this are earlier versions of the function which are not important to this discussion. They are kept for reference since they have properties of interest for future calculations.

The function <code>convert_functions_to_points_continuous</code> takes continuous functions given by a Python function name, eg:

<syntaxhighlight lang="python">def weibull(k, lamb, x):
    w = (k/lamb)*(x/lamb)**(k-1.0)*math.exp(-(x/lamb)**k)
    return w

def Pd0(x):
    k = 3.0
    lamb = 0.2
    return weibull(k, lamb, x)</syntaxhighlight>
and converts them to a discrete set of (x, p) points depending on the <code>misc_input</code>, eg:

<syntaxhighlight lang="python">misc_input = {'xmin': 0.0, 'xmax': 1.0, 'n': 10}</syntaxhighlight>
The continuous function is sampled at each of the <code>n</code> intervals between <code>xmin</code> and <code>xmax</code> to create the corresponding set of discrete points. If the input is already a set of discrete points, <code>convert_functions_to_points_continuous</code> does nothing.

The function <code>set_initial_intermediate_results_sapccti</code> provides an empty set of <code>intermediate_results</code> to nodes (leaf nodes) that do not already have them:

<syntaxhighlight lang="python">def set_initial_intermediate_results_sapccti(data):
    for comp in data.components:
        if(comp.intermediate_results == []):
            x_points_zero = comp.opinion.pdf_points[0]
            pdf_points_zero = [0.0]*len(x_points_zero)
            comp.intermediate_results = [[comp.opinion.pdf_points[0], comp.opinion.pdf_points[1]], [x_points_zero, pdf_points_zero], 0]</syntaxhighlight>
The last line in the above provides the <code>intermediate_results</code> for a particular node, that is the (x, p) points for its personal opinion, the (x, p) points of the nonexistent children where p is a list of zeros, and the population of the children which, in this case, is zero.

<code>modify_prob_data_for_continuous_complex_trust_intermediate</code> is an outer function for doing the probability modification using the complex trust. It breaks up each node’s <code>intermediate_results</code> into its corresponding personal opinion and the summed opinion of its children. The personal opinion is then modified using the judgement trust portion of the <code>trust_factor</code> and the summed children opinions are modified using the communication portion of the <code>trust_factor</code>. Since in most cases the population of the children will be greater than 1, the summed probability of the children is divided by this population to create an average probability for the trust-modification. After modification, the result is multiplied by the population to recreate the sum for continued calculation:

<syntaxhighlight lang="python">def modify_prob_data_for_continuous_complex_trust_intermediate(data):
    for comp in data.components:
        # personal opinion
        xp_points_personal = comp.intermediate_results[0]
        x_points_personal = xp_points_personal[0]
        pdf_points_personal = xp_points_personal[1]
        T_personal = comp.trust_factor[0] #use Tj
        comp.intermediate_results[0][1] = calc_pdfmod_from_complex_trust_intermediate(x_points_personal, pdf_points_personal, T_personal)
        # computed opinion of children
        xp_points_children = comp.intermediate_results[1]
        x_points_children = xp_points_children[0]
        pdf_points_children = xp_points_children[1] #this is really a sum
        T_children = comp.trust_factor[1] #use Tc
        Npop = comp.intermediate_results[2]
        #Need to modify the function to handle Npop > 1 PM_090623
        if(Npop == 1):
            comp.intermediate_results[1][1] = calc_pdfmod_from_complex_trust_intermediate(x_points_children, pdf_points_children, T_children, Npop)
        elif(Npop > 1):
            #account for the fact that we have a sum here, not an individual probability
            pdf_points_children = (np.array(pdf_points_children) / Npop).tolist()
            inter_results = calc_pdfmod_from_complex_trust_intermediate(x_points_children, pdf_points_children, T_children, Npop)
            inter_results = (np.array(inter_results) * Npop).tolist()
            comp.intermediate_results[1][1] = inter_results
        else: #(Npop == 0)
            continue
</syntaxhighlight>
Within this function another function, <code>calc_pdfmod_from_complex_trust_intermediate</code> is called to perform the actual modification calculation. This follows exactly the description set out in the [[Notes on setting up and using the sandbox|last post]] starting at the section titled “Trust modification for continuous algorithms using complex trust factors”.

The function <code>sum_kids_intermediate_results_and_putinto_parent</code> goes through each child node and adds up all their <code>intermediate_results</code>. This sum then becomes this parent’s child portion of its <code>intermediate_results</code>, alongside its personal opinion:

<syntaxhighlight lang="python">def sum_kids_intermediate_results_and_putinto_parent(data):
    #parent is the first one (index 0) and kids are the rest
    x_points = data.components[0].opinion.pdf_points[0]
    kids = data.components[1:]
    tot_kids = [0.0]*len(x_points)
    tot_npop_kids = 0
    for kid in kids:
        sum_kid = addlists(kid.intermediate_results[0][1], kid.intermediate_results[1][1])
        npop_kid = 1 + kid.intermediate_results[2]
        tot_kids = addlists(tot_kids, sum_kid)
        tot_npop_kids = tot_npop_kids + npop_kid
    #put this into the parent intermediate result
    data.components[0].intermediate_results[1][1] = tot_kids
    data.components[0].intermediate_results[2] = tot_npop_kids</syntaxhighlight>
Given a finalized set of <code>intermediate_results</code>, the only remaining task is to take the average for the node which is done in <code>ave_points_continuous_complex_trust_intermediate4</code>. This is fairly straightforward since it only involves adding up the node’s personal opinion with the already calculated sum of its children and dividing by the total population:

<syntaxhighlight lang="python">def ave_points_continuous_complex_trust_intermediate4(data):
    inter_results_parent = data.components[0].intermediate_results
    x_data = inter_results_parent[0][0]
    p_parent = inter_results_parent[0][1]
    p_sum_kids = inter_results_parent[1][1]
    npop_kids = inter_results_parent[2]
    p_sum = addlists(p_parent, p_sum_kids)
    npop = npop_kids + 1
    p_ave = (np.array(p_sum) / npop).tolist()
    return [x_data, p_ave] </syntaxhighlight>