Information Rating System Wiki - User contributions [en]

Main Page

2026-03-26T13:54:01Z

Lem: Reverted edit by RickieGordon8 (talk) to last revision by Pete

<strong>Welcome to the Information Rating System Wiki.</strong>

[https://peerverity.info/ PeerVerity] is on a mission to combat disinformation by empowering individuals with tools to share and rate information. The project described in these pages is a peer-to-peer [[ratings system]]. Given the overwhelming quantity of online information, and the ease with which it can manipulate its audience, we seek to develop a better means to qualify, filter, and analyze it. The basic idea is that by using a network of users, and [[Aggregation techniques|aggregating their opinion]], we can more accurately (and quickly) assess information and make better decisions from it.

The system will allow users to create networks of other users who, in turn, have their own networks. Therefore the system can be several levels deep. A question can be asked (eg should the Fed reduce interest rates?) and folks on the network would answer it in a privacy preserving manner. An aggregator would collect the answers, compute an average (or some other aggregate calculation), and return the result to the person asking the question. The questioner can assign [[trust]] levels to individuals in their network and use these in the aggregate calculation. This system, as described, is personal in nature and can be viewed as a [[The subjective and community ratings system|Subjective Ratings System (SRS)]].

Some people on the network may choose to be public and allow anyone to link to them for their [[opinion]]. Therefore, there will be private nodes, who remain unknown to all but the nodes they choose to link with, and public nodes who are known to everyone.

We anticipate that like-minded people will join together to form communities and create a variant of the public SRS, known as a [[The subjective and community ratings system|Community Ratings System (CRS)]]. This ratings system will enable a voting system which will do a better job of understanding people's needs and allow them to participate in policy-making. In this manner, we anticipate a [[direct democracy]] where members themselves would vote on policy issues, pass laws, etc. The technology behind the ratings system would make direct democracy a practical and efficient mechanism for public participation. The CRS might also establish organizations and recognize experts who can help provide policy information in a public manner. Such organizations might then form the implementation arm for substantive policy decisions.

There are other important aspects of the ratings system. The software and its algorithms will be open to all. Users and communities will be able to tweak the settings of current algorithms or come up with new ones of their own (eg for aggregation). The software will include features for [[debate]], an important part of investigating the truth of contentious issues and making policy in a direct democracy. It will also include a robust package of socio-economic simulation tools, based on engineering simulation. These would allow communities to study policy options in detail from a rational, unbiased perspective. Through the software, ordinary people will have the tools to understand the truth and select an optimized choice from many options.

We thus see the formation of a “ratings-based society” (RBS) which can transform our current atmosphere of misinformation and dysfunctional governance. It is hoped that doing so will allow all members to benefit from a new [[consensual reality]] where high quality information is the norm and which complements rational decision-making.

Community and libertarianism

2024-10-09T19:26:54Z

Lem:

{{Main|Community}}

One aspect of our culture is a strong emphasis on individual agency and rights. We may not all be formally libertarians but a [[Philosophy of John Rawls#Libertarianism|libertarian ethos]] pervades our views and, as noted above, is strongly connected to the [[Cryptocurrency|crypto community]] where the idea of a [[ratings system]] might find its first adopters.

There is inevitably a conflict between a [[Ratings system|ratings system]], [[Community|communities]], and libertarianism. There is further conflict when we add concerns about [[Privacy, identity, and fraud in the ratings system|privacy]], which I would argue is an important component of libertarianism. After all, keeping the state out of our affairs is much easier when it doesn’t <i>know</i> about our affairs.

But we have defined our communities, and any ratings system they might have, as voluntary in nature. Thus the individual, the libertarian if you will, can choose the [[community]] and ratings system they wish to join. Once in a community, they have continuing influence over its ratings system. In some ways we can view the ratings system as the glue that makes a libertarian society work since it can be tuned to be as obtrusive or unobtrusive as its members want. The ratings system can also be as individualistic or as communitarian as members choose. And members can, of course, leave the community and join another one whenever they want.

Is this enough? Have we reconciled the libertarian with the community? Maybe. We will explore this a little further as we go along. Libertarians will certainly be attracted to joining communities of choice rather than the one they were born in, for which they had no choice and have almost no influence over. [[wikipedia:Nation state|Nation-states]] are not, to say the least, libertarian societies. But communities with ratings systems where everyone is judged must seem like quite an imposition on the libertarian mind.

Instead of answering this directly let’s point to some of the advantages of libertarian philosophy in any community. The greatest one, it would appear, is that in a conflict between flawed social obligation vs. personal morality, the libertarian would favor the latter. Libertarians don’t commit [[wikipedia:Genocide|genocide]] or knowingly design faulty airplanes. We can expect their sense of personal morality and agency to assert itself in cultures that try to promote questionable behavior.

Healthy communities require dissent and make progress as a result of it. [[wikipedia:Slavery|Slavery]] existed for thousands of years before dissenters realized that it was wrong and started campaigning to get rid of it. But it took a long time to accomplish and many vested interests had to be defeated in order to do it. Today we are experiencing a large scale evolution to a non-majority white Christian society and many people are against it. We are also changing our views on [[wikipedia:Greenhouse gas emissions|carbon emissions]] for [[Environmental policy in communities|environmental]] reasons but, again, there will be plenty of opposition and it will take a long time.

It would help if we had a ratings system that could produce required cultural changes faster. By [[Debate|giving voice to dissenting views]] and simply having a quicker feedback loop, we can reduce the time needed for social change. It would also help to have small communities that can pioneer ideas and see if they work before they are adopted more generally. In this way, both the community and the dissenter are an active part of change.

Avoiding feedback

2024-09-25T15:43:44Z

Lem:

{{Main|Technical overview of the ratings system}}

<span id="methods-of-preventing-feedback"></span>
= Methods of preventing feedback =

Say we have a fully-connected [[trust]] network that looks like:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
alice -> bob [dir="both"];
alice -> carol [dir="both"];
carol -> bob [dir="both"];

}
</kroki>
And ''alice'' wants ''bob''’s [[Computed opinion|computed opinion]]. A careless implementation could run into an infinite loop, where ''alice'' asks ''bob'' for his computed [[opinion]], who asks ''carol'' for her computed opinion, who asks ''alice'' for her computed opinion, who asks ''bob''…

This also leads to the fact that ''alice'' can get ''bob'' and ''carol''’s opinion twice – once directly, once indirectly through the other. This is acceptable, and probably desirable. If we add some [[Trust factor|trust factors]] to this network like so:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
alice -> bob [label="trust=0.8",dir="both"];
alice -> carol [label="trust=0.2",dir="both"];
carol -> bob [label="trust=0.8",dir="both"];

}
</kroki>
We can see that ''alice'' doesn’t [[Trust|trust]] ''carol'' much, so she will assign a low weight ''carol''’s direct opinions. But when she asks ''bob'' for his opinion, ''bob'' rates ''carol'' highly so his computed opinion will include ''carol''’s opinion, and ''carol''’s influence on ''alice''’s computed opinion will probably be significantly higher through ''bob'' than when presented directly to ''alice''. But we can say that ''alice'' has assigned high trust in ''bob'' because ''bob''’s computed opinions are usually right, regardless of how he comes up with them. She doesn’t know that he trusts ''carol'', and doesn’t need to. So getting ''carol''’s opinion through ''bob'' is probably a good thing, as her low direct trust in ''carol'' may be a mistake. Or maybe this is a situation where we have different trust factors for different domains; where ''alice'' is asking a car question and ''bob'' knows that ''carol'' is a mechanic so has a high trust factor on automotive [[Predicate|predicate]]s, but ''alice'' just knows her as someone with a reputation for being prone to [[wikipedia:Groupthink|groupthink]] on cultrue-war issues.

<span id="depth-limit"></span>
== Depth Limit ==

A simple way to avoid [[wikipedia:Recursion|infinite recursion]] is to put in a depth limit. When ''alice'' asks ''bob'' and ''carol'' for their computed opinions, she will set a depth limit of, say, <math display="inline">depth = 4</math>. Then, when ''bob'' gets the request, he would ask the nodes he trusts for their computed opinions, but pass <math display="inline">depth - 1</math> to ''alice'' and ''carol'', and so on. Eventually, if you are asked for a computation with <math display="inline">depth = 0</math>, you just return your own personal opinion, or <code>null</code> if you don’t have one. The initial depth limit would need to be set to be roughly the depth of an acyclic version of your network so there aren’t any nodes you can’t reach because they’re too far away.

Pros:

* simple to implement
* doesn’t require any node identifiers
* low network overhead (1 byte?)/low computing overhead
* results are stable – if you ask the same question on the same network multiple times, you’ll get the same answer.

Cons:

* peoples opinions are counted multiple times. my gut feeling is people near you tend to get relatively more weight than those farther away from you.
* you have to balance your initial value of <math display="inline">depth</math> somewhere between cutting off distant sources of information, and over-valuing nearby information

Notes:

* The ideal value for the three-person fully-connected network would be <math display="inline">depth = 1</math>, and even that may have ''alice'' getting her own personal opinion fed back to her unless the algorithm has a way for alice to tell bob not to turn around and ask her right back, which would be a step towards a node identifier.
* For [[Privacy|privacy]], you may want to stop at <math display="inline">depth = 1</math> in the algorithm as described above because allowing <math display="inline">depth = 0</math> requests directly exposes your personal opinion)
* My ten-seconds-of-effort google search turned up these statistics on [[Social network|social networks]]: https://miro.medium.com/v2/resize:fit:4800/format:webp/1*rkITSNe7ngMh8vmQexayQw.png The highlight being that for twitter, the average path length was 2.6 and the diameter was 18. Other services had somewhat higher average path lengths and somewhat lower diameters, but the difference was still at least a factor of 3.

<span id="unique-identifier-for-each-inquiry"></span>
== Unique identifier for each inquiry ==

This has a few variants. The property they all have in common is that when ''alice'' starts trying to figure out her computed opinion, she assigns a [[wikipedia:Universally unique identifier|unique identifier]] to the computation, and passes that identifier to ''bob'' and ''carol'' with her requests. If a node gets a request with an identifier it has never seen before, it does a full computation. If it gets a request with an identifier it has seen before, it does something less. That could be:

* returning <code>null</code>, effectively saying “I have no opinion on this, because my opinion has already been accounted for in this calculation”
* returning the same (cached) result they returned the first when they did the full computation
* returning their own personal opinion

Pros:

* low network overhead (16 bytes). low computational overhead
* doesn’t require node identifiers
* doesn’t loop back and count someone’s opinions multiple times

Cons:

* unstable. results change depending on which order nodes receive requests
* some variants wouldn’t allow someone’s opinion to count multiple times, which we convinced ourselves was desirable

<span id="passing-the-path"></span>
== Passing the path ==

In this method, each time you send a request to a new node, you include the list of all upstream nodes, so it knows not to loop back. When a node receives a request and all of the nodes they trust are listed in the request’s upstream nodes, it will just return its personal opinion, or null if it has none.

<kroki lang="seqdiag">
seqdiag {
alice -> bob [label = "request(ignore=alice)"];
bob -> carol [label = "request(ignore=alice,bob)"];
bob <-- carol [label = "carol's personal opinion"];
alice <-- bob [label = "bob's computed opinion"];
alice -> carol [label = "request(ignore=alice)"];
carol -> bob [label = "request(ignore=alice,carol)"];
carol <-- bob [label = "bob's personal opinion"];
alice <-- carol [label = "carol's computed opinion"];
}
</kroki>
Pros:

* stable
* only counts opinions twice when we want it to
* doesn’t double-count anyone or loop back

Cons:

* requires node identifiers
* higher network overhead for the list of node identifiers
* list of upstream nodes leaks a lot of information

<span id="privacy-preserving-variant"></span>
=== Privacy-preserving variant ===

To make the above more palatable, we can change it as follows:

* Every time a node makes a request
** they create a random [[wikipedia:Universally unique identifier|UUID]] to identify themselves for this computation. They store it in a short-term cache so they’ll know it if they see it again.
** they also generate a number of other decoy UUIDs that help obscure the trust paths. Maybe between 3 and 10 decoys per request.
** then they send their request with both the real and decoy UUIDs (sorted lexicographically, say)
* When a node receives a request, it checks to see if any of the UUIDs in the request are in its short-term cache of ids that represent itself.
** if so, it returns <code>null</code>
** if not, it computes its opinion normally. When it has to send requests to other nodes for, it does the same thing the originating node did: create and cache a real UUID, create several decoys, and append them to the list of UUIDs in the request it’s processing

Pros:

* makes it much harder to figure out who is whom

Cons:

* makes requests much larger on the network (figure what, maybe another 512 bytes on average?)
* also increases the number of requests (a full new set of leaf nodes). In the non-privacy version, if ''carol'' was processing <code>request(ignore=alice,bob)</code>, she knew not to contact ''alice''. In the privacy variant, she needs to contact ''alice'' because she doesn’t know if any of the UUIDs represent alice.

Attenuation in trust networks

2024-09-25T15:25:04Z

Lem:

{{Main|Technical overview of the ratings system}}

<h2>Review of Cycling</h2>

To summarize [[Effect of cycling in trust networks|cycling]] briefly, if [[Trust]]=1.0 for all nodes, then cycling will rapidly lead to a confidence of 1 (100%) for a given question (if probabilities for most of the nodes are above 50%). However, if [[Trust|Trust]] < 1.0 then the confidence will [[wikipedia:Asymptote|asymptotically]] approach some limit below 1. The question was asked, why? Why the asymptote? I hadn’t, and still really haven’t, done the math to answer this (See APPENDIX below for an attempt) so I offered up the following qualitative explanation:

# The confidence goes to 100% in a cycling network with Trust=1 (as long as our probabilities are above 50%). If Trust=0, the confidence is simply the confidence of the head node, which trusts itself. We would think that if Trust is between 0 and 1 that the confidence would also go to 100% if enough cycling were to take place, just like the case when Trust=1.0. But this would imply a sharp discontinuity between Trust=0 and everything else. Usually, when given a choice between continuous and discontinuous, you should pick continuous. Nature just seems to work that way, at least from a human-scale point of view. It is more reasonable that a continuous variation exists between Trust=0 and Trust=1.
# The asymptote is the result of two forces fighting each other: one being the multiple counting of the same nodes over and over (cycling) and the other being the attenuation of the trust as the nodes get further away from the top-most node. When trust=1, no attenuation occurs so the full effect of multiple counting takes over. When Trust < 1, nodes further out exert less and less influence on the final answer because they have to go through multiple trust layers and, hence, become attenuated. A node with Trust=0 has no influence on the final answer which is almost the same as the influence a distant node has. Since its influence is almost nothing we could view that as being effectively the same as a trust of zero. Hence the term “trust attenuation”. As nodes get farther away from the top node their “effective” trust declines more and more.

<h2>Effect of Trust Attenuation</h2>

Let’s note first that this has nothing to do with cycling. Even in networks where no cycling takes place, if the trust of multiple nodes is below 1, attenuation will cause the last node to have very little influence on the final answer.

Let’s look at the implications of this. If a node that’s many levels down has low trust (due to attenuation) but is the only node that has any informed knowledge of a subject, how do we factor that in without completely washing out the information it provides? It seems a shame to discount the only node that knows anything because of attenuation.

Cases of this would be any question that is difficult to answer and will require an extensive dive into the network to find an authoritative source. In this situation most of the network doesn’t know anything but there’s one guy who does, somewhere deep down. Examples would be serious questions we might really pose: Is Bibi, from Israel, who just contacted me for a business deal, a good guy? Is it better to get a liver transplant in India or the US? No one in my immediate network knows the answer to these questions so it’s going to take a few levels to get there. The problem is that the network path to anyone who knows is long and has trust factors built in to every node which will attenuate the result as we go along. Take a look at the following example:

[[File:354dda5b3fb3b7a7d8220df8a448cb44_image.png|image]]

Here all nodes have a trust of 70% for their succeeding node and Nodes 0-7 have P=50%, meaning they don’t know anything and their contribution to the overall answer is nil. The only node with any real knowledge is 8, which we represent as P=100%. If we use [https://gitlab.syncad.com/peerverity/trust-model-playground/-/wikis/uploads/6484e59605fd81085eaa127bd98bb9ed/sapienza_trusttree.py sapienza_trusttree.py] to compute this case we arrive at Ptot = 53% as shown. This is not a very good result, just slightly better than random.

Let’s contrast this with a network composed of two nodes, where there is a direct link between 0 and 8:

[[File:6945896d20cb2f044417fa6bb1420df0_image.png|image]]

Here, by eliminating all the nodes that contribute nothing to the answer except attenuation, we obtain the far more reasonable result of 85%.

<h2>Ideas for Dealing with Trust Attenuation</h2>

To handle this we could allow the user to assign trust factors of 1 to nodes that don’t know the answer (ie ones that have 50% confidence). Trust seems like a less likely issue in cases where people simply don’t know but are willing to pass along information they’ve received. We could even automate this – the node could just pass the answer from their child up without doing anything if they, in fact, have no knowledge and, presumably, no stake in the outcome:

[[File:16f4acbb814cbb110a2afa9eff0f51e0_image.png|image]]

Here we’ve cleanly passed up Node 8’s confidence to the top node without that pesky trust attenuation problem getting in the way. But we’ve also introduced another obvious problem in that we are simply taking Node 8’s word for it on its 100% confidence level. Clearly we need to reality-check this with a trust factor but we’ve just assigned them all to 1 to avoid attenuation.

We might call this “the biased source problem”. Frequently the people who know anything about an obscure subject are biased because they make their business that subject and have a stake in the outcome. Our best source on Bibi is his golfing buddy at the chamber of commerce. The best source on liver transplants in India is an Indian hepatologist who’d like to do the transplant himself. The results are unrealistically confident answers.

So a variant on this idea is to allow the trust factor of the next-to-last node to stand but ignore the rest by assigning them to 1. This would preserve the trust information of the guy who knows the guy who knows. Here, Node 7’s trust for Node 8 stands (at 70%) but the rest of the nodes are set to 100%:

[[File:6a34ceceeac7399d5f077c25fabefc7b_image.png|image]]

Another option would be to allow Node 0 to assign a single trust factor to the whole network, perhaps based on how far away the source node (Node 8) is.

In any event, we are short-circuiting our way to the nodes that really matter, the one that knows the guy who knows, and the guy who knows. I think our system should allow this in some form or other. The user would then get an unfiltered [[Opinion|opinion]] on the question at hand.

<h2>APPENDIX -- Some math on the asymptote</h2>
This is a continuation of the first paragraph of this post. Please read that before trying to understand what's going on here. Also, this is strictly a nerd's eye view of a (very) specific problem. Read it if you're having insomnia.

I don’t have an elegant proof because it’s a crazy amount of algebra, but we can go over some thoughts:

Let’s take a look at the following network:

0 – 1 – 2 – 3

where P=0.6 for all nodes and T=0.7 for all nodes. To roll up the confidence of Node 0 for any given [[predicate]] question we can start by computing the confidence of Node 2 after taking into account Node 3. This is just the [https://ceur-ws.org/Vol-1664/w9.pdf Bayes eqn. as modified by Sapienza’s trust factor]:

P2 = 0.6*(0.5 + (0.6-0.5)''T) / (0.6''(0.5 + (0.6-0.5)''T) + 0.4''(0.5+(0.4-0.5)*T)) = 0.6654

We continue by calculating the confidence of Node 1 using the above, just calculated, confidence of Node 2:

P1 = 0.6*(0.5 + (0.6654-0.5)''T) / (0.6''(0.5 + (0.6654-0.5)''T) + 0.4''(0.5+(1-0.6654-0.5)*T)) = 0.7062

and so on until we’ve calculated Node 0. If we just substitute T=0.7 into this, we can derive a [[wikipedia:Recurrence relation|recurrence relation]] of the form:

Pnew = (.09 + .42''P ) / (.43 + .14''P)

That is, the Probability of the next level (new) is a function of the probability of the previous level. The other numbers are just constants associated with the trust (0.7 to keep things simple) and the Pnom (0.5 for a [[Predicate|predicate]] question). P will vary from 0-1, so we can construct a table of Pnew as a function of P:

{|
! When T=0.7
!
|-
| P
| Pnew
|-
| 0
| 0.209302326
|-
| 0.1
| 0.297297297
|-
| 0.2
| 0.379912664
|-
| 0.3
| 0.457627119
|-
| 0.4
| 0.530864198
|-
| 0.5
| 0.6
|-
| 0.6
| 0.66536965
|-
| 0.7
| 0.727272727
|-
| 0.8
| 0.78597786
|-
| 0.9
| 0.841726619
|-
| 1
| 0.894736842
|}

We see here that P equal to and below 0.7 results in Pnew > P. Numbers equal to and above 0.8 results in Pnew < P. Therefore, no matter where we start, the recurrence relation will converge somewhere between 0.7-0.8. If we run sapienza_trusttree.py for many nodes (Level = 15) we will obtain P=0.7668. We can also just say Pnew = P in the equation above and solve to get the same result.

When Trust=1, we get the following recurrence relation:

Pnew = 0.6''P / (0.2''P + 0.4)

Which leads to a table like this:

{|
! When T=1
!
|-
| P
| Pnew
|-
| 0
| 0
|-
| 0.1
| 0.142857143
|-
| 0.2
| 0.272727273
|-
| 0.3
| 0.391304348
|-
| 0.4
| 0.5
|-
| 0.5
| 0.6
|-
| 0.6
| 0.692307692
|-
| 0.7
| 0.777777778
|-
| 0.8
| 0.857142857
|-
| 0.9
| 0.931034483
|-
| 1
| 1
|-
| 1.1
| 1.064516129
|-
| 1.2
| 1.125
|-
| 1.3
| 1.181818182
|}

This shows that all values of P above 0 but below 1 increase the resulting Pnew. The asymptote here, of course, is 1 as we already know.

The only difference between these recurrence relations is Trust. Therefore it is trust below 1 which generates a recurrence relation that leads to an asymptote below 1.

Based on this, we can write a general equation as a function of T and P:

Pnew = (0.3 + 0.6''T''(P-0.5)) / ( (0.3 + 0.6''T''(P-0.5)) + 0.2 + 0.4*(0.5-P)*T )

If we set Pnew = P, we obtain, after some algebra,

P**2 + P*(2.5/T - 3.5) + (1.5 - 1.5/T) = 0

The solution of this equation defines the asymptote (Pasymp), ie the highest probability we can achieve given the trust. It can be solved numerically or using the quadratic formula:

P=( (3.5-2.5/T) +- SQRT((2.5/T - 3.5)*''2 - 4''(1.5-1.5/T)) ) / 2

{|
! T
! Pasymp
|-
| 0.01
| 0.60096891
|-
| 0.1
| 0.610567768
|-
| 0.2
| 0.623475383
|-
| 0.3
| 0.639520137
|-
| 0.4
| 0.659852575
|-
| 0.5
| 0.686140662
|-
| 0.6
| 0.72075922
|-
| 0.7
| 0.766864466
|-
| 0.8
| 0.827934423
|-
| 0.9
| 0.906150469
|-
| 1
| 1
|}

[[File:96bd3aa31a128c2ea22634e90459d987_image.png|image]]

This is not quite a proof but it gives us a more rigorous picture of what’s going on. When trust falls to 0, the confidence of the head node, 0.6, is as good as it gets. If Trust is 1, we will eventually reach P=1 after enough nodes have been factored in. For all T in between, we have a continuously varying degree of P. This makes sense and confirms our initial intuition that a continuous variation in P will result from varying T.

Allowing for more than predicate questions in the trust-weighted histogram (TWH) algorithm

2024-09-25T14:55:43Z

Lem:

{{Main|[[Trust-weighted histogram|Trust-weighted histograms]]}}

The <code>trust_weighted_histogram</code> (TWH) algorithm was discussed last time but only handles [[predicate]] questions. Indeed, it only handles single valued probabilities under the assumption that the other [[Probability|probability]] in a [[Predicate|predicate]] [[wikipedia:Probability distribution|distribution]] is (1-P). Although the examples that exercise it are actually provided as a two-valued distribution [P, 1-P], the algorithm ignores the 2nd value.

This was corrected using the <code>trust_weighted_histogram_sets</code> algorithm where each “set” corresponds to a different probability and results in a different [[Histogram|histogram]]. In essence the algorithm does the same calculation as before for multiple probabilities and presents its results as a set of histograms, one for each probability.

The setup and example are exactly the same as in the [[Internal:FromGitlab/Notes_on_using_Lem's_algorithm_interface|previous discussion]] and [[Internal:FromGitlab/Dan's_proposal_for_trust_weighted_histograms|Eric’s description of the TWH algorithm]]:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
1 [label="1, P=20%"]
2 [label="2, P=30%"]
3 [label="3, P=40%"]
4 [label="4, P=45%"]
5 [label="5, P=60%"]
6 [label="6, P=90%"]
7 [label="7, P=55%"]
8 [label="8, P=65%"]
9 [label="9, P=70%"]
10 [label="10, P=80%"]
1 -> 2 [label="T=0.9"];
1 -> 3 [label="T=0.9"];
1 -> 4 [label="T=0.9"];
2 -> 5 [label="T=0.9"];
2 -> 6 [label="T=0.9"];
3 -> 7 [label="T=0.9"];
3 -> 8 [label="T=0.9"];
4 -> 9 [label="T=0.9"];
4 -> 10 [label="T=0.9"];
}
</kroki>
The output for both the actual and intermediate results has a 2nd set of values representing the 2nd probability in the predicate distribution:

<pre>[[Trust|trust]]_weighted_histogram_sets test -- TEST1:
Trust Weighted Histogram output256 = [[0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.9, 0.0, 0.0, 0.9], [0.0, 0.9, 0.0, 0.0, 0.9, 0.0, 0.0, 1.0, 0.0, 0.0]]
Trust Weighted Histogram output378 = [[0.0, 0.0, 0.0, 0.0, 1.0, 0.9, 0.9, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.9, 0.9, 0.0, 1.0, 0.0, 0.0, 0.0]]
Trust Weighted Histogram output4910 = [[0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.9, 0.9, 0.0], [0.0, 0.0, 0.9, 0.9, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0]]
Trust Weighted Histogram Overall output1234 = [[0.0, 0.0, 0.5555555555555556, 0.5, 1.0, 0.45, 0.9, 0.45, 0.45, 0.45], [0.0, 0.5, 0.5, 1.0, 1.0, 0.5555555555555556, 0.5555555555555556, 0.5555555555555556, 0.6172839506172839, 0.0]]</pre>
This is, of course, still a predicate question but any number of probabilities can now be handled. Another test (TEST2) of this algorithm is provided in the [https://gitlab.syncad.com/peerverity/trust-model-playground/-/snippets/148 snippet ]which adds a 3rd probability:

<pre>trust_weighted_histogram_sets test -- TEST2:
Add a 3rd probability
Trust Weighted Histogram output256 = [[0.0, 0.0, 1.0, 0.0, 0.0, 0.9, 0.0, 0.0, 0.9, 0.0], [0.0, 0.9, 0.0, 0.0, 0.9, 0.0, 0.0, 1.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]
Trust Weighted Histogram output378 = [[0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.4736842105263158, 0.0, 0.0], [0.0, 0.0, 0.0, 0.9, 0.0, 0.9, 1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]
Trust Weighted Histogram output4910 = [[0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.9, 0.9, 0.0, 0.0], [0.0, 0.0, 0.9, 0.9, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]
Trust Weighted Histogram Overall output1234 = [[0.0, 0.5555555555555556, 0.5, 1.0, 0.0, 0.45, 0.45, 0.6868421052631579, 0.45, 0.0], [0.0, 0.4736842105263158, 0.4736842105263158, 0.9473684210526316, 0.4736842105263158, 1.0, 0.5263157894736842, 0.5263157894736842, 0.5847953216374269, 0.0], [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]</pre>
The functions for this algorithm are denoted by the suffix <code>_sets</code>. They are largely the same as the functions for [[Internal:FromGitlab/Dan's_proposal_for_trust_weighted_histograms|TWH ]] algorithm discussed previously except modified to handle one extra dimension. This is mainly seen in any list/array operation which had to be modified to be a “list of lists” rather than a simple list.

For example the functions GetHForLeafNode and GetHForLeafNode_sets differ in exactly this manner:

<syntaxhighlight lang="python">def GetHForLeafNode(opinion, bins):
P = opinion.pdf_points
H = []
for binx in bins:
if(P[0] >= binx[0] and P[0] < binx[1]):
H.append(1.0)
else:
H.append(0.0)
return H</syntaxhighlight>
<syntaxhighlight lang="python">def GetHForLeafNode_sets(opinion, bins):
P = opinion.pdf_points
H = []
Hlist = []
for p in P:
for binx in bins:
if(p >= binx[0] and p < binx[1]):
H.append(1.0)
else:
H.append(0.0)
Hlist.append(H)
H = []
return Hlist</syntaxhighlight>
This function takes the [[Opinion|opinion]] of a leaf node and determines which probability bin it corresponds to. It places a 1 in the proper bin and 0 in the other bins, creating a histogram. The first function ignores the fact that P is multi-valued and simply uses the first value, P[0]. The second iterates over all the P values in an outer loop. An <code>Hlist</code> is used to store each <code>H</code> so created. Many of the other functions in the algorithm were modified to create an outer loop along these lines.

No modifications to the <code>algorithms.py</code> interface were required although it should be noted that the output [[opinion]] is now a list of lists (as you can see above) rather than a simple list. The pdf_points field in OpinionData is specified as a <code>list[float]</code> but apparently it doesn’t care that the actual data type is a list of lists of floats.

<span id="trustprobabilitypopulation-graphs-algorithm"></span>

Argument evaluation and scoring

2024-09-25T13:53:49Z

Lem:

<h2>Some thoughts on evaluating [[Argument|argument]]s</h2>

In our last meeting Dan asked how we might evaluate an argument made by a respondent instead of simply relying on the given [[Probability|probability]] (as we've been doing so far). An argument, assuming it is made public, could then be evaluated by the questioner and others independently to find a more accurate probability. This opens up a new idea in our work, that of assessing the truth by evaluating the reasoning put forth in an [[Opinion|opinion]].

One idea for doing this starts with a simple model for argument construction. The argument consists of [[Supporting statement|supporting statement]]s which are tied together with [[Logic|logic]] to form a conclusion. The conclusion is the answer to the overall question being asked of the network. Each supporting statement and the [[logic]] can be evaluated independently to determine the extent to which the conclusion is true. The following diagram illustrates this:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="Answer/Conclusion, Pc"]
1 [label="Logic, Pl"]
2 [label="Support. Stmt. 1, Ps1"]
3 [label="Support. Stmt. 2, Ps2"]
4 [label="Support. Stmt. 3, Ps3"]
0 -> 1 [dir="back"];
1 -> 2 [dir="back"];
1 -> 3 [dir="back"];
1 -> 4 [dir="back"];
}
</kroki>

The probability of the supporting statements can be combined in a [[Bayes' theorem|Bayes]]ian manner. This is in keeping with the Bayesian idea of modifying prior probabilities given new evidence (ie supporting statements). These probabilities can be [[Trust|trust]]-modified as Sapienza proposed (https://ceur-ws.org/Vol-1664/w9.pdf) but since they are likely being assigned by the questioner, we will assume that [[trust]] is already built into them. Of more importance is the [[Relevance|relevance]] of the supporting statements. They can range from completely irrelevant to completely relevant. A completely relevant statement will take the full value of the probability it was originally assigned. A completely irrelevant statement would reduce the probability to 50%, where it will have no influence on the outcome. In that sense relevance functions in the same way trust does to modify the probability:

<math>
P_{mod} = P_{nom} + R(P - P_{nom})
</math>

where

<math>R</math> is relevance (0.0 - 1.0)

<math>P_{mod}</math> is relevance-modified probability

<math>P_{nom}</math> is the nominal probability (=0.5 for a [[Predicate|predicate]] question)

<math>P</math> is the unmodified probability

After the relevance-modification, each supporting statement is combined in the usual manner via Bayes. For the first two statements,

<math>
P_{comb1,2} = {P_{s1}P_{s2}\over {P_{s1}P_{s2} + (1-P_{s1})(1-P_{s2})}}
</math>

and so on for each additional statement. Here it is to be understood that <math>P_{s1}</math>, etc. is the value <i>after</i> the relevance modification.

Logic will also have a probability assigned to it to represent its quality. A fully illogical argument would receive a 0, which when combined via Bayes with the supporting statements would render the probability of the entire argument 0. This makes sense because a completely illogical argument, regardless of the strength of its supporting statements, destroys itself. A fully logical argument, however, will not receive a 1 but rather a 0.5. When combined with the supporting statements a 1 would render the final probability a 1, which is not reasonable. A 0.5, however, would do nothing and the final probability would be the combined probability of the supporting statements. Thus we assume that perfect logic is neutral and less than perfect logic reduces the combined probability of the statements. Again, this seems reasonable. We expect, by default, logical arguments which then rest on the strength of their supporting statements. If we notice flaws in the logic we discount the strength of the argument accordingly.

Let's try an example with these ideas:

Question: Are humans causing frog populations to decline?<br>

Answer / Conclusion: Yes, mankind is causing a fall in frog populations.<br>

Logic: Mankind is causing the fall of frog populations if we can show that frog populations are decreasing over time and can show a human behavior that causes the decline.<br>

Supporting Statements:<br>

# [https://en.wikipedia.org/wiki/Frog#:~:text=Frog%20populations%20have%20declined%20significantly%20since%20the%201950s. Frog populations have declined since the 1950's.]
# My wife complained that she doesn't see frogs anymore.
# [https://wwf.panda.org/discover/our_focus/freshwater_practice/freshwater_biodiversity_222/ Scientists say] that the loss of freshwater habitats has affected frog populations.

We start by judging the quality of the supporting statements. 1 seems like a well substantiated statement (a high P) but is not completely relevant because it only hints at human involvement. 2 is completely true but mostly irrelevant. 3 is a contributor but seems less substantiated than 1 and contains no human cause. We proceed by assigning probability and relevance values:

<math>P_{s1}=0.9</math>
<math>R_{s1}=0.7</math>
<math>P_{s1mod} = 0.5 + 0.7(0.9 - 0.5) = 0.78</math>

<math>P_{s2}=1.0</math>
<math>R_{s2}=0.0</math>
<math>P_{s2mod} = 0.5 + 0.0(1.0 - 0.5) = 0.5</math>

<math>P_{s3}=0.75</math>
<math>R_{s3}=0.5</math>
<math>P_{s3mod} = 0.5 + 0.5(0.75 - 0.5) = 0.625</math>

Since s2 won't count in the Bayesian calculation we can ignore it and:

<math>P_{comb,s} = {(0.78)(0.625) \over {0.78(0.625)+0.22(0.375)}} = 0.855</math>

The logic/conclusion in this case is reasonably strong so we will assign it a high value, say <math>P_l = 0.45</math> (remember, out of 0.5). It could be improved by observing that the word "behavior" is too general and should be replaced by, say, "policy choice" (ie urban growth into ecologically important wetlands). We note here that logic is more than just the mathematical construction of an argument. Since we are speaking a human language, logic might also be flawed because it uses imprecise wording.

Putting <math>P_{comb,s}</math> together with <math>P_l</math> using Bayes we obtain a concluding probability:

<math>P_c = {0.855(0.45) \over {{0.855(0.45) + 0.145(0.55)}}} = 0.82</math>

One potential pitfall of this model is that repetitive supporting statements of high probability will quickly render a combined probability near 1.0. As we've seen in the past, this is simply the result of the Bayes equation. The user would need to watch for attempts like these to distort the answer by removing repetitive statements or making them irrelevant.

<h2>Scoring of individual arguments</h2>

Arguments can be scored on [[Veracity|veracity]], impact, relevance, clarity, and informal quality (lack of fallacies):

- Veracity is how true the argument is based on source information. Source information itself will be scored: <math>V</math>

- Impact & Relevance is how deeply the argument affects the main contention of the [[debate]] (or the argument immediately above): <math>R</math>

- Clarity is how understandable the argument is: <math>C</math>

- Informal quality (lack of fallacies) is whether the argument commits any logical fallacies of its own. A list of informal fallacies (and formal ones) will be provided to help users select appropriately: <math>F</math>

Since Impact and Relevance are closely related concepts we will merge these into one, Relevance. The simplest method for combining these is to average them, or weighted average them:

<math>
S = w_vV + w_rR + w_cC + w_fF
</math>

where

<math>w_x</math> is a weighting for category X (eg Veracity, Relevance, etc)

<math>w_v + w_r + w_c + w_f = 1</math>

This seems reasonable and if we believe that certain criteria should weigh more (such as Veracity) we can easily make the weighting factors reflect this. However, intuitively it seems that a category such as Veracity should not only weigh more but have the power to take down the whole argument. After all, if the argument is a straightforward lie, it should receive a score of zero, regardless of its other attributes (such as relevance, clarity, etc):

<i>Who is the best choice for President? X is the best choice because he will land a person on Mars in his first year.</i>

This argument is a lie and although it is clear, has no evident fallacies, and is relevant to the question at hand, it should be thrown out.

The same can be said of Relevance. A completely irrelevant argument should also have the power to render the whole argument moot:

<i>Who is the best choice for President? X is the best choice because he likes pizza and so do I.</i>

With this in mind, we can propose the following equation, which we will dub the "VRFC equation":

<math>
S = VR(w_fF + w_cC)
</math>

Where

<math>S</math> = Score for the argument which varies from 0-1

<math>w_f</math> = weighting factor for Fallacies.

<math>w_c</math> = weighting factor for Clarity.

<math>w_f + w_c = 1</math>

and each of the constituent variables (<math>V, R, F, C</math>) has a range 0-1.

In this equation either Veracity or Relevance have the power to nullify the entire argument. Similarly a combination of Fallacies and lack of Clarity can do the same. However, a fallacious argument alone seems like it could still have merit, as would an argument whose only flaw was lack of clarity:

<i>We should support Czechoslovakia because if the Nazi's prevail they will conquer the world.</i>

This argument commits the slippery slope [[Fallacy|fallacy]] but is not entirely invalid. Similarly an unclear argument can still manage to make a point:

<i>We should support Europe because first the Sudetenland, then the Czechs, and soon enough it's over when all the Brits had to do was get rid of that weakling sooner.</i>

It would seem that a fallacious argument should weigh more than an unclear one. Proposed weights might be:

<math>w_f = 0.7, w_c = 0.3</math>

<h3>Rolling up the score of argument trees</h3>

The equation above applies to a single argument but, as we've seen, most arguments have sub-arguments below, sub-sub-arguments, and so forth. They are really trees in which each individual argument can be scored separately.

Here we develop a proposed equation for rolling up the score for an argument based on its own score and that of its sub-arguments. In doing so we emphasize that any argument can stand on its own and be scored in the absence of sub-arguments. This creates an interesting dynamic. The sub-argument may bolster or detract from the parent argument but the extent to which it does should be limited.

Furthermore, once the sub-argument becomes weaker than a certain threshold, it should stop influencing the parent argument altogether. Here, we will set this threshold at 0.5. Thus only Pro sub-arguments that score 0.5 or better will have any influence on the parent argument. For Con sub-arguments we will use the same threshold but first modify the sub-argument score by <math>1-S</math>. Thus a strong Con sub-argument, scoring say 0.9, would enter the calculation with a score of 0.1. The result is a range of scores 0-1 of which 0-0.5 is Con and 0.5-1 is Pro. Scores of exactly 0.5 are neutral.

Let's consider the case with one argument and one pro sub-argument and one Con sub-argument.

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="Argument, s = 0.9"]
1 [label="Pro sub-argument, xp = 0.7"]
2 [label="Con sub-argument, xc = 0.7"]
0 -> 1 [dir="both"];
0 -> 2 [dir="both"];
}
</kroki>

In this case the argument's score is 0.9, and both the Pro/Con sub-argument score is 0.7. These numbers would normally be arrived at by using the VRFC eqn above, but we will just assume them for now. The first sub-argument, in this case, bolsters the argument because it is a Pro argument and has a score (0.7) greater than 0.5. The second sub-argument, with the same score, detracts from the argument because it is on the Con side. We emphasize that if these scores were at or below 0.5 they would have no effect on the argument.

The general equation governing this situation is as follows:

For <math>s > 0.5</math> and <math>x > 0.5</math>

<math>
s_{mod} = 2(1-s)fx + s - (1-s)f
</math>

For <math>s > 0.5</math> and <math>x <= 0.5</math>

<math>
s_{mod} = s
</math>

For <math>s < 0.5</math> and </math>x > 0.5</math>

<math>
s_{mod} = 2sfx + s - sf
</math>

For <math>s < 0.5</math> and <math>x <= 0.5</math>

<math>
s_{mod} = s
</math>

where

<math>s</math> = score for parent argument

<math>x = x_p</math> = score for Pro arguments

<math>x = 1-x_c</math> = score for Con arguments

<math>f</math> = maximum possible fraction of increase possible, 0-1

The variable <math>f</math> is a user selected number between 0-1 and represents the extent to which the sub-argument score can be affected. For example, a sub-argument with x = 0.9, as in the diagram above, can be improved by 0.1 to a maximum of 1. Then <math>f</math> represents the fraction of 0.1 that we will allow for our improvement. If <math>f = 0.25</math>, for instance, then the maximum range around 0.9 that the sub-argument can affect is <math>(0.25)(0.1) = 0.025</math>. Thus the maximum score the argument can have is 0.925 and the minimum is 0.875.

For the argument above:

<math>s = 0.9</math>

<math>f = 0.25</math> User input

For the Pro sub-argument:

<math>x = x_p = 0.7</math>

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.9)(0.25)(0.7) + 0.9 - (1-0.9)(0.25) = 0.91</math>

For the Con sub-argument:

<math>x = (1-x_c) = (1-0.7) = 0.3</math>

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.9)(0.25)(0.3) + 0.9 - (1-0.9)(0.25) = 0.89</math>

We can see here that the Pro and Con sub-arguments exactly balance each other since they both have the same score.

The equation above is piecewise linear and can be visualized as follows:

![image](uploads/7f62b22b9eb19771503d654db29cab92/image.png)

One important property of this equation is that the stronger (or weaker) an argument becomes, it becomes harder for a sub-argument to change it. This is because the maximum allowed movement is <math>1-s</math> if <math>s > 0.5</math> or simply <math>s</math> if <math>s <= 0.5</math>. The idea behind this property is that very strong arguments should be harder to dislodge precisely because they have covered themselves well. A weaker argument, for instance one that fails to mention an obvious supporting fact, is in a position to be bolstered more by a sub-argument which mentions the fact. Similarly a very weak argument should be difficult to bolster. If the argument is a lie or irrelevant, for instance, there isn't much that can be done to rescue it.

This property has the further consequence that <math>s</math> cannot be changed by the sub-arguments if it is 1 or 0. A truly perfect argument, <math>s = 1</math>, cannot be weakened no matter how strong its Con sub-argument. Similarly a perfectly flawed argument, <math>s = 0</math> cannot be bolstered with any Pro sub-argument. We will discuss below a method to deal with the fact that, regardless of the quality of the argument, users may still vote to score arguments 1 or 0.

<h4>Population adjustments</h4>

The algorithm described above assumes a single vote for the argument and sub-arguments. In fact, this will rarely be the case, because multiple users will be voting on each. The effect of a sub-argument on its parent should be weighed by the population of users who voted for the sub-argument and parent argument.

Here we propose a simple modification factor, based on the ratio of users voting for each argument/sub-argument:

<math>s_{mod,pop} = (s_{mod} - s){p_s\over p} + s</math>

where

<math>s_{mod,pop}</math> is the population modified score

<math>s_{mod}</math> is the modified score without population modifications (see above)

<math>p_s</math> is the population voting for the sub-argument

<math>p</math> is the population voting for the parent argument

<math>s</math> is the original score of the parent argument

Usually we expect that sub-arguments will receive fewer votes than parent arguments, so <math>{{p_s\over p} <= 1}</math> in general. For the case when <math>p_s > p</math> we will force <math>{p_s\over p} = 1</math>. Therefore there is no danger that a sub-argument can overwhelm a parent argument by voting power alone. This is in keeping with our philosophy that sub-arguments can have at best a limited effect on parent arguments.

<h4>Example calculation</h4>

Let's do a problem with the following argument tree and <math>f=0.25</math>:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis Statement"]
1 [label="1, Pro argument, s = 0.9, p = 96"]
2 [label="2, Pro sub-argument, xp = 0.7, p = 55"]
3 [label="3, Pro sub-argument, xp = 0.8, p = 26"]
4 [label="4, Con sub-argument, xc = 0.6, p = 30"]
5 [label="5, Con sub-argument, xc = 0.7, p = 43"]
6 [label="6, Pro sub-argument, xp = 0.85, p = 19"]
7 [label="7, Con sub-argument, xc = 0.95, p = 28"]
8 [label="8, Con argument, ...."]
0 -> 1 [dir="both"];
1 -> 2 [dir="both"];
1 -> 5 [dir="both"];
2 -> 3 [dir="both"];
2 -> 4 [dir="both"];
5 -> 6 [dir="both"];
5 -> 7 [dir="both"];
0 -> 8 [dir="both"];
}
</kroki>

Our objective here is to roll up the score for the Pro side of this tree. The Con side would be calculated similarly and we will skip this for the sake of brevity. Note that <math>s</math> stands for the score and <math>p</math> is the population voting to produce that score. We start at the bottom, with the 2-3-4 portion of the tree, and for the sake of consistency with the above calculation we will recast <math>x_p</math> for 2 as <math>s</math> and label the population of the sub-arguments as <math>p_s</math>.

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
2 [label="2, Pro argument, s = 0.7, p = 55"]
3 [label="3, Pro sub-argument, xp = 0.8, ps = 26"]
4 [label="4, Con sub-argument, xc = 0.6, ps = 30"]
2 -> 3 [dir="both"];
2 -> 4 [dir="both"];
}
</kroki>

For the Pro sub-argument, we write:

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.7)(0.25)(0.8) + 0.7 - (1-0.7)(0.25) = 0.745</math>

We modify this by the respective populations:

<math>s_{mod,pop23} = (s_{mod} - s){p_s\over p} + s = (0.745 - 0.7){26\over 55} + 0.7 = 0.721</math>

For the Con sub-argument we first modify its score,

<math>x = 1 - x_c = 0.4</math>

and write

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.7)(0.25)(0.4) + 0.7 - (1-0.7)(0.25) = 0.685</math>

and modify by the respective population,

<math>s_{mod,pop24} = (s_{mod} - s){p_s\over p} + s = (0.685 - 0.7){30\over 55} + 0.7 = 0.692</math>

These two values of <math>s_{mod,pop}</math> can now be combined to create a new <math>s</math> for the Pro argument:

<math>s_{mod,tot} = (s_{mod,pop23} - s) + (s_{mod,pop24} - s) + s = (0.721 - 0.7) + (0.692 - 0.7) + 0.7 = 0.713</math>

We note here that the Pro argument got a little stronger as a result of its sub-arguments. The Pro sub-argument was substantially stronger than the Con sub-argument and, although fewer people voted for it, the population difference was not large.

For the Con sub-argument 5-6-7 we have the following situation:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
5 [label="5, Con argument, s = 0.7, p = 43"]
6 [label="6, Pro sub-argument, xp = 0.85, ps = 19"]
7 [label="7, Con sub-argument, xc = 0.95, ps = 28"]
5 -> 6 [dir="both"];
5 -> 7 [dir="both"];
}
</kroki>

Here, for the Pro sub-argument, we first modify its score since it is the opposite of its parent. It is as if the parent were a Pro argument and the child were a Con argument.

<math>x = 1 - x_p = 1 - 0.85 = 0.15</math>

We then proceed as usual with the calculation:

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.7)(0.25)(0.15) + 0.7 - (1-0.7)(0.25) = 0.6475</math>

<math>s_{mod,pop56} = (s_{mod} - s){p_s\over p} + s = (0.6475 - 0.7){19\over 43} + 0.7 = 0.677</math>

For the Con sub-argument <math>x = x_c = 0.95</math> since the parent argument is also Con:

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.7)(0.25)(0.95) + 0.7 - (1-0.7)(0.25) = 0.7675</math>

<math>s_{mod,pop57} = (s_{mod} - s){p_s\over p} + s = (0.7675 - 0.7){28\over 43} + 0.7 = 0.744</math>

We combine these two results in the same manner as above:

<math>s_{mod,tot} = (s_{mod,pop56} - s) + (s_{mod,pop57} - s) + s = (0.677 - 0.7) + (0.744 - 0.7) + 0.7 = 0.721</math>

With the bottom layer of the tree calculated, we have the following situation:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis Statement"]
1 [label="1, Pro argument, s = 0.9, p = 96"]
2 [label="2, Pro sub-argument, xp = 0.713, ps = 55"]
5 [label="5, Con sub-argument, xc = 0.721, ps = 43"]
8 [label="8, Con argument, ...."]
0 -> 1 [dir="both"];
1 -> 2 [dir="both"];
1 -> 5 [dir="both"];
0 -> 8 [dir="both"];
}
</kroki>

All that remains is to calculate the 1-2-5 portion, which is very similar to the 2-3-4 calculation performed above. Therefore will skip the details of this and simply report the results:

<math>s_{mod,pop12} = 0.906</math>

<math>s_{mod,pop15} = 0.895</math>

<math>s_{mod, tot} = 0.901</math>

We see here that the final result is not much different than the original <math>s = 0.9</math>. This is a result of the Pro sub-arguments essentially being cancelled by the Con sub-arguments. Such a result is to be expected in many cases.

In this example, we are skipping the Con side of the overall argument (node 8 in the tree above) because it would be exactly the same as what we have shown. If it had been calculated we would then combine the result for 1 and 8 to produce an overall score for the argument.

The calculations above can be performed with the [attached snippet](https://gitlab.syncad.com/[[Peer|peer]]verity/trust-model-playground/-/snippets/164). The user input portion of the snippet is set up for the calculation we did immediately above:

<syntaxhighlight lang="python">
#User input
side_parent = 'pro' #side, pro or con, that the parent argument is on
s = 0.9 #score for the parent argument
mf = 0.25 #max fraction that parent argument can be changed in terms of (1-s) or (s-0)
p = 96.0 #population voting for the parent argument
x_pro_arr = [0.713] #score for the pro children
x_con_arr = [0.721] #score for the con children
ps_pro_arr = [55.0] #population voting for each pro child sub-argument
ps_con_arr = [43.0] #pop voting for each con child sub-argument
mods_if1or0 = True #True if we want scores of 1 or 0 to be modified to near 1 or 0 (otherwise they can't be adjusted by this calculation)
</syntaxhighlight>

To note, the snippet contains arrays to handle any number of child arguments. These are combined in the same way we combined the single Pro and Con argument above.

Another variable, `mods_if1or0` controls whether we allow <math>s</math> to be modified when it is set to 1 or 0. As discussed above, arguments where <math>s = 1</math> or <math>s = 0</math> are perfect, or perfectly flawed, and thus cannot be changed with sub-arguments. This idea may be theoretically plausible but it wouldn't stop users from voting 1 or 0 for arguments. In such cases the `mods_if1or0` switch, when True, changes <math>s</math> to 0.99 and 0.01 respectively.

As a side note, this property is similar to Bayesian probabilities of 1 or 0 which also cannot be changed. We have discussed this problem in earlier posts under the guise that 1 or 0 probabilities don't really exist because they would require an infinite sample size. In the same way, a perfect argument (or perfectly imperfect) argument cannot exist because it would, at some point, run into the same issues that Bayesian probabilities do.

For example, suppose we've invented a pill that cures cancer. It is one dose, costs 10 cents to make, has no side effects, has no environmental impact due to manufacture, and is certain to cure someone's cancer. The argument for a cancer patient taking the pill is, for all practical purposes, perfect. There is simply no plausible argument against it. We could score such an argument a 1 until we remember our probabilities. We only know the pill works and has no side effects on a limited population, say 100,000 patients. We don't know what effect it will have on the 100,001st patient. So the best we can say is that the drug is 0.99999 effective. Given that the argument is really predicated on the effectiveness of the drug we could say its score is also 0.99999.

<h3>[https://slashdot.org/ Slashdot]</h3>

Slashdot offers a system for content moderation summarized by the following from [wikipedia:Slashot|wikipedia]:

::<i>Slashdot's editors are primarily responsible for selecting and editing the primary stories that are posted daily by submitters. The editors provide a one-paragraph summary for each story and a link to an external website where the story originated. Each story becomes the topic for a threaded discussion among the site's users. A user-based moderation system is employed to filter out abusive or offensive comments.[63] Every comment is initially given a score of −1 to +2, with a default score of +1 for registered users, 0 for anonymous users (Anonymous Coward), +2 for users with high "karma", or −1 for users with low "karma". As [[Moderator|moderator]]s read comments attached to articles, they click to moderate the comment, either up (+1) or down (−1). Moderators may choose to attach a particular descriptor to the comments as well, such as "normal", "offtopic", "flamebait", "troll", "redundant", "insightful", "interesting", "informative", "funny", "overrated", or "underrated", with each corresponding to a −1 or +1 rating. So a comment may be seen to have a rating of "+1 insightful" or "−1 troll".[57] Comments are very rarely deleted, even if they contain hateful remarks.

::Starting in August 2019 anonymous comments and postings have been disabled.

::Moderation points add to a user's rating, which is known as "karma" on Slashdot. Users with high "karma" are eligible to become moderators themselves. The system does not promote regular users as "moderators" and instead assigns five moderation points at a time to users based on the number of comments they have entered in the system – once a user's moderation points are used up, they can no longer moderate articles (though they can be assigned more moderation points at a later date). Paid staff editors have an unlimited number of moderation points. A given comment can have any integer score from −1 to +5, and registered users of Slashdot can set a personal threshold so that no comments with a lesser score are displayed. For instance, a user reading Slashdot at level +5 will only see the highest rated comments, while a user reading at level −1 will see a more "unfiltered, anarchic version". A meta-moderation system was implemented on September 7, 1999,to moderate the moderators and help contain abuses in the moderation system. Meta-moderators are presented with a set of moderations that they may rate as either fair or unfair. For each moderation, the meta-moderator sees the original comment and the reason assigned by the moderator (e.g. troll, funny), and the meta-moderator can click to see the context of comments surrounding the one that was moderated.</i>

Slashdot's purpose is to promote high quality discussion which is somewhat similar to our purpose of promoting high quality arguments. In particular, the [[Reputation|reputation]] (karma) of the moderators is an interesting concept. We could use a similar system to weight voters with a good reputation higher in their argument scoring. Another interesting idea is the use of word descriptors to match scores. In our system descriptors such as "Completely irrelevant", "somewhat irrelevant", etc. could be a useful way to break up corresponding numerical ranges in our 0-1 scoring system.

<h2>Refining our Argument Score with Reputation/Trust</h2>

[[#Scoring of individual arguments|Above]] we discussed an equation to score arguments on the basis of Veracity, Relevance, Freedom from Fallacies, and Clarity:

<math>
S = VR(w_fF + w_cC)
</math>

Where

<math>S</math> is overall score

<math>V</math> is Veracity

<math>R</math> is Relevance

<math>F</math> is Fallacies (ie freedom from fallacies)

<math>C</math> is Clarity

<math>w_f</math> is weighting for Fallacies, eg 0.7

<math>w_c</math> is weighting for Clarity, eg 0.3

<math>w_f + w_c = 1</math>

Each user would vote on each category and the resulting <math>S</math> would be calculated. However, each user may have a different reputation/trust for their ability to judge these four criteria. We can take this into account by simply adding a weighting factor for Trust:

<math>
S = (T_vV)(T_rR)(w_fT_fF + w_cT_cC)
</math>

where

<math>T_x</math> = Trust in user's ability to evaluate each category <math>x</math> (Veracity, Relevance, Fallacies, and Clarity)

Since the user evaluates trust in multiple categories, it would be useful to generate a composite trust for all the categories:

<math>
T_{comp} = {(T_vV)(T_rR)(w_fT_fF + w_cT_cC) \over{VR(w_fF + w_cC)}}
</math>

Where

<math>T_{comp}</math> is the composite trust for all categories.

Note that <math>T_{comp}</math> can also be seen as an "average" trust, ie the single factor that produces the same argument score as that resulting from the multiple trust factors.

Once we have <math>T_{comp}</math> we can use it to generate an average <math>S</math> for all users:

<math>
S_{ave} = {\sum S \over{\sum T_{comp}}}
</math>

That is, instead of dividing by the number of people voting, we divide by the total of how much they "count". This is similar to the [trust-weighted average scheme]([[A trust weighted averaging technique to supplement straight averaging and Bayes]]) we have proposed before. It is this average <math>S</math> that we will take to be the score for the argument.

<h3>Rhetorical vs. [[Practical argument|Practical Argument]]s</h3>

So far our scoring methodology has been focused mainly on rhetorical aspects of arguments. Is the argument true, relevant to its parent contention, clear, and logical? These criteria certainly touch on the practical impact an argument may have and so far we have been merging Impact with Relevance since it is hard to distinguish the two. Consider the following:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Argument: Biden is a good President because he got an infrastructure \n bill passed that will do good things for the whole country."]
1 [label="1, Sub-argument: The bill provides $2.9 million \n to Roanoke airport for improvements."]
2 [label = "2, Sub-argument: The bill provides $150 billion \n to combat climate change."]
0 -> 1 [dir="forward"];
0 -> 2 [dir="forward"];
}
</kroki>

Here it would seem appropriate to roll Impact into Relevance. Presumably most voters will recognize Sub-argument 2 as being the more relevant one simply because it is, on a practical level, the more impactful one.

However, usually Relevance is a rhetorical quality, not a practical one. In this case, as a matter of rhetoric, both supporting arguments are relevant to the topic at hand. They are both, without question, part of Biden's infrastructure bill. They are both true, clear, and free of fallacies. But even so, although it is clear what is going on, it would be better to separate the issue of whether Biden is a good president from the issue of which infrastructure allocations add the most value.

One reason for this separation, in terms of the [math laid out previously](Argument scoring), is that a weak sub-argument stops having any influence once its score goes below 0.5. However, if we are scoring Impact, either implicitly or explicitly, this rule would seem inappropriate. The rule makes many small sub-arguments, like the one about Roanoke airport, stop having any value. Perhaps the Roanoke airport argument alone is negligible but it still has a positive impact and, when added to all the other similar projects around the country, would amount to a sizeable contribution. Therefore it wouldn't be correct to nullify it altogether.

We also wouldn't want arguments from Impact to prematurely influence necessary [[Rhetorical argument|rhetorical argument]]s. Let's take a favorite from moral philosophy, that of a healthy young person, John who comes in for a routine checkup at a clinic which has 5 critical patients in need of organ transplants. John has the organs they need and is a match for all of them. The doctors, using a purely utilitarian argument decide to kill John and harvest his organs. It makes sense: 1 person dies and 5 live so we're ahead. Let's for the moment disregard legal and other social artifacts which might persuade the doctors otherwise. This approach contrasts with a deontological perspective which argues that the ends do not justify the means and that, indeed, the means in this case are all important. But we can only ferret out the deontological argument by actually having it, within the context of a rhetorical debate. We would hope such a debate would successfully preclude any utilitarian considerations whatsoever.

This leads us to the more general reason why separating the scoring techniques is appropriate. A score for Impact is essentially a [[Cost-benefit-risk analysis|cost-benefit-risk analysis]] for which established techniques exist and which would be confusing if scored together with the rhetorical argument. Indeed, by the time we reach an argument where impact is of interest we have usually dispensed with the rhetorical nature of the argument:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis: We have $150 billion to spend and \n should spend it on climate change"]
1 [label="1, Electric cars and charging stations.\n I = 0.5"]
2 [label="2, Home and building insulation, solar, heat pumps, etc.\n I = 0.3"]
3 [label="3, Carbon capture.\n I = 0.2"]
0 -> 1 [dir="forward"];
0 -> 2 [dir="forward"];
0 -> 3 [dir="forward"];
}
</kroki>

We can see that this argument is likely the outcome of previous arguments where basic points have been agreed to (or at least settled) such as whether climate change is real, Biden is a good president, etc. Here we are at the final stages of the argument, which consists of a resolution to move forward with some practical course of action.

In this case each sub-argument is, in effect, a lobbying effort for the money. All the sub-arguments are equal in terms of their Veracity, rhetorical Relevance, freedom from Fallacies, and Clarity. The only point of dispute is whether the money is better spent on one option or another. In a situation like this, <math>\sum I = 1</math> would be defined as the effect of all plausible infrastructural investments we could make to impact climate change. In this respect let's assume we are limited to the three options above. The voters, presumably armed with engineering studies, would then weigh in to assess the impact of each proposal.

It is important to emphasize that the argument at this point is a technical one. It is numerical in nature and hinges on scientific rigor. Our system for assessing trust in the people who can reasonably provide input at this level will be important. One can easily imagine how interested parties (and the merely ignorant) could skew the results with their vote. At the same time we want to encourage participants to move toward this type of debate since it leads to practical benefits and, by its nature, tends to reduce partisan rancor.

The idea outlined here stands apart, by design, from the method that scores the rhetorical quality (eg the VRFC equation) of the argument. Rhetoric is designed to convince you of the argument as a whole but arguments using Impact are designed to forward a specific recommendation. Clearly, bundling the Impact with the VRFC equation is inappropriate.

In many cases arguments will have a hard time getting to this level of practicality. They tend to remain mired in the basic rhetoric that governs them:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis: Does God exist?"]
1 [label="1, Yes, because I can speak to Him and \n He responds by doing good things for me."]
2 [label="2, No, because there is no objective \n evidence that there is anyone listening."]
0 -> 1 [dir="forward"];
0 -> 2 [dir="forward"];
}
</kroki>

This argument is clearly truncated but we can see where it is going. It is, in essence, one about the Veracity of personal experience vs. demonstrable evidence, which is a philosophical debate. It is hard to ascribe Impact to it because it doesn't ever get to the point of enumerating proposals.

But it could eventually transform itself into one that did. Let's assume the participants agree to settle, or at least table, the philosophical debate and concentrate instead on a test of how to improve your life. Both the religious and secular sides propose certain practices:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis: Does God exist?"]
1 [label="1, Yes, because I can speak to Him and \n He responds by doing good things for me."]
2 [label="2, No, because there is no objective \n evidence that there is anyone listening"]
3 [label="3 ...more debate..."]
4 [label="4 ...more debate..."]
5 [label="5, Modified Thesis: Is your life best improved by religious or secular practices?"]
6 [label="6, Religious"]
7 [label="7, Secular"]
8 [label="8, Pray for 20 minutes \n every night and ask \n for what you need."]
9 [label="9, Go to your religious \n service every week and \n perform the rituals."]
10 [label="10, Study for 20 minutes \n every night in an area \n where your problems are."]
11 [label="11, Find a support group \n and meet with them \n every week."]
0 -> 1 [dir="forward"];
0 -> 2 [dir="forward"];
1 -> 3 [dir="forward"];
2 -> 4 [dir="forward"];
3 -> 5 [dir="forward"];
4 -> 5 [dir="forward"];
5 -> 6 [dir="forward"];
5 -> 7 [dir="forward"];
6 -> 8 [dir="forward"];
6 -> 9 [dir="forward"];
7 -> 10 [dir="forward"];
7 -> 11 [dir="forward"];
}
</kroki>

In this case, the reader is being invited to try the techniques on offer and an evaluation of Impact might thus be made. Note that in this case we have suspended our evaluation of the rhetorical qualities of each argument and begun a new one which seeks a utilitarian appraisal of which side might offer a better personal outcome. This is in keeping with the notion of Impact as separate from the rhetorical qualities of the argument.

We could, in theory, envision someone who is not religious adopting religious practices because he concludes that it does him good. Perhaps he tried both sides and decided the religious side was the one yielding the greatest benefit. This possibility is no doubt one motivation for religious debaters to drop their metaphysical convictions and adopt a practical way to approach their disagreement with the secular side. In any event, this progression seems like a healthy outcome since metaphysical debates usually have little hope of resolution.

Impact is necessarily focused on some particular goal. If the goal is material benefit and you pray and get rich you might think prayer works. But if the goal is broader than that, you might be disturbed by the fact that you are doing something you don't really believe is true. Your goal might be to get rich without compromising philosophical integrity. In this case the object of our Impact changes to become more than simply material wealth. A clear statement of the argument thesis in terms of goals is obviously important here.

In spite of our attempts at separating impact from rhetoric we will often find ourselves with a mix of the two:
[[File:Procon.png|center|frame]]
Here we've scored the Pro argument weaker than the Con argument using our standard rhetorical measures (VRFC). Arguably the Pro argument speculates more (a type of fallacy) about what would happen if we stopped supporting Ukraine. The Con argument isn't perfect either since it seems to assume that the money saved would actually be used in some constructive way. Still, money not spent is certainly money saved so we'll mark it down only slightly. That said, it would be ridiculous to stop the argument after concluding the Con side "won". The argument is not really a rhetorical argument at all but rather a statement of Impact. One side argues for the impact of saving the money. The other argues for the impact of failing to spend the money. We may not know how events would play out in this situation but we acknowledge the risk of catastrophic consequences for failure to act. The impact score for the Pro side is thus much higher.

This is a particular case where the argument should be separated out into one that is explicitly about impact but it is not clear how best to achieve that. One way would be to allow participants to intervene by asking questions or proposing to move the debate in a more fruitful direction, perhaps by suggesting a new main contention (ie thesis).
[[File:Procon2.png|center|frame]]
A basically new debate ensues. Incentives to move the debate might include reputational points for agreeing on a more productive direction.

Let's look at an example of what this "productive" argument might look like in terms of cost-benefit-risk analysis:

[[File:Productiveargument.png|center|frame]]
The red boxes are Con arguments and the green box is the Pro argument. We can see right away why the Pro argument might have stiff opposition since it leads to an infinite cost-benefit ratio and is virtually certain to occur (it is our current policy). It is possible that other scenarios could play out within the context of supporting Ukraine but let's leave these aside for the moment.

So the Pro side looks bad until we start looking at the Con scenarios. In Scenario 1 we envision taking the $75 billion spent on Ukraine aid and providing free [[community]] college instead. Doing so provides an economic benefit in the long run so we provide an estimate for that. However, military and policy experts have said that ignoring Ukraine would result in having to contain a newly resurgent Russia and this could double our defense costs in the near term ($800 billion). The resulting cost benefit ratio is 2.9, a positive number which is undesirable. It also is the highest probability scenario, at 70%. Other scenarios involve some type of war with Russia and would involve an even greater outlay of funds, not to mention the sheer human toll of war. Only Scenario 4 envisions a minor outlay to contain a victorious Russia which would be offset by the benefit of free community college. This scenario is desirable but unlikely.

These scenarios are much like sub-arguments but stripped of any need to assess their rhetorical quality. By looking at CB ratios and probabilities we can determine which policy direction to take.

<h3>Interaction effects between arguments and sub-arguments</h3>

Although arguments have been presented as standalone entities, users may often score them after reading, and accounting for, their sub-arguments. In that case, the sub-argument's influence on the parent argument would be counted twice -- once due to the mathematical effect discussed [[#Scoring of individual arguments|above]] and twice due to the the influence the sub-argument has on the user's scoring of the parent argument.

This effect is clearly undesirable and efforts should be made to control it. The software could, for instance, be equipped with the following checks:

* If it detects that a user voted for a sub-argument and subsequently voted for an argument, it can flag the sub-argument score so it does not participate in the mathematical effect it would otherwise have on the argument. We are assuming here, of course, that a user who has voted for a sub-argument will be unable to avoid having it influence his vote for the parent argument.
* If it detects a vote for an argument but not a vote for its sub-argument, it doesn't know if the user has read the sub-argument in a way that would influence their vote for the parent argument. In such a case, the user can simply be asked if the sub-argument was read and, if so, to flag any subsequent vote by the user for the sub-argument as a non-participant in its mathematical influence on the parent argument.
* If the sub-argument does not yet exist when the vote for the parent argument is cast, the software will flag a subsequent vote for any newly developed sub-argument as a legitimate participant in the mathematical influence it has on the parent argument.

It is probably difficult to make a system like this foolproof. A user might report not having read a sub-argument that they have, in fact, read. Tracking features could, in theory, be developed to check whether this is the case and react accordingly. However, it would still be difficult to know for sure how deeply the user understands the sub-argument just based on a record that they clicked on it or had it "open". It also seems like it would be easy to overdo tracking of this kind to the point where it simply turns off an otherwise enthusiastic user. Another interesting idea is the use of word descriptors to match scores. In our system descriptors such as "Completely irrelevant", "somewhat irrelevant", etc. could be a useful way to break up corresponding numerical ranges in our 0-1 scoring system.

Argument evaluation and scoring

2024-09-25T13:52:17Z

Lem:

<h2>Some thoughts on evaluating arguments</h2>

In our last meeting Dan asked how we might evaluate an argument made by a respondent instead of simply relying on the given probability (as we've been doing so far). An argument, assuming it is made public, could then be evaluated by the questioner and others independently to find a more accurate probability. This opens up a new idea in our work, that of assessing the truth by evaluating the reasoning put forth in an [[opinion]].

One idea for doing this starts with a simple model for argument construction. The argument consists of supporting statements which are tied together with [[logic]] to form a conclusion. The conclusion is the answer to the overall question being asked of the network. Each supporting statement and the logic can be evaluated independently to determine the extent to which the conclusion is true. The following diagram illustrates this:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="Answer/Conclusion, Pc"]
1 [label="Logic, Pl"]
2 [label="Support. Stmt. 1, Ps1"]
3 [label="Support. Stmt. 2, Ps2"]
4 [label="Support. Stmt. 3, Ps3"]
0 -> 1 [dir="back"];
1 -> 2 [dir="back"];
1 -> 3 [dir="back"];
1 -> 4 [dir="back"];
}
</kroki>

The probability of the supporting statements can be combined in a Bayesian manner. This is in keeping with the Bayesian idea of modifying prior probabilities given new evidence (ie supporting statements). These probabilities can be [[trust]]-modified as Sapienza proposed (https://ceur-ws.org/Vol-1664/w9.pdf) but since they are likely being assigned by the questioner, we will assume that trust is already built into them. Of more importance is the relevance of the supporting statements. They can range from completely irrelevant to completely relevant. A completely relevant statement will take the full value of the probability it was originally assigned. A completely irrelevant statement would reduce the probability to 50%, where it will have no influence on the outcome. In that sense relevance functions in the same way trust does to modify the probability:

<math>
P_{mod} = P_{nom} + R(P - P_{nom})
</math>

where

<math>R</math> is relevance (0.0 - 1.0)

<math>P_{mod}</math> is relevance-modified probability

<math>P_{nom}</math> is the nominal probability (=0.5 for a [[predicate]] question)

<math>P</math> is the unmodified probability

After the relevance-modification, each supporting statement is combined in the usual manner via Bayes. For the first two statements,

<math>
P_{comb1,2} = {P_{s1}P_{s2}\over {P_{s1}P_{s2} + (1-P_{s1})(1-P_{s2})}}
</math>

and so on for each additional statement. Here it is to be understood that <math>P_{s1}</math>, etc. is the value <i>after</i> the relevance modification.

Logic will also have a probability assigned to it to represent its quality. A fully illogical argument would receive a 0, which when combined via Bayes with the supporting statements would render the probability of the entire argument 0. This makes sense because a completely illogical argument, regardless of the strength of its supporting statements, destroys itself. A fully logical argument, however, will not receive a 1 but rather a 0.5. When combined with the supporting statements a 1 would render the final probability a 1, which is not reasonable. A 0.5, however, would do nothing and the final probability would be the combined probability of the supporting statements. Thus we assume that perfect logic is neutral and less than perfect logic reduces the combined probability of the statements. Again, this seems reasonable. We expect, by default, logical arguments which then rest on the strength of their supporting statements. If we notice flaws in the logic we discount the strength of the argument accordingly.

Let's try an example with these ideas:

Question: Are humans causing frog populations to decline?<br>

Answer / Conclusion: Yes, mankind is causing a fall in frog populations.<br>

Logic: Mankind is causing the fall of frog populations if we can show that frog populations are decreasing over time and can show a human behavior that causes the decline.<br>

Supporting Statements:<br>

# [https://en.wikipedia.org/wiki/Frog#:~:text=Frog%20populations%20have%20declined%20significantly%20since%20the%201950s. Frog populations have declined since the 1950's.]
# My wife complained that she doesn't see frogs anymore.
# [https://wwf.panda.org/discover/our_focus/freshwater_practice/freshwater_biodiversity_222/ Scientists say] that the loss of freshwater habitats has affected frog populations.

We start by judging the quality of the supporting statements. 1 seems like a well substantiated statement (a high P) but is not completely relevant because it only hints at human involvement. 2 is completely true but mostly irrelevant. 3 is a contributor but seems less substantiated than 1 and contains no human cause. We proceed by assigning probability and relevance values:

<math>P_{s1}=0.9</math>
<math>R_{s1}=0.7</math>
<math>P_{s1mod} = 0.5 + 0.7(0.9 - 0.5) = 0.78</math>

<math>P_{s2}=1.0</math>
<math>R_{s2}=0.0</math>
<math>P_{s2mod} = 0.5 + 0.0(1.0 - 0.5) = 0.5</math>

<math>P_{s3}=0.75</math>
<math>R_{s3}=0.5</math>
<math>P_{s3mod} = 0.5 + 0.5(0.75 - 0.5) = 0.625</math>

Since s2 won't count in the Bayesian calculation we can ignore it and:

<math>P_{comb,s} = {(0.78)(0.625) \over {0.78(0.625)+0.22(0.375)}} = 0.855</math>

The logic/conclusion in this case is reasonably strong so we will assign it a high value, say <math>P_l = 0.45</math> (remember, out of 0.5). It could be improved by observing that the word "behavior" is too general and should be replaced by, say, "policy choice" (ie urban growth into ecologically important wetlands). We note here that logic is more than just the mathematical construction of an argument. Since we are speaking a human language, logic might also be flawed because it uses imprecise wording.

Putting <math>P_{comb,s}</math> together with <math>P_l</math> using Bayes we obtain a concluding probability:

<math>P_c = {0.855(0.45) \over {{0.855(0.45) + 0.145(0.55)}}} = 0.82</math>

One potential pitfall of this model is that repetitive supporting statements of high probability will quickly render a combined probability near 1.0. As we've seen in the past, this is simply the result of the Bayes equation. The user would need to watch for attempts like these to distort the answer by removing repetitive statements or making them irrelevant.

<h2>Scoring of individual arguments</h2>

Arguments can be scored on veracity, impact, relevance, clarity, and informal quality (lack of fallacies):

- Veracity is how true the argument is based on source information. Source information itself will be scored: <math>V</math>

- Impact & Relevance is how deeply the argument affects the main contention of the [[debate]] (or the argument immediately above): <math>R</math>

- Clarity is how understandable the argument is: <math>C</math>

- Informal quality (lack of fallacies) is whether the argument commits any logical fallacies of its own. A list of informal fallacies (and formal ones) will be provided to help users select appropriately: <math>F</math>

Since Impact and Relevance are closely related concepts we will merge these into one, Relevance. The simplest method for combining these is to average them, or weighted average them:

<math>
S = w_vV + w_rR + w_cC + w_fF
</math>

where

<math>w_x</math> is a weighting for category X (eg Veracity, Relevance, etc)

<math>w_v + w_r + w_c + w_f = 1</math>

This seems reasonable and if we believe that certain criteria should weigh more (such as Veracity) we can easily make the weighting factors reflect this. However, intuitively it seems that a category such as Veracity should not only weigh more but have the power to take down the whole argument. After all, if the argument is a straightforward lie, it should receive a score of zero, regardless of its other attributes (such as relevance, clarity, etc):

<i>Who is the best choice for President? X is the best choice because he will land a person on Mars in his first year.</i>

This argument is a lie and although it is clear, has no evident fallacies, and is relevant to the question at hand, it should be thrown out.

The same can be said of Relevance. A completely irrelevant argument should also have the power to render the whole argument moot:

<i>Who is the best choice for President? X is the best choice because he likes pizza and so do I.</i>

With this in mind, we can propose the following equation, which we will dub the "VRFC equation":

<math>
S = VR(w_fF + w_cC)
</math>

Where

<math>S</math> = Score for the argument which varies from 0-1

<math>w_f</math> = weighting factor for Fallacies.

<math>w_c</math> = weighting factor for Clarity.

<math>w_f + w_c = 1</math>

and each of the constituent variables (<math>V, R, F, C</math>) has a range 0-1.

In this equation either Veracity or Relevance have the power to nullify the entire argument. Similarly a combination of Fallacies and lack of Clarity can do the same. However, a fallacious argument alone seems like it could still have merit, as would an argument whose only flaw was lack of clarity:

<i>We should support Czechoslovakia because if the Nazi's prevail they will conquer the world.</i>

This argument commits the slippery slope fallacy but is not entirely invalid. Similarly an unclear argument can still manage to make a point:

<i>We should support Europe because first the Sudetenland, then the Czechs, and soon enough it's over when all the Brits had to do was get rid of that weakling sooner.</i>

It would seem that a fallacious argument should weigh more than an unclear one. Proposed weights might be:

<math>w_f = 0.7, w_c = 0.3</math>

<h3>Rolling up the score of argument trees</h3>

The equation above applies to a single argument but, as we've seen, most arguments have sub-arguments below, sub-sub-arguments, and so forth. They are really trees in which each individual argument can be scored separately.

Here we develop a proposed equation for rolling up the score for an argument based on its own score and that of its sub-arguments. In doing so we emphasize that any argument can stand on its own and be scored in the absence of sub-arguments. This creates an interesting dynamic. The sub-argument may bolster or detract from the parent argument but the extent to which it does should be limited.

Furthermore, once the sub-argument becomes weaker than a certain threshold, it should stop influencing the parent argument altogether. Here, we will set this threshold at 0.5. Thus only Pro sub-arguments that score 0.5 or better will have any influence on the parent argument. For Con sub-arguments we will use the same threshold but first modify the sub-argument score by <math>1-S</math>. Thus a strong Con sub-argument, scoring say 0.9, would enter the calculation with a score of 0.1. The result is a range of scores 0-1 of which 0-0.5 is Con and 0.5-1 is Pro. Scores of exactly 0.5 are neutral.

Let's consider the case with one argument and one pro sub-argument and one Con sub-argument.

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="Argument, s = 0.9"]
1 [label="Pro sub-argument, xp = 0.7"]
2 [label="Con sub-argument, xc = 0.7"]
0 -> 1 [dir="both"];
0 -> 2 [dir="both"];
}
</kroki>

In this case the argument's score is 0.9, and both the Pro/Con sub-argument score is 0.7. These numbers would normally be arrived at by using the VRFC eqn above, but we will just assume them for now. The first sub-argument, in this case, bolsters the argument because it is a Pro argument and has a score (0.7) greater than 0.5. The second sub-argument, with the same score, detracts from the argument because it is on the Con side. We emphasize that if these scores were at or below 0.5 they would have no effect on the argument.

The general equation governing this situation is as follows:

For <math>s > 0.5</math> and <math>x > 0.5</math>

<math>
s_{mod} = 2(1-s)fx + s - (1-s)f
</math>

For <math>s > 0.5</math> and <math>x <= 0.5</math>

<math>
s_{mod} = s
</math>

For <math>s < 0.5</math> and </math>x > 0.5</math>

<math>
s_{mod} = 2sfx + s - sf
</math>

For <math>s < 0.5</math> and <math>x <= 0.5</math>

<math>
s_{mod} = s
</math>

where

<math>s</math> = score for parent argument

<math>x = x_p</math> = score for Pro arguments

<math>x = 1-x_c</math> = score for Con arguments

<math>f</math> = maximum possible fraction of increase possible, 0-1

The variable <math>f</math> is a user selected number between 0-1 and represents the extent to which the sub-argument score can be affected. For example, a sub-argument with x = 0.9, as in the diagram above, can be improved by 0.1 to a maximum of 1. Then <math>f</math> represents the fraction of 0.1 that we will allow for our improvement. If <math>f = 0.25</math>, for instance, then the maximum range around 0.9 that the sub-argument can affect is <math>(0.25)(0.1) = 0.025</math>. Thus the maximum score the argument can have is 0.925 and the minimum is 0.875.

For the argument above:

<math>s = 0.9</math>

<math>f = 0.25</math> User input

For the Pro sub-argument:

<math>x = x_p = 0.7</math>

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.9)(0.25)(0.7) + 0.9 - (1-0.9)(0.25) = 0.91</math>

For the Con sub-argument:

<math>x = (1-x_c) = (1-0.7) = 0.3</math>

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.9)(0.25)(0.3) + 0.9 - (1-0.9)(0.25) = 0.89</math>

We can see here that the Pro and Con sub-arguments exactly balance each other since they both have the same score.

The equation above is piecewise linear and can be visualized as follows:

![image](uploads/7f62b22b9eb19771503d654db29cab92/image.png)

One important property of this equation is that the stronger (or weaker) an argument becomes, it becomes harder for a sub-argument to change it. This is because the maximum allowed movement is <math>1-s</math> if <math>s > 0.5</math> or simply <math>s</math> if <math>s <= 0.5</math>. The idea behind this property is that very strong arguments should be harder to dislodge precisely because they have covered themselves well. A weaker argument, for instance one that fails to mention an obvious supporting fact, is in a position to be bolstered more by a sub-argument which mentions the fact. Similarly a very weak argument should be difficult to bolster. If the argument is a lie or irrelevant, for instance, there isn't much that can be done to rescue it.

This property has the further consequence that <math>s</math> cannot be changed by the sub-arguments if it is 1 or 0. A truly perfect argument, <math>s = 1</math>, cannot be weakened no matter how strong its Con sub-argument. Similarly a perfectly flawed argument, <math>s = 0</math> cannot be bolstered with any Pro sub-argument. We will discuss below a method to deal with the fact that, regardless of the quality of the argument, users may still vote to score arguments 1 or 0.

<h4>Population adjustments</h4>

The algorithm described above assumes a single vote for the argument and sub-arguments. In fact, this will rarely be the case, because multiple users will be voting on each. The effect of a sub-argument on its parent should be weighed by the population of users who voted for the sub-argument and parent argument.

Here we propose a simple modification factor, based on the ratio of users voting for each argument/sub-argument:

<math>s_{mod,pop} = (s_{mod} - s){p_s\over p} + s</math>

where

<math>s_{mod,pop}</math> is the population modified score

<math>s_{mod}</math> is the modified score without population modifications (see above)

<math>p_s</math> is the population voting for the sub-argument

<math>p</math> is the population voting for the parent argument

<math>s</math> is the original score of the parent argument

Usually we expect that sub-arguments will receive fewer votes than parent arguments, so <math>{{p_s\over p} <= 1}</math> in general. For the case when <math>p_s > p</math> we will force <math>{p_s\over p} = 1</math>. Therefore there is no danger that a sub-argument can overwhelm a parent argument by voting power alone. This is in keeping with our philosophy that sub-arguments can have at best a limited effect on parent arguments.

<h4>Example calculation</h4>

Let's do a problem with the following argument tree and <math>f=0.25</math>:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis Statement"]
1 [label="1, Pro argument, s = 0.9, p = 96"]
2 [label="2, Pro sub-argument, xp = 0.7, p = 55"]
3 [label="3, Pro sub-argument, xp = 0.8, p = 26"]
4 [label="4, Con sub-argument, xc = 0.6, p = 30"]
5 [label="5, Con sub-argument, xc = 0.7, p = 43"]
6 [label="6, Pro sub-argument, xp = 0.85, p = 19"]
7 [label="7, Con sub-argument, xc = 0.95, p = 28"]
8 [label="8, Con argument, ...."]
0 -> 1 [dir="both"];
1 -> 2 [dir="both"];
1 -> 5 [dir="both"];
2 -> 3 [dir="both"];
2 -> 4 [dir="both"];
5 -> 6 [dir="both"];
5 -> 7 [dir="both"];
0 -> 8 [dir="both"];
}
</kroki>

Our objective here is to roll up the score for the Pro side of this tree. The Con side would be calculated similarly and we will skip this for the sake of brevity. Note that <math>s</math> stands for the score and <math>p</math> is the population voting to produce that score. We start at the bottom, with the 2-3-4 portion of the tree, and for the sake of consistency with the above calculation we will recast <math>x_p</math> for 2 as <math>s</math> and label the population of the sub-arguments as <math>p_s</math>.

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
2 [label="2, Pro argument, s = 0.7, p = 55"]
3 [label="3, Pro sub-argument, xp = 0.8, ps = 26"]
4 [label="4, Con sub-argument, xc = 0.6, ps = 30"]
2 -> 3 [dir="both"];
2 -> 4 [dir="both"];
}
</kroki>

For the Pro sub-argument, we write:

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.7)(0.25)(0.8) + 0.7 - (1-0.7)(0.25) = 0.745</math>

We modify this by the respective populations:

<math>s_{mod,pop23} = (s_{mod} - s){p_s\over p} + s = (0.745 - 0.7){26\over 55} + 0.7 = 0.721</math>

For the Con sub-argument we first modify its score,

<math>x = 1 - x_c = 0.4</math>

and write

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.7)(0.25)(0.4) + 0.7 - (1-0.7)(0.25) = 0.685</math>

and modify by the respective population,

<math>s_{mod,pop24} = (s_{mod} - s){p_s\over p} + s = (0.685 - 0.7){30\over 55} + 0.7 = 0.692</math>

These two values of <math>s_{mod,pop}</math> can now be combined to create a new <math>s</math> for the Pro argument:

<math>s_{mod,tot} = (s_{mod,pop23} - s) + (s_{mod,pop24} - s) + s = (0.721 - 0.7) + (0.692 - 0.7) + 0.7 = 0.713</math>

We note here that the Pro argument got a little stronger as a result of its sub-arguments. The Pro sub-argument was substantially stronger than the Con sub-argument and, although fewer people voted for it, the population difference was not large.

For the Con sub-argument 5-6-7 we have the following situation:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
5 [label="5, Con argument, s = 0.7, p = 43"]
6 [label="6, Pro sub-argument, xp = 0.85, ps = 19"]
7 [label="7, Con sub-argument, xc = 0.95, ps = 28"]
5 -> 6 [dir="both"];
5 -> 7 [dir="both"];
}
</kroki>

Here, for the Pro sub-argument, we first modify its score since it is the opposite of its parent. It is as if the parent were a Pro argument and the child were a Con argument.

<math>x = 1 - x_p = 1 - 0.85 = 0.15</math>

We then proceed as usual with the calculation:

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.7)(0.25)(0.15) + 0.7 - (1-0.7)(0.25) = 0.6475</math>

<math>s_{mod,pop56} = (s_{mod} - s){p_s\over p} + s = (0.6475 - 0.7){19\over 43} + 0.7 = 0.677</math>

For the Con sub-argument <math>x = x_c = 0.95</math> since the parent argument is also Con:

<math>s_{mod} = 2(1-s)fx + s - (1-s)f = 2(1-0.7)(0.25)(0.95) + 0.7 - (1-0.7)(0.25) = 0.7675</math>

<math>s_{mod,pop57} = (s_{mod} - s){p_s\over p} + s = (0.7675 - 0.7){28\over 43} + 0.7 = 0.744</math>

We combine these two results in the same manner as above:

<math>s_{mod,tot} = (s_{mod,pop56} - s) + (s_{mod,pop57} - s) + s = (0.677 - 0.7) + (0.744 - 0.7) + 0.7 = 0.721</math>

With the bottom layer of the tree calculated, we have the following situation:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis Statement"]
1 [label="1, Pro argument, s = 0.9, p = 96"]
2 [label="2, Pro sub-argument, xp = 0.713, ps = 55"]
5 [label="5, Con sub-argument, xc = 0.721, ps = 43"]
8 [label="8, Con argument, ...."]
0 -> 1 [dir="both"];
1 -> 2 [dir="both"];
1 -> 5 [dir="both"];
0 -> 8 [dir="both"];
}
</kroki>

All that remains is to calculate the 1-2-5 portion, which is very similar to the 2-3-4 calculation performed above. Therefore will skip the details of this and simply report the results:

<math>s_{mod,pop12} = 0.906</math>

<math>s_{mod,pop15} = 0.895</math>

<math>s_{mod, tot} = 0.901</math>

We see here that the final result is not much different than the original <math>s = 0.9</math>. This is a result of the Pro sub-arguments essentially being cancelled by the Con sub-arguments. Such a result is to be expected in many cases.

In this example, we are skipping the Con side of the overall argument (node 8 in the tree above) because it would be exactly the same as what we have shown. If it had been calculated we would then combine the result for 1 and 8 to produce an overall score for the argument.

The calculations above can be performed with the [attached snippet](https://gitlab.syncad.com/peerverity/trust-model-playground/-/snippets/164). The user input portion of the snippet is set up for the calculation we did immediately above:

<syntaxhighlight lang="python">
#User input
side_parent = 'pro' #side, pro or con, that the parent argument is on
s = 0.9 #score for the parent argument
mf = 0.25 #max fraction that parent argument can be changed in terms of (1-s) or (s-0)
p = 96.0 #population voting for the parent argument
x_pro_arr = [0.713] #score for the pro children
x_con_arr = [0.721] #score for the con children
ps_pro_arr = [55.0] #population voting for each pro child sub-argument
ps_con_arr = [43.0] #pop voting for each con child sub-argument
mods_if1or0 = True #True if we want scores of 1 or 0 to be modified to near 1 or 0 (otherwise they can't be adjusted by this calculation)
</syntaxhighlight>

To note, the snippet contains arrays to handle any number of child arguments. These are combined in the same way we combined the single Pro and Con argument above.

Another variable, `mods_if1or0` controls whether we allow <math>s</math> to be modified when it is set to 1 or 0. As discussed above, arguments where <math>s = 1</math> or <math>s = 0</math> are perfect, or perfectly flawed, and thus cannot be changed with sub-arguments. This idea may be theoretically plausible but it wouldn't stop users from voting 1 or 0 for arguments. In such cases the `mods_if1or0` switch, when True, changes <math>s</math> to 0.99 and 0.01 respectively.

As a side note, this property is similar to Bayesian probabilities of 1 or 0 which also cannot be changed. We have discussed this problem in earlier posts under the guise that 1 or 0 probabilities don't really exist because they would require an infinite sample size. In the same way, a perfect argument (or perfectly imperfect) argument cannot exist because it would, at some point, run into the same issues that Bayesian probabilities do.

For example, suppose we've invented a pill that cures cancer. It is one dose, costs 10 cents to make, has no side effects, has no environmental impact due to manufacture, and is certain to cure someone's cancer. The argument for a cancer patient taking the pill is, for all practical purposes, perfect. There is simply no plausible argument against it. We could score such an argument a 1 until we remember our probabilities. We only know the pill works and has no side effects on a limited population, say 100,000 patients. We don't know what effect it will have on the 100,001st patient. So the best we can say is that the drug is 0.99999 effective. Given that the argument is really predicated on the effectiveness of the drug we could say its score is also 0.99999.

<h3>[https://slashdot.org/ Slashdot]</h3>

Slashdot offers a system for content moderation summarized by the following from [wikipedia:Slashot|wikipedia]:

::<i>Slashdot's editors are primarily responsible for selecting and editing the primary stories that are posted daily by submitters. The editors provide a one-paragraph summary for each story and a link to an external website where the story originated. Each story becomes the topic for a threaded discussion among the site's users. A user-based moderation system is employed to filter out abusive or offensive comments.[63] Every comment is initially given a score of −1 to +2, with a default score of +1 for registered users, 0 for anonymous users (Anonymous Coward), +2 for users with high "karma", or −1 for users with low "karma". As moderators read comments attached to articles, they click to moderate the comment, either up (+1) or down (−1). Moderators may choose to attach a particular descriptor to the comments as well, such as "normal", "offtopic", "flamebait", "troll", "redundant", "insightful", "interesting", "informative", "funny", "overrated", or "underrated", with each corresponding to a −1 or +1 rating. So a comment may be seen to have a rating of "+1 insightful" or "−1 troll".[57] Comments are very rarely deleted, even if they contain hateful remarks.

::Starting in August 2019 anonymous comments and postings have been disabled.

::Moderation points add to a user's rating, which is known as "karma" on Slashdot. Users with high "karma" are eligible to become moderators themselves. The system does not promote regular users as "moderators" and instead assigns five moderation points at a time to users based on the number of comments they have entered in the system – once a user's moderation points are used up, they can no longer moderate articles (though they can be assigned more moderation points at a later date). Paid staff editors have an unlimited number of moderation points. A given comment can have any integer score from −1 to +5, and registered users of Slashdot can set a personal threshold so that no comments with a lesser score are displayed. For instance, a user reading Slashdot at level +5 will only see the highest rated comments, while a user reading at level −1 will see a more "unfiltered, anarchic version". A meta-moderation system was implemented on September 7, 1999,to moderate the moderators and help contain abuses in the moderation system. Meta-moderators are presented with a set of moderations that they may rate as either fair or unfair. For each moderation, the meta-moderator sees the original comment and the reason assigned by the moderator (e.g. troll, funny), and the meta-moderator can click to see the context of comments surrounding the one that was moderated.</i>

Slashdot's purpose is to promote high quality discussion which is somewhat similar to our purpose of promoting high quality arguments. In particular, the reputation (karma) of the moderators is an interesting concept. We could use a similar system to weight voters with a good reputation higher in their argument scoring. Another interesting idea is the use of word descriptors to match scores. In our system descriptors such as "Completely irrelevant", "somewhat irrelevant", etc. could be a useful way to break up corresponding numerical ranges in our 0-1 scoring system.

<h2>Refining our Argument Score with Reputation/Trust</h2>

[[#Scoring of individual arguments|Above]] we discussed an equation to score arguments on the basis of Veracity, Relevance, Freedom from Fallacies, and Clarity:

<math>
S = VR(w_fF + w_cC)
</math>

Where

<math>S</math> is overall score

<math>V</math> is Veracity

<math>R</math> is Relevance

<math>F</math> is Fallacies (ie freedom from fallacies)

<math>C</math> is Clarity

<math>w_f</math> is weighting for Fallacies, eg 0.7

<math>w_c</math> is weighting for Clarity, eg 0.3

<math>w_f + w_c = 1</math>

Each user would vote on each category and the resulting <math>S</math> would be calculated. However, each user may have a different reputation/trust for their ability to judge these four criteria. We can take this into account by simply adding a weighting factor for Trust:

<math>
S = (T_vV)(T_rR)(w_fT_fF + w_cT_cC)
</math>

where

<math>T_x</math> = Trust in user's ability to evaluate each category <math>x</math> (Veracity, Relevance, Fallacies, and Clarity)

Since the user evaluates trust in multiple categories, it would be useful to generate a composite trust for all the categories:

<math>
T_{comp} = {(T_vV)(T_rR)(w_fT_fF + w_cT_cC) \over{VR(w_fF + w_cC)}}
</math>

Where

<math>T_{comp}</math> is the composite trust for all categories.

Note that <math>T_{comp}</math> can also be seen as an "average" trust, ie the single factor that produces the same argument score as that resulting from the multiple trust factors.

Once we have <math>T_{comp}</math> we can use it to generate an average <math>S</math> for all users:

<math>
S_{ave} = {\sum S \over{\sum T_{comp}}}
</math>

That is, instead of dividing by the number of people voting, we divide by the total of how much they "count". This is similar to the [trust-weighted average scheme]([[A trust weighted averaging technique to supplement straight averaging and Bayes]]) we have proposed before. It is this average <math>S</math> that we will take to be the score for the argument.

<h3>Rhetorical vs. Practical Arguments</h3>

So far our scoring methodology has been focused mainly on rhetorical aspects of arguments. Is the argument true, relevant to its parent contention, clear, and logical? These criteria certainly touch on the practical impact an argument may have and so far we have been merging Impact with Relevance since it is hard to distinguish the two. Consider the following:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Argument: Biden is a good President because he got an infrastructure \n bill passed that will do good things for the whole country."]
1 [label="1, Sub-argument: The bill provides $2.9 million \n to Roanoke airport for improvements."]
2 [label = "2, Sub-argument: The bill provides $150 billion \n to combat climate change."]
0 -> 1 [dir="forward"];
0 -> 2 [dir="forward"];
}
</kroki>

Here it would seem appropriate to roll Impact into Relevance. Presumably most voters will recognize Sub-argument 2 as being the more relevant one simply because it is, on a practical level, the more impactful one.

However, usually Relevance is a rhetorical quality, not a practical one. In this case, as a matter of rhetoric, both supporting arguments are relevant to the topic at hand. They are both, without question, part of Biden's infrastructure bill. They are both true, clear, and free of fallacies. But even so, although it is clear what is going on, it would be better to separate the issue of whether Biden is a good president from the issue of which infrastructure allocations add the most value.

One reason for this separation, in terms of the [math laid out previously](Argument scoring), is that a weak sub-argument stops having any influence once its score goes below 0.5. However, if we are scoring Impact, either implicitly or explicitly, this rule would seem inappropriate. The rule makes many small sub-arguments, like the one about Roanoke airport, stop having any value. Perhaps the Roanoke airport argument alone is negligible but it still has a positive impact and, when added to all the other similar projects around the country, would amount to a sizeable contribution. Therefore it wouldn't be correct to nullify it altogether.

We also wouldn't want arguments from Impact to prematurely influence necessary rhetorical arguments. Let's take a favorite from moral philosophy, that of a healthy young person, John who comes in for a routine checkup at a clinic which has 5 critical patients in need of organ transplants. John has the organs they need and is a match for all of them. The doctors, using a purely utilitarian argument decide to kill John and harvest his organs. It makes sense: 1 person dies and 5 live so we're ahead. Let's for the moment disregard legal and other social artifacts which might persuade the doctors otherwise. This approach contrasts with a deontological perspective which argues that the ends do not justify the means and that, indeed, the means in this case are all important. But we can only ferret out the deontological argument by actually having it, within the context of a rhetorical debate. We would hope such a debate would successfully preclude any utilitarian considerations whatsoever.

This leads us to the more general reason why separating the scoring techniques is appropriate. A score for Impact is essentially a cost-benefit-risk analysis for which established techniques exist and which would be confusing if scored together with the rhetorical argument. Indeed, by the time we reach an argument where impact is of interest we have usually dispensed with the rhetorical nature of the argument:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis: We have $150 billion to spend and \n should spend it on climate change"]
1 [label="1, Electric cars and charging stations.\n I = 0.5"]
2 [label="2, Home and building insulation, solar, heat pumps, etc.\n I = 0.3"]
3 [label="3, Carbon capture.\n I = 0.2"]
0 -> 1 [dir="forward"];
0 -> 2 [dir="forward"];
0 -> 3 [dir="forward"];
}
</kroki>

We can see that this argument is likely the outcome of previous arguments where basic points have been agreed to (or at least settled) such as whether climate change is real, Biden is a good president, etc. Here we are at the final stages of the argument, which consists of a resolution to move forward with some practical course of action.

In this case each sub-argument is, in effect, a lobbying effort for the money. All the sub-arguments are equal in terms of their Veracity, rhetorical Relevance, freedom from Fallacies, and Clarity. The only point of dispute is whether the money is better spent on one option or another. In a situation like this, <math>\sum I = 1</math> would be defined as the effect of all plausible infrastructural investments we could make to impact climate change. In this respect let's assume we are limited to the three options above. The voters, presumably armed with engineering studies, would then weigh in to assess the impact of each proposal.

It is important to emphasize that the argument at this point is a technical one. It is numerical in nature and hinges on scientific rigor. Our system for assessing trust in the people who can reasonably provide input at this level will be important. One can easily imagine how interested parties (and the merely ignorant) could skew the results with their vote. At the same time we want to encourage participants to move toward this type of debate since it leads to practical benefits and, by its nature, tends to reduce partisan rancor.

The idea outlined here stands apart, by design, from the method that scores the rhetorical quality (eg the VRFC equation) of the argument. Rhetoric is designed to convince you of the argument as a whole but arguments using Impact are designed to forward a specific recommendation. Clearly, bundling the Impact with the VRFC equation is inappropriate.

In many cases arguments will have a hard time getting to this level of practicality. They tend to remain mired in the basic rhetoric that governs them:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis: Does God exist?"]
1 [label="1, Yes, because I can speak to Him and \n He responds by doing good things for me."]
2 [label="2, No, because there is no objective \n evidence that there is anyone listening."]
0 -> 1 [dir="forward"];
0 -> 2 [dir="forward"];
}
</kroki>

This argument is clearly truncated but we can see where it is going. It is, in essence, one about the Veracity of personal experience vs. demonstrable evidence, which is a philosophical debate. It is hard to ascribe Impact to it because it doesn't ever get to the point of enumerating proposals.

But it could eventually transform itself into one that did. Let's assume the participants agree to settle, or at least table, the philosophical debate and concentrate instead on a test of how to improve your life. Both the religious and secular sides propose certain practices:

<kroki lang="graphviz">
digraph G {
fontname="Helvetica,Arial,sans-serif"
node [fontname="Helvetica,Arial,sans-serif"]
edge [fontname="Helvetica,Arial,sans-serif"]
layout=dot
0 [label="0, Thesis: Does God exist?"]
1 [label="1, Yes, because I can speak to Him and \n He responds by doing good things for me."]
2 [label="2, No, because there is no objective \n evidence that there is anyone listening"]
3 [label="3 ...more debate..."]
4 [label="4 ...more debate..."]
5 [label="5, Modified Thesis: Is your life best improved by religious or secular practices?"]
6 [label="6, Religious"]
7 [label="7, Secular"]
8 [label="8, Pray for 20 minutes \n every night and ask \n for what you need."]
9 [label="9, Go to your religious \n service every week and \n perform the rituals."]
10 [label="10, Study for 20 minutes \n every night in an area \n where your problems are."]
11 [label="11, Find a support group \n and meet with them \n every week."]
0 -> 1 [dir="forward"];
0 -> 2 [dir="forward"];
1 -> 3 [dir="forward"];
2 -> 4 [dir="forward"];
3 -> 5 [dir="forward"];
4 -> 5 [dir="forward"];
5 -> 6 [dir="forward"];
5 -> 7 [dir="forward"];
6 -> 8 [dir="forward"];
6 -> 9 [dir="forward"];
7 -> 10 [dir="forward"];
7 -> 11 [dir="forward"];
}
</kroki>

In this case, the reader is being invited to try the techniques on offer and an evaluation of Impact might thus be made. Note that in this case we have suspended our evaluation of the rhetorical qualities of each argument and begun a new one which seeks a utilitarian appraisal of which side might offer a better personal outcome. This is in keeping with the notion of Impact as separate from the rhetorical qualities of the argument.

We could, in theory, envision someone who is not religious adopting religious practices because he concludes that it does him good. Perhaps he tried both sides and decided the religious side was the one yielding the greatest benefit. This possibility is no doubt one motivation for religious debaters to drop their metaphysical convictions and adopt a practical way to approach their disagreement with the secular side. In any event, this progression seems like a healthy outcome since metaphysical debates usually have little hope of resolution.

Impact is necessarily focused on some particular goal. If the goal is material benefit and you pray and get rich you might think prayer works. But if the goal is broader than that, you might be disturbed by the fact that you are doing something you don't really believe is true. Your goal might be to get rich without compromising philosophical integrity. In this case the object of our Impact changes to become more than simply material wealth. A clear statement of the argument thesis in terms of goals is obviously important here.

In spite of our attempts at separating impact from rhetoric we will often find ourselves with a mix of the two:
[[File:Procon.png|center|frame]]
Here we've scored the Pro argument weaker than the Con argument using our standard rhetorical measures (VRFC). Arguably the Pro argument speculates more (a type of fallacy) about what would happen if we stopped supporting Ukraine. The Con argument isn't perfect either since it seems to assume that the money saved would actually be used in some constructive way. Still, money not spent is certainly money saved so we'll mark it down only slightly. That said, it would be ridiculous to stop the argument after concluding the Con side "won". The argument is not really a rhetorical argument at all but rather a statement of Impact. One side argues for the impact of saving the money. The other argues for the impact of failing to spend the money. We may not know how events would play out in this situation but we acknowledge the risk of catastrophic consequences for failure to act. The impact score for the Pro side is thus much higher.

This is a particular case where the argument should be separated out into one that is explicitly about impact but it is not clear how best to achieve that. One way would be to allow participants to intervene by asking questions or proposing to move the debate in a more fruitful direction, perhaps by suggesting a new main contention (ie thesis).
[[File:Procon2.png|center|frame]]
A basically new debate ensues. Incentives to move the debate might include reputational points for agreeing on a more productive direction.

Let's look at an example of what this "productive" argument might look like in terms of cost-benefit-risk analysis:

[[File:Productiveargument.png|center|frame]]
The red boxes are Con arguments and the green box is the Pro argument. We can see right away why the Pro argument might have stiff opposition since it leads to an infinite cost-benefit ratio and is virtually certain to occur (it is our current policy). It is possible that other scenarios could play out within the context of supporting Ukraine but let's leave these aside for the moment.

So the Pro side looks bad until we start looking at the Con scenarios. In Scenario 1 we envision taking the $75 billion spent on Ukraine aid and providing free [[community]] college instead. Doing so provides an economic benefit in the long run so we provide an estimate for that. However, military and policy experts have said that ignoring Ukraine would result in having to contain a newly resurgent Russia and this could double our defense costs in the near term ($800 billion). The resulting cost benefit ratio is 2.9, a positive number which is undesirable. It also is the highest probability scenario, at 70%. Other scenarios involve some type of war with Russia and would involve an even greater outlay of funds, not to mention the sheer human toll of war. Only Scenario 4 envisions a minor outlay to contain a victorious Russia which would be offset by the benefit of free community college. This scenario is desirable but unlikely.

These scenarios are much like sub-arguments but stripped of any need to assess their rhetorical quality. By looking at CB ratios and probabilities we can determine which policy direction to take.

<h3>Interaction effects between arguments and sub-arguments</h3>

Although arguments have been presented as standalone entities, users may often score them after reading, and accounting for, their sub-arguments. In that case, the sub-argument's influence on the parent argument would be counted twice -- once due to the mathematical effect discussed [[#Scoring of individual arguments|above]] and twice due to the the influence the sub-argument has on the user's scoring of the parent argument.

This effect is clearly undesirable and efforts should be made to control it. The software could, for instance, be equipped with the following checks:

* If it detects that a user voted for a sub-argument and subsequently voted for an argument, it can flag the sub-argument score so it does not participate in the mathematical effect it would otherwise have on the argument. We are assuming here, of course, that a user who has voted for a sub-argument will be unable to avoid having it influence his vote for the parent argument.
* If it detects a vote for an argument but not a vote for its sub-argument, it doesn't know if the user has read the sub-argument in a way that would influence their vote for the parent argument. In such a case, the user can simply be asked if the sub-argument was read and, if so, to flag any subsequent vote by the user for the sub-argument as a non-participant in its mathematical influence on the parent argument.
* If the sub-argument does not yet exist when the vote for the parent argument is cast, the software will flag a subsequent vote for any newly developed sub-argument as a legitimate participant in the mathematical influence it has on the parent argument.

It is probably difficult to make a system like this foolproof. A user might report not having read a sub-argument that they have, in fact, read. Tracking features could, in theory, be developed to check whether this is the case and react accordingly. However, it would still be difficult to know for sure how deeply the user understands the sub-argument just based on a record that they clicked on it or had it "open". It also seems like it would be easy to overdo tracking of this kind to the point where it simply turns off an otherwise enthusiastic user. Another interesting idea is the use of word descriptors to match scores. In our system descriptors such as "Completely irrelevant", "somewhat irrelevant", etc. could be a useful way to break up corresponding numerical ranges in our 0-1 scoring system.

Argument evaluation and scoring

2024-09-25T13:49:16Z

Lem: Undo revision 2137 by Pete (talk)

Template:Transparent

2024-08-20T16:17:13Z

Lem:

Template:Transparent

2024-08-20T16:16:25Z

Lem: Created page with "<span class="mw-kroki">{{{0}}}</span>"

MediaWiki:Common.css

2024-08-19T17:44:34Z

Lem:

/* CSS placed here will be applied to all skins */

/* BEGIN: Dark Mode Maths */
/* Here we fix <math> and <kroki> tags for dark modes in Poncho and Citizen skins. */

:is(.poncho-dark-mode, .skin-citizen-dark) :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display),
:is(.poncho-dark-mode, .skin-citizen-dark) .mw-kroki img {
filter: hue-rotate(180deg) invert(1);
}

@media (prefers-color-scheme: dark) {
.skin-citizen-auto :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display),
.skin-citizen-auto .mw-kroki img {
filter: hue-rotate(180deg) invert(1);
}
}
/* END: Dark Mode Maths */

/* BEGIN: Disable site subtitles */
#siteSub {
display: none;
}
/* END: Disable site subtitles */

MediaWiki:Common.css

2024-08-19T16:32:17Z

Lem:

/* CSS placed here will be applied to all skins */

/* BEGIN: Dark Mode Maths */
/* Here we fix <math> and <kroki> tags for dark modes in Poncho and Citizen skins. */

:is(.poncho-dark-mode, .skin-citizen-dark) :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display) {
/*:is(.poncho-dark-mode, .skin-citizen-dark) .mw-kroki * {*/
filter: hue-rotate(180deg) invert(1);
}

@media (prefers-color-scheme: dark) {
.skin-citizen-auto :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display) {
/* .skin-citizen-auto .mw-kroki * {*/
filter: hue-rotate(180deg) invert(1);
}
}
/* END: Dark Mode Maths */

/* BEGIN: Disable site subtitles */
#siteSub {
display: none;
}
/* END: Disable site subtitles */

MediaWiki:Common.css

2024-08-19T16:09:03Z

Lem:

/* CSS placed here will be applied to all skins */

/* BEGIN: Dark Mode Maths */
/* Here we fix <math> and <kroki> tags for dark modes in Poncho and Citizen skins. */

:is(.poncho-dark-mode, .skin-citizen-dark) :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display),
:is(.poncho-dark-mode, .skin-citizen-dark) .mw-kroki * {
filter: hue-rotate(180deg) invert(1);
}

@media (prefers-color-scheme: dark) {
.skin-citizen-auto :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display),
.skin-citizen-auto .mw-kroki * {
filter: hue-rotate(180deg) invert(1);
}
}
/* END: Dark Mode Maths */

/* BEGIN: Disable site subtitles */
#siteSub {
display: none;
}
/* END: Disable site subtitles */

MediaWiki:Common.css

2024-08-19T16:02:01Z

Lem:

/* CSS placed here will be applied to all skins */

/* BEGIN: Dark Mode Maths */
/* Here we fix <math> tags for dark modes in Poncho and Citizen skins. */

:is(.poncho-dark-mode, .skin-citizen-dark) :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display, .mw-kroki),
:is(.poncho-dark-mode, .skin-citizen-dark) :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display, .mw-kroki) *
{
filter: hue-rotate(180deg) invert(1);
}

@media (prefers-color-scheme: dark) {
.skin-citizen-auto :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display),
.skin-citizen-auto :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display) * {
filter: hue-rotate(180deg) invert(1);
}
}
/* END: Dark Mode Maths */

/* BEGIN: Disable site subtitles */
#siteSub {
display: none;
}
/* END: Disable site subtitles */

MediaWiki:Common.css

2024-08-19T15:59:27Z

Lem:

/* CSS placed here will be applied to all skins */

/* BEGIN: Dark Mode Maths */
/* Here we fix <math> tags for dark modes in Poncho and Citizen skins. */

:is(.poncho-dark-mode, .skin-citizen-dark) :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display, .mw-kroki) *
{
filter: hue-rotate(180deg) invert(1);
}

@media (prefers-color-scheme: dark) {
.skin-citizen-auto :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display) * {
filter: hue-rotate(180deg) invert(1);
}
}
/* END: Dark Mode Maths */

/* BEGIN: Disable site subtitles */
#siteSub {
display: none;
}
/* END: Disable site subtitles */

MediaWiki:Common.css

2024-08-19T15:49:29Z

Lem:

/* CSS placed here will be applied to all skins */

/* BEGIN: Dark Mode Maths */
/* Here we fix <math> tags for dark modes in Poncho and Citizen skins. */

:is(.poncho-dark-mode, .skin-citizen-dark) :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display, .mw-kroki)
{
filter: hue-rotate(180deg) invert(1);
}

@media (prefers-color-scheme: dark) {
.skin-citizen-auto :is(.mwe-math-fallback-image-inline, .mwe-math-fallback-image-display) {
filter: hue-rotate(180deg) invert(1);
}
}
/* END: Dark Mode Maths */

/* BEGIN: Disable site subtitles */
#siteSub {
display: none;
}
/* END: Disable site subtitles */

File:A52b326fa655929fd493d2be0a7306c5 image.png

2024-08-01T18:22:31Z