Organizations as raters

Organizations as Raters

Last time we discussed Freedom House as a ratings system for countries. We noted that people will likely delegate most of their ratings “weight” to trusted organizations (or individuals) that have the expertise in a particular area or capacity to do detailed studies that ordinary people can’t do. In fact, it is likely that many different ratings categories will be delegated to some organization or another. We might even use a ratings service of this kind for people, especially those we don’t know, to find out if they are trustworthy.

Our own society has several ratings systems both for individuals and organizations. We have a significant number of organizations that perform ratings of academic institutions, health-care facilities, consumer goods, etc. We have a large number of informal customer ratings for services and products. We are constantly being asked to fill out surveys to rate our experience in transactions. And we also have more formal systems: financial credit ratings for individuals, health ratings for individuals receiving government medical benefits, no-fly lists, a “good character” rating used by the Australian govt. to reject immigrants, etc.

Right now, the internet itself serves in many ways as a type of informal ratings system. If we want information on someone we don’t know, we can easily google them and see if anything negative turns up. We can also see if they have a LinkedIn or social media presence and make some judgements based on that. Other folks frequently provide recommendations on LinkedIn to people they have worked with, although these seem to be universally biased toward the positive. It’s not a ratings system per se but it gives us something to go on. If we’re really suspicious we can look up cases in the court system and see if the person has been arrested or convicted of any crimes. States generally maintain searchable websites of this kind.

Rating the Rater and Unfair Ratings

Our ratings system has always had the notion that raters can be rated on their ratings and their ability to rate. This is supposed to keep the system honest. But it is unlikely that members will spend a great deal of time assessing other people’s ability to rate others. In a system where ratings will take up a fair bit of time anyway, the requirement to rate the rater seems like it would be too much of an ask. But there are exceptions and these would include a rating for oneself or someone close that is deemed unfair. Then the rater can be expected to be put under quite a lot of scrutiny. In situations like these, a process would need to be developed to resolve differences. For example:

Direct communication between the ratee and the rater to discuss the disagreement over the rating.
A complaint lodged with an organization that rates other raters (as discussed above) expressing that the rating should be reviewed.
Investigation by the organization with results delivered to both parties.
If parties wish to continue, the complaint and investigative results are referred to “ratings” court for further investigation and hearing (ie “trial”).
If the trial court rules in favor of the aggrieved ratee, then the rating is expunged from the ratings system. The court may further investigate and rule on the reasons for the incorrect rating. If it is found that the rater acted out of malice or some corrupt self-interested reason, a sanction may be placed on them. If the rater simply made a mistake, or isn’t versed enough in ratings to contribute meaningfully, then an educational course of study might be recommended. Our ratings-based society will presumably educate its members on how to properly rate others from an early age. This will be a core function of the civics education in such a society.

Controlling Organizations that Rate

Let’s return for a moment to organizations that rate and ask how an organizational ratings system be controlled and kept in check? By the ratings system itself (of course). We have spoken before of a continuous circle of ratings throughout society. We presume to be using the system in the context of a direct democracy where the ratings system plays a central part in the organization of civic life. Any organization having as its function the rating of others will be subject to especially stringent review by other raters and, indeed, other organizations. In fact, such organizations will and probably should be subject to laws intended to control their behavior.

What might such laws be? The first might be to register with the community as an “official” ratings organization and to provide evidence of expertise in certain areas. The idea would be to show that the ratings organization is somehow more qualified than a random collection of the citizenry to conduct ratings on a particular topic. The idea that the organization would limit its ratings to certain areas where it had expertise would obviously be a requirement. Other rules might dictate that the organization be free of political or ideological bias. This could be assessed by some objective measure (eg membership in the Federalist society would be a disqualifier for court appointments) or by the community itself. If an organization’s bias rating gets too high they could be disqualified from their official status as a ratings organization. Other rules concerning transparency of operations, financial integrity, etc. could be put in place for these and other types of organizations which claim to serve in the public interest.

We mentioned that such organizations might also be used to rate individuals, mostly for the sake of facilitating trustworthy interactions between people who don’t know each other. This is a sensitive topic and highlights even more the need for scrutiny and rules. Such an organization would probably be able to find data on individuals from public sources, such as court records, and statistics that are willingly provided by people to the community government. Other methods might be to tap into the ratings of other individuals who know the individual in question. They might have to solicit such data or perhaps they could find it if people opened their ratings information to the public. Clearly a system such as this, given its power to disrupt the lives of community members, would require a highly disciplined system of checks.