Bureau Comparisons: It’s about Value, not Quantity


Bureaux all over the world are compared based on the quantity of information held. More often than not, the value that the data contributes towards reducing the risk is overlooked. In this article we will provide you with guidelines of how to compare bureaux data with one another to establish which is truly the bureau that will contribute the most to your decision process.

Where to Start?

To compare the volume and quality of data that each Credit Reference Agency (CRA) holds, a CRA comparison should be conducted. To do this, a representative sample of recent applications can be extracted. For each account, identical information (Identity Number, name, address, etc.) should be supplied to both credit reference agencies. ScoreSharp recommends a minimum of 5,000 applications should be extracted. When performing comparisons, no ‘footprint’ of a credit application is left on the CRA databases, so there is no impact on the consumer. This analysis will only indicate which bureau has the most information as at today. More information does however not necessarily mean more predictive information, i.e. the ability of the data to rank between good and bad paying customers. ScoreSharp therefore recommends that in addition to the above mentioned comparison a “Retrospective Analysis” be undertaken.

This analysis involves the extraction of a sample of mature accounts, i.e. accounts that have been on the books for longer than 12 months that can be classified as Goods and Bads, and previously declined accounts. These accounts should be processed against the various CRAs’ retrospective databases. The retrospective run will use the date of application and return only the information that was available as at that date. ScoreSharp recommends a minimum of 2500 Good Accounts, 2500 Bad Accounts and 2500 Declined Accounts be extracted for this analysis. This form of analysis is critical in understanding how effectively the data held by the CRA predicts risk on a particular portfolio.

Evaluating the Data

ScoreSharp recommends that the following analysis be conducted: –

  • Quantity of Data held on the CRAs
  • Similarity Analysis
  • Predictive Value of the Data
  • Comparing the CRA Scores (if available)
  • Other factors that should be considered

Quantity of Data held on the CRAs

Data quantity is the amount of information returned by the CRAs on your customer base, i.e. Number of Judgement Records returned etc. An example is given in Table 1 below. ScoreSharp recommends that the following be compared in this analysis:

  • Hit Rates
  • IDs Verified
  • Total Number of Judgement Records
  • Number of Judgement Records in the Last 365 Days
  • Total Number of Adverse Records
  • Number of Adverse Records in the Last 365 Days
  • Total Number of Enquiries
  • Total Number of Enquiries in the Last 365 Days
  • Number of Open Payment Profiles (CPA Members Only)
  • Number of Closed Payment Profiles (CPA Members Only)
  • Worst Status on Any Payment Profile (CPA Members Only)
  • Worst Status on Open Payment Profile (CPA Members Only)

Table 1
Although CRA1 has a lower Hit Rate more judgements were returned from this bureau then from CRA2.Figure 1

Similarity Analysis

Table 2 below demonstrates that although hit rates / data quantity may be very similar between CRAs, it does not mean that the same accounts have data hits. CRA1 had 4.44% of the judgements compared to 3.96% for CRA2. Table 2 shows that of the 444 accounts with a judgement record on CRA1 only 252 (63.64%) of them also had a judgement on CRA2. These comparisons are extremely important for the determination of a bureau strategy, i.e. single or dual calls to the CRAs as well as the risks associated with only using one CRA.

Table 2
CRA1 (Horizontal) vs. CRA2 (Vertical)
Figure 2

Predictive Value of the Data

It is important to investigate the value of CRA data in the light of assessing good and bad credit risk. An example to consider might be if a customer has a judgement against his name. The value of this reference is indicated in the tendency of the customer to be a good or a bad credit risk with your portfolio. Certainly, there are customers who have a judgement and are not bad. Thus, the question to ask is: “How effective is this information in discriminating between good and bad?”

ScoreSharp measures the predictability of bureau criteria using a formula that is based on the good / bad odds and the percentage of the population to which the information applies. This measure is referred to as the criterion strength. Strengths of 0 indicate that a criterion has no predictive strength, whereas strengths of 100+ would be predictive information. The formulas for assessing the attribute and criterion strength are given below:

Figure 3

Some examples of criterion strengths are given in Table 3.

Table 3
Criterion strengths of some bureau characteristics

Table 3To summarise, this table indicates that there are obviously considerable differences in the value of information supplied by the CRAs. This demonstrates the value and necessity of separate scoring models developed specifically to assess risk using a particular CRA’s data. This comparison will also assist in determining which CRAs’ data is more relevant, i.e. more predictive on your portfolio for assessing risk.

Comparing the CRA Scores

CRA Scores analyse the consumer credit bureau file information and produces a numeric score that is indicative of the risk associated with extending credit to an individual consumer. The CRA Scores are available online as a component of a credit report, or in batch for account management or pre-screen applications.

The most effective way of comparing the predictability of CRA Scores against each other is through the comparison of their respective Gini Coefficients. The Gini Coefficient is a measure of a scorecard’s ability to separate good accounts from bad accounts. The Gini Coefficient is a number between 0 and 100, where 0 means no ability to separate between good and bad accounts. The higher the Gini Coefficient the greater the ability of the CRA Score to separate between good and bad accounts. From Figure 5 it can be seen that at 50% of the Goods and 26.4% of the Bads will be accepted if CRA1 is used compared to 30% if CRA2 is used. CRA1 thus has a more predictive score for your portfolio.

Figure 2
Gini Coefficient Comparison

Demo Behavioural Workshop v1.0.doc

Other Considerations

When comparing CRAs the following should also be taken into consideration when deciding on which CRA to use predominantly:

  • Service Levels and Service Level Agreements
  • Price charged per enquiry
  • Speed of response
  • Values added services and products provided

Should you require any further information please do not hesitate to contact ScoreSharp to assist you in determining the most effective means of carrying out a CRA comparison: scoresharp@compuscan.co.za