Appendix B: Journal Quality Index | Department of Economics Handbook | Kent State University

Appendix B: Journal Quality Index

I.   Summary

This document describes the metric used for assessing the quality of publications in economics and business journals in the department of economics at Kent State University. We believe the metric is informative, comprehensive, consistent, easily constructed and interpreted, and externally validated.

The metric is the Journal Quality Index (JQI), a measure that assigns values to journals from 0 to 10.  The present index covers 2,869 peer-reviewed journals in economics and business, providing a substantially more comprehensive list than the previous College of Business Journal Quality list.  Journals that are externally validated by accepted metrics receive a score greater than 0 while journals that are not externally validated receive a 0, thus the index distinguishes between journals with and without external validation.  Approximately, 43% of the journals on the JQI receive a score of 0 while the remaining journals range from 0.5 to 10.

The current index was computed based on metrics from 2014.  The index will be recomputed every three years, using the most up-to-date metrics.  The formulas for computing the index may be revised as well, subject to approval from the economics department Faculty Advisory Committee.

II.   Journal Quality Index (JQI) Construction

The JQI is constructed based on two main metrics: the Article Influence (AI) percentile and the national rankings of economics and business journals in the United Kingdom (ABS) and France (CNRS).  The index also utilizes information from the national list of Australian (ABDC) to create a comprehensive list of journals, as well as the journal rankings on journal lists of peer institutions (see below for a description of the data sources).

Basic Methodology

We begin by merging the ABDC, ABS, CNRS, peer lists and the AI list.  From that, we keep all journals that appear on at least one of the ABDC, ABS, CNRS or peer lists.  In practice, this is the journals on the ABDC list which is by far the most comprehensive list.  This effectively provides the universe of economics and business journals (the remaining journals on the AI list being from outside these disciplines).  Currently, this provides a list of 2,869 journals.

We then convert both the ABS and CNRS into 5-point scales where 5 represent the extraordinarily recognized journals.  Due to missing data from journals not appearing on each list, we construct a measure Scale5 as the average of the two 5-point scales or the available scale when only one is available.   Scale5 is set to 0 if missing.

We convert the AI score to a percentile (AIP ) of the entire distribution (science and social science).  We use the entire distribution because it will more easily allow future interdisciplinary work to be evaluated. AIP is set to 0 if missing.

In the simple form, the Journal Quality Index is calculated as


which ranges from 0 to 10.  A score of 0 means that there is no outside validation of the journal quality.  Any journal with a score greater than 0 does have external validation, with varying ranges of quality.

The primary problem with the JQI is that there are some journals for which there is only information on either Scale5 or AIP which will have 0 for part of the calculation.  To account for this, missing metrics are imputed using the available Article Influence percentile, ABS/CNRS rating, and the peer institution lists.  Two imputations are calculated and the final score is the max of the non-imputed score or the average of the two imputations.  The details of the imputation are described below.  In practice the imputation procedure has little effect on the JQI scores, with over 90% of journals without missing data having an imputed value less than 1 point different than the simple index.

III.          External Data Sources

1.  Article Influence

Article influence is a measure of the influence that an article in a specific journal has on academic knowledge.  It is based on citation reports from the Journal Citation Report (JCR) for all journals in science and social sciences that are large enough to have a 5-year impact factor. The AI score is now included in the Journal Citation Reports which, in 2012, had 10,845 journals of which 9,345 had an AI influence score.  The provided AI score is normed to have an average value of 1 (just as the 5-year impact factor) but it is highly skewed. 

This metric is useful because it is comprehensive across disciplines (allowing for researchers provide documentation for interdisciplinary work) and provides a continuous measure of the influence that an article is expected to have in the universe of scientific thought.

2.  ABS/CNRS

These are two national lists of journals in economics and business that were constructed using a variety of criteria.  Each list utilizes a modified 4 point scale where journals are all given a 1-4 score (4 is high for ABS, 1 is high for CNRS) but each list denotes certain top-level journals which are distinguished for being extraordinarily recognized or influential.  There are 528 journals on both lists with an additional 332 journals only on the ABS list and 303 only on the CNRS list.

These lists provide accepted categorizations of journals that help look beyond simple metrics and consider the relative influence within different areas of research (each assigns journals to different areas of research). 

ABDC

This is the national journal list of Australian business schools.  It is by far the most comprehensive list including over 2,600 journals across a range of economics and business disciplines.  This list was utilized only to help establish the universe of economics and business journals.

3.  Peer lists

These are lists collected from peer institutions.  The lists are:

Institution

Categorization

Florida International University

Uses the Financial Times list; 45 targeted journals.

Texas Tech

Uses an A+, A, B+ system (B journals are any peer-reviewed system).

University of Connecticut

A list of targeted journals.

University of Memphis

Uses a 1, 2, 3 tier system (1 high).

University of Oklahoma

Uses an A, A- system.

University of Pittsburgh

Uses a main list and a “Plus” list (secondary journals).

Washington State University

Uses an A, A-, B+ system.

IV.  Imputation of Missing Metrics

For some journals there is either only information on Scale5 or AIP.  As a result they will have 0 for part of the calculation.  We address this problem by producing final JQI scores using imputations procedures.  There are many different ways to impute missing data. The approach outlined below was chosen for its relative simplicity.   Available data on Scale5 and AIP are combined with data from peer institution lists to impute missing scores.  This is done allowing for the imputation to be different across disciplines while consistently utilizing the available metrics.  Additionally, the imputation is done in a way that journals with both Scale5 and AIP do not lose relative ranking to other journals. 

For every journal that has at least a Scale5 or AI:

1.   We calculate the non-imputed index  where missing data is set to 0.

IV.          Discussion

Interpretation

The JQI is easy to interpret and is much more informative than the current CoB classification system.  First, the list attempts to provide close to the universe of economics and business research (much larger than the current list) but assigns a score of 0 to any journal that does not have an Article Influence Score and does not appear on either the ABS, CNRS or any peer institution list.  Thus, while these are journals in the economics and business universe, they are not journals that are recognized by any of the metrics utilized.  These journals comprise 43% of the list in 2014 (see Figure 1).

Figure 1: Distribution of Journal Quality Index, 2014

 

Second, interpreting the quality of journals with an index greater than 0 is straightforward and intuitive.  Journals that contribute substantially to scientific knowledge (high AI percentile) and are recognized highly (high ranks on ABS/CNRS) will have high scores on the JQI.  Journals that do not contribute as much to scientific knowledge or are not as widely regarded receive scores closer to 0.  Journals will have intermediate scores if they are in the middle of the AI and ABS/CNRS scales or in cases where there is a disagreement between the two metrics.  Summary statistics of the JQI are presented in Table 1.

Table 1: Summary Statistics of the Journal Quality Index, 2014

 

All Journals on JQI

Journals with JQI>0

N

2869

1634

Mean

2.6

4.6

 

 

 

Min

0

0.7

10th percentile

0

1.6

25th percentile

0

2.3

50th percentile

1.7

4

75th percentile

4.7

6.5

90th percentile

7.5

8.3

Max

10

10

It is important to note that a journal that receives a low score, but strictly greater than 0, is externally validated and thus should be considered a recognized avenue for research.  Figure 1 demonstrates that there is a higher density of journals in the 0.5 – 4 range and decreasing density as the index goes to 10, consistent with being more prestigious journals. 

Updates and Journals Not on the List

The JQI is easily updated as the ABS/CNRS lists (updated periodically), AI scores (updated annually) and peer lists (updated infrequently) change over time.  The current list is based on data as of March, 2014.  The index will be recalculated every three years to reflect changes in that time period.

Journals that are not on the list would need to be validated by some form of documentation.  These journals will tend to be either: 1) very new in which case there are not yet external metrics used to construct the index and has not been recognized on any economics or business list, or 2) outside the typical universe of economics and business research.  In the latter case, which would incorporate a wide range of interdisciplinary work, journals can be generally placed on the JQI if they are one of the 8,000 or so science or social sciences with an article influence score.