Gain-probability diagrams as an alternative to significance testing in economics and finance

David Trafimow (Department of Psychology, New Mexico State University, Las Cruces, New Mexico, USA)
Ziyuan Wang (University of Lynchburg, Lynchburg, Virginia, USA)
Tingting Tong (Department of Mathematical Sciences, New Mexico State University, Las Cruces, New Mexico, USA)
Tonghui Wang (Department of Mathematical Sciences, New Mexico State University, Las Cruces, New Mexico, USA)

Asian Journal of Economics and Banking

ISSN: 2615-9821

Article publication date: 4 July 2023

Issue publication date: 14 November 2023

591

Abstract

Purpose

The purpose of this article is to show the gains that can be made if researchers were to use gain-probability (G-P) diagrams.

Design/methodology/approach

The authors present relevant mathematical equations, invented examples and real data examples.

Findings

G-P diagrams provide a more nuanced understanding of the data than typical summary statistics, effect sizes or significance tests.

Practical implications

Gain-probability diagrams provided a much better basis for making decisions than typical summary statistics, effect sizes or significance tests.

Originality/value

G-P diagrams provide a completely new way to traverse the distance from data to decision-making implications.

Keywords

Citation

Trafimow, D., Wang, Z., Tong, T. and Wang, T. (2023), "Gain-probability diagrams as an alternative to significance testing in economics and finance", Asian Journal of Economics and Banking, Vol. 7 No. 3, pp. 333-357. https://doi.org/10.1108/AJEB-05-2023-0045

Publisher

:

Emerald Publishing Limited

Copyright © 2023, David Trafimow, Ziyuan Wang, Tingting Tong and Tonghui Wang

License

Published in Asian Journal of Economics and Banking. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode


1. Introduction

Consider a medical scenario implying a government financing decision. Suppose researchers invent a medicine to treat a disease. Should the government invest to render the medicine more widely available to its citizens? At present, a widely accepted procedure would be to perform a significance test to determine if the medicine is effective at a statistically significant level. If so, that would constitute a reason to make the investment; if not, the government would not invest. However, significance testing is a poor way to evaluate the effectiveness of the medicine. It is quite possible for a medicine not to be effective, but nevertheless achieve statistical significance, if the sample of cases is sufficiently large. This is because statistical significance depends on two items: the size of the effect and the sample size. It is possible for a sufficiently large sample size to compensate for even an extremely small effect. The p-values upon which significance testing depends confound the effect sizes researchers obtain with the sample sizes they collect. In fact, McQuitty (2004, 2018) famously argued against too large sample sizes on the grounds that they ensure statistical significance even when there is not much of an effect. Of course, a problem with this argument is that statistical significance is not the only matter of concern. There is the much more important issue of the extent to which sample sizes are sufficiently large to engender confidence that the sample statistics provide good estimates of population parameters. The larger the sample size, the better the estimate. Thus, from the point of view of estimation, McQuitty's famous advice is contraindicated. In addition, the fact that the advice was so well received by business and economics researchers and cited over 1,000 times and received an accolade by Journal of Global Scholars of Marketing Science, points to one of many problems with significance testing thinking (Trafimow et al., 2021). There have been many criticisms of significance testing, including several reviews (Hubbard, 2016; Ziliak and McCloskey, 2016; Trafimow, 2019) and 43 articles in the 2019 special issue of The American Statistician, the highly respected journal of The American Statistical Association. Because the disadvantages of significance testing have been covered so extensively, there is no point in rehashing them here. Rather, the present goal is to present a more useful alternative.

In the medical scenario, there are many factors to consider. These include the seriousness of the disease, the cost of the medicine and many others. In addition, and crucially, there is the issue of how well the medicine works. Put in the form of a question, what is the probability that a randomly selected person who takes the medicine will be better off, or worse off, and by how much, with the medicine than without it?

Nor must we restrict ourselves to economics issues pertaining to medicine. If an economic intervention is proposed to increase people's living space, what is the probability that a randomly selected person will be better off, or worse off, and by how much living space, with the intervention than without it? Or if a government wishes to institute a policy to increase income in a poor area, what is the probability that a person will be better off, or worse off, and by how much income, with the new policy than without it? The issue of probabilities of being better off, or worse off, by varying degrees, is relevant to many potential economics applications.

To estimate the probability of being better off, or worse off, by varying degrees, with the intervention or policy change than without it, it is necessary to look carefully at the data to determine the distribution. We will consider two families of distributions: skew normal distributions, which include normal distributions, and delta log-skew-normal (LSN) distributions, which include lognormal distributions. The subsequent section includes crucial equations, with subsections for skew normal and delta LSN distributions. Following that, we provide examples of how to use the equations to construct gain-probability (G-P) diagrams to draw conclusions far beyond that which significance tests allow (Tong et al., 2022; Trafimow et al., 2022; Wang et al., 2022).

2. Probability of being better off or worse off by how much? Skew normal distributions

Whereas normal distributions have two parameters, mean μ and standard deviation σ; skew normal distributions have three parameters. The location ξ replaces the mean, the scale ω replaces the standard deviation, and there is a shape (or skewness) parameter α. When the shape parameter equals 0 the distribution is normal, and the location equals the mean and the scale equals the standard deviation. But when the shape parameter does not equal 0, the location does not equal the mean and the scale does not equal the standard deviation. In symbols, when α = 0, ξ = μ and ω = σ; but when α ≠ 0, ξμ and ωσ. The family of normal distributions is a subset of the family of skew normal distributions. Thus, the family of skew normal distributions is more generally applicable than the family of normal distributions.

Let us now derive the probability that a randomly chosen person from one skew normal population has a higher score on the dependent variable than a randomly chosen person from another skew normal population. We will assume dependent populations at any level of correlation between −1 and +1, with independent populations (correlation = 0) as a special case.

D2.1.

Azzalini and Valle (1996) A random vector X=(X1,,Xk) is said to have an k-dimensional multivariate skew normal distribution with the vector of location parameter μ=(μ1,,μk)Rk, the scale parameter of the positive definite matrix Σ, and the vector of skewness (shape) parameters α=(α1,,αk), denoted as XSNk(μ, Σ, α), if its density function (pdf) is given by

(2.1) fX(x)=2ϕk(x;μ,Σ)ΦαΣ1/2(xμ),
where ϕk(x; μ, Σ) is the density of the k-dimensional multivariate normal distribution Nk(μ, Σ) with mean μ and covariance matrix Σ, and Φ(z) is the cumulative distribution function (cdf) of the standard normal random variable ZN(0, 1). Here xis the transpose of the vector xRk.

2.1 Calculation of δab = P(Z > 0) in independent distributions

As we mentioned above, Let XSN(ξ1, ω1, α1), YSN(ξ2, ω2, α2) and we assume that X and Y are independent. We want to obtain the distribution of the linear combination, Z = X + aY + b, of X and Y for specified aR and bR and then calculate δab = P(Z > 0). We need to find the distribution of Z first, which is given below.

There are two advantages of using Z = X + aY + b. Both advantages stem from the possibility that a researcher might be interested in more than simply a probabilistic advantage for one group over another. A researcher might be interested in the probabilities of being better off, or worse off, to varying degrees. The equation provides two ways to assess this. One way is by setting a = −1 and varying b, to obtain the probability that a randomly selected person from one condition will score higher than a randomly selected person from another condition, by the amount the researcher specifies by setting b to that value. This is the most straightforward advantage.

A second advantage is that the equation provides a way to assess the probability of being better off, or worse off, by varying degrees, in terms of multiples. For example, what is the probability that a randomly selected person from one group will score twice as much, thrice as much and so on, as a randomly selected person from the other group? The equation facilitates such an assessment. The researcher merely sets b = 0 and lets a vary at multiples of interest.

Theorem 2.1.

Let XSN(ξ1,ω12,α1), YSN(ξ2,ω22,α2). First we assume that two skew normal populations are independent. Then the probability density function (pdf) of Z = X + aY + b is

(2.2) fZ(z)=cϕ(z;ν,τ2)Φ2[B(zν);02,Δ],
where
c=Φ21(02;02,Δ+τ2BB),ν=ξ1+aξ2+b,τ2=ω12+a2ω22,B=d/τ2,
02=00,Δ=1+α12001+α22ddτ2,d=α1ω1aα2ω2.

Here ϕ(z; ν, τ2) is the pdf of the normal distribution N(ν, τ2) with mean ν and variance τ2, and Φ2(u; 02, Δ) is the cumulative distribution function (cdf) of the bivariate normal distribution with mean vector 02 and covariance Δ.

The proof of Theorem 2.1 is given in Appendix. Note that the pdf of Z given in Equation (2.2) is the special case of the closed skew normal density. For details of closed skew normal distributions, see Gupta et al. and Zhu et al.

Now we can compute the probability P(Z > 0), which is given by

(2.3)δab=Pr(Z>0)=0cϕ(z;ν,τ2)Φ2[B(zν);02,Δ]dz.

Density curves of Z for different parameters and their corresponding P(Z > 0) are given in Figures 1 and 2, respectively. These values can be instantiated into the equations presented earlier, but there is an easier way too. We provide a freely available online calculator at https://probab.shinyapps.io/inde_prob/

2.2 Calculation of δρab = P(U > 0) in dependent distributions

Now we consider the bivariate skew normal random vector X=(X1,X2)SN2(ξ,Σ,α), with location parameter ξ, scale parameter Σ and skewness parameter α given by

ξ=ξ1ξ2,Σ=ω12ρω1ω2ρω1ω2ω22,α=α1α2.

Then we have the following result and its proof.

Theorem 2.2.

Let XSN2(ξ, Σ, α) given above and consider U = X1 + aX2 + b with a,bR. Then the pdf of U is

(2.4) fU(u)=2ϕ(u;ξ1+aξ2+b,ω12+a2ω22+2aρω1ω2)×Φα*(uξ1aξ2b)(ω12+a2ω22+2aρω1ω2)1/2,
where
α*=α1d1+α2d2(1α12α22)k2σ2(α1d1+α2d2)2

with d1=ω12+ω1ω2(1ρ2+aρ), d2=aω22+ω1ω2(a1ρ2ρ), k=ω12+ω22+21ρ2ω1ω2 and σ2=ω12+a2ω22+2aρω1ω2.

The proof of Theorem 2.2 is given in Appendix.

The probability δρab = P(U > 0) is given by

(2.5)δρab=P(U>0)=0fU(u)du.

Density curves of U for different parameters and their corresponding probability P(U > 0) are given in Figures 3 and 4, respectively. For example, in Figure 3, we can see that the density curves are affected by the location parameter ξ2 = 0, 2, 4, 2 with other parameters specified and ρ = 0.25, together with their corresponding probabilities P(U > 0) = 0.433816, 0.566184, 0.226627, 0.10565, respectively. We provide a different online calculator, still freely available online at https://probab.shinyapps.io/ProbU/

2.3 Estimation of δab = P(Z > 0) and δρab = P(U > 0)

The above two theorems imply that if the location, scale and shape parameters are known in two populations, one can calculate the probabilities δab = P(Z > 0) and δρab = P(U > 0). But in most situations in real life the population parameters are not known. Thus estimating δab and δρab is necessary to be studied. The parametric estimation of P(X > Y) for normal distributions in the context of probabilistic environmental risk assignment was discussed by Jacobs et al. Here, we consider the method of moment estimator (MME) and maximum likelihood estimator (MLE). The estimators obtained in this two ways are denoted by δ^abMME and δ^abMLE respectively.

Method of moment estimator (MME) Equation (2.6), below, relates location to mean and scale to standard deviation.

(2.6)ξ=μ2πδωandω2=σ212πδ21,
where δ=α/1+α2. To obtain parameter estimators from samples, it is necessary to obtain an estimate of delta δ, which is the moment estimate δ^ in Equation (2.7) below.
(2.7)|δ^|=π2|γ^1|23|γ^1|23+((4π)/2)23
where γ^1 is the sample skewness and the sign of δ^ is the same as the sign of γ^1. In turn, it is easy to obtain an estimate of the shape parameter α using Equation (2.8):
(2.8)α^=δ^1δ^2.

Rewriting Equation (2.6) in terms of sample estimates, as opposed to population parameters, renders Equation (2.9).

(2.9)ξ^=X¯2πδ^ω^andω^2=S212πδ^21.

Then we can obtain the δ^abMME by substituting the MMEs of ξ1, ξ2, ω1, ω2, α1 and α2 in Equation (2.3).

Maximum likelihood estimator (MLE) The log-likelihood function for the skew normal distribution is given by

(2.10)log(L)=nlog2ωn2log(2π)12i=1nxiξω2+i=1nlogΦαxiξω

To derive the MLEs of the parameters in skew normal distribution, we take the partial derivatives of the lnL functions with respect to parameters of interest and equal them to zero. Then the corresponding MLEs are obtained by solving the following equations:

(2.11)logLξ=i=1nxiξωαi=1nϕαxiξωΦαxiξω=0,logLω=n+i=1nxiξω2αi=1nϕαxiξωΦαxiξωxiξω=0,logLα=i=1nϕαxiξωΦαxiξωxiξω=0.

By the invariance property of MLEs, we obtain δ^abMLE by substituting the MLEs of ξ1, ξ2, ω1, ω2, α1 and α2 in Equation (2.3).

To derive δ^ρabMME and δ^ρabMLE under matched data setting, instead of estimating ξ1, ξ2, ω1, ω2, α1 and α2 we estimate ξ = ξ1 + 2 + b, σ2=cΣc=ω12+a2ω22+2aρω1ω2 and

(2.12)α*=δ*1δ*2=α1d1+α2d2(1α12α22)k2σ2(α1d1+α2d2)2

by using the same methods as in the independent setting. Then by substituting the MMEs and MLEs of ξ, σ2 and α in Equation (2.5), we can get δ^ρabMME and δ^ρabMLE.

3. Probability of being better off or worse off by how much? Delta log-skew-normal distributions

The log-normal distributions is very popular in both theory and applications of probability and statistics. However, when modeling complex random phenomena in many applied areas, there are a real need of more flexible models which extend the log-normal. Lin and Stoyanov introduced the LSN distribution which is an extension for modeling positive data using log-normal models. A positive random variable X is LSNly distributed (i.e. XLSN(μσ, α)) if the logarithm of X is skew normally distributed with location parameter μ, scale parameter σ and slant parameter α. Let φ(.) and Φ(.) be respectively probability density function (pdf) and cumulative distribution function (cdf) of the standard normal distribution, then we have that

(3.1)fX(x)=2xσφ(logx;μ,σ2)Φαlogxμσ,x>0.

Density curves, fX(x), of X for parameters μ = 0, σ = 0.5 with different values of α = −1, 0, 1 are given in Figure 5. From Figure 5, we can see that the slant parameter α plays an important role in LSN family. The mean and variance of X are respectively given by

E(X)=2expμ+σ2/2Φ(ασ),V(X)=2exp2μ+σ2exp(σ2)Φ(2ασ)2Φ2(ασ).

The following is the definition of the multivariate LSN distribution.

D3.1.

The random vector X = (X1, …, Xp)′ is said to have a p-dimensional LSN distribution if log  XSNp(μ, Σ, α), the multivariate skew-normal distribution with location parameter μ, scale matrix Σ, and the skewness parameter α, denoted by XLSNp(μ, Σ, α). The pdf of X is

(3.2) h(x)=2i=1pxi|Σ|12φp(logx;μ,Σ)Φ[αΣ1(logxμ)],ifxRp+,0otherwise,
where x = (x1, …, xp)′, log  x = (log  x1, …, log  xp)′, and Rp+={x|xi>0 for i=1,,p}.

Note that, in this paper, we only consider the family of the bivariate distributions, p = 2, for both dependent and independent cases.

In life sciences it is common for data to contain a relatively large number of zeros. In many real situations, it is appropriate to model such data using a LSN distribution for the positive values, together with an additional probability mass at all zeros. This type of distribution is commonly referred to as the delta LSN, defined below.

D3.2.

The random variable X is said to have a delta-LSN distribution with parameters δ, μ, σ, and α, denoted as XDLSN(δ, μ, σ2, α), if the pdf of X is given by

(3.3) gX(x)=0ifx<0,δifx=0,(1δ)fX(x),ifx>0,
where fX(x) is given in Equation (3.1). Similarly, the random vector (X1,X2) is said to have a bivariate delta LSN distribution with parameters δ=(δ0,δ1,δ2), μ, Σ, and α, denoted as XDLSN2(δ, μ, Σ, α) if its cdf of X is given by
(3.4) P(X1<0,X2<0)=0,ifx1<0,x2<0,P(X1=0,X2=0)=δ0,ifx1=0,x2=0,P(X1x1,X2=0)=δ1F(x1),ifx1>0,x2=0,P(X1=0,X2x2)=δ2G(x2),ifx1=0,x2>0,P(X1x1,X2x2)=1i=02δiH(x1,x2),ifx1>0,x2>0,
where 0 ≤ δi < 1, i = 0, 1, 2, and 1i=02δi>0, F(x1) and G(x2) are the cdf's of X1LSN(μ1,σ12,α1) and X2LSN(μ2,σ22,α2), respectively, and H(x1, x2) is the joint cdf of XLSN2(μ, Σ, α).

Note that the requirement of 1i=02δi>0 is necessary as XLSN2(μ, Σ, α) is assumed. Also if X1 and X2 are independent, then from Equation (3.4), we can obtain that

X1DLSN(δ*1,μ1,σ12,α1)andX2DLSN(δ*2,μ2,σ22,α2)

with δ0 = δ*1δ*2, δ1 = (1 − δ*1)δ*2, δ2 = δ*1(1 − δ*2) and H(x1, x2) = F(x1)G(x2) for all x1, x2 > 0.

3.1 Linear combination of independent delta log-skew-normal distributions

Let XDLSN(δ1,μ1,σ12,α1), YDLSN(δ2,μ2,σ22,α2) and we assume that X and Y are independent. We want to obtain the distribution of the linear combination, Z = X + aY + b, of X and Y for specified constants, a ≠ 0 and bR; and then find P(Z > 0), the probability of Z > 0.

Theorem 3.1.

Let XDLSN(δ1,μ1,σ12,α1), YDLSN(δ2,μ2,σ22,α2) and assume that X and Y are independent. Then the pdf of Z = X + aY + b is

  • (i) for a < 0,

(3.5) fZ(z)=zbafX(zayb)fY(y)dy1afYzba,ifz<b,δ1δ2ifz=b,0fX(zayb)fY(y)dy+fX(zb),ifz>b.
  • (ii) for a > 0,

(3.6) fZ(z)=0ifz<b,δ1δ2ifz=b,0zbafX(zayb)fY(y)dy+1afYzba+fX(zb),ifz>b.
where fX(x), fY(y) are pdf's of X when x > 0 and Y when y > 0, respectively.

The proof of Theorem 3.1 is given in Appendix. Note that if a = 0, then Z = X + b so that we only have the univariate case.

Now we can compute the probability P(Z > 0), which is given by.

  • (i) for a < 0,

(3.7)P(Z>0)=00fX(zayb)fY(y)dydz+0fX(zb)dz,ifb<0,b0fX(zayb)fY(y)dydz+bfX(zb)dz+0bzbafX(zayb)fY(y)dydz0b1afYzbadz,ifb0.
  • (ii) for a > 0,

(3.8)P(Z>0)=00zbafX(zayb)fY(y)dydz+01afYzbadz+0fX(zb)dz,ifb<0,b0zbafX(zayb)fY(y)dydz+b1afYzbadz+bfX(zb)dz,ifb0.

Here is a link of shiny apps and its corresponding R code is available upon the request: https://dlnprobability.shinyapps.io/dlsn_independent/.

3.2 Bivariate delta log-skew-normal distribution

Now we consider the bivariate delta LSN random vector X=(X1,X2)DLSN2(δ,μ,Σ,α), given by (3.4) with parameters given by

δ=δ0δ1δ2,μ=μ1μ2,Σ=σ12ρσ1σ2ρσ1σ2σ22,α=α1α2.

Then we have the following result and its proof is given in Appendix.

Theorem 3.2.

Let XDLSN2(δ, μ, Σ, α) given above and consider U = X1 + aX2 + b with constants a ≠ 0 and bR. Then the pdf of U is

  • (i) for a < 0

(3.9) fU(u)=uba(1δ0δ1δ2)h(uax2b,x2)dx2δ2aguba,ifu<b,δ0ifu=b,0(1δ0δ1δ2)h(uax2b,x2)dx2+δ1f(ub),ifu>b.
  • (ii) for a > 0

(3.10) fU(u)=0ifu<b,δ0ifu=b,0uba(1δ0δ1δ2)h(uax2b,x2)dx2+δ2aguba+δ1f(ub),ifu>b.
where 0 ≤ δi < 1, i = 0, 1, 2, and 1i=02δi>0, f(x1) and g(x2) are the pdf's of X1LSN(μ1,σ12,α1) and X2LSN(μ2,σ22,α2), respectively, and h(x1, x2) is the joint pdf of XLSN2(μ, Σ, α).

Here is a link of shiny apps and its corresponding code is available upon the request: https://dlnprobability.shinyapps.io/dlsn_dependent/.

4. Applications: invented examples

To explain the possibilities, we invented examples. First, we examine skew normal examples and then lognormal examples.

4.1 Skew normal examples

Suppose an economist is interested in comparing salaries of males versus females in a particular geographical area. In this geographical area, based on large samples of randomly selected males and females, suppose the mean salaries are $45,000 for both males and females, the standard deviations are $1,000 for each, with skews of 0.5 and −0.5 for males and females, respectively. Because the mean salary is the same for males and females, there seems no reason to promote a policy change to redress inequality in salaries.

However, let us convert the foregoing statistics to estimates of skew normal parameters: location, scale and shape. After making the conversion, the estimated locations are $43,947.79 and $46,052.21 for males and females, respectively; the estimated scales are $1,451.601 for both males and females (the squared value is $2107144.694); and the estimated distribution shapes are 2.173758 and − 2.173758 for males and females, respectively. We saw earlier that, going by means, there is no relative advantage for males or females. However, we now see that going by locations, there is a relative disadvantage for males relative to females. The difference in means implies a neutral policy whereas the difference in locations implies a policy to increase male salaries to bring them to the level of female salaries. However, neither difference answers the real question, which is: What is the probability of males being advantaged or disadvantaged, relative to females and by how much? Fortunately, the foregoing skew normal mathematical equations allow us to answer this question, especially with the aid of the program that can be accessed via the following link: https://probab.shinyapps.io/inde_prob/.

Figure 6 contains a G-P diagram that provides the probabilities that males will be advantaged (black bars) or disadvantaged (gray bars) relative to females. As will be seen presently, G-P diagrams provide the advantage that researchers can assess probabilistic advantages or disadvantages, at varying degrees of extremity. Note that each gray bar is higher than its corresponding black bar, demonstrating a consistent advantage for females over males. For example, the probability that a randomly selected male would have an advantage of 0–500 is 0.134 whereas the probability that a randomly selected female would have a similar advantage is 0.143. In general, there is asymmetry favoring females over males. Thus, using skew normal statistics to construct a G-P diagram can lead to different conclusions than remaining with normal statistics. Note, too, that if we perform a significance test on the means, the typical t-test would come out consistent with the null hypothesis of no difference whereas Figure 1 shows a very clear difference.

Or consider another example to illustrate the potential importance of seeming trivial skewness. Consider the deductible for doctor visits based on two insurance companies labeled INSURE-A and INSURE-B. For both insurance companies, the location is $60 and the scale is $10 (squared scale is $100). The difference is that the skewness for INSURE-A is 0.10 whereas it is −0.10 for INSURE-B. The small skewness magnitude is so small for both companies that the distributions are very near normal, and typical statisticians and data analysts would treat the distributions as normal. The obvious conclusion is that the two insurance companies are equivalent. However, constructing a G-P diagram suggests a very different conclusion.

Figure 7 shows that despite the two insurance companies having equivalent locations, scales and skews seemingly only trivially different from 0, the G-P diagram shows that a randomly selected doctor's visit is much more likely to be varying degrees more expensive with INSURE-A than with INSURE-B. Each black bar, representing INSURE-A resulting in more expensive payments, is substantially higher than its corresponding gray bar, representing INSURE-B resulting in more expensive payments. The G-P diagram is quite asymmetric; the black bar (INSURE-A) probabilities are higher than corresponding gray bar (INSURE-B) probabilities. Thus, if the goal is to reduce the expected payment amount, the G-P diagram indicates INSURE-B is a substantially statistically better bet than is INSURE-A, despite the seeming equivalence based on similar locations. More generally, Figure 7 shows that seeming trivial skewness differences can have extreme consequences.

4.2 Lognormal examples

Consider time wasted per hour at work in two different industries. Suppose that time wasted is lognormally distributed in companies in Business A and Business B. In Business A, the mean time wasted is 11.373, the standard deviation is 2.888 and the variance is 8.342. In Business B, the mean is 11.078, the standard deviation is 12.376 and the variance is 153.158. The effect size is 0.04, which is trivial, so the obvious conclusion is that type of business has little to do with time wasted per hour. Too, with such a small effect size, a significance test likely would come out insignificant with typical sample sizes.

However, let us now consider lognormal statistics. The log transformed mean and standard deviation for Business A is 2.4 and 0.25 (variance is 0.0625), respectively. These values are 2.0 and 0.9 (variance is 0.81) for Business B. Thus, the probability that a randomly selected person in Business A will waste more time than a randomly selected person in Business B is approximately 0.67, despite the miniscule conventional effect size in the previous paragraph. In addition, it is possible to construct a G-P diagram with more nuanced information. One capability of G-P diagrams is that effects can be reported as multiples. For example, it is possible to concern ourselves with the probability that time-wasting is twice, thrice, etc. that in one business relative to the other. The G-P diagram renders clear that which would be obscured otherwise; time-wasting is much more of a problem for Business A than for Business B (see Figure 8).

Imagine two large cities, City A and City B and an economist is concerned with comparing commuting times. Suppose commuting times are lognormally distributed and the mean commuting time for City A is 41.264 min, the standard deviation is 74.054 and the variance is 5484.041. For City B, these values are 30.114, 33.641 and 1131.691, respectively. There seems an obvious advantage for City B over City A in that the mean commute is shorter for City B than for City A by 11.15 min.

The mean of the logarithmically transformed distribution for City A is 3.0, the standard deviation is 1.2 and the variance is 1.44. For City B, these values are 3.0, 0.90 and 0.81, respectively. The probability that a randomly selected person from City A would have a shorter commuting time than a randomly selected person from City B is 0.50; despite appearances based on the previous paragraph, there is no probabilistic advantage for City A over City B. Moreover, the G-P diagram is symmetric; as each black bar in Figure 9 is the same size as its corresponding gray bar, there is no probabilistic advantage for either city, no matter the multiple under consideration.

Consider another example, using the same log normal parameter values as the previous example, but with the addition of considering shape. Suppose that the shape parameter for City A is 0.10 and the shape parameter for City B is −0.10. In that case, the probability that a randomly selected person in City A would have a longer commuting tie than a randomly selected person in City B is 0.54, not 0.50. And the probability that a randomly selected person in City B would have a longer commuting time than a randomly selected person in City A is 0.46. Thus, in contrast to the previous example, the slight difference in shape parameters leads to a clear probabilistic disadvantage for City A relative to City B. In addition, in contrast to Figure 9 that is symmetrical, Figure 10 that illustrates the present example is asymmetric.

5. Application: real data

In the previous section, we examined invented examples to demonstrate the lessons that can be learned by constructing G-P diagrams. In this section, we present analyses of real data.

5.1 NBA guards (2022–2023 regular season)

We downloaded data from the following website: https://www.espn.com/nba/stats/player/_/season/2023/seasontype/2/table/defensive/sort/avgSteals/dir/desc. Among other types of information, it contains data for minutes played by point guards and shooting guards. Let us compare them using traditional normal statistics versus skew normal statistics.

The means for point guards and shooting guards are 24.0 and 19.8, respectively. The standard deviations are 8.8 and 9.4, respectively. In addition, the skews are −0.3 and 0.2, respectively. Thus, point guards average 4.2 more minutes per game than shooting guards.

We might ask about how likely a randomly chosen point guard would average more minutes than a randomly chosen shooting guard. Based on the difference in averages, we might guess that the probability that randomly chosen point guard would average more minutes than a randomly chosen shooting guard would be greater than 0.50, but perhaps not too much more than that. But rather than guess, let us run out the calculations.

As usual, the first step is to find the skew normal values for location, scale and shape. For point guards these are 31.4, 11.5 (scale squared is 131) and −1.4, respectively. For shooting guards, these are 13.1, 11.6 (scale squared is 134) and 1.1, respectively. Based on these values, the probability that a randomly selected point guard would play more minutes per game than a randomly selected shooting guard is 0.63. Figure 11 provides a G-P diagram. An interesting aspect of the figure is that the middle two bars are not very far apart, thereby indicating that at small amounts of difference in minutes played, there is only a slight probabilistic advantage for point guards over shooting guards. However, comparing black against gray bars at greater extremes indicates a strong probabilistic advantage for point guards over shooting guards. The G-P diagram suggests a more subtle substantive story than does a mere difference in means or difference in locations.

5.2 Living space for males versus females

Hyman et al. (2002) obtained data comparing living space for their male and female participants, in square feet. The mean amount of living space for men is 2300 and the standard deviation is 705. The mean amount of living space for women is 2186 and the standard deviation is 671. The effect size is miniscule and not statistically significant: Cohen's d = 0.04, t(564) < 1. However, the data are lognormally distributed and a G-P diagram suggests a very different conclusion. To commence, let us find the means and standard deviations after a logarithmic transformation. The transformed mean and standard deviation for males is 7.69 and 0.31 (variance = 0.0961), respectively. The transformed mean and standard deviation for females is 7.64 and 0.32 (variance = 0.1024), respectively. Thus, the probability that a randomly selected male will have more living space than a randomly selected female is 0.54 whereas the probability that a randomly selected female will have more living space than a randomly selected male is only 0.46, a more impressive difference than is suggested by the Cohen's d or the nonsignificant p-value.

For a more fine-grained analysis, Figure 12 provides the G-P diagram. Because it is convenient to express gender effects with respect to living space in multiples, Figure 12 is in terms of multiples. That is, the black bars provide the probabilities that a randomly selected male will have 1.0 to 1.5 times as much living space as a randomly selected female, 1.5 to 2.0 times as much living space and so on. The gray bars provide analogous probabilistic advantages for females. Each black bar is larger than its corresponding gray bar thereby indicating that although the probabilistic advantage for males over females is not large, it is not trivial either. Again, we see that a G-P diagram provides both more valid and more nuanced conclusions than typical differences between means, Cohen's d, or significance tests.

Consider precipitation in Buffalo for 306 days starting in June in 2019 versus 2020 [https://www.ncdc.noaa.gov/cdo-web/]. Because there are days when there is no precipitation, the data fall under the category of delta skew lognormal distributions. Thus, there are the following bullet-listed summary parameter estimates for each year:

  • locations are 2.8712 for 2019 and 2.8327 for 2020,

  • squared scales are 3.884841 for 2019 and 3.806401 for 2020,

  • shapes are −2.929 for 2019 and -3.019 for 2020 and

  • probabilities of zeroes (lack of precipitation for a day) are 0.49346 for 2019 and 0.59150 for 2020.

This is an especially interesting example because (a) it engages all the parameters of delta skew lognormal distributions and (b) ties are possible. A tie can occur if there is no precipitation on a randomly selected day in 2019 or 2020. The probability of more precipitation on a randomly selected day in 2019 than on a randomly selected day in 2020 is 0.404. The probability of more precipitation on a randomly selected day in 2020 than on a randomly selected day in 2019 is 0.304 and the probability of a tie is 0.292. Thus, there is a general probabilistic advantage for 2019 over 2020, assuming precipitation is positive. (If precipitation is negative, then there is a probabilistic disadvantage for 2019 relative to 2020. However, we will assume that precipitation is positive.)

Figure 13 shows a G-P diagram. However, the diagram differs from the others because ties are possible. The black bar in the middle shows the probability of ties. The dark gray bars show the probability of more precipitation in 2019 than in 2020, by varying millimeters. And the light gray bars show the probability of more precipitation in 2020 than in 2019, by varying millimeters. The figure shows that the probabilistic advantage for 2019 over 2020 is clear both at small amounts and at large ones.

6. Discussion

The G-P diagrams based on both invented and real data indicate important lessons for economics researchers. One lesson is that both summary statistics and significance tests can be very misleading. It is possible for a difference in means to hide that there is no probabilistic advantage for either group over the other. However, it is possible, too, for a lack of difference in means to hide a very sizable probabilistic advantage for one group over the other. One solution is to use more appropriate summary statistics. For example, if the data are obtained from a skew-normal distribution, locations, scales and shapes can be better than means and standard deviations. However, even distribution-appropriate sample statistics can be misleading. For instance, we have seen an example where the locations and scales of two groups are the same and the difference in skews very small. Nevertheless, the G-P diagram (Figure 7) indicated a very large probabilistic difference.

Although it is dramatic to show that traditional statistics and G-P diagrams can come to opposing conclusions, G-P diagrams have another advantage. Specifically, G-P diagrams can support subtle conclusions that cannot be addressed by either summary statistics or significance tests. Figure 11 provides a nice case in point. It shows that at relatively small differences in minutes played, there is only a miniscule probabilistic advantage for point guards over shooting guards. But at more extreme differences in minutes played, the probabilistic advantage for point guards increases substantially.

In conclusion, there is no need for economics researchers to constrain themselves by depending solely on summary statistics and null hypothesis significance tests. As we have seen, G-P diagrams provide opportunities to contradict the seeming implications of summary statistics or null hypothesis significance tests. In addition, even in cases where there is no contradiction, such as the data pertaining to minutes played by National Basketball Association (NBA) point guards and shooting guards, G-P diagrams provide much more nuanced information than do summary statistics and null hypothesis significance tests. We hope and expect that future economics researchers will avail themselves of the potential advantages to be enjoyed by exploiting the capabilities of G-P diagrams for providing subtle probabilistic information about comparisons of interest.

Figures

The density of Z when location parameters of both populations equal 0, the scale parameters for first and second population equal 1 and 2 respectively

Figure 1

The density of Z when location parameters of both populations equal 0, the scale parameters for first and second population equal 1 and 2 respectively

The density of Z when location parameters for first and second population equal −2 and 0 respectively, the scale parameters for first and second population equal 1 and 2 respectively

Figure 2

The density of Z when location parameters for first and second population equal −2 and 0 respectively, the scale parameters for first and second population equal 1 and 2 respectively

The density of U when location parameters of the first population equal 0, the scale parameters for first and second population equal 1 and 2 respectively

Figure 3

The density of U when location parameters of the first population equal 0, the scale parameters for first and second population equal 1 and 2 respectively

The density of U when location parameters of the first and second population equal 0 and 4, the scale parameters for first and second population equal 1 and 2 respectively

Figure 4

The density of U when location parameters of the first and second population equal 0 and 4, the scale parameters for first and second population equal 1 and 2 respectively

The density curves of log-skew normal distribution with μ = 0, σ = 0.5 and values of α = 1, 0, − 1

Figure 5

The density curves of log-skew normal distribution with μ = 0, σ = 0.5 and values of α = 1, 0, − 1

The probability of males or females being better off, or worse off, to varying degrees, with respect to salaries

Figure 6

The probability of males or females being better off, or worse off, to varying degrees, with respect to salaries

The probability of being better off, or worse off, by varying degrees, with respect to co-payments for doctor visits

Figure 7

The probability of being better off, or worse off, by varying degrees, with respect to co-payments for doctor visits

The probability of being better off or worse off, by differing multiples

Figure 8

The probability of being better off or worse off, by differing multiples

The probability of being better off by varying multiples, with respect to commuting times, depending on city

Figure 9

The probability of being better off by varying multiples, with respect to commuting times, depending on city

Probability of being better off or worse off, by varying multiples, with respect to commuting times, depending on city and taking shape parameters into account

Figure 10

Probability of being better off or worse off, by varying multiples, with respect to commuting times, depending on city and taking shape parameters into account

Probability of varying degrees of advantage (or disadvantage) for shooting guards or point guards with respect to minutes played

Figure 11

Probability of varying degrees of advantage (or disadvantage) for shooting guards or point guards with respect to minutes played

Probability that males or females are better off, by varying multiples, with respect to living space

Figure 12

Probability that males or females are better off, by varying multiples, with respect to living space

Probability that there is a precipitation advantage, by varying degrees, for 2019 or 2020, or that there is no advantage for either year (black bar)

Figure 13

Probability that there is a precipitation advantage, by varying degrees, for 2019 or 2020, or that there is no advantage for either year (black bar)

Appendix

Source(s): Appendix by authors

In this section, we provide detailed proofs of Theorem 2.1, 2.2, 3.1, Theorem 3.2.

A.1 . Proof of Theorem 2.1

The joint pdf of XSN(ξ1, ω1, α1) and YSN(ξ2, ω2, α2) is

f(x,y)=4ϕ(x;ξ1,ω12)ϕ(y;ξ2,ω22)Φα1xξ1ω1Φα2yξ2ω2.

Let Z = X + aY + b, Y = Y, then X = Z − aY − b, Y = Y and it is easy to see that the Jacobian J of this transformation is 1. Thus, we obtain the pdf of Z as

fZ(z)=fX(zayb)fY(y)dy,

which is

(A.1)fZ(z)=4ϕ(zayb;ξ1,ω12)ϕ(y;ξ2,ω22)Φα1zaybξ1ω1Φα2yξ2ω2=2πω1ω2e(zaybξ1)22ω12+(yξ2)22ω22Φ2α1zaybξ1ω1,α2yξ2ω2;0,I2dy=4ω*ϕ(z;ξ1+aξ2+b,ω12+a2ω22)ϕ(y;c,ω*2)×Φ2α1zaybξ1ω1,α2yξ2ω2;0,I2dy,
where
ω*=ω1ω2ω12+a2ω22andc=aω22(zbξ1)+ω12ξ22ω12+a2ω22.

Let T=ycω*. Then Equation(A.1) will be

(A.2)fZ(z)=4ϕ(z;ξ1+aξ2+b,ω12+a2ω22)×ETΦ2α1za(tω*+c)ξ1bω1,α2tω*+a+cξ2ω2;0,I2

with TN(0, 1). If we denote X1=α1za(tω*+c)ξ1bω1 and X2=α2tω*+a+cξ2ω2, then EṪ in Equation (A.2) can be simplified as

(A.3)ETΦ2(X1,X2),0,I2=ET[P(U1X1,U2X2|X1,X2)]=P(U1X1,U2X2)=P(U1X10,U2X20),
where U1, U2 and T are independent standard normal random variables. Note that E(Ui − Xi) = −E(Xi) and Var(Ui − Xi) = 1 + Var(Xi) for i = 1, 2. Additionally,
Cov(U1X1,U2X2)=Cov(X1,X2)=aα1α2ω*2ω1ω2.

Therefore, the joint distribution of (U1X1,U2X2) is

(U1X1,U2X2)N2(μ,ΣJ),
where
μ=α1zacξ1bω1α2cξ2ω2andΣJ=1+α12a2ω*2ω12aα1α2ω*2ω1ω2aα1α2ω*2ω1ω21+α22ω*2ω22.
Then Equation(A.3) can be written to be
ETΦ2(X1,X2),0,I2=Φ2(μ;0,ΣJ),
and therefore, Equation (A.1) will be
(A.4)f1,2(z)=4ϕ(z;ξ1+aξ2+b,ω12+a2ω22)Φ2[B(z(ξ1+aξ2+b));02,ΣJ],
where B=α1ω1ω12+a2ω22,aα2ω2ω12+a2ω22.

A.2 . Proof of Theorem 2.2

Let c = (1,a)′, we are trying to find the distribution of U = X1 + aX2 + b = cX + b. First we derive the moment generating function (mgf) of U:

(A.5)MU(t)=E[exp(tcX+tb)]=2expt(cξ+b)+12t2cΣcΦ(δΣ1/2ct).

It is easy to obtain that cξ + b = ξ1 + 2 + b and σ2=cΣc=ω12+a2ω22+2aρω1ω2. From Wang et al.,

Σ1/2=ω12+1ρ2ω1ω2kρω1ω2kρω1ω2kω22+1ρ2ω1ω2k,
where
k=ω12+ω22+21ρ2ω1ω2.

Thus, δ = δ′Σ1/2c(c′Σc)−1/2. By equation (2.1), we obtain that USN(ξ1 + 2 + b, σ2, α) with

(A.6)α*=δ*1δ*2=α1d1+α2d2(1α12α22)k2σ2(α1d1+α2d2)2,
where d1=ω12+ω1ω2(1ρ2+aρ) and d2=aω22+ω1ω2(a1ρ2+ρ). Then the density of U is given by
(A.7)fU(u)=2ϕ(u;cξ+b,σ2)Φα*(ubξb)σ
=2ϕ(u;ξ1+aξ2+b,ω12+a2ω22+2aρω1ω2)
(A.8)×Φα*(uξ1aξ2b)(ω12+a2ω22+2aρω1ω2)1/2.

A.3 . Proof of Theorem 3.1

The joint pdf of XΔ(δ1,μ1,σ12) and YΔ(δ2,μ2,σ22) is

(A.9)g(x,y)=δ1δ2ifx=0,y=0(1δ1)(1δ2)fX(x)fY(y)ifx>0,y>0δ1(1δ2)fY(y)ifx=0,y>0δ2(1δ1)fX(x)ifx>0,y=0
where fX(x), fY(y) are the probability distribution functions of X when x > 0 and Y when y > 0, respectively.
  1. For x > 0, y > 0, let Z = X + aY + b, Y = Y so that X = Z − aY − b, Y = Y. It is easy to see that the Jacobian J of this transformation is 1. Thus, the pdf of Z as

fZ(z)=fX(zayb)fY(y)dy,
  1. For x = 0, y > 0, we have x = 0, Z = aY + b and the pdf of Z is

fZ(z)=δ1(1δ2)fYzba1|a|.
  1. Similarly, for x > 0, y = 0, the pdf of Z is

fZ(z)=δ2(1δ1)fX(zb).

Note that the sign of a determines the range of Z. If a < 0, then the pdf of Z = X + aY + b is

(A.10)fZ(z)=zbafX(zayb)fY(y)dy1afYzbaifz<bδ1δ2ifz=b0fX(zayb)fY(y)dy+fX(zb)ifz>b.

Also if a > 0, the pdf of Z is given by

(A.11)fZ(z)=δ1δ2ifz=b0zbafX(zayb)fY(y)dy+1afYzba+fX(zb)ifz>b

so that desired result follows.

A.4 . Proof of Theorem 3.2

Note that from Equation (3.4), we know that the joint pdf of (X1,X2)Δ2(δ,μ,Σ) is given by

(A.12)gX(x1,x2)=δ0ifx1=0,x2=0δ1f(x1)ifx1>0,x2=0δ2g(x2)ifx1=0,x2>0(1δ0δ1δ2)h(x1,x2)ifx1>0,x2>0,
where f(x1), g(x2) are pdf's of log-normal distributions LN(μ1,σ12) and LN(μ2,σ22), respectively, and h(x1, x2) is the joint pdf of (X1,X2)LN2(μ,Σ).
  1. For x1 > 0, x2 > 0, let U = X1 + aX2 + b, X2 = X2, then X1 = U − aX2 − b, X2 = X2. It is easy to see the Jacobian J of this transformation is 1. Thus, we obtain the marginal pdf of U as

fU(u)=(1δ0δ1δ2)h(uax2b,x2)dx2,
  1. For x1 = 0, x2 > 0, let x1 = 0, U = aX2 + b and the Jacobian of this transformation is 1/a so that the pdf of U is

fU(u)=δ2guba1|a|.
  1. Similarly for x1 > 0, x2 = 0, the pdf of U is

fU(u)=δ1f(ub).

Therefore the pdf of U = X1 + aX2 + b is, for a < 0,

(A.13)fU(u)=uba(1δ0δ1δ2)h(uax2b,x2)dx2δ2agubaifu<bδ0ifu=b0(1δ0δ1δ2)h(uax2b,x2)dx2+δ1f(ub)ifu>b

and for a > 0, the pdf of U is

(A.14)fU(u)=δ0ifu=b0uba(1δ0δ1δ2)h(uax2b,x2)dx2+δ2aguba+δ1f(ub)ifu>b.

Thus the proof of Theorem 3.2 is completed.

References

Azzalini, A. and Valle, A.D. (1996), “The multivariate skew normal distribution”, Biometrika, Vol. 83 No. 4, pp. 715-726, doi: 10.1093/biomet/83.4.715.

Hubbard, R. (2016), Corrupt Research: the Case for Reconceptualizing Empirical Management and Social Science, Sage Publications, Los Angeles, CA.

Hyman, M.R., Ganesh, G. and McQuitty, S. (2002), “Augmenting the household affluence construct”, Journal of Marketing Theory and Practice, Vol. 10 No. 3, pp. 13-31, doi: 10.1080/10696679.2002.11501917.

McQuitty, S. (2004), “Statistical power and structural equation models in business research”, Journal of Business Research, Vol. 57 No. 2, pp. 175-183, doi: 10.1016/S0148-2963(01)00301-0.

McQuitty, S. (2018), “Reflections on ‘Statistical power and structural equation models in business research’”, Journal of Global Scholars of Marketing Science, Vol. 28 No. 3, pp. 272-277, doi: 10.1080/21639159.2018.1434806.

Tong, T., Wang, T., Trafimow, D. and Wang, C. (2022), “The probability of being better or worse off, and by how much, depending on experimental conditions with skew normal populations”, in SriboonchittaKreinovich, W.Y.E.S.V. (Ed.), Credible Asset allocation, Optimal Transport Methods, and Related Topics (141-149), Springer-Verlag.

Trafimow, D. (2019), “A frequentist alternative to significance testing, p-values, and confidence intervals”, Econometrics, Vol. 7 No. 2, pp. 1-14, available at: https://www.mdpi.com/2225-1146/7/2/26

Trafimow, D., Hyman, M.R., Kostyk, A., Wang, C. and Wang, T. (2021), “The harmful effect of null hypothesis significance testing on marketing research: an example”, Journal of Business Research, Vol. 125, pp. 39-44, doi: 10.1016/j.jbusres.2020.11.069.

Trafimow, D., Hyman, M.R., Kostyk, A., Wang, Z., Tong, T., Wang, T. and Wang, C. (2022), “Gain-probability diagrams in consumer research”, International Journal of Market Research, Vol. 64 No. 4, pp. 470-483, doi: 10.1177/14707853221085509.

Wang, Z., Wang, T., Trafimow, D. and Xu, Z. (2022), “A different kind of effect size based on samples from two populations with delta log-skew-normal distributions”, in Ngoc Thach, N., Ha, D.T., Trung, N.D. and Kreinovich, V. (Eds), Prediction and Causality in Econometrics and Related Topics. ECONVN 2021. Studies in Computational Intelligence, Springer, Cham, Vol. 983, doi: 10.1007/978-3-030-77094-5_10.

Ziliak, S.T. and McCloskey, D.N. (2016), The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives, The University of Michigan Press, Ann Arbor, MI.

Acknowledgements

It has come to the attention of the publisher that the article, Trafimow, D., Wang, Z., Tong, T. and Wang, T. “Gain-probability diagrams as an alternative to significance testing in economics and finance” published in Asian Journal of Economics and Banking https://doi.org/10.1108/AJEB-05-2023-0045 gave an incorrect location for Trafimow’s affiliation. This error was introduced in the editorial process and has now been corrected in the online version. The publisher sincerely apologises for this error.

Corresponding author

David Trafimow can be contacted at: dtrafimo@nmsu.edu

Related articles