Positive, right-skewed data, such as time to an event, hospital costs, income or energy use, almost never have constant variance: the larger the mean, the larger the spread. The gamma distribution models exactly that. This article proves two facts that explain why it is so convenient. First, as the precision parameter \(\phi\) grows, the gamma converges to a normal, but a normal that keeps the mean–variance relationship of the data. Second, with the log link (which is not the canonical one), the estimation algorithm becomes a sequence of ordinary, unweighted least squares fits, and several properties of linear regression carry over almost literally. It ends with a numerical check and a verdict on the phrase often used to sum up the second fact.
§1The parametrization
We write \(Y \sim \text{Gamma}(\mu, \phi)\) with density
that is, shape \(\phi\) and rate \(\phi/\mu\). Hence
The coefficient of variation is constant: it does not depend on \(\mu\). That is the natural assumption for data whose “error” is proportional to the level: a $10,000 cost varies more, in dollars, than a $100 one, but the same in percentage terms. In GLM language, the variance function is \(V(\mu) = \mu^2\) and the dispersion is \(\phi^{-1}\).
§2As \(\phi \to \infty\), \(Y\) approaches \(N(\mu,\ \mu^2/\phi)\)
2.1 Statement
Let \(Y_\phi \sim \text{Gamma}(\mu, \phi)\) and define the standardized version
Then \(Z_\phi \xrightarrow{d} N(0,1)\) as \(\phi\to\infty\). In practice, for large \(\phi\), \(Y \approx N(\mu,\ \mu^2/\phi)\).
2.2 Proof via the moment generating function
The MGF of the gamma with shape \(\phi\) and rate \(\phi/\mu\) is
Since \(Z = \frac{\sqrt\phi}{\mu}Y - \sqrt\phi\),
Note that \(\mu\) is gone: the distribution of \(Z\) depends on \(\phi\) only. Taking logs and using \(-\log(1-x) = x + \tfrac{x^2}{2} + \tfrac{x^3}{3} + \cdots\) for \(|x|<1\), with \(x = s/\sqrt\phi\):
For fixed \(s\), every term after \(s^2/2\) goes to zero as \(\phi\to\infty\). So \(M_Z(s)\to e^{s^2/2}\), the MGF of \(N(0,1)\), and by Lévy's continuity theorem \(Z_\phi \xrightarrow{d} N(0,1)\). \(\blacksquare\)
Another reading. If \(\phi = k\) is an integer, \(Y\) has the same distribution as \(\frac{1}{k}\sum_{j=1}^{k} E_j\), with \(E_j\) independent exponentials of mean \(\mu\). That is a mean of \(k\) i.i.d. variables with variance \(\mu^2\), and the Central Limit Theorem gives \(N(\mu, \mu^2/k)\) directly. The precision \(\phi\) acts like a “sample size” hidden inside each observation.
2.3 How fast the approximation improves
Expansion (3) shows that the first departure from normality is the cubic term, of order \(\phi^{-1/2}\). It is the skewness:
| \(\phi\) | CV | skewness \(2/\sqrt\phi\) | \(P(\text{approximating normal} < 0) = \Phi(-\sqrt\phi)\) |
|---|---|---|---|
| 1 | 1.00 | 2.00 | 0.159 |
| 5 | 0.45 | 0.89 | 0.013 |
| 10 | 0.32 | 0.63 | 0.0008 |
| 20 | 0.22 | 0.45 | 0.000004 |
| 100 | 0.10 | 0.20 | ≈ 0 |
With \(\phi \gtrsim 10\), the approximating normal puts practically no mass on negative values and the skewness is already moderate. For small \(\phi\) (CV near 1), the approximation is poor and the gamma has to be used as a gamma. A much better approximation for moderate \(\phi\) is Wilson–Hilferty, \((Y/\mu)^{1/3} \approx N\!\left(1-\frac{1}{9\phi},\ \frac{1}{9\phi}\right)\): the cube root removes almost all of the skewness.
2.4 And on the log scale
Write \(Y = \mu\,G\), with \(G\sim\text{Gamma}(1,\phi)\), of mean 1. Then \(\log Y = \log\mu + \log G\), and
where \(\psi\) and \(\psi'\) are the digamma and trigamma functions. By the delta method, \(\log Y \approx N(\log\mu,\ 1/\phi)\): on the log scale the variance is constant. That fact is the bridge to §3.
2.5 What the approximation buys you
- Heteroscedasticity stays in the model. The limiting normal is not the homoscedastic \(N(\mu,\sigma^2)\); it is \(N(\mu, \mu^2/\phi)\). Even in the “nearly normal” regime, the gamma still says that the standard deviation grows in proportion to the mean. An ordinary normal regression gets exactly this wrong.
- Reliable Wald inference. Intervals like \(\hat\mu\,(1 \pm z_{\alpha/2}/\sqrt{\hat\phi})\) and normal-based tests become accurate.
- Small-dispersion asymptotics. The scaled deviance \(\phi\,D\) approaches \(\chi^2_{n-p}\) as \(\phi\) grows, even with \(n\) fixed. Likelihood-ratio and goodness-of-fit tests become accurate without huge samples.
- Simple diagnostics. Pearson residuals \((y-\hat\mu)/\hat\mu\) and deviance residuals become approximately normal, and normal QQ plots become informative.
- Choosing between similar models matters little. With large \(\phi\), the gamma, the log-normal and a “normal with constant CV” are nearly indistinguishable, and the estimates of \(\boldsymbol\beta\) barely move. With small \(\phi\) the choice matters, and the gamma has the advantage of modelling \(\mathbb E[Y]\) directly.
- It justifies familiar shortcuts, such as weighted least squares with weights \(1/\hat\mu^2\) or a linear regression of \(\log y\) (§3.5).
§3The gamma GLM with a log link
3.1 The model
For independent observations \(Y_i \sim \text{Gamma}(\mu_i,\phi)\):
The canonical link of the gamma is the reciprocal, \(g(\mu) = 1/\mu\). It has a serious practical problem: \(\mu\) must be positive, but nothing stops \(\mathbf x^\top\boldsymbol\beta\) from becoming negative or zero during fitting. The log link guarantees \(\mu_i > 0\) for any \(\boldsymbol\beta\) and gives coefficients that read as multiplicative effects: a one-unit increase in \(x_j\) multiplies the mean by \(e^{\beta_j}\).
3.2 Score and information
The log-likelihood, up to terms without \(\boldsymbol\beta\), is
Differentiating with respect to \(\eta_i\):
By the chain rule, with \(\partial\eta_i/\partial\boldsymbol\beta = \mathbf x_i\):
\(\mathbf J\) is the observed information; \(\mathcal I\) is the expected one, because \(\mathbb E[Y_i/\mu_i] = 1\). This is the key result: the Fisher information is \(\phi\,\mathbf X^\top\mathbf X\). It does not depend on \(\boldsymbol\beta\) and has the same form as in linear regression, where \(\mathcal I = \sigma^{-2}\mathbf X^\top\mathbf X\).
3.3 IRLS becomes ordinary least squares
In any GLM, Fisher scoring can be written as iteratively reweighted least squares:
For the gamma with a log link, \(V(\mu) = \mu^2\) and \(g'(\mu) = 1/\mu\), so
The weights vanish, and each iteration is
that is, an ordinary linear regression (OLS) of the working response \(\mathbf z\) on \(\mathbf X\). The matrix \((\mathbf X^\top\mathbf X)^{-1}\) is computed once; only the vector \(\mathbf z\) changes between iterations.
Why does this happen? The weights are constant exactly when \(V(\mu)\,g'(\mu)^2\) is constant, that is, when \(g'(\mu)\propto V(\mu)^{-1/2}\). Integrating, \(g(\mu) \propto \int V(\mu)^{-1/2}\,d\mu\), which is the variance-stabilizing transformation. For \(V(\mu)=\mu^2\), \(\int \mu^{-1}d\mu = \log\mu\). The log link is to the gamma what the square root is to the Poisson, which also gives constant weights (\(w_i = 4\)).
3.4 Properties inherited from linear regression
All of them follow directly from \(\mathbf W = \mathbf I\) and \(\mathcal I = \phi\,\mathbf X^\top\mathbf X\):
| Property | Linear regression | Gamma with log link |
|---|---|---|
| Normal equations | \(\mathbf X^\top\mathbf e = \mathbf 0\) | \(\mathbf X^\top\mathbf r = \mathbf 0\), with \(r_i = (y_i-\hat\mu_i)/\hat\mu_i\), the Pearson residual |
| With an intercept | \(\sum e_i = 0\) | \(\sum r_i = 0 \iff \frac1n\sum y_i/\hat\mu_i = 1\) |
| Covariance of \(\hat{\boldsymbol\beta}\) | \(\sigma^2(\mathbf X^\top\mathbf X)^{-1}\), exact | \(\phi^{-1}(\mathbf X^\top\mathbf X)^{-1}\), asymptotic |
| Hat matrix | \(\mathbf H = \mathbf X(\mathbf X^\top\mathbf X)^{-1}\mathbf X^\top\) | The same, not depending on \(\hat{\boldsymbol\beta}\) |
| Dispersion estimator | \(s^2 = \sum e_i^2/(n-p)\) | \(1/\hat\phi = \sum r_i^2/(n-p)\) (Pearson) |
| Does \(\hat{\boldsymbol\beta}\) depend on the error scale? | No | It does not depend on \(\phi\) |
Some practical consequences:
- Standard errors without \(\hat\mu\). The estimated covariance depends on the data only through \(\hat\phi\). The design \(\mathbf X\) alone fixes the relative precision of the coefficients, as in a designed linear-regression experiment.
- Leverages known before fitting. The \(h_{ii}\) are the same as in a linear regression with the same \(\mathbf X\), which simplifies Cook's distance, DFBETAs and experimental design.
- Scale invariance. Replacing \(y\) with \(c\,y\) (dollars to thousands of dollars, say) only adds \(\log c\) to the intercept. The other coefficients and all standard errors are unchanged.
- Orthogonality of \(\boldsymbol\beta\) and \(\phi\), as in any GLM: estimating \(\phi\) does not disturb inference on \(\boldsymbol\beta\).
3.5 The bridge to a linear regression of \(\log y\)
From §2.4, \(\log Y_i = \mathbf x_i^\top\boldsymbol\beta + \log G_i\), with \(G_i\) i.i.d., because the distribution of \(G\) does not depend on \(\mu_i\). This is an exact linear model with i.i.d., non-normal errors:
So OLS of \(\log y\) on \(\mathbf X\) estimates the slopes \(\beta_1,\dots,\beta_p\) without bias; estimates the intercept with bias \(\psi(\phi)-\log\phi\approx -1/(2\phi)\); and has covariance \(\psi'(\phi)(\mathbf X^\top\mathbf X)^{-1}\), against \(\phi^{-1}(\mathbf X^\top\mathbf X)^{-1}\) for gamma maximum likelihood. The relative efficiency of the slopes is \(1/(\phi\,\psi'(\phi))\):
| \(\phi\) | 1 | 2 | 5 | 10 | 20 | 100 |
|---|---|---|---|---|---|---|
| efficiency of OLS on \(\log y\) | 0.61 | 0.78 | 0.90 | 0.95 | 0.975 | 0.995 |
With large \(\phi\) the two methods practically coincide, which is §2 seen from another angle. With small \(\phi\) the gamma GLM is much more efficient. It also estimates \(\mathbb E[Y]\) directly, without the retransformation (smearing) correction that OLS on \(\log y\) needs. OLS on \(\log y\) is, by the way, an excellent starting value for IRLS.
3.6 Numerical check
I simulated \(n=500\) observations with \(\phi=5\), \(\beta_0 = 1\), \(\beta_1 = 0.5\) and \(x\sim N(0,1)\), and ran IRLS with unit weights, that is, an OLS of \(\mathbf z\) on \(\mathbf X\) at each step:
- it converged in 5 iterations to \(\hat{\boldsymbol\beta} = (0.9535,\ 0.4830)\), identical to
statsmodels(GLM, gamma family, log link); - \(\mathbf X^\top\mathbf r \approx (4\times10^{-13},\ 2\times10^{-10})\) and \(\frac1n\sum y_i/\hat\mu_i = 1.0000\);
- the standard errors \(\sqrt{\operatorname{diag}\big((\mathbf X^\top\mathbf X)^{-1}/\hat\phi\big)}\) match
statsmodelsto every digit; - OLS on \(\log y\) gave a slope of \(0.4805\), practically the same, and an intercept of \(0.842\), about \(0.11\) lower, as the bias \(\psi(5)-\log 5 = -0.103\) predicts.
§4Verdict: “like a simple linear regression”
You often read that, in the gamma GLM with a log link, estimation is “like a simple linear regression, with important properties”. In essence the phrase is true, with three caveats.
What is true: each IRLS step is literally an ordinary, unweighted linear regression of the working response \(\mathbf z\) on \(\mathbf X\); the Fisher information \(\phi\,\mathbf X^\top\mathbf X\), the covariance \(\phi^{-1}(\mathbf X^\top\mathbf X)^{-1}\) and the hat matrix have exactly the linear-regression form and do not depend on \(\boldsymbol\beta\); the Pearson residuals are orthogonal to the columns of \(\mathbf X\) and sum to zero with an intercept, like OLS residuals. The structural reason is that \(\log\) is the variance-stabilizing transformation for \(V(\mu)=\mu^2\).
The caveats:
- “Simple” only in the sense of “ordinary, unweighted”. The result holds for any number of covariates. It has nothing to do with simple linear regression in the technical sense, the one with a single covariate.
- It is not a single regression. It is iterative: \(\mathbf z\) depends on \(\hat{\boldsymbol\mu}\), there is no closed form, and the covariance and normality of \(\hat{\boldsymbol\beta}\) are asymptotic, not exact as in the normal linear model.
- Fisher scoring is not Newton–Raphson here. Because the link is not canonical, the observed information \(\phi\sum (y_i/\mu_i)\,\mathbf x_i\mathbf x_i^\top\) differs from the expected one. Both algorithms converge to the same \(\hat{\boldsymbol\beta}\), but along different paths, and standard errors based on the observed information differ slightly from \(\phi^{-1}(\mathbf X^\top\mathbf X)^{-1}\). The canonical-link property \(\sum y_i = \sum\hat\mu_i\) is also lost; in its place, \(\sum y_i/\hat\mu_i = n\).
In short: the gamma with a log link offers the computational and diagnostic structure of a linear regression, with variance proportional to \(\mu^2\), means that are always positive and multiplicative effects. When \(\phi\) is large, it practically coincides with a linear regression of \(\log y\).
§5Further reading
- P. McCullagh and J. A. Nelder (1989). Generalized Linear Models, 2nd ed., ch. 8. Chapman & Hall. Data with a constant coefficient of variation.
- B. Jørgensen (1997). The Theory of Dispersion Models. Chapman & Hall. Small-dispersion asymptotics.
- D. Firth (1988). Multiplicative errors: log-normal or gamma? Journal of the Royal Statistical Society B, 50(2), 266–268. Efficiency of the gamma against the log-normal.
- E. B. Wilson and M. M. Hilferty (1931). The distribution of chi-square. Proceedings of the National Academy of Sciences, 17(12), 684–688.