#01-12

INVESTIGATING THE PROBABILITY OF SIGN INCONSISTENCY IN THE REGRESSION COEFFICIENTS OF MARKERS FLANKING QTL

by

J. T. Gene Hwang and Dan Nettleton

Cornell University and Iowa State University

ABSTRACT

Estimates of the locations and effects of quantitative trait loci (QTL) can be obtained by regressing phenotype on marker genotype. Under certain basic conditions, the signs of regression coefficients flanking QTL must be the same. There is no guarantee, however, that the signs of the regression coefficient {\em estimates} will be the same. We use {\em sign inconsistency} to describe the situation in which there is disagreement between the signs of the estimated regression coefficients flanking QTL. The presence of sign inconsistency can undermine the effectiveness of QTL mapping strategies that presume intervals whose markers have regression coefficient estimates of differing sign to be devoid of QTL. This paper investigates the likelihood of sign inconsistency under various conditions. We derive an analytic expression for the approximate probability of sign inconsistency in the single QTL case. We also examine sign inconsistency probabilities when multiple QTL are present through simulation. We have discovered that the probability of sign inconsistency can be unacceptably high, even when the conditions for QTL detection are otherwise quite favorable.