We assessed sex differences in fractional anisotropy (FA) and mean diffusion (MD) of temporal lobe white matter (TLWMFA, TLWMMD), TLWM asymmetry, and the relationship between TLWM asymmetry and FA and MD of the corpus callosum (CCFA, CCMD) using diffusion tensor tractography. We found sex differences in the MD of both inferior longitudinal fasciculus (ILF); however, it disappeared after statistical adjustments for intracranial volume and CCMD variability. FA of the superior longitudinal fasciculus (SLFFA) and FA and MD of the ILF (ILFFA, ILFMD) showed leftward asymmetry. In females, the asymmetry indices of SLFFA and the MD of the SLF were significantly correlated with CCFA and CCMD, respectively, which may be related to sex differences in the hemispheric specialization for language.