Abstract
We study the unusual behavior of the incommensurate lattice formed by Li on the anisotropic trough-like surface of Mo(112). Starting at a coverage of 0.66, we found a continuous transformation of a rectangular into an oblique unit cell. This oblique cell transforms back into a rectangular cell at coverages higher than 0.85. Both transitions are explained by an instability in the shear modulus of the layers at the respective critical coverages. It leads to the development of smectic-like fluctuations and finally to a melting transition. For the oblique phase below melting, long-range correlations along the troughs, but correlations over less than 10 lattice constants normal to the troughs are observed in experiment. The experimental results are in reasonable agreement with the theoretical model proposed.