Nonmagnetic Ground States and a Possible Quadrupolar Phase in 4d and 5d Lacunar Spinel Selenides GaM_{4}Se_{8} (M=Nb, Ta).

Structural and magnetic properties of cubic spinel selenides GaM_{4}Se_{8} (M=Nb, Ta), which are candidates for the molecular J_{eff}=3/2 Mott insulators, are investigated. The effective magnetic moments are reduced compared to the spin only value, indicating the presence of sizable spin-orbit coupling. GaNb_{4}Se_{8} and GaTa_{4}Se_{8} exhibit phase transitions into the nonmagnetic ground states with orthorhombic and tetragonal structures, respectively, which are robust against magnetic field up to at least 60 T. A cubic-cubic phase transition is observed in GaNb_{4}Se_{8} preceding the magnetic transition, suggesting the existence of a quadrupolar-ordered phase theoretically predicted in the J_{eff}=3/2 Mott insulator.