Nanoparticles and in particular gold nanorods have interesting optical properties arising from two well-differentiated plasmon modes. The frequency of such modes can be altered by their chemical environment and coupling with neighboring rods. This study investigates new composite materials made of gold nanorods adsorbed on thermoresponsive poly(N-isopropylacrylamide) (PNIPAM) microgels. It is shown that the thermally induced collapse of the polymer network inside the particles leads to a red shift of the longitudinal plasmon band of the gold rods, which is found to be fully reversible.