Do I have to worry about temperature with neodymium magnets?

Yes.  Neodymium Iron Boron magnets are sensitive to heat.  If a magnet heated above its maximum operating temperature (176°F (80°C) for standard N grades) the magnet will permanently lose a fraction of its magnetic strength.  If they are heated above their Curie temperature (590°F (310°C) for standard N grades), they will lose all of their magnetic properties.  Different grades of neodymium different maximum operating and Curie temperatures.