Do I have to worry about temperature with neodymium magnets?

Yes.  Neodymium Iron Boron magnets are sensitive to heat.  If a magnet heated above its maximum operating temperature (176°F (80°C) for standard N grades) the magnet will permanently lose a fraction of its magnetic strength.  If they are heated above their Curie temperature (590°F (310°C) for standard N grades), they will lose all of their magnetic properties.  Different grades of neodymium different maximum operating and Curie temperatures. Yes.  Neodymium Iron Boron magnets are sensitive to heat.  If a magnet heated above its maximum operating temperature (176°F (80°C) for standard N grades) the magnet will permanently lose a fraction of its magnetic strength.  If they are heated above their Curie temperature (590°F (310°C) for standard N grades), they will lose all of their magnetic properties.  Different grades of neodymium different maximum operating and Curie temperatures. 

Will my neodymium magnets lose strength over time?

Very little. Neodymium magnets are the strongest and most permanent magnets known to man. If they are not overheated or physically damaged, neodymium magnets will lose less than 1% of their strength over 10 years – not enough for you to notice unless you have very sensitive measuring equipment. They won't even lose their strength if they are held in repelling or attracting positions with other magnets over long periods of time.