I can reflect the light from a laser pointer back to the source using a mirror. Are you saying that the mirror has to have a higher temperature than the source for that to occur?
Like with current in electricity, there has to be a difference in potential for energy to flow. Second law of thermodynamics.
Don't confuse laws of thermodynamics with principles of electricity and magnetism.