It's much more than an 'idea', Ards. Naturally they memorise landscapes - if you think about it, that is how they learned to survive in ancient times. Certain landscapes represented good feeding grounds, others where they might be ambushed, others where there was water. Just like all animals, and us - we remember a place and the emotion we felt there. Horses see a place and remember the experience they had there, too. Unfortunately, they also very quickly memorise where the horsebox parks (horsebox = go home = eat nice hay) and exits are, hence the occasional tanking-off minus jockey!
Never underestimate a horse's reaction to a venue (landscape) where something unpleasant occurred. He'd have been swerving that by miles, given the choice. Not given the choice, be prepared for a negative reaction.