It can be either, there are different types of "dirt", "dirt" itself is a generic term for anything brown really - for example Turfway are using polytrack as Woodbine are planning on doing and a few other tracks, whilst California racing board are requiring Santa Anita, Golden Gate, Bay Meadows, Del Mar, Hollywood Park etc to all use an artificial "polytrack like" surface by the end of 2007.
Like turf when it rains the ground is effected, it becomes more sloppy and muddy and thus there are a number of going descriptions - fast, good, sloppy and muddy amongst others. An "off track" is anything that's not fast and you'll see some horses being described as "off track specialists" pretty much like those who prefer heavy ground to good or faster. This is as a result of most US races as far as I know being run on a fast dirt track, if it rains too much the turf races on a card may well be switched to the dirt track - including some stakes events.