I think the definition, strictly speaking, between 'vermin' and 'pest' is laid down by DEFRA, certainly not the hunts. Vermin are considered to be vectors (carriers) of diseases, and pests are just annoying, but not likely to carry disease. A pest, for example, is the aphid. It can be killed by ladybirds, who are not pests, or it can be killed by an insecticide (not a good idea, since you then kill the ladybirds and other beetles and bugs).
Vermin include many rodents, but not all, and in some cities pigeons have been culled because they're considered such on account of the mites in their feathers and their copious amounts of poo. If DEFRA ever decides once and for all that TB is spread to cattle by badgers, then you'll see much, much bigger badger culls than they've been doing already, as they'll be branded as 'vermin'. Cue badger cull sabbers...
And a hearty 'well done' to the ALF for releasing mink into the defenceless countryside, where they continue, as Ven says, their brutal decimation of the helpless, native water vole, as well as toads, newts, lizards, and and as many dormice and other small creatures that they can shred in a day. But, as mink hounds are now off the hunting menu, they can continue to wreak havoc in their non-native environment. Just as long as they don't get made into coats, that's all right, then.