Victoria plum is susceptible to all sorts of things, more than other varieties. But what you describe is not the dreaded silverleaf or plum pox and does not sound like a major problem, in fact I wonder if it is something like the "June drop" on apples, when fruit stay small and then fall off. Trees mostly know how many fruit they can support, although plums less than apples… Is it a young tree, in poor soil perhaps, struggling a bit and feeling dry at the moment?