Dave, it is a little unclear as to whether you are filling beds from new every year? The issue is not contamination at least.
From your description of the fine shavings in your manure, I would be happy to use the manure as surface mulch, but not mixed with your clay. Mixing wood with soil can lead to soil nutrients, mainly nitrogen, being ‘grabbed’ by wood and anything carboniferous, to help itself rot down. But this happens little with any wood on the surface, where nitrogen from the air is also available to it.
It may even work to fill a bed with such manure if the proportion of wood to animal droppings is not high, and some breakdown has happened already. I can’t say for sure, you could try it and see.