Andrew I would leave all that stuff there, it has become part of your soil, full of organisms I expect and your best route to improve matters is surface organic matter.
So make a hole large enough for the tree roots (Not necessarily 2ftx2ft, may be more or less, backfill with soil only and then organic matter on top.
It has worked well here on builders rubble. My whole small garden is like that.
