Hi A,
You will have to remove the Cobbles if you want to grow anything on that site as an inch of soil is no where near deep enough for anything except grass perhaps.
You could alternatively go the raised bed route, i.e., build a series of wood (or other material) sided beds on top of your ground. In this case you will still have to consider drainage if you leave the cobbles in place under the raised beds.
Don.