Nice looking car!
Does the noise get louder when you turn? If so, then you could have a CV joint giving up. The reason why it goes quiet when you engage the clutch is because you are taking the load off the gearbox, and thus the drive shafts.
May be your clutch release bearing, but this would normally grind when you engaged the clutch.
Also, check the oil level in your gearbox. If it is low, then that may have caused an internal problem in the gearbox.