Surrendering UK sovereignty of the Chagos Islands is an “irresponsible act”, a Labour former minister has said, backing calls for the Government to withdraw from a deal over the territory.