If you reached point B from point A - and you tell someone "I would like to go back", then you are expecting to go back to A. Not some intermediate, arbitrarily chosen point C.