The number of people who voted for Leave for the primary reason of controlling immigrant numbers is quite small - only 12% of the electorate (or 9% of the total Population). Thus the referendum can NOT be used to justify a Brexit where controlling immigration is the dominant criterion for an acceptable deal. The figures are illustrated here - the purple bar marked "*" represents those voted for Leave for the primary reason of controlling immigrant.
The figures behind this graphic are as follows.
The figures come from the electoral commission, and the most recent House of Commons Library briefing paper on migration. The estimate for the proportion of leave voters for whom controlling immigration was the most important factor were inferred from the polls Lord Ashcroft funded at the time (exit polls in this case).
See also the Telegraph's article "Did Britain really vote Brexit to cut immigration?" on the 29th June 2016.