the simplest way to bring in these annotations would be via the GEMINI toolsuite, just that
a) you’ll have to stop working with VCF format at that point because GEMINI converts VCF to an sqlite database during the annotation step and keeps using this format for all further processing and querying
b) with the current version of GEMINI, you’ll only have ExAC release 0.3 available
c) you would have to work on usegalaxy.eu for now because GEMINI, for technical reasons, is currently not available on usegalxy.org (though hopefully this will get fixed in the not so distant future)
If you want to try to annotate your VCF directly (SnpSift annotate may or may not work for the task - I never tried to use it for anything but dbSNP data) the data you’re looking for is available from:
The files you find there have been compressed with bgzip, but if you specify data type “VCF” on import Galaxy should unpack them for you.