@TGoddard Checking the custom genome/build is a great place to start. Most tools do not work well when the fasta identifier lines “>” contain any description content. Plus, if you are incorporating reference annotation while mapping – if it doesn’t match the reference genome that can also lead to problems (the chromosome names/format must be exactly the same between those two inputs). If you are not incorporating reference annotation yet, you’ll still need to check it – most if not all downstream tools are sensitive to that kind of chromosome mismatch issue. The job may not even fail – just produce odd results.
FAQs that may help. I’m guessing you found these already, but the mismatch issue comes up so often, will post again for reference for others reading.
Once you have confirmed/resolved all of the above, and the job still fails, especially with a resource-related error, it may be that your custom genome is simply too large to process at the public site. Fragmented assemblies (over 100 or so “chromosomes”) and large assemblies (most mammals, certain plants) are the usual underlying reason for the “genome too large” types of failures when using a custom genome/build. The original troubleshooting FAQ I posted has more details about what you can try next.
If you want a 2nd opinion on the CG/annotation match or content, before trying other ways to use Galaxy, please send in a bug report from the error and leave all inputs/outputs intact. Paste a link to this topic in the bug report comments for context, then post back here so we know when to look for it. But I hope it works out before then 
