This tool uses very simple inputs, most in
Expand the input datasets and if you do not see any warnings about a mismatched datatype, then the data is probably OK. You could also “redetect” the datatype to double-check (pencil icon > Edit Attributes > Datatype > Redetect datatype).
That said, it sounds like the job may be running out of resources due to the number of reads (probably memory resources during indexing). Downsampling/subsampling the reads is one option. Setting up a Galaxy server with more dedicated resources is another.
Using the latest version of the tool is also strongly recommended (check with the “Versions” pull-down menu at the top of the tool form).
For parameters: remove all filters, review the results (hits), then tune from there.
The Vsearch publication itself might be easier to interpret. Then the manual can be used as a specific reference. There is also a Google forum for this tool suite: https://groups.google.com/forum/#!forum/vsearch-forum
The Galaxy metagenomics tutorials may also be a useful reference (although none cover this tool suite specifically!): https://training.galaxyproject.org/training-material/topics/metagenomics/