Panaroo 1.8.0+galaxy0 fails on GFF3 collection with FileNotFoundError: input_directory/*.gff

usegalaxy.eu support
, ,
1

Hello,

I encountered what appears to be a wrapper issue in Panaroo (Galaxy wrapper version 1.8.0+galaxy0 on usegalaxy.eu).

Input:

- Collection of 19 GFF3 files

- All files were generated by Bakta

- Collection datatype: gff3

The job fails immediately with:

FileNotFoundError: [Errno 2] No such file or directory: ‘input_directory/*.gff’

Relevant traceback:

Traceback (most recent call last):

File “/usr/local/bin/panaroo”, line 10, in

sys.exit(main())

File “/usr/local/lib/python3.11/site-packages/panaroo/_main_.py”, line 318, in main

with open(args.input_files[0], “r”) as file:

FileNotFoundError: [Errno 2] No such file or directory: ‘input_directory/*.gff’

From the generated command line I can see that the wrapper creates symlinks such as:

input_directory/SRR38521030_Staphylococcus_aureus_ST8_CC8

(without .gff extension)

but Panaroo is invoked with:

-i input_directory/*.gff

This suggests that no files match the wildcard and Panaroo receives the literal string ‘input_directory/*.gff’.

Could this be a wrapper issue related to handling GFF3 collections?

Thank you.

2

Hello @Loc_Le

Yes, this tool is a bit tricky to use. A full adjustment at the tool wrapper level is pending but for now, the solution is to include .gff on the collection identifiers.

You can recreate the collection (retaining the file extensions) or use the Extract Element identifiers, text editing tools, then Relabel identifiers functions to add these back in. And, for HTP use cases, Apply Rules can make this sort of adjustment too.

Recreating the collection is the easiest, so try that first if this seems overly complicated.

Hope this helps and follow up questions are welcome! :slight_smile:

3

Dear Prof. Jennifer Hillman-Jackson.

Thank you for your earlier suggestion, and my apologies for a very delayed follow-up.

I tried recreating the collection in Galaxy and deliberately left the “remove file extension” option unchecked. Unfortunately, the job still failed.

I then followed your alternative suggestion using the Extract Element Identifiers and Relabel Identifiers tools to restore the file extensions. Unfortunately, this did not resolve the issue either.

The generated symlinks are now created with the .gff3 suffix, for example: input_directory/SRR38521030_Staphylococcus*aureus_ST8_CC8.gff3

However, the wrapper still invokes Panaroo with: -i input_directory/*.gff

Since all generated files end with .gff3, the wildcard appears not to match any files and Panaroo receives the literal string input_directory/*.gff, leading to: FileNotFoundError: input_directory/*.gff

From the generated command line it looks like the wrapper may currently create .gff3 links while still expecting .gff inputs.

I have included the full Galaxy command line and error message below in case the additional details are helpful for debugging.

Thank you again for your time and assistance.

Best regards,

Le Tuan Loc

Run 1: not using Relabel Identifiers

CLI: mkdir outdir && mkdir input_directory && ln -fs ‘/data/dnb13/galaxy_db/files/b/7/5/dataset_b754a17d-3319-444d-88d1-912474ce52e1.dat’ ‘input_directory/AP017922.1_Staphylococcus_aureus_ST398_CC398_ISU926.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/b/a/0/dataset_ba0dd125-511e-431a-882b-ce2d13f17f57.dat’ ‘input_directory/AP017922.1_Staphylococcus_aureus_ST6272_CC72_JP080.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/8/3/f/dataset_83f6d967-07b2-4872-a04f-c8fb3c7ef3e9.dat’ ‘input_directory/ERR107794_Staphylococcus_aureus_ST36_CC30.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/6/e/dataset_a6e9cdca-174a-4fae-9bc9-99e67e5628c4.dat’ ‘input_directory/ERR107822_Staphylococcus_aureus_ST22_CC22.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/d/1/c/dataset_d1c435c6-a785-4a15-81ff-38597c91e088.dat’ ‘input_directory/ERR10900234_Staphylococcus_aureus_ST121_CC121.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/9/4/c/dataset_94c4a8fd-1792-494b-af7b-595d7eabff6d.dat’ ‘input_directory/ERR10900246_Staphylococcus_aureus_ST121_CC121.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/3/7/c/dataset_37c5e3d0-2e5a-4a4e-9a31-2f589435bbd8.dat’ ‘input_directory/ERS1179830_Sraphylococcus_aureus_ST36_CC30_NRS740_TN-82.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/e/2/c/dataset_e2c0dca6-e97a-436b-8fed-9a076aa591b1.dat’ ‘input_directory/GCF_000695215.1_Staphylococcus_aureus_ST22_CC22.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/9/8/7/dataset_9871073f-9be5-4d4b-a7dd-c763fc1d646a.dat’ ‘input_directory/GCF_000695875.1_Staphylococcus_aureus_ST8_CC8.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/d/2/b/dataset_d2b1e361-cbef-40ec-8e65-c49bebec048b.dat’ ‘input_directory/GCF_001887075.1_Staphylococcus_aureus_ST398_CC398_E154.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/1/4/4/dataset_144edb93-f972-4657-8fa9-adbcf74fede7.dat’ ‘input_directory/GCF_003031485.1_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/0/4/2/dataset_0425a6f0-5a7b-4913-9450-15401500ba94.dat’ ‘input_directory/GCF_009811395.1_Staphylococcus_aureus_ST8_CC8.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/1/1/dataset_a11e4b51-788e-422e-9d6a-88ca56f78c02.dat’ ‘input_directory/GCF_030342405.2_Staphylococcus_aureus_ST45_CC45_BSN01_Closure_genomic.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/5/8/1/dataset_5810ba99-0879-4e83-900c-7105d704773f.dat’ ‘input_directory/GCF_041356355.1_Staphylococcus_aureus_ST5_CC5_MRSA-16.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/8/f/3/dataset_8f395a45-d303-49bb-9bbb-607f4d7a52de.dat’ ‘input_directory/GCF_053749775.1_Staphylococcus_aureus_ST45_CC_45_SAIE_GP_04.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/2/0/0/dataset_200b6581-969d-4055-b3b7-228ed3b16da9.dat’ ‘input_directory/SRR1159867_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/4/0/dataset_a4015227-7905-43f9-b9b5-6113deb8c1b5.dat’ ‘input_directory/SRR13194715_Staphylococcus_aureus_ST30_CC_30_144KP_subject_1135.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/b/7/e/dataset_b7ea8b8c-2b42-4870-8921-3db7fbd8d4d1.dat’ ‘input_directory/SRR27644380_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/c/b/7/dataset_cb7a1890-be3c-4e0a-951a-0d7ae68e894d.dat’ ‘input_directory/SRR38521030_Staphylococcus_aureus_ST8_CC8.gff3’ && panaroo --clean-mode ‘strict’ --codon-table ‘11’ --remove-invalid-genes --threshold ‘0.98’ --family_threshold ‘0.7’ --len_dif_percent ‘0.98’ --family_len_dif_percent ‘0.0’ --search_radius ‘5000’ --refind_prop_match ‘0.2’ --refind-mode ‘default’ --min_trailing_support ‘2’ --trailing_recursive ‘1’ --edge_support_threshold ‘1.0’ --remove_by_consensus ‘False’ --high_var_flag ‘5’ --min_edge_support_sv ‘2’ --alignment ‘core’ --aligner ‘mafft’ --core_entropy_filter 0.1 -i input_directory/*.gff -o outdir -t ${GALAXY_SLOTS:-8} && mv outdir/gene_presence_absence.Rtab outdir/gene_presence_absence_rtab.Rtab && mv outdir/combined_protein_cdhit_out.txt outdir/combined_protein_cdhit_out.fa

Error: Traceback (most recent call last):
File “/usr/local/bin/panaroo”, line 10, in
sys.exit(main())
^^^^^^
File “/usr/local/lib/python3.11/site-packages/panaroo/_main_.py”, line 318, in main
with open(args.input_files[0], “r”) as file:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: ‘input_directory/*.gff’

Run 02: using Relabel identifier

CLI: mkdir outdir && mkdir input_directory && ln -fs ‘/data/dnb13/galaxy_db/files/b/7/5/dataset_b754a17d-3319-444d-88d1-912474ce52e1.dat’ ‘input_directory/AP017922.1_Staphylococcus_aureus_ST398_CC398_ISU926.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/b/a/0/dataset_ba0dd125-511e-431a-882b-ce2d13f17f57.dat’ ‘input_directory/AP017922.1_Staphylococcus_aureus_ST6272_CC72_JP080.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/8/3/f/dataset_83f6d967-07b2-4872-a04f-c8fb3c7ef3e9.dat’ ‘input_directory/ERR107794_Staphylococcus_aureus_ST36_CC30.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/6/e/dataset_a6e9cdca-174a-4fae-9bc9-99e67e5628c4.dat’ ‘input_directory/ERR107822_Staphylococcus_aureus_ST22_CC22.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/d/1/c/dataset_d1c435c6-a785-4a15-81ff-38597c91e088.dat’ ‘input_directory/ERR10900234_Staphylococcus_aureus_ST121_CC121.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/9/4/c/dataset_94c4a8fd-1792-494b-af7b-595d7eabff6d.dat’ ‘input_directory/ERR10900246_Staphylococcus_aureus_ST121_CC121.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/3/7/c/dataset_37c5e3d0-2e5a-4a4e-9a31-2f589435bbd8.dat’ ‘input_directory/ERS1179830_Sraphylococcus_aureus_ST36_CC30_NRS740_TN-82.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/e/2/c/dataset_e2c0dca6-e97a-436b-8fed-9a076aa591b1.dat’ ‘input_directory/GCF_000695215.1_Staphylococcus_aureus_ST22_CC22.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/9/8/7/dataset_9871073f-9be5-4d4b-a7dd-c763fc1d646a.dat’ ‘input_directory/GCF_000695875.1_Staphylococcus_aureus_ST8_CC8.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/d/2/b/dataset_d2b1e361-cbef-40ec-8e65-c49bebec048b.dat’ ‘input_directory/GCF_001887075.1_Staphylococcus_aureus_ST398_CC398_E154.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/1/4/4/dataset_144edb93-f972-4657-8fa9-adbcf74fede7.dat’ ‘input_directory/GCF_003031485.1_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/0/4/2/dataset_0425a6f0-5a7b-4913-9450-15401500ba94.dat’ ‘input_directory/GCF_009811395.1_Staphylococcus_aureus_ST8_CC8.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/1/1/dataset_a11e4b51-788e-422e-9d6a-88ca56f78c02.dat’ ‘input_directory/GCF_030342405.2_Staphylococcus_aureus_ST45_CC45_BSN01_Closure_genomic.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/5/8/1/dataset_5810ba99-0879-4e83-900c-7105d704773f.dat’ ‘input_directory/GCF_041356355.1_Staphylococcus_aureus_ST5_CC5_MRSA-16.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/8/f/3/dataset_8f395a45-d303-49bb-9bbb-607f4d7a52de.dat’ ‘input_directory/GCF_053749775.1_Staphylococcus_aureus_ST45_CC_45_SAIE_GP_04.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/2/0/0/dataset_200b6581-969d-4055-b3b7-228ed3b16da9.dat’ ‘input_directory/SRR1159867_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/4/0/dataset_a4015227-7905-43f9-b9b5-6113deb8c1b5.dat’ ‘input_directory/SRR13194715_Staphylococcus_aureus_ST30_CC_30_144KP_subject_1135.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/b/7/e/dataset_b7ea8b8c-2b42-4870-8921-3db7fbd8d4d1.dat’ ‘input_directory/SRR27644380_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/c/b/7/dataset_cb7a1890-be3c-4e0a-951a-0d7ae68e894d.dat’ ‘input_directory/SRR38521030_Staphylococcus_aureus_ST8_CC8.gff3’ && panaroo --clean-mode ‘strict’ --codon-table ‘11’ --remove-invalid-genes --threshold ‘0.98’ --family_threshold ‘0.7’ --len_dif_percent ‘0.98’ --family_len_dif_percent ‘0.0’ --search_radius ‘5000’ --refind_prop_match ‘0.2’ --refind-mode ‘default’ --min_trailing_support ‘2’ --trailing_recursive ‘1’ --edge_support_threshold ‘1.0’ --remove_by_consensus ‘False’ --high_var_flag ‘5’ --min_edge_support_sv ‘2’ --alignment ‘core’ --aligner ‘mafft’ --core_entropy_filter 0.1 -i input_directory/*.gff -o outdir -t ${GALAXY_SLOTS:-8} && mv outdir/gene_presence_absence.Rtab outdir/gene_presence_absence_rtab.Rtab && mv outdir/combined_protein_cdhit_out.txt outdir/combined_protein_cdhit_out.fa

Error: Traceback (most recent call last):
File “/usr/local/bin/panaroo”, line 10, in
sys.exit(main())
^^^^^^
File “/usr/local/lib/python3.11/site-packages/panaroo/_main_.py”, line 318, in main
with open(args.input_files[0], “r”) as file:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: ‘input_directory/*.gff’

4

Hi @Loc_Le

Thanks for trying to make the changes!

I agree that the .gff3 added to the element identifiers is the root problem. Since your original files had “gff3”, the preservation of the full original file name during collection creation was not enough.

I would suggest changing these to be simply sample.gff.

Example: if the current collection element identifier is

AP017922.1_Staphylococcus_aureus_ST6272_CC72_JP080.gff3

Change this to be

AP017922.1_Staphylococcus_aureus_ST6272_CC72_JP080.gff

Or, possibly just this

AP017922.1.gff`

Would you like to share a history with a few of these files in it? Three is enough. We can come up with the exact transformation and give it a test run together.

I’ll watch for your reply!

Update: I decided to create an example with some demo files, then I put those steps into simple reusable workflow that I think should work for you as well! You should be able to input the collection with the .gff3 extensions. To test it, try toggling to send the output to a new history to isolate the changes from your main history, then test panaroo with it. If it works, you can move back to your main history to continue (or, modify your workflow to include these steps?).

Feel free to modify your copy of the workflow any way you want! If I was using this, I would probably hide all of the intermediate files and include it as a subworkflow in my main workflow that runs Pararoo. This would bundle all the uninteresting technical transformations as part of running the tool. :slight_smile:

Please give this a try!

6

Dear Prof. Jennifer Hillman-Jackson,

Thank you very much for your valuable help.

I have received your reply describing both solutions: (1) renaming .gff3 identifiers to .gff, and (2) the proposed workflow.

I have successfully tested solution (1), and it appears that Panaroo works correctly once the file name extensions are changed accordingly on the local PC before compiled into Galaxy collection.

The downstream SNP-sites and IQ-TREE steps also ran without any issues.

For reference, here is the shared history you requested. Unfortunately, I am unable to post a direct link in this reply box, so I had to replace all forward slashes (/) with backslashes (\) and all periods (.) with commas (,). My sincere apologies for the inconveniences.

https:\usegalaxy,eu\u\tuanlocle\h\unnamed-history

Please feel free to inspect or reuse any of the datasets. They are all public genomes obtained from NCBI.

At the moment, I am testing solution (2), which incorporates the workflow you proposed. I will send you the results as soon as possible.

Thank you again for your time and assistance.

Best regards,

Le Tuan Loc

7

Dear Prof Jennifer Hillman-Jackson.

I have performed the second test using the tools in the workflow you sent.

Look like it really generated the gff files in the form usable for Panaroo.

Here is the history of that test. Unfortunately, I am unable to post a direct link in this reply box, so I had to replace all forward slashes (/) with backslashes (\) and all periods (.) with commas (,). My sincere apologies for the inconveniences.

https:\usegalaxy,eu\u\tuanlocle\h\unnamed-history-1

I am very grateful for your help and assistance so far. Thank you very much for your help.

Sincerely,

Le Tuan Loc

9

:rocket:
The result looks great! I’m really glad this worked for you. Do you think this is resolved? We are still looking into adjusting the Galaxy wrapper around the tool but this one is a bit tricky since the underlying code is very strict about the file naming, and the way we organize files is abstracted in an object store.

So, a technical issue to solve, but we are aware of the complication. If anyone is reading this later, please sure the details of what your files are currently named and we can help to tune a small workflow to rename in a way the tool can interpret. I created a new ticket to track this specifically since it seems to have been missed when making the prior adjustments.

And .. We’ve had to be fairly aggressive for the spam filtering but maybe we are currently too strict! Or, am I misunderstanding? Are you posting at the forum directly or through some other portal (AI chat? Which one? Maybe this can be mitigated.). I’d like to review and thanks for letting us know this was a problem. Share links should always be accepted here, even for brand new users.