Panaroo 1.8.0+galaxy0 fails on GFF3 collection with FileNotFoundError: input_directory/*.gff

usegalaxy.eu support
, ,
1

Hello,

I encountered what appears to be a wrapper issue in Panaroo (Galaxy wrapper version 1.8.0+galaxy0 on usegalaxy.eu).

Input:

- Collection of 19 GFF3 files

- All files were generated by Bakta

- Collection datatype: gff3

The job fails immediately with:

FileNotFoundError: [Errno 2] No such file or directory: ‘input_directory/*.gff’

Relevant traceback:

Traceback (most recent call last):

File “/usr/local/bin/panaroo”, line 10, in

sys.exit(main())

File “/usr/local/lib/python3.11/site-packages/panaroo/_main_.py”, line 318, in main

with open(args.input_files[0], “r”) as file:

FileNotFoundError: [Errno 2] No such file or directory: ‘input_directory/*.gff’

From the generated command line I can see that the wrapper creates symlinks such as:

input_directory/SRR38521030_Staphylococcus_aureus_ST8_CC8

(without .gff extension)

but Panaroo is invoked with:

-i input_directory/*.gff

This suggests that no files match the wildcard and Panaroo receives the literal string ‘input_directory/*.gff’.

Could this be a wrapper issue related to handling GFF3 collections?

Thank you.

2

Hello @Loc_Le

Yes, this tool is a bit tricky to use. A full adjustment at the tool wrapper level is pending but for now, the solution is to include .gff on the collection identifiers.

You can recreate the collection (retaining the file extensions) or use the Extract Element identifiers, text editing tools, then Relabel identifiers functions to add these back in. And, for HTP use cases, Apply Rules can make this sort of adjustment too.

Recreating the collection is the easiest, so try that first if this seems overly complicated.

Hope this helps and follow up questions are welcome! :slight_smile:

3

Dear Prof. Jennifer Hillman-Jackson.

Thank you for your earlier suggestion, and my apologies for a very delayed follow-up.

I tried recreating the collection in Galaxy and deliberately left the “remove file extension” option unchecked. Unfortunately, the job still failed.

I then followed your alternative suggestion using the Extract Element Identifiers and Relabel Identifiers tools to restore the file extensions. Unfortunately, this did not resolve the issue either.

The generated symlinks are now created with the .gff3 suffix, for example: input_directory/SRR38521030_Staphylococcus*aureus_ST8_CC8.gff3

However, the wrapper still invokes Panaroo with: -i input_directory/*.gff

Since all generated files end with .gff3, the wildcard appears not to match any files and Panaroo receives the literal string input_directory/*.gff, leading to: FileNotFoundError: input_directory/*.gff

From the generated command line it looks like the wrapper may currently create .gff3 links while still expecting .gff inputs.

I have included the full Galaxy command line and error message below in case the additional details are helpful for debugging.

Thank you again for your time and assistance.

Best regards,

Le Tuan Loc

Run 1: not using Relabel Identifiers

CLI: mkdir outdir && mkdir input_directory && ln -fs ‘/data/dnb13/galaxy_db/files/b/7/5/dataset_b754a17d-3319-444d-88d1-912474ce52e1.dat’ ‘input_directory/AP017922.1_Staphylococcus_aureus_ST398_CC398_ISU926.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/b/a/0/dataset_ba0dd125-511e-431a-882b-ce2d13f17f57.dat’ ‘input_directory/AP017922.1_Staphylococcus_aureus_ST6272_CC72_JP080.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/8/3/f/dataset_83f6d967-07b2-4872-a04f-c8fb3c7ef3e9.dat’ ‘input_directory/ERR107794_Staphylococcus_aureus_ST36_CC30.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/6/e/dataset_a6e9cdca-174a-4fae-9bc9-99e67e5628c4.dat’ ‘input_directory/ERR107822_Staphylococcus_aureus_ST22_CC22.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/d/1/c/dataset_d1c435c6-a785-4a15-81ff-38597c91e088.dat’ ‘input_directory/ERR10900234_Staphylococcus_aureus_ST121_CC121.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/9/4/c/dataset_94c4a8fd-1792-494b-af7b-595d7eabff6d.dat’ ‘input_directory/ERR10900246_Staphylococcus_aureus_ST121_CC121.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/3/7/c/dataset_37c5e3d0-2e5a-4a4e-9a31-2f589435bbd8.dat’ ‘input_directory/ERS1179830_Sraphylococcus_aureus_ST36_CC30_NRS740_TN-82.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/e/2/c/dataset_e2c0dca6-e97a-436b-8fed-9a076aa591b1.dat’ ‘input_directory/GCF_000695215.1_Staphylococcus_aureus_ST22_CC22.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/9/8/7/dataset_9871073f-9be5-4d4b-a7dd-c763fc1d646a.dat’ ‘input_directory/GCF_000695875.1_Staphylococcus_aureus_ST8_CC8.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/d/2/b/dataset_d2b1e361-cbef-40ec-8e65-c49bebec048b.dat’ ‘input_directory/GCF_001887075.1_Staphylococcus_aureus_ST398_CC398_E154.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/1/4/4/dataset_144edb93-f972-4657-8fa9-adbcf74fede7.dat’ ‘input_directory/GCF_003031485.1_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/0/4/2/dataset_0425a6f0-5a7b-4913-9450-15401500ba94.dat’ ‘input_directory/GCF_009811395.1_Staphylococcus_aureus_ST8_CC8.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/1/1/dataset_a11e4b51-788e-422e-9d6a-88ca56f78c02.dat’ ‘input_directory/GCF_030342405.2_Staphylococcus_aureus_ST45_CC45_BSN01_Closure_genomic.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/5/8/1/dataset_5810ba99-0879-4e83-900c-7105d704773f.dat’ ‘input_directory/GCF_041356355.1_Staphylococcus_aureus_ST5_CC5_MRSA-16.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/8/f/3/dataset_8f395a45-d303-49bb-9bbb-607f4d7a52de.dat’ ‘input_directory/GCF_053749775.1_Staphylococcus_aureus_ST45_CC_45_SAIE_GP_04.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/2/0/0/dataset_200b6581-969d-4055-b3b7-228ed3b16da9.dat’ ‘input_directory/SRR1159867_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/4/0/dataset_a4015227-7905-43f9-b9b5-6113deb8c1b5.dat’ ‘input_directory/SRR13194715_Staphylococcus_aureus_ST30_CC_30_144KP_subject_1135.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/b/7/e/dataset_b7ea8b8c-2b42-4870-8921-3db7fbd8d4d1.dat’ ‘input_directory/SRR27644380_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/c/b/7/dataset_cb7a1890-be3c-4e0a-951a-0d7ae68e894d.dat’ ‘input_directory/SRR38521030_Staphylococcus_aureus_ST8_CC8.gff3’ && panaroo --clean-mode ‘strict’ --codon-table ‘11’ --remove-invalid-genes --threshold ‘0.98’ --family_threshold ‘0.7’ --len_dif_percent ‘0.98’ --family_len_dif_percent ‘0.0’ --search_radius ‘5000’ --refind_prop_match ‘0.2’ --refind-mode ‘default’ --min_trailing_support ‘2’ --trailing_recursive ‘1’ --edge_support_threshold ‘1.0’ --remove_by_consensus ‘False’ --high_var_flag ‘5’ --min_edge_support_sv ‘2’ --alignment ‘core’ --aligner ‘mafft’ --core_entropy_filter 0.1 -i input_directory/*.gff -o outdir -t ${GALAXY_SLOTS:-8} && mv outdir/gene_presence_absence.Rtab outdir/gene_presence_absence_rtab.Rtab && mv outdir/combined_protein_cdhit_out.txt outdir/combined_protein_cdhit_out.fa

Error: Traceback (most recent call last):
File “/usr/local/bin/panaroo”, line 10, in
sys.exit(main())
^^^^^^
File “/usr/local/lib/python3.11/site-packages/panaroo/_main_.py”, line 318, in main
with open(args.input_files[0], “r”) as file:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: ‘input_directory/*.gff’

Run 02: using Relabel identifier

CLI: mkdir outdir && mkdir input_directory && ln -fs ‘/data/dnb13/galaxy_db/files/b/7/5/dataset_b754a17d-3319-444d-88d1-912474ce52e1.dat’ ‘input_directory/AP017922.1_Staphylococcus_aureus_ST398_CC398_ISU926.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/b/a/0/dataset_ba0dd125-511e-431a-882b-ce2d13f17f57.dat’ ‘input_directory/AP017922.1_Staphylococcus_aureus_ST6272_CC72_JP080.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/8/3/f/dataset_83f6d967-07b2-4872-a04f-c8fb3c7ef3e9.dat’ ‘input_directory/ERR107794_Staphylococcus_aureus_ST36_CC30.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/6/e/dataset_a6e9cdca-174a-4fae-9bc9-99e67e5628c4.dat’ ‘input_directory/ERR107822_Staphylococcus_aureus_ST22_CC22.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/d/1/c/dataset_d1c435c6-a785-4a15-81ff-38597c91e088.dat’ ‘input_directory/ERR10900234_Staphylococcus_aureus_ST121_CC121.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/9/4/c/dataset_94c4a8fd-1792-494b-af7b-595d7eabff6d.dat’ ‘input_directory/ERR10900246_Staphylococcus_aureus_ST121_CC121.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/3/7/c/dataset_37c5e3d0-2e5a-4a4e-9a31-2f589435bbd8.dat’ ‘input_directory/ERS1179830_Sraphylococcus_aureus_ST36_CC30_NRS740_TN-82.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/e/2/c/dataset_e2c0dca6-e97a-436b-8fed-9a076aa591b1.dat’ ‘input_directory/GCF_000695215.1_Staphylococcus_aureus_ST22_CC22.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/9/8/7/dataset_9871073f-9be5-4d4b-a7dd-c763fc1d646a.dat’ ‘input_directory/GCF_000695875.1_Staphylococcus_aureus_ST8_CC8.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/d/2/b/dataset_d2b1e361-cbef-40ec-8e65-c49bebec048b.dat’ ‘input_directory/GCF_001887075.1_Staphylococcus_aureus_ST398_CC398_E154.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/1/4/4/dataset_144edb93-f972-4657-8fa9-adbcf74fede7.dat’ ‘input_directory/GCF_003031485.1_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/0/4/2/dataset_0425a6f0-5a7b-4913-9450-15401500ba94.dat’ ‘input_directory/GCF_009811395.1_Staphylococcus_aureus_ST8_CC8.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/1/1/dataset_a11e4b51-788e-422e-9d6a-88ca56f78c02.dat’ ‘input_directory/GCF_030342405.2_Staphylococcus_aureus_ST45_CC45_BSN01_Closure_genomic.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/5/8/1/dataset_5810ba99-0879-4e83-900c-7105d704773f.dat’ ‘input_directory/GCF_041356355.1_Staphylococcus_aureus_ST5_CC5_MRSA-16.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/8/f/3/dataset_8f395a45-d303-49bb-9bbb-607f4d7a52de.dat’ ‘input_directory/GCF_053749775.1_Staphylococcus_aureus_ST45_CC_45_SAIE_GP_04.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/2/0/0/dataset_200b6581-969d-4055-b3b7-228ed3b16da9.dat’ ‘input_directory/SRR1159867_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/a/4/0/dataset_a4015227-7905-43f9-b9b5-6113deb8c1b5.dat’ ‘input_directory/SRR13194715_Staphylococcus_aureus_ST30_CC_30_144KP_subject_1135.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/b/7/e/dataset_b7ea8b8c-2b42-4870-8921-3db7fbd8d4d1.dat’ ‘input_directory/SRR27644380_Staphylococcus_aureus_ST5_CC5.gff3’ && ln -fs ‘/data/dnb13/galaxy_db/files/c/b/7/dataset_cb7a1890-be3c-4e0a-951a-0d7ae68e894d.dat’ ‘input_directory/SRR38521030_Staphylococcus_aureus_ST8_CC8.gff3’ && panaroo --clean-mode ‘strict’ --codon-table ‘11’ --remove-invalid-genes --threshold ‘0.98’ --family_threshold ‘0.7’ --len_dif_percent ‘0.98’ --family_len_dif_percent ‘0.0’ --search_radius ‘5000’ --refind_prop_match ‘0.2’ --refind-mode ‘default’ --min_trailing_support ‘2’ --trailing_recursive ‘1’ --edge_support_threshold ‘1.0’ --remove_by_consensus ‘False’ --high_var_flag ‘5’ --min_edge_support_sv ‘2’ --alignment ‘core’ --aligner ‘mafft’ --core_entropy_filter 0.1 -i input_directory/*.gff -o outdir -t ${GALAXY_SLOTS:-8} && mv outdir/gene_presence_absence.Rtab outdir/gene_presence_absence_rtab.Rtab && mv outdir/combined_protein_cdhit_out.txt outdir/combined_protein_cdhit_out.fa

Error: Traceback (most recent call last):
File “/usr/local/bin/panaroo”, line 10, in
sys.exit(main())
^^^^^^
File “/usr/local/lib/python3.11/site-packages/panaroo/_main_.py”, line 318, in main
with open(args.input_files[0], “r”) as file:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: ‘input_directory/*.gff’

4

Hi @Loc_Le

Thanks for trying to make the changes!

I agree that the .gff3 added to the element identifiers is the root problem. Since your original files had “gff3”, the preservation of the full original file name during collection creation was not enough.

I would suggest changing these to be simply sample.gff.

Example: if the current collection element identifier is

AP017922.1_Staphylococcus_aureus_ST6272_CC72_JP080.gff3

Change this to be

AP017922.1_Staphylococcus_aureus_ST6272_CC72_JP080.gff

Or, possibly just this

AP017922.1.gff`

Would you like to share a history with a few of these files in it? Three is enough. We can come up with the exact transformation and give it a test run together.

I’ll watch for your reply!

Update: I decided to create an example with some demo files, then I put those steps into simple reusable workflow that I think should work for you as well! You should be able to input the collection with the .gff3 extensions. To test it, try toggling to send the output to a new history to isolate the changes from your main history, then test panaroo with it. If it works, you can move back to your main history to continue (or, modify your workflow to include these steps?).

Feel free to modify your copy of the workflow any way you want! If I was using this, I would probably hide all of the intermediate files and include it as a subworkflow in my main workflow that runs Pararoo. This would bundle all the uninteresting technical transformations as part of running the tool. :slight_smile:

Please give this a try!