I have a workflow that is working fine, the only issue is that when I run it on multiple datasets, it outputs everything for all jobs to the same history. This is not optimal and I would prefer that the output from each dataset goes into a separate history. Yes, I can tag each dataset so that I know which outputs came from which inputs, but still I would rather have separate histories. Checking the box that says send to a new history still sends everything to the same one. Anyone have any ideas? Thanks!
Hi @kam132132
Yes, you found the available option. Everything can be written to a new history and tagged. Then you can copy datasets into new histories if you want to later. What you see in a history as a “dataset” is just a pointer to the actual dataset – so you can have as many copies as you want across histories, and it is all still just that original file (and only counted once against any quota limits, or storage location).
One workflow invocation + one workflow version + one output history
- All the original inputs are mapped over into that output history if it is different than the starting data history. This is intentional. Together, these function as a reproducibility “log” for what happened with all the details.
- Tagging the original samples will propagate that tag down into all the outputs.
- Search the history with a tag (or click on that tag on any dataset) to filter that history view for just the associated data with that same tag.
If the reason you want to split up the outputs was for data storage reasons, you can split where the data is stored between storage locations when executing the workflow. We have a topic here with some screenshots can explain how to use that function. → Result files were deposited in different storage (from 30-day storage to permanent storage) - #2 by jennaj
One more option: If you are running large batches, you could execute the workflow as a batch. Run it once per sample in a loop. You would need to use the API for this custom implementation. All the results would write back into a history for application review or sharing for later. You can run jobs using the API against public servers with your standard account.
Hope this helps! 