Old bam files not running in GATK or deeptools after moving files to new server

HI,

We just recently moved all of our Galaxy data to a new server with python 2.7 so that we could upgrade to the latest version of galaxy. Everything seems to be working pretty well except that when I try to re-run old GATK or deeptools tasks, the job fails.

I have checked/compared “moved data” versus “re-uploaded data” using these two methods:

  1. moved bam data: job errors: checked file locations to make sure they match, checked permissions, run jobs with new data (error)
  2. re-uploaded" bam data: job success: copied data (from the new server to my local machine) and re-uploaded them. Jobs run successfully with this data creating the same output as the old jobs

I am wondering if anyone might know why this is happening with method 1. There does not seem to be anything wrong with the bam files or the tools.

Any ideas would be great, thanks!

Below are the errors output by some of the tools after BAM were only moved (not re-uploaded) using method 1

Deeptools bam compare output

Fatal error: Exit code 1 ()
The file one.bam does not exist

GATK haplotype caller

##### ERROR MESSAGE: File /data/galaxy/tmp/tmp-gatk-PBNEqR/gatk_input_0.bam.bai 
is malformed: Premature end-of-file while reading BAM index file /data/galaxy/tmp/tmp-gatk-
PBNEqR/gatk_input_0.bam.bai. 
It's likely that this file is truncated or corrupt -- 
Please try re-indexing the corresponding BAM file.

Indel realigner

##### ERROR MESSAGE: File /data/galaxy/tmp/tmp-gatk-tXbYgP/gatk_input.bam.bai is malformed: 
Premature end-of-file while reading BAM index file /data/galaxy/tmp/tmp-gatk-tXbYgP/gatk_input.bam.bai. 
It's likely that this file is truncated or corrupt -- 
Please try re-indexing the corresponding BAM file.

Note: Moderator reformatted post for clarity

2 Likes

Update* I have located the bam indexes on our original server and discovered that the _metadata files (containing the indexes) in the files dir were not copied over with the rest of the files. Copying these files over has fixed this issue.

2 Likes

Great, glad you found the problem and thanks for posting back the solution!