fixing corrupted images (raw, tiff, or jpeg)

Canon EF 70-200mm f/4.0L USM

10/16/2003 01:36:26 PM · #1

Is anyone familiar with software which attempts to fix corrupted image files? I'm not interested in data recovery tools that are fat32-specific, such as those used to retrieve "lost" images from cf cards and such. The damage I am looking at lies entirely within the file itself; I could be more specific, but it probably would be meaningless to most.

RHoldenSr

Canon EOS-5D

10/16/2003 01:39:12 PM · #2

What is exactly wrong with the file?

It won't load?
the EXIF data is corrupted?

Can you give more details.

Calvus

10/16/2003 02:33:09 PM · #3

Originally posted by Calvus:

What is exactly wrong with the file?

Some bits that aren't supposed to be there, are.

Originally posted by Calvus:

It won't load?

Loading is generally not a problem, though that generally depends on the software used.

Originally posted by Calvus:

the EXIF data is corrupted?

Usually, this depends on the file format, but it's safe to assume that the EXIF data is not corrupted.

Originally posted by Calvus:

Can you give more details.

The problem seems related to the data found in (some) blocks, and, as far as I can tell, only those blocks referenced by indirect inodes. Each block has up to 1 pair of bytes inserted, with the last two bytes being truncated (lost forever perhaps). The byte pairs inserted are usually duplicates: if the correct data is the byte tuple {b_1, b_2, ..., B_N}, then the corrupt data is a tuple that looks like {b_1, b_2, ..., b_i, b_i+1, b_j, b_j++1, ..., b_N-2} and much of the time b_j = b_i, b_j+1 = b_i+1. Otherwise (almost always), one of b_j or b_j+1 is the same as it's i'th counterpart and the other is nearly bit-wise equivalent to its counterpart.

I'm sure that helps.

Sigma 12-24mm f/4.5-5.6 EX Aspherical DG for Canon

10/17/2003 07:55:35 AM · #4

robsmith

Canon EOS-300D Rebel

10/17/2003 08:20:05 AM · #5

Originally posted by dwoolridge:

Originally posted by Calvus:
What is exactly wrong with the file?

Some bits that aren't supposed to be there, are.

Originally posted by Calvus:
It won't load?

Loading is generally not a problem, though that generally depends on the software used.

Originally posted by Calvus:
the EXIF data is corrupted?

Usually, this depends on the file format, but it's safe to assume that the EXIF data is not corrupted.

Originally posted by Calvus:
Can you give more details.

The problem seems related to the data found in (some) blocks, and, as far as I can tell, only those blocks referenced by indirect inodes. Each block has up to 1 pair of bytes inserted, with the last two bytes being truncated (lost forever perhaps). The byte pairs inserted are usually duplicates: if the correct data is the byte tuple {b_1, b_2, ..., B_N}, then the corrupt data is a tuple that looks like {b_1, b_2, ..., b_i, b_i+1, b_j, b_j++1, ..., b_N-2} and much of the time b_j = b_i, b_j+1 = b_i+1. Otherwise (almost always), one of b_j or b_j+1 is the same as it's i'th counterpart and the other is nearly bit-wise equivalent to its counterpart.

I'm sure that helps.

Not wishing to sound too technical (or pompous), so in laymans terms I'd say "you're screwed"

10/17/2003 08:27:22 AM · #6

Is it RAW, TIFF or JPEG files you are trying to recover ?

Is it a specific file or are you just asking generally ?

10/17/2003 08:47:04 AM · #7

Originally posted by Gordon:

Is it RAW, TIFF or JPEG files you are trying to recover ?

All three.

Originally posted by Gordon:

Is it a specific file or are you just asking generally ?

No, it's not just one file, perhaps a few hundred.

10/17/2003 09:17:03 AM · #8

For RAW files, I'd suggest maybe trying several different RAW converters if you have access to them or trial versions

(e.g., breezebrowser, C1, Photoshop) that use different loader libraries. Some may be more tolerant of the particular corruption in your files than in others.

For the other formats, a search on google for TIFF file repair and JPEG file repair generated a lot of hits, probably of variable quality.

If you've got corrupted parts of the files, you will probably lose some of each photo, there isn't a whole lot of redundancy in the file format for a JPEG for example.

10/17/2003 10:02:00 AM · #9

Originally posted by Gordon:

For RAW files, I'd suggest maybe trying several different RAW converters if you have access to them or trial versions

I tried Canon's sw, breezebrowser, and photoshop acr, all to no avail. Only photoshop acr was graceful enough not to completely bail out. I haven't tried the crw tool yet, but I'll leave it for later.

Originally posted by Gordon:

For the other formats, a search on google for TIFF file repair and JPEG file repair generated a lot of hits, probably of variable quality.

Yeah, and almost completely useless. Naturally, this was my first (well, second) avenue, but the JPEG tools in particular fall into 1 or 3 camps: file recovery from "loss"; JPEG header recovery/repair; and JPEG artifacts (low quality) "repair".

I hadn't yet looked at tiff recovery tools.

For JPEG, because the problem is block-level byte (pair) shifts, you get interesting and predictable results. By correcting these shifts, I expect to lose some quality in a small area of data (whatever the JPEG compression area unit is in these cases). I can't predict what the RAW results will be and I can't remember if the D60 uses compressed RAW data now or not.

10/17/2003 10:14:38 AM · #10

The RAW files are losslessly compressed from what I can tell. They are certainly image content dependant so compressed, but probably just run-length encoded

As for repairing them, if they are all corrupted in a predictable fashion it should be possible to write tools to repair them - something like perl or python could do it quite easily.

I doubt you'll find an off the shelf tool that deals with what you are seeing in this specific case though...

What did you do to them in the first place ? :)

Canon EF 70-200mm f/4.0L USM

10/17/2003 10:33:49 AM · #11

Originally posted by Gordon:

As for repairing them, if they are all corrupted in a predictable fashion it should be possible to write tools to repair them - something like perl or python could do it quite easily.

Well, when I say predictable, I mean the block-level analysis matches what one sees visually, though the results are more severe as the file size increases. Anyway, I've already started writing a tool. I find Perl and python are too slow for me; I prefer C. Since predicting byte pair corruption in blocks is not very discriminating, I'd really like a visual front end that does the same thing using multiple passes.

Maybe I'll write a photoshop plug-in if I can find the time. I doubt it would be much use to anyone but me though.

Originally posted by Gordon:

What did you do to them in the first place ? :)

They got fsck'd. Personally, I think it's a bug in FreeBSD'd fsck and normally I'd be quite keen to hunt it down, but my data corruption is more important to me at the moment.

The really sad part is I use an IDE-based RAID mirroring hardware device and I also made a full mirror of the corrupted data (what the hell was I thinking?).

All the software/hardware in the world can't save me from myself.

RHoldenSr

Canon EOS-5D

10/17/2003 10:44:08 AM · #12

If they were this way when you downloaded them from your camera:

I would look at the storage card.
Has it over heated (In a car)?
How old is it?

If you used a card reader I would also review how you used it?
Did you unplug it before inserting or removing the card?
Some require both or one of these methods.

If they were ok when you got them off the camera I would say you possibly have a hard drive problem, i.e. fragmentation causing cross linked files or etc.

Have you defragged the hard drive lately?

Is the hard drive giving you any errors?
Is the OS locking up on you?
Blue screen of death (Windows)

There are other reasons this could have happened, it is hard to diagnose without being able to physically check everything.

I sure hope you find the answer and recover them.

Calvus

Originally posted by dwoolridge:

No, it's not just one file, perhaps a few hundred.