Anyone converting their archives to DNG?

01/05/2009 10:46:02 PM · #1

I'm seriously considering switching my archival files to DNG...I'm a dedicated LR user now, I've had 4 different SLRs (RAW formats), and I'm getting a bit tired of having twice as many files in a folder due to the sidecar files. When you have over a terabyte of image files, the sidecar files are wasteful and can lead to disk fragmentation. Not to mention, since they are not supported by the OS, any program looking at the folder sees the hodgepodge of files there.

Other than them not being legal as proof files in DPC (I'm talking about my archive here, and I always make a DVD backup with the RAW files as soon as I shoot), can anyone share their experiences and rationale about switching (or not)?

Sigma 18-300mm f3.5-6.3 DC Macro OS HSM

01/06/2009 10:11:21 AM · #2

* bump *

jhomrighaus

Nikon D5300

01/06/2009 10:27:38 AM · #3

Not sure what you mean by sidecar files, I only convert out of raw the images I want to work on, the rest just stay as raw until I want them, of course a lot of my photos are crap so that may be part of it. If you have the original Raw files on DVD then you are covered to a certain extent. Why not just compress the raw images into Zip files in your archive then you can pull them back up full later. Or for that matter why not store them as lossless Jpegs which I thought were smaller than raws anyway.

I am not overly familiar with DNG format myself but I will take a look at it and see whats what.

Five_Seat

Canon EOS-6D

Canon EF 24-105mm f/4.0L IS

01/06/2009 10:29:56 AM · #4

I started to at one point, did it for maybe a month or two, then stopped. I did it mainly for the archival benefits provided by the file standard, but I also found that it cut down on the size of my raw files from my 20D by around 2MB a file which was nice.

I stopped because for DPC, I would need the RAW file, and I was really trying to limit the amount of redundancy in workflow/free up some space on my HDD, but I bought an external HDD and space is no longer really an issue for me. It sort of became a hassle to do the convert.

Also, this is probably a topic for another thread, but I have some relatively unfounded feeling that when you convert something to DNG, you are losing something. I have to investigate this further and run some comparions between Canon DPP and Aperture, which is the main program I use.

01/06/2009 10:33:53 AM · #5

from a technical standpoint, after reading the DNG spec, I can't find a reasonable reason to bother doing it, other than if you want to try to make life easier for Adobe. That's the only company that I see really getting helped by DNG. It saves them having to reverse engineer the format, while still letting the vendors add any old proprietary stuff they like, into the DNG format (e.g., you could still encrypt your white balance, while saving it in DNG format, if you wanted to, like Nikon tried)

You still have to assume that Adobe got the reverse engineering of the proprietary formats correct in the first place, when you convert to DNG - and if they have, there is no point doing it. Otherwise, you can always store the image as DNG, with the original embedded in the DNG as well, which just means you've doubled your disk space for no advantage. Maybe in 6 months Adobe reverse engineer another useful bit of the CR2 format that they didn't previously understand, and now you can get better RAW conversions from those files (the RAW converters keep getting better after all) and that information wasn't properly translated into the DNG format. Seems possible, given that this is the very problem DNG is claiming to be a solution for in the future when everyone adopts it.

Maybe there will be a reason for it in the future, when Canon and Adobe have gone out of business and nobody supports Canon RAW formats. Then it might make sense to convert the images to a format that is readable.

DNG is mostly a bag (an image data storage format), not an image format (much like TIFF) - it defines where stuff goes, not what goes in it.

I haven't yet seen a convincing argument for users. There's good financial reasons for it if you have shares in Adobe though.

I like using sidecar files for a fairly specific reason - I'd rather the image files are being touched as little as possible. A program crashes (which LR does on occasion) and leaves an image file open, maybe you lose an original if it is manipulating the DNG file at the time. With an original RAW and sidecar, the RAW file is always just read only.

I think DNG has potential in the future, if it evolves to be a standard that is used by the camera manufacturers to store their data in an open, specified way. For now though, it is a convenience for the software writers, in that they can potentially find the bits of information in a RAW file more easily, but it still doesn't do enough to define what that information actually should be and how it should be stored. You could very easily use DNG and create an entirely proprietary RAW format, within the open DNG format and nobody would be any better off, other than the companies like Adobe.

E.g., from the DNG spec, pages 15 and page 37

Proprietary Data
Camera manufacturers may want to include proprietary data in a raw file for use by their own
raw converter. DNG allows proprietary data to be stored using private tags, private IFDs,
and/or a private MakerNote.
It is recommended that manufacturers use the DNGPrivateData and MakerNoteSafety tags to
ensure that programs that edit DNG files preserve this proprietary data. See Chapter 4, â��DNG
Tagsâ�� on page 17 for more information on the DNGPrivateData and MakerNoteSafety tags.

and:

DNGPrivateData
Description:
DNGPrivateData provides a way for camera manufacturers to store private data in the DNG
file for use by their own raw converters, and to have that data preserved by programs that edit DNG files.

Message edited by author 2009-01-06 10:49:53.

Canon EF-S 10-22mm f/3.5-4.5 USM

01/06/2009 10:35:36 AM · #6

dup

Message edited by author 2009-01-06 10:40:46.

undieyatch

Olympus E-10

01/06/2009 12:30:21 PM · #7

I like it. A great concept & it works.
Some camera makers already use the format DNG (Leica) or have included it (Pentax) as a format choice in recent camera models.
It is the only universal format as any raw file can be accommodated. It is a perfect & versatile archive for collections of various raw files.

01/06/2009 12:33:23 PM · #8

Thought I'd walk through pros and cons "aloud" here. May change my mind:

Pros:

- I don't have the extra sidecar files (XMP)
o That means you can use "other" programs to manage your files and not get the XMP and RAW files out of sync, e.g., move files in ACDSee (of course your LR DB then needs to be resync'd)
o A lot of little XMP files removed, means less fragmentation of your disk and directories
- Fewer file formats to support (I have CRW, CR2, and two different NEF formats.... so far!)
- LR may be speedier with DNGs (and it canb store a larger preview; not sure yet if this helps!)
- Saves you from upgrading PS CS all the time, per Adobe's forced policy that the RAW converter isn't updated for older versions of PS (twice I've bought a new camera only to find that if I didn't upgrade CS, I wouldn't be able to use RAW).

Cons:

- Making adjustments in LR means that the large DNG files need to be backed up again--rather than the tiny XMP files! (that's a big CON!)
- File dates changed during conversion--lose the external display of capture time (which is otherwise preserved when I copy the files from my media)
(This is just a converter limitation--they obviously don't think it's important to give you a preserve file date option, but I do!)
- Potential bugs in DNG converter means that one day I may have to go back to the "backup", and restore my NEF files, extract the XMP from the DNG, and redo!

Hmmm, that first con might have just convinced me not to go with DNG.

Pentax D-FA* 70-200mm f/2.8 ED DC AW

01/06/2009 12:45:35 PM · #9

You can, if you wanted, configure lightroom not to use sidecar files for metadata, and just keep it in the Lightroom Catalog. You can do this with the RAW files copied into the catalog, or left where they are too. Just means you have to back up the catalog and can't share the metadata so easily with other Adobe tools.

Assuming you are using a recent file system (FAT32/ NTFS) the XMP files will only be taking up a small amount of space on the disk. 32Kb blocks are fine on a FAT32 system, so they don't waste much space at all and as you mentioned, get backed up quickly.

The main reason Adobe use XMP is that they don't properly understand the CR2/RAW/ NEF formats well enough to want to extend/ modify them - by extension, you'd think they wouldn't be so sure they could parse everything perfectly to create a complete DNG either. To avoid this problem, many people embed the original RAW file into the DNG, effectively doubling the size of every negative. If Adobe didn't have enough doubts to add that feature in, I doubt it'd be there.

Certainly converting everything to DNG would save you from needing PS to know the RAW formats. I don't think I've used the PS RAW converter at all since Lightroom came out though - I do the initial conversion in LR then open in PS. You'd still need to update whatever tool you were converting to DNG with I suppose. Cheaper than PS maybe.

Message edited by author 2009-01-06 12:46:11.

K3Master

Pentax K-3

01/06/2009 12:51:32 PM · #10

So, DPC doesn't accept DNG as originals then? My camera can shoot DNG natively. If I chose to do so, would I be screwed?

Pentax D-FA* 70-200mm f/2.8 ED DC AW

01/06/2009 01:01:33 PM · #11

K10DGuy: DNG is acceptable as proof when it's the native capture format. Not when you convert after the fact though.

Gordon: As far as forcing upgrades, actually, they do that not only for PS but LR too. Though I always convert and process in LR too. I just didn't like the fact when I was stuck on CS 1 that I could not open my camera RAW files from the XT, or my Nikons even if I wanted to. I use NTFS, but I still worry that having a lot of little files can lead to MFT fragmentation.

I have decided not to convert, based on the backup issue. Unfortunately, I'm not thrilled about the idea of keeping the edits in the database either, since it's easy for the DB's association with the file to break, and of course, it makes an incremental backup strategy less robust for preserving info at a low bandwidth.

I did convert a few folders already, and I'll just leave them converted as a bridge to nowhere ;)

K3Master

Pentax K-3

01/06/2009 01:02:58 PM · #12

Originally posted by nshapiro:

K10DGuy: DNG is acceptable as proof when it's the native capture format. Not when you convert after the fact though.

Ah. Gotcha.