How to make sure your photos are digitally safe for a 100 years
In 2003 I thought I was starting a family photo scanning project which was going to be just 11 albums containing about 1,100 photos. The idea was to scan them, describe them and put them on the web for family, extended family and friends to see. What I ended up with was over 8,000 photos, two dozen films, 3 website incarnations and more to come.
Little did I realise how big the project would be come, how big was the challenge, how rewarding and entertaining it would be for those looking at the end result. My task was much larger than most readers would face, but I didn't know that when I started. I was able to unearth professional portrait photos from the late 1890's, glass slides from the early 1900's, celluloid negatives from the 1920's, photos and films from my mother and her family from the 1930's and 40's, then more photos and negatives from my own family as my older brothers were taught basic photography by my mother who worked as a professional photographer before the war.
My aim here is to write up what I did and how I would do it if I was starting today. I am writing this as a note to self and to help anyone embarking on a similar project. We will be looking at:
I used various Apple Macintosh computers for this project.
I knew I needed a scanner capable of handling prints, slides and negatives of all sizes. The only scanner I found at the time capable of scanning large 120, larger negatives and glass slides was an Epson 3200 which was top of the consumer line at the time. After a number of tests I chose to scan prints at 600 dpi, older negatives at 1,200 dpi, newer and 35 mm negs and slides at 2,400 dpi which was about double the prevailing recommendations at the time.
I wanted to ensure the originals would never need to be rescanned, so I was only happy when I knew that going any higher with the DPI count would not make any difference to the quality of the scan.
Scanning at such a high resolution was always going to create a problem with storage. Ideally I wanted to store the scans in a lossless format, but this was not practical. So, after more tests I decided I would save in JPEG format at 100% after making the most basic corrections. I found no visible degradation of the images. If I was starting now I would consider saving the images in either the PNG or a Raw format. I would be mindful that PNG files are very much larger than JPEGs (See about formats).
There are a few points to consider when scanning. A flat bed scanner used for scanning negatives and slides is a compromise, but you are highly unlikely to find a slide scanner capable of scanning large negatives.
Many of the early cameras leaked light so what may have been a good photo could easily be blown out and weren't printed. Often adjusting the scanning area to just the good portion of the photo will obtain a good result. Scanning a negative is preferred over a print, but often the negatives will be scratched making a difficult choice as to what to keep - print or negative.
Many of the negatives you scan were probably never printed, so you obtain photos that have never been seen. Regardless of whether it is a good or bad photo, keep it as it may give joy regardless of its quality. I found myself many times thinking a scan should be deleted to have someone think it was great when seeing the same photo.
The chances are the same photo will be scanned numerous times. Don't worry about that until you have finished the scanning project when you will be in a better position to find duplicates and pick the best scans.
Make sure you scanner is set up where you can also do other things. Many a scan was made in front of the TV.
Summary
I briefly mentioned file formats and how I used the highest quality JPEG format. I also mentioned that if I was starting the project today I may use either PNG or DNG. Below is a discussion about image file formats. Read this before you start scanning.
JPEG
If you open a JPEG, edit it, save it - you loose quality. If you intend doing a lot of edits on a JPEG, convert it to a lossless format, do your work there and then save it as a new JPEG if required. Editing the metadata which also includes rotation doesn’t need to re-save the image, so doesn’t degrade quality.
A newer JPEG2000 standard is like JPEG, but better and it supports lossless compression. Unfortunately JPEG2000 has gained little traction and is probably best avoided for archiving.
TIFF
Can support both lossy and lossless compression, but with larger files than JPEG. Overtime the format has become fragmented, so it is best to keep clear of it.
PNG
Supports lossless compression and is more efficient than TIFF. According to the DigiKam developers it is an ideal format for archiving photos.
PGF
Developed as a JPEG replacement, but not very popular.
RAW
Raw sounds great, but there isn’t a standard for raw between camera manufacturers and they change their own Raw specifications from time to time. It is not a file format that can be used for archiving. Fortunately, Adobe has an open Raw format called DNG.
DNG
Stands for Digital Negative. It is an open standard that is well supported. The digiKam developers recommend that Camera Raw files be converted to this format as it is future proof and keeps the original intact after editing the metadata.
Summary
Unless you are very lucky, you will probably find you are given a box of prints and negatives without any semblance of order. Sort out the negs from the prints and just start scanning.
My process was to scan the photos, type up descriptions into a text document as I scanned. Noting the file number, description, people and date. Often the date was a rough estimate, but sometimes you would find the date developed stamped on the photos. I then created low resolution copies of the photos and imported these into a Filemaker Pro database with the descriptions and file names. After time I worked out I wanted the file name, keywords, headline, caption, people and date. I was then able to use this database to quickly show family members the photos, add and correct the descriptions and look for duplicates. Later I was able to import the information into the photos, sort them by period, rename the photos and file them away.
Many times I would have family over to look at the newest scans to work on the descriptions. Sometimes it would be a visit to older relatives to do the same.
If I was starting today I would scan directly into Adobe Bridge or Graphics Converter and enter the descriptions directly into the images. If I thought the descriptions where common amongst many photos, I would probably use a spreadsheet and import the data later using Graphics Converter import CSV metadata functions.
Summary
Every digital photo you take contains MetaData. Open a photo in Preview and go to Tools/Inspector (command i) and you will see it. This applies to just about any graphics program. They should show:
File Properties - General
Exif - Camera Data
IPTC Core
IPTC Extension
For more details go to:
https://photometadata.org/META-Resources-Field-Guide-to-Metadata
Not all software shows all the metadata and the metadata can be stored in the file or outside it in what is called a sidecar. JPEGs can store it in the photo, while Camera raw files cannot. In addition, non destructive software will sometimes store the metadata in the sidecar or somewhere else.
Not all software use the same titles for metadata fields. For example the photographer field in Adobe Bridge and Apple Preview is called the Creator, while Graphics Converter uses Byline. The same goes for the Creators title. Just be aware they may differ.
The main items we are concerned with is the IPTC information to do with:
Caption/Description, Caption/Description Writer, Headline, Object Name/Title, Keywords, People, Copyright and contact details.
Why is this important?
If you want to distribute and/or properly archive your photos you need to make sure the correct information is embedded into the photos - just like writing on the back of a print. Not all software handles metadata the same. Given the right software you can create web galleries and high quality PDF’s which display the metadata. In addition, the metadata specifications have changed and will continue to change over time.
Summary
We are mainly concerned with with:
You should make your own enquires about copyright. What I found was that many of the image hosting sites attempt to lay claim to your photos. It could be outright ownership or the ability to use your photos for their own purposes without consent. Many websites will strip out the metadata you have so carefully imbedded into the photo, so often photographers will watermark their photos.
Regardless, it is important to embed your copyright information such as © 2020 James Clarke. If you are posting the work online you should think about watermarking your work and you might want to even watermark the description and headline metadata fields into the photos as I've done with the Esther Eccles family photo project.
Summary
You are going to need software to scan and capture the images, quickly edit and save the them, work with the metadata, catalogue and then display them.
The Scanner
Apple ImageCapture - should work well with most scanners, but not all.
Your scanners software - should work with your scanner, but may not always be updated to the latest system software. It is always worth comparing the Apple ImageCapture to the software that came with your scanner.
VueScan - from https://www.hamrick.com. Is probably the best scanning software you can purchase. At prices from $60 to $130 it isn't cheap.
Some software will be very clever at detecting the image. The Epson software was very good at detecting strips of negatives, while the Apple software was not.
Editing the Image
Although Apple's Preview program allows quite a bit of photoshop type manipulation, I would first look at Graphics Converter from https://www.lemkesoft.de/en/products/graphicconverter which is available as shareware, so free with nagging or at a good price of $40 US.
This software is very powerful with image manipulation, metadata and batch processing. You will end up using it for your project, but the user interface is difficult and you may prefer using something like Pixelmator or Adobe Photoshop Elements for image manipulation and Graphics Converter to deal with the metadata. Regardless, you will need Graphics Converter.
Metadata
Your choices here are limited. You will find Graphics Converter the best choice for metadata and you may use it in conjunction with Adobe Bridge which is very good at finding photos based on the metadata. Graphics converter has literally 100's, if not 1,000's of functions to manipulate your images. Here are the ones I found essential.
Adobe Bridge is excellent with metadata, but lacks the batch processing abilities of Graphics Converter. Bridge's strength is with organising.
Summary
Assuming you have picked the right file format, high quality JPEG, PNG or DNG, embedded your metadata, scanned at a high resolution, made quick edits - you are on your way to protecting your photos for the next 100 years. This section is about making sure you are not locked into a proprietary system of filing and ensuring your work can always be found.
There are essentially 2 types of Digital Asset Managers (DAMs) for organising your photos.
Catalog software copies your photos and metadata to a proprietary database for organising and sorting.
Browser software looks at your own folder structure and reads the images and metadata as required.
For archiving purposes Browser software is favoured as it doesn't lock you into a proprietary system, but it is worth understanding how both work. Some software such as DigiKam straddles both systems.
Catalog Software
Photos, iPhoto, Aperture and Adobe Lightroom are under the umbrella of Catalog software. They look at the metadata when the photos are imported and store this information in a separate database. Any changes to the metadata are stored in the database and Apple software only embeds changes into the photos when they are exported. Others, like Lightroom can sync the meta data easily back to the original photo. The separate database allows the creation of folders, albums, events and more. Importantly a photo may reside in multiple folders and albums without duplicating the photo, just like a shortcut or alias does.
Photos and iPhoto keep the user well away from the photos. Most DAMs keep track of any changes made to the photos - never altering the original photo file. They keep a log of changes made and apply them to the photos as required. So, if I change a photo, Photos does not alter the original file. It will display the changes in Photos, but if you look at the original photo, you will not see the changes. The only time it embeds the changes is when you export a copy of the photo. Apple Photos is also very limited in what metadata fields you can see and access.
Printing descriptions, titles etc is also limited in Photos - iPhoto had a good HTML Web Gallery, but this was dropped from Photos and it doesn’t print descriptions on photos unless it is a contact sheet. So, apart from being quite nice to use it really has some big limitations.
Photos is great with facial recognition, creating folders and albums. Some of the other software also does this, but not as easily.
In the end I became very disenfranchised with Apple's iPhotos and Photos because of it's metadata limitations and closed nature. See my article on how to escape Apple Photos.
Browser Software
Adobe Bridge, digiKam and most other DAMs I’ve looked at rely on the metadata - They fall under the category of Browser software. Any edits to the metadata are immediately embedded into the photo. These systems don’t organise the photos as easily as Apple Photos as they rely on the metadata for sorting and the use of folders, favourites and directories. They can’t show albums and events in the same way as Photos does and are not as friendly to use, but they do a lot and are future proofed as the photos are not hidden from the user.
Software Choices
Adobe Bridge - This is free from Adobe and the program I favour, but you do need an account with Adobe. This software is their simple photo management software which allows printing, contact sheets, editing and batch editing of metadata. It is pretty hopeless with printing the metadata with your photos, but good for bulk changing your metadata.
It has a feature called collections and smart collections which is similar to Apple Photos Album and folders feature. Any changes to a photo’s metadata are made immediately and reflect in the collections. This looks like the best one to use and has fantastic search functions which can be used over multiple folders.
digiKam - This is an open source software that has a confusing interface, but is very powerful. It has great PDF and HTML printing with Headline and Description fields. The manual is well worth a read as it covers file formats, metadata and keeping your files safe.
Very, very full featured. Full support of metadata. Very complicated and hard to get your head around. This is very well regarded, but I’d have to say a bit too complicated. Worth installing even if you don't use it as your DAM, but good for Web Galleries.
Picktorial - This software looks great and works fast. Unlike Apple Photos it doesn’t rely on a database, but it does remember where you folders are, including on external drives. It has the ability to create new folders and delete files and folders directly in the finder. So, if you delete something it is in the Mac’s rubbish bin.
Good metadata display, editing with the ability to copy metadata from one photo to another. Where possible changes are within the file, or in a sidecar if required - eg for Raw. It has good searches, keywords, but doesn’t display names. Not many do which is why it is a good idea to copy the names into the keywords area. It has non destructive editing which is limited in the free version to a few edits. Star ratings ***** carry over to Bridge and Graphics Converter.
Summary
My main aim was to create a web gallery that at the least was easy to setup, navigate and displayed both the headlines and descriptions from the metadata. My search has been extensive, but largely fruitless. Let me explain the problems I encountered and my eventual solution.
Web Galleries
I have over 7,000 scans. I wanted to display, the good photos and the not so good ones showing at least the headlines and descriptions from the metadata. Some of my audience are luddites when it comes to computers, so the interface had to be very straight forward.
iPhoto - was my first port of call. It has a reasonable web gallery system which can show the necessary metadata. The galleries are ugly and tricky to navigate, but it was all I could find and served the first few incarnations of the project. I suspect quite a few of my relatives gave up on trying to navigate the project, but I know quite a few persisted. Although it was workable, it is no longer an option being replaced by Apple Photos. Unfortunately, Apple Photos dropped the web gallery feature.
Adobe Bridge used to have a well regarded Web Gallery feature, but it has been removed from the program.
Graphics Converter has quite a good Web Gallery along the same lines as iPhoto, but it is a bit tricky to navigate, limited with the metadata and on the ugly side.
DigiKam has a good looking Web Gallery, but will not display the required metadata.
Open Source CPanel programs. I looked at quite a few of these, but quite frankly most were beyond me. They require more research.
The Solution to Web Galleries
In the end I couldn't find any good solutions, so I cobbled together my own. My site is built using the Mac only program Sparkle from https://sparkleapp.com. It happily handles image galleries with 50 images per gallery and 4 or more galleries per page, but it strips out the metadata.
Printing
Most of the programs mentioned will allow you to create PDF pages with the metadata you might require. If you need more control you could use the export metadata to CSV function in Graphics Converter to extract the metadata for copy and pasting into your page layout program. In any case you will need to read up on Colour Calibration and be aware that displaying an in image in RGB is very different to printing which used CMYK.
I think we can safely say the software and computers we are using today won't be used by our descendants. We can only hope the file formats we choose are readable by future computer systems. Whatever photo work you do with your archive needs to be as open as possible with embedded data.
Please contact me with any corrections, comments or thoughts and if you use Apple’s iPhoto or Photos read the next article - No to Apple Photos.
James Clarke