thingelstad

Jamie Thingelstad's personal website

Month: November 2005 (page 1 of 2)

Home Vacation

I’m taking this week off, using up some of my remaining vacation time before it vanishes. Tammy and I are not feeling adventurous enough to go on a big trip with Mazie, so we are staying home for the week. I’ve picked a couple of projects to work on (like I need more of them!) and spending time relaxing and hanging out with Tammy and Mazie. We have some babysitting arranged so we can head out and have some outside fun as well.

I’m feeling really pumped about what I’m working on at the office lately (read: change the world), and it’s hard to disconnect. But, I know it’s healthy and I will come back after a week with even more focus and clearer vision.

'tis the Season

The holiday season is now officially upon us. We were at Southdale today and Mazie got her first picture with Santa! She didn’t have a lot of things on her list, but I clearly heard Santa mentioning something about the Teletubbies.

She had a great time today going around to different places and getting carried around by Mom and Dad. Such a cutie!

<rant>
Santa’s little stand there had several print options, along with the option of getting a CD with the “Hi-Res Digital Photo” on it. I asked what resolution it was and none of Santa’s little helpers knew. They were shooting with a Canon Digital Rebel, so I asked if it was just native resolution of the Rebel. They figured yes. $26 later I have a CD with a 3.15 megapixel image on it. Now, that may be “Hi-Res” to some, but it’s practically a camera-phone in my book. Grrr…
</rant>

Miss… TiVo… So… Much…

We just had our roof redone, and of course they had to rip the DirecTV satelite dish off the roof. Last night, for the first time in 6 years (at least), I had to actually sit down and watch a TV show at the time it was on and with commercials! It was stunning.

Technology is interesting. It starts as a nice plus, and faster than you can imagine turns into a must have. I wanted to catch this show and I realized I only knew (and this was vague) that it was on Monday night. I didn’t know what channel, and wasn’t sure what time. I had to go to TVGuide.com (I felt like I needed a shower afterwards) and look up when it was on.

Then, had to rush to the TV and watch it at the time the network wants me to. Man, this whole thing just blew.

Thankfully the dish gets put back up tomorrow morning, and TiVo will spark back to life and make our world a much happier place again. Whew!

Restaurant: Naar Grille

Tammy and I went to our first movie since Mazie was born today. We saw Walk the Line if you are interested (and yes, it was very good). We also went to a new restaurant for lunch before the movie. We tried out Naar Grille. They opened up about 2 weeks ago. If location really matters I worry for them. They are not in an ideal location, there is no major road with people zipping by. The food however was very good. We went on Sunday which we did not know was buffet only, but it was quiet tasty.

The food is billed as Mediterranean fare. The gyros were very good, and everything looked tasty. I’d encourage people to give it a try – particularly if you like this cuisine.

Google Analytics – Overload?

I was really excited to see the release of Google Analytics this week. I’ve been looking for a better way to track site usage and Weblog Expert just isn’t cutting it anymore. I would suggest that it’s essentially impossible to do site analytics using log files anymore. Spiders, robots, spammers, etc. make your log files into total garbage. Even with significant filtering, you still get mountains of trash.

I was moments away from plunking down “real money” on Visistat to solve this problem, and then Google Analytics came along free of charge.

I’ve been impressed with the data display and am really excited about it, but Google (for the first time I can recall) seems to be completely overwhelmed with traffic. It’s Sunday morning and my reports are still showing Thursday morning data. They target a 24 hour turnaround in report data, but this is a bit crazy. This is what they are telling customers.

Waiting for Data
Analytics has been successfully installed and data is being gathered now. The demand for Google Analytics surpassed even our highest expectations and as a result some customers may temporarily experience report-update delays. All data continues to be collected and no data has been lost. We are currently adding resources to ensure high-quality service. We apologize for any inconvenience.

I’m eagerly waiting for them to deal with the load. Maybe they need to add a few thousand more servers <grin/>. However, if they can’t get this thing working soon I think they may end up with a real PR problem. Especially considering the target audience, web builders!

Terabytes of Disneyworld

A couple of years ago when Tammy and I visited Disneyworld I was struck by the pervasive use of digital cameras and digital video equipment. I started to wonder, as I watched everyone around me merrily clicking away megabytes of storage, how much data is all this?

This train of thought simply would not leave my head, and I ended up doing some analysis on my own digital photography behavior, and ultimately trying to answer the question: How much storage does Disneyworld use everyday?

For this and a wonderful analysis of the storage issues with digital photography, read my article Digital Photograpy Storage Explosion.

Digital Photography Storage Explosion

This article is also available in PDF format
You can download the raw data used in this analysis (2.28MB)

Abstract

Digital cameras have spread at an amazing rate. There are almost no film cameras being developed anymore, having been relegated to antique status. In fact, at the most recent photographer’s expo, for the first time in the history of the event, no 35mm film cameras were introduced. Canon, Nikon and others are only focused in the digital realm. Even for high-end photography, digital has now become the “king of the hill”.

While digital photography has huge benefits, it comes with a data management challenge that is an iceberg we are just now seeing the top of on the horizon. The average home computer user is ill-equipped to deal with these challenges, and even worse, most do not even realize the problem exists. This data problem ranges from photo organization, to storage and disaster recovery. These issues, particularly the last two, have the potential to be the Achilles heal of digital photography.

This document looks deeper into the trends regarding the storage requirements of digital photos, and some of the ancillary challenges that need to be addressed.

The Problem

I got my very first digital camera, a Fuji MX-700, in June of 1999 and instantly fell in love with it. That camera awoke me to the wonders of digital photography. Instant gratification after taking the picture and the ability to easily share images over the Internet were just the beginning. I had for years owned the most popular film camera ever made, the Pentax K-1000. It collected dust for years, and I recently released it to a better owner through an eBay auction.

With its very small sensor and file sizes that camera couldn’t compare to the quality of film. Many people scoffed at the idea that digital would be a real format for photography. The same fate fell on the first MP3 players. The failure in this logic is not accounting for the rapid pace that digital technology advances at. It was, and continues to be, only a matter of time before digital mediums eclipse their historical “analog” counterparts.

Manufacturer Model Sensor Size
Fuji Film MX-700 1.5 million
Fuji Film MX-2700 2.3
Toshiba PDR-M70 3.37
Canon PowerShot G2 4.0
Canon PowerShot G3 4.0
Canon Digital Rebel 6.3
Canon S500 5.0
Canon EOS 20D 8.2

Every photo I’ve taken since that day in 1999 has been digital.

While much about this transition has been wonderful, and I certainly would not return to film, there are challenges. In the period from June 9, 1999 to October 27, 2005, a span of 2,332 days, I’ve taken over 20,000 pictures. That represents over 8 pictures a day for several years, a frequency that is unfathomable in film photography. This collection now represents nearly 50 gigabytes of storage that is more precious than any collection of spreadsheets or word processing documents I can imagine. Vacation memories, the birth of a child, weddings are just some examples of this immensely important data. Figure 1 shows the storage requirement of these photos over time.


Figure 1

The explosion of storage required to accommodate increasing numbers of photos at higher resolutions is a genuine challenge. This data management problem was, until recently, inconceivable to the average person and was only an issue that organizations with specialized technical staff had to deal with. The idea of backing up 50 GB of data on anything other than a spare hard drive is preposterous; optical media is too small (including DVD) and tape is cost prohibitive, hard to use and prone to failure. Even the hard drive option has only recently become an option with the advent of USB2 and Firewire, before that there really was no option.

The amazing thing about this storage issue is the rate at which it is growing. It took 299 days for me to collect my first gigabyte of photos. I didn’t reach the five gigabyte point until 1,144 days of photography. That is approximately half of the time I’ve had a digital camera. In the 1,188 days following that I’ve added another 41 gigabytes of photos, all while continuing to accelerate the storage needs. Figure 2 shows the storage size of photos taken by year (only showing years with complete data).


Figure 2

This may sound huge, but luckily this technical problem has been helped by hard drive manufacturers. Figure 3 below compares the growth of my digital photos to Moore’s Law[1] and Kryder’s Law[2]. Both of these laws relate to the growth rates of technology. Moore’s Law, reduced, states that computing power doubles every 18 months. Kryder’s Law is a similar law for storage, but doubling even faster at 13 months.


Figure 3 – Plotted on logarithmic scale.

It is interesting to note that both Moore’s and Kryder’s growth rates were well ahead of me for nearly the entire first year of photography. Since then, I’ve steadily outpaced Moore’s and am knocking aggressively on the top-end of Kryder’s growth rate.

The casual observer could stop reading here and think there is no problem. The growth of storage (theoretically at least) is growing at the same pace as digital photography. However, that isn’t the entire story. If we make the assumption that the average person consumes technology at the average rate (dictated by Moore’s Law) what we find is that on average, people have to buy more technology than they would have historically purchased to deal with storing all this data. In the past the default hard drive options at Dell.com would be fine, but as digital photography explodes the average user will be buying more sophisticated systems that come with a variety of technical challenges. This is a hidden, and mandatory cost and still doesn’t resolve the issue of backing up this data.

Digital Behavior

Digital photography introduces people to an entirely new way of using photos, with one of the biggest implication being how many pictures you actually take.

Take a moment to look at Figure 4 below showing the number of photos taken each month with a 6-month moving average.


Figure 4

You can see a clear growth in the tendency to simply take more and more photos. I have been taking 36% more pictures every year since starting with digital photography. Compounding that problem is the growth of megapixels, and with it file-sizes, in digital cameras. The storage space required for the increasingly growing collection of photos is growing geometrically.

Figure 5 is interesting for other reasons as it identifies how events like vacation (July 2002, July 2004) affect these trends, as well as major life events like the birth of a child (June 2005). Going to Europe in July 2004 now carries with it a permanent issue of keeping 4.5 gigabytes of data around forever. That is notably more challenging than putting some prints in a photo album and placing it on the shelf. One is an active process of sophisticated data management, the other a passive matter of organizing physical documents. As we continue to use digital technology to capture events in our life we may begin to look at the digital cost of those events. Going on vacation will cost $2,000 and have a data load of 3GB, my childs school play will have a data load of 1.5GB, etc. All the while being additive of course, we are never removing old content.


Figure 5

Let’s take a moment to look at our digital photography behavior. This is somewhat tangential to the issue at hand, but is interesting nonetheless and provides some depth to the trend of taking more photos over time. Figure 6 below shows the number of digital photos taken each month during the sample period.


Figure 6

It is clear that the summer months represent big picture taking times. December also reflects a high volume of pictures, no doubt due to the Christmas holiday. Figures 7 and 8 show photographic activity by day and hour.


Figure 7


Figure 8
The results are not terribly surprising. The majority of photos are taken on the weekends and there are very few photos taken while sleeping. It is a little surprising that the afternoon is higher than the early evening hours.

Terabytes of Disney World

So clearly this is a growing mountain of data, and this is just one families archive of photos. The impact of this on a large scale, with hundreds of millions of photographers throughout the world is stunning. On a recent vacation to Disney World I was particularly taken with the pervasiveness of digital photography and digital video[3] equipment. I started to wonder what the storage requirement of the thousands of vacationers visiting Disney World everyday was.

I contacted Disney to find out how many visitors on average attend Disney World each day but they do not disclose that number.

Date: Fri, 21 Oct 2005 09:16:30 -0700
To:
Subject: Re: Request for General Information about Walt Disney World Resort
From: Walt Disney World Guest Mail
Reply-To: Walt Disney World Guest Mail

Dear Jamie,

Thank you for contacting the Walt Disney World Resort.

We appreciate your interest in obtaining information about guest attendance. Regrettably, this type of specific information is proprietary and cannot be released. We are sorry that we cannot be of more assistance, and we appreciate your understanding.

If you have questions or need further assistance, feel free to contact us.

Sincerely,

Ronda
WDW Online Communications

Denied the true information, I did find some estimates of attendance on the web[4]. Using this estimate I concluded that Disney World averaged 38,462 visitors a day. According to at least one survey 54% of Americans are using digital cameras[5]. It seems straightforward to me that the average visitor to Disney World is also more likely to have a digital camera, but we’ll use the average anyway.

The next challenge is determining the distribution of camera types. I used a standard bell curve distribution from 2 megapixel to 8 megapixel (shown on the right), and also assumed that everyone is taking a JPEG file. The last bit of information needed is an estimate of how many pictures each one of these digitally equipped visitors would take at Disney World. I estimated 100 pictures, which I would suggest is high, but realistic given that you are on vacation, at a special place and have a high desire to capture “life’s moments”. For comparison, I took 241 pictures in two days at Disney World, averaging 120 pictures a day.

Given these assumptions, the attendees to Disney World generate 3,869 GB, or 3.78 terabytes, of data every single day. An equally interesting way to look at this is bandwidth. If you used the average cable modem available in the US today, and transmitted data at the maximum rate every minute that Disney World was open[6], you would have to have 420 cable modems running in parallel (839 Mbps) to handle the data.

Challenges

The challenges posed by dealing with such a large volume of data for both the individual, and the technology world as a whole, are significant. When you add to the equation that these are your memories, some of the most precious things you have, the criticality of the problem becomes clear.

Backing up data of this size is not easy. My solution thus far has been a plan of massive redundancy. I have four computers in my house that have large hard drives and every night an automated task runs to replicate any new images onto all of the backup points. This provides great redundancy and protection, however it’s expensive using an additional 200 GB of storage for redundant copies, and growing every day. Realistically this solution will not be viable in a couple of years without further investment in high-end technology.

An additional problem with this is that it is all in one physical location so I’m not protected from catastrophic failure (e.g., fire burns down my house). It would be great to leverage a broadband connection to backup to a service provider over the Internet, however the pricing models for this make it unreasonable, costing hundreds of dollars a month. Alternatively, some technically sophisticated families can adopt a buddy system mirroring their photos between each other using backup programs with FTP options. This is a nice option, with relatively low costs but is hard to setup and issues like encryption must be addressed to insure privacy.

In the meantime, the best solution for offsite storage seems to be a removable USB or Firewire hard drive pair that you rotate to a safe deposit box. This is cumbersome and requires manual intervention which will only work for the most diligent of people.

We may think that the storage requirement problem is easily solved by Kryder’s law and the hard drive manufacturers; there are further challenges that will make storage requirements grow even faster.

The Raw Problem

Almost all digital photography is done straight to a JPEG file. This is an easy format that is usable on nearly every computer that exists, but may be a bad decision for pictures that you really care about. Most cameras support a file format called RAW which captures the photo in a different way, allowing you much more flexibility in the future. To keep it simple, a RAW file can be considered analogous to a negative[7], with shooting to JPEG being similar to a Polaroid. You can then digitally develop that RAW file and improve the picture significantly. However, this comes at a cost – storage. It should be assumed that in the relatively near future camera manufacturers will migrate away from JPEG as consumers look for higher quality and more post-processing capability.

Raw files are compressed, but not as much as JPEG files. The resulting file is approximately 300% larger than a comparable JPEG file. Figure 9 shows the theoretical effect of using RAW photography as opposed to JPEG.


Figure 9 – Plotted on logarithmic scale.

You can see the problem right away! We break above Kryder’s law immediately and stay well above it. At the end of our sample period our storage needs have tipled from 46 GB to 143 GB. Let’s look at this growth on a linear scale in Figure 10, as opposed to the logarithmic scale in Figure 9. The huge disparity in storage needs versus growth are profound.


Figure 10

While it is appealing to do photography only in RAW, the storage implications make it very challenging.

Other People’s

Another interesting phenomenon in digital photography is the impact of other people pictures. I have a couple thousand pictures that others have taken that I keep in my collection. Note, those pictures are excluded from all the analysis in this document. These photos further increase my growth rates.

I call this trend “group photography”. This is very typical with extended family outings or holidays. Several people with digital cameras will take pictures throughout the day and then one person will collect the photos from everyone. They will then synchronize the timestamps of the photos and distribute the entire collection of photos to everyone. This is a wonderful thing, but amplifies even further the massive storage issues related to digital photography. Now that trip to Disney World results in several hundred photos instead of the smaller hundred that you took by yourself.

Conclusions

I believe that I am seeing the challenges of this problem sooner than the average person. I may take more pictures, and I’ve been doing digital photography for a long-time. However, it is clear that the industry needs to provide both software and services to help the average consumer with this problem, or run the risk of huge issues as people start losing more photos and precious memories and digital photography experiences a customer backlash. It is only a matter of time before large numbers of people start running out of hard drive space, or have a virus delete their entire summer vacation. Given the huge storage sizes it is unlikely that people will have backups around. After all, how many people can really honestly say they practice good data management processes even with small files?

I already know several people that print all their digital pictures for this very reason. They simply do not trust their computer, or their own skills at managing the data, and print everything for archival. This seems cumbersome to me, but I understand the comfort that having those photos in storage would provide. Far too many people have been clobbered by an Internet virus and lost data. They never want to lose their memories.

There is a rather simple suggestion to make this problem better and that is editing. Most people tend to keep every digital photo they ever take, never deleting anything. I do not like the idea of deleting photos unless they are plainly out-of-focus or otherwise bad. I suggest this problem should be solved by the technology that created it.

There are several things that companies like Canon and Nikon should be doing to assure this problem is managed. Digital camera makers should be pushing the burgeoning online backup service market to lower it’s prices for large amounts of storage. There are hundreds of companies that will backup data for you to a managed service over a broadband connection. The problem is cost. These services still operate in terms of megabytes, not gigabytes. They all would charge several hundred dollars a month for this service.

Once solving the backup problem with cheap, reliable service provider model we need to address the software problem. There are no photo management programs that do well managing 20,000 photos. Programs like iPhoto and Picasa became unusable with very large photo collections. Additionally, the metadata that you attach to photos for indexing and searching needs to be self-contained in the photo files themselves, like the EXIF data that the camera embeds. The industry cannot try to lock people in to specific software packages by storing metadata in proprietary formats.

There are independent software developers working on the software problem, but it is not moving fast enough. The managed services market has held it’s prices unreasonably high even while storage costs continue to decline. The is an opportunity for services like Flickr to provide options to help these problems, but informed customers will be hesitant to turn over such important data to a service like that.

In the meantime, the most cost-effective solution is to leverage a “buddy system” with others that have broadband connections. Swap hard drives and use software to backup encrypted copies of your digital photo collection to their network. As is too often the case though, this solution is only viable for the technically sophisticated user with their own infrastructure.

END


[1] For more information on Moore’s Law see Wikipedia at http://en.wikipedia.org/wiki/Moore%27s_Law.[2] For more information on Kryder’s Law see Wikipedia at http://en.wikipedia.org/wiki/Kryder%27s_law.

[3] This document doesn’t attempt to address the storage issues related to digital video, however, it is worth noting that they are orders of magnitude bigger than digital photography.

[4] Article at http://www.wdisneyw.co.uk/attend.html.

[5] Article at http://www.letsgodigital.org/en/news/articles/story_4267.html.

[6] Disney World Magic Kingdom is open from 9:00 am to 8:00 pm.

[7] For more information on the RAW format see http://www.luminous-landscape.com/tutorials/understanding-series/u-raw-files.shtml.

Construction Zone

Our house has been a construction zone this last week, and will hopefully start to wind down after this next week. We scheduled a number of projects to occur right now.

  • We are completing the 3rd phase of our landscaping project and they are installing a stone path with stairs in the back. This is totally smooth and hardly impacts us at all since it’s outside. They should finish this coming week and so far it looks great!
  • On Tuesday we had a new furnace and air conditioner installed. Our old one was going and we wanted to put in a higher efficiency unit. This went well and is now done after a couple of follow-up visits to make sure everything was done. We have a 96.2% efficient furnace and a 15+ SEER air conditioner now.
  • We had new shower doors put in the master bathroom. We now have the glass-style doors like nicer hotels have. This is a nice upgrade and completed on time.
  • We are having a new roof put on as well. This one started out very poorly with bad communication with this contractor and the crew not showing up for the first two days. Things are turning better though as of today. More communication and it looks like they should (weather permitting) finish by Wednesday. I really underestimated what a headache this one would be, it’s a huge mess.
  • The roof is in preparation for the installation of solar heat next. We wanted to get the roof redone before doing the solar projects (we are also considering solar electric) so we wouldn’t have to redo it all in a couple of years when it will be a lot harder. The solar heat will go on in mid-December but they are prepping the roof now with the appropriate pipes and moving things around.

I don’t know how people put up with remodeling their entire house. I’d be a basket case.

Mazie in Jump-Up

I took some video of Mazie having fun in her Jump-Up and decided to try to get it online. I’m still pretty bad at this video thing, but gotta start somewhere. Click to watch the video (Windows Media 9, 4.5MB).

San Jose Trip

I spent last weekend in the San Jose area for my friend Scott’s wedding. Scott and I met when we were freshman in college. We hit it off right away sharing many techie, geek hobbies and a general interest in how things work (Scott was a EE major). Many a night were spent jawing on and on about computers, cars and numerous other topics. Scott moved to San Jose a number of years ago so we don’t see each other as often anymore. As is typical with guys we are not real active pen pals, so it was great to spend some time with him and be a part of his big day!

I flew into San Jose airport with a sense of deja vu. The last time I had flown into that airport was for my friend Phil’s wedding. Seems SJC is exclusively reserved for wedding use in my book. :-) I landed on Friday around noon with enough time to drop my junk off at the hotel and then go meet the guys for a groom’s day out.

We went to Lemans Karting in Fremont (where Scott and Cara live) and had a blast! 15 of us suited up, got our helmets on, and headed out on the track. The karts hit about 25 mph, but the scale sure makes it seem like a lot faster. I was really surprised by how much force it took to steer them through the hairpin turns. All of us had pretty sore hands and arms afterwards. I did pretty poorly in the first heat placing 9th out of 10 with a best lap of 26.013 seconds. However, I improved greatly in the second heat placing 3rd out of 10 with a best lap of 24.230 seconds. After that we went to City Beach for drinks, appetizers and dinner. Good food, good conversation and a really fun bunch of guys.

On Saturday morning, the big day, Scott and I rode with his friend Roger up to Club Sportiva in San Francisco. Cara arranged for Scott to have a Ferrari 308GTSi to drive for the day and I was accompanying him up to go get it. As an aside, Roger drove a Toyota Prius and I sure found that thing cool. It was a “shiny pebble” moment extraordinaire!

Club Sportiva is a pretty interesting concept. If you have a penchant for exotic and interesting cars, but lack the wallet to own them, you should check it out. We picked up the “Magnum Ferrari” and Scott drove us out on 280 and then up to Skyline drive to stretch out the car a bit. Very nice ride. One of the funnier moments was when a guy in a Ferrari 550 Maranello pulled up next to us, blipped his horn to say “Hi”, clicked his paddle shifters down twice and disappeared in a puff.

We stopped midway on Skyline Drive to have lunch at Alice’s Restaurant. It’s a little unclear if it is the Alice’s Restaurant. It does not appear to be “a half-a-mile from the railroad tracks”. On their menu they specify that it was created around the same time as the song, and the owner’s name was Alice. Good enough for me.

The wedding itself was in Half Moon Bay. Photos were on the beach with the waves pounding in and the ceremony itself up the coast shortly looking out on the ocean. It was a small, casual wedding. A very nice time with lots of laughter and joy. Congrats to Scott and Cara!

Older posts

© 2014 thingelstad

Theme by Anders NorenUp ↑