Go Back   Steve's Digicams Forums > Digicam Help > General Discussion

Reply
 
Thread Tools Search this Thread
Old Feb 10, 2014, 12:10 AM   #1
Senior Member
 
ItDontMeanAThing's Avatar
 
Join Date: Oct 2010
Location: Hua Hin, Thailand
Posts: 403
Default Critique my IS testing method

I tested IS to see how slow I could go without blur. Couldn't find an 'IS testing for dummies' so my test plan is based on various blog posts plus educated guesses.

Why test? I read a post recently which distinguished between gripping a rangefinder style mirrorless camera and supporting it. I started to play with some different ways to hold my travel camera (EOS M) when I realized I need a baseline for comparing how steady each was, or which was the steadiest.

Two questions. I identified 2 shutter speeds as the slowest ones without blur, 1/125 for IS off and 1/50 for IS on. First, did I identify the correct speeds? Between 59 year old eyes and a laptop screen, I need a second opinion. Next, are there any parts of the test procedure can be improved to make judging the results more obvious? Just to be clear, I'm interested in testing methods and interpreting results, not in different ways to hold a camera.

Test conditions - Flat, eye-level newsprint target. Tv mode. ISO Auto (though it never budged off 100), AF in spot mode, 55mm focal length (the long end of my kit zoom). Two sequences: IS on, IS off. Shutter speeds: 500, 250, 125, 80, 50, 30, 20, 13, 8, 5, 3. Target and I were both in open shade, bright sun elsewhere, negligible wind. I held the camera the way I do when I'm shooting at 1/100 or faster. Yes, I have more stable ways to hold my camera for slow shutter speeds. I'll test those methods and some new ones after getting feedback on my testing method.

Changes for the next test. Image quality degraded because of small apertures and overexposure when the aperture could go no smaller. Next test will use manual ISO. First I'll make test exposures at the fastest and slowest shutter speeds to be tested. Then select a starting manual ISO which will give decent IQ at the fastest speeds but, after lowering ISO with decreasing shutter speed, not be over exposed at the slowest. I'm guessing 400 - 800 to start.

Images are 100% crops. Only processing is a bit of sharpening. Left side photos are IS off. There are 2 pairs photos.

IS Off no blur - 1/125


IS Off with blur - 1/80


IS On no blur - 1/50


IS on with blur - 1/30


All the test shots: http://www.flickr.com/photos/9834191...7640676691735/
ItDontMeanAThing is offline   Reply With Quote
Sponsored Links
Old Feb 10, 2014, 6:33 AM   #2
Senior Member
 
TCav's Avatar
 
Join Date: Sep 2005
Location: Washington, DC, Metro Area, Maryland
Posts: 13,544
Default

The key element of any test should be repeatability. Your test relies on you holding the camera, and you can't rely on you being just as shaky from one test series to the next. I've tried things like mounting the camera on a tripod and placing that tripod on the hood of a car with the engine running at idle. That worked, but since the oscillations that occur in the human body occur at a frequency of around 8-12 Hz, with the most severe oscillations happening at the lower frequencies, that test wasn't very useful regardless of how successful it was.

If you're going to rely on your own shake instead of something more repeatable, I suggest you repeat your test at least 3 times, alternating from "IS On" to "IS Off" for each series. In other words, don't do 3 test with IS On, then 3 tests with IS Off. Instead, do 6 tests, the first with IS On, the second with IS Off, the third with IS On, etc. And don't average the results, but use the worst result as your guide.
__________________
  • The lens is the thing.
  • 'Full Frame' is the new 'Medium Format'.
  • "One good test is worth a thousand expert opinions." - Tex Johnston, Boeing 707 test pilot.

Last edited by TCav; Feb 10, 2014 at 6:36 AM.
TCav is offline   Reply With Quote
Old Feb 10, 2014, 8:03 AM   #3
Senior Member
 
TCav's Avatar
 
Join Date: Sep 2005
Location: Washington, DC, Metro Area, Maryland
Posts: 13,544
Default

And if your intent is to determine for yourself how much you can get away with, hold the camera the best way you know how, not the way that you think will produce the greatest motion blur.

Since your results rely on your shake and your gear, they aren't of use to others, so make them most useful for you.
__________________
  • The lens is the thing.
  • 'Full Frame' is the new 'Medium Format'.
  • "One good test is worth a thousand expert opinions." - Tex Johnston, Boeing 707 test pilot.
TCav is offline   Reply With Quote
Old Feb 10, 2014, 9:52 AM   #4
Senior Member
 
VTphotog's Avatar
 
Join Date: Mar 2005
Location: Extreme Northeastern Vermont, USA
Posts: 4,214
Default

If the method works for you, then it works, for you. Humans vary so much in their methods of holding cameras, inherent stability, sleep, and caffeine intake, that what you find to be the slowest speed you can use under some conditions, may be beyond others' capabilities, and faster than some.

brian
VTphotog is online now   Reply With Quote
Old Feb 10, 2014, 6:37 PM   #5
Senior Member
 
ItDontMeanAThing's Avatar
 
Join Date: Oct 2010
Location: Hua Hin, Thailand
Posts: 403
Default

Quote:
Originally Posted by TCav View Post
The key element of any test should be repeatability. Your test relies on you holding the camera, and you can't rely on you being just as shaky from one test series to the next.
True and obvious. If I wanted to test the IS system independent of my holding, I'd look at the CIPA results.


Quote:
Originally Posted by TCav View Post
If you're going to rely on your own shake instead of something more repeatable, I suggest you repeat your test at least 3 times, alternating from "IS On" to "IS Off" for each series. In other words, don't do 3 test with IS On, then 3 tests with IS Off. Instead, do 6 tests, the first with IS On, the second with IS Off, the third with IS On, etc. And don't average the results, but use the worst result as your guide.
Hadn't thought about alternating testing between 'on' and 'off'. Thanks. Only reason I tested while 'off' was curiosity about IS effectiveness, so I won't be repeating that.

Both my education and career in data analysis say ignore the outliers and average the rest, assuming there is a central tendency. I'm looking for a personal rule of thumb, not a precise measure. If a shot is critical I rest the camera on something solid or bring a tripod.
ItDontMeanAThing is offline   Reply With Quote
Old Feb 10, 2014, 6:43 PM   #6
Senior Member
 
ItDontMeanAThing's Avatar
 
Join Date: Oct 2010
Location: Hua Hin, Thailand
Posts: 403
Default

Quote:
Originally Posted by TCav View Post
And if your intent is to determine for yourself how much you can get away with, hold the camera the best way you know how, not the way that you think will produce the greatest motion blur.

Since your results rely on your shake and your gear, they aren't of use to others, so make them most useful for you.

I addressed those.

Quote:
Originally Posted by ItDontMeanAThing View Post
I tested IS to see how slow I could go without blur.
... snip ...
Yes, I have more stable ways to hold my camera for slow shutter speeds. I'll test those methods and some new ones after getting feedback on my testing method.
ItDontMeanAThing is offline   Reply With Quote
Old Feb 10, 2014, 6:46 PM   #7
Senior Member
 
ItDontMeanAThing's Avatar
 
Join Date: Oct 2010
Location: Hua Hin, Thailand
Posts: 403
Default

Quote:
Originally Posted by VTphotog View Post
If the method works for you, then it works, for you. Humans vary so much in their methods of holding cameras, inherent stability, sleep, and caffeine intake, that what you find to be the slowest speed you can use under some conditions, may be beyond others' capabilities, and faster than some.

brian
Looks like this wasn't clear enough:
Quote:
Originally Posted by ItDontMeanAThing View Post
Just to be clear, I'm interested in testing methods and interpreting results, not in different ways to hold a camera.
ItDontMeanAThing is offline   Reply With Quote
Old Feb 11, 2014, 12:06 AM   #8
Senior Member
 
VTphotog's Avatar
 
Join Date: Mar 2005
Location: Extreme Northeastern Vermont, USA
Posts: 4,214
Default

Looks like I wasn't clear enough. If you are comparing IS systems or cameras, you need a standardized, repeatable method. But that isn't what you are doing, so whatever method you develop is relevant only to yourself, under the conditions you set. There are too many variables for it to mean anything to anyone else.

brian
VTphotog is online now   Reply With Quote
Old Feb 11, 2014, 1:11 AM   #9
Senior Member
 
ItDontMeanAThing's Avatar
 
Join Date: Oct 2010
Location: Hua Hin, Thailand
Posts: 403
Default

Quote:
Originally Posted by VTphotog View Post
Looks like I wasn't clear enough. If you are comparing IS systems or cameras, you need a standardized, repeatable method. But that isn't what you are doing, so whatever method you develop is relevant only to yourself, under the conditions you set. There are too many variables for it to mean anything to anyone else.

brian
You were perfectly clear. My response said your contribution had nothing to do with what I requested in post #1. Was I not clear?
ItDontMeanAThing is offline   Reply With Quote
Old Feb 11, 2014, 5:37 AM   #10
Senior Member
 
TCav's Avatar
 
Join Date: Sep 2005
Location: Washington, DC, Metro Area, Maryland
Posts: 13,544
Default

Quote:
Originally Posted by ItDontMeanAThing View Post
Hadn't thought about alternating testing between 'on' and 'off'. Thanks. Only reason I tested while 'off' was curiosity about IS effectiveness, so I won't be repeating that.

Both my education and career in data analysis say ignore the outliers and average the rest, assuming there is a central tendency. ...
I'm a big fan of the Standard Deviation. With only one test, the standard deviation is infinite. And with only three tests, chances are one of the tests will be outside one standard deviation from the mean. You can define an "Outlier" anyway you want, but my preference is that, for a very small sample size, an outlier is outside two standard deviations from the mean. If you're only going to test three times, it's unlikely that you'd get any outliers.

And since your objective is ...
Quote:
Originally Posted by ItDontMeanAThing View Post
... to see how slow [you] could go without blur ...
... any results you get are significant because you got them, and shouldn't be discarded.
__________________
  • The lens is the thing.
  • 'Full Frame' is the new 'Medium Format'.
  • "One good test is worth a thousand expert opinions." - Tex Johnston, Boeing 707 test pilot.
TCav is offline   Reply With Quote
 
Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off



All times are GMT -5. The time now is 3:55 PM.