Steve's Digicams Forums

Steve's Digicams Forums (https://forums.steves-digicams.com/)
-   General Discussion (https://forums.steves-digicams.com/general-discussion-11/)
-   -   Critique my IS testing method (https://forums.steves-digicams.com/general-discussion/210027-critique-my-testing-method.html)

ItDontMeanAThing Feb 10, 2014 12:10 AM

Critique my IS testing method
 
I tested IS to see how slow I could go without blur. Couldn't find an 'IS testing for dummies' so my test plan is based on various blog posts plus educated guesses.

Why test? I read a post recently which distinguished between gripping a rangefinder style mirrorless camera and supporting it. I started to play with some different ways to hold my travel camera (EOS M) when I realized I need a baseline for comparing how steady each was, or which was the steadiest.

Two questions. I identified 2 shutter speeds as the slowest ones without blur, 1/125 for IS off and 1/50 for IS on. First, did I identify the correct speeds? Between 59 year old eyes and a laptop screen, I need a second opinion. Next, are there any parts of the test procedure can be improved to make judging the results more obvious? Just to be clear, I'm interested in testing methods and interpreting results, not in different ways to hold a camera.

Test conditions - Flat, eye-level newsprint target. Tv mode. ISO Auto (though it never budged off 100), AF in spot mode, 55mm focal length (the long end of my kit zoom). Two sequences: IS on, IS off. Shutter speeds: 500, 250, 125, 80, 50, 30, 20, 13, 8, 5, 3. Target and I were both in open shade, bright sun elsewhere, negligible wind. I held the camera the way I do when I'm shooting at 1/100 or faster. Yes, I have more stable ways to hold my camera for slow shutter speeds. I'll test those methods and some new ones after getting feedback on my testing method.

Changes for the next test. Image quality degraded because of small apertures and overexposure when the aperture could go no smaller. Next test will use manual ISO. First I'll make test exposures at the fastest and slowest shutter speeds to be tested. Then select a starting manual ISO which will give decent IQ at the fastest speeds but, after lowering ISO with decreasing shutter speed, not be over exposed at the slowest. I'm guessing 400 - 800 to start.

Images are 100% crops. Only processing is a bit of sharpening. Left side photos are IS off. There are 2 pairs photos.

IS Off no blur - 1/125
http://4.static.img-dpreview.com/fil...0QfMGnWUssE%3D

IS Off with blur - 1/80
http://1.static.img-dpreview.com/fil...EcnPQPcY4gw%3D

IS On no blur - 1/50
http://4.static.img-dpreview.com/fil...jGC0TpeKt9Q%3D

IS on with blur - 1/30
http://3.static.img-dpreview.com/fil...fYZUsTbcLlM%3D

All the test shots: http://www.flickr.com/photos/9834191...7640676691735/

TCav Feb 10, 2014 6:33 AM

The key element of any test should be repeatability. Your test relies on you holding the camera, and you can't rely on you being just as shaky from one test series to the next. I've tried things like mounting the camera on a tripod and placing that tripod on the hood of a car with the engine running at idle. That worked, but since the oscillations that occur in the human body occur at a frequency of around 8-12 Hz, with the most severe oscillations happening at the lower frequencies, that test wasn't very useful regardless of how successful it was.

If you're going to rely on your own shake instead of something more repeatable, I suggest you repeat your test at least 3 times, alternating from "IS On" to "IS Off" for each series. In other words, don't do 3 test with IS On, then 3 tests with IS Off. Instead, do 6 tests, the first with IS On, the second with IS Off, the third with IS On, etc. And don't average the results, but use the worst result as your guide.

TCav Feb 10, 2014 8:03 AM

And if your intent is to determine for yourself how much you can get away with, hold the camera the best way you know how, not the way that you think will produce the greatest motion blur.

Since your results rely on your shake and your gear, they aren't of use to others, so make them most useful for you.

VTphotog Feb 10, 2014 9:52 AM

If the method works for you, then it works, for you. Humans vary so much in their methods of holding cameras, inherent stability, sleep, and caffeine intake, that what you find to be the slowest speed you can use under some conditions, may be beyond others' capabilities, and faster than some.

brian

ItDontMeanAThing Feb 10, 2014 6:37 PM

Quote:

Originally Posted by TCav (Post 1367917)
The key element of any test should be repeatability. Your test relies on you holding the camera, and you can't rely on you being just as shaky from one test series to the next.

True and obvious. If I wanted to test the IS system independent of my holding, I'd look at the CIPA results.


Quote:

Originally Posted by TCav (Post 1367917)
If you're going to rely on your own shake instead of something more repeatable, I suggest you repeat your test at least 3 times, alternating from "IS On" to "IS Off" for each series. In other words, don't do 3 test with IS On, then 3 tests with IS Off. Instead, do 6 tests, the first with IS On, the second with IS Off, the third with IS On, etc. And don't average the results, but use the worst result as your guide.

Hadn't thought about alternating testing between 'on' and 'off'. Thanks. Only reason I tested while 'off' was curiosity about IS effectiveness, so I won't be repeating that.

Both my education and career in data analysis say ignore the outliers and average the rest, assuming there is a central tendency. I'm looking for a personal rule of thumb, not a precise measure. If a shot is critical I rest the camera on something solid or bring a tripod.

ItDontMeanAThing Feb 10, 2014 6:43 PM

Quote:

Originally Posted by TCav (Post 1367922)
And if your intent is to determine for yourself how much you can get away with, hold the camera the best way you know how, not the way that you think will produce the greatest motion blur.

Since your results rely on your shake and your gear, they aren't of use to others, so make them most useful for you.


I addressed those.

Quote:

Originally Posted by ItDontMeanAThing (Post 1367900)
I tested IS to see how slow I could go without blur.
... snip ...
Yes, I have more stable ways to hold my camera for slow shutter speeds. I'll test those methods and some new ones after getting feedback on my testing method.


ItDontMeanAThing Feb 10, 2014 6:46 PM

Quote:

Originally Posted by VTphotog (Post 1367926)
If the method works for you, then it works, for you. Humans vary so much in their methods of holding cameras, inherent stability, sleep, and caffeine intake, that what you find to be the slowest speed you can use under some conditions, may be beyond others' capabilities, and faster than some.

brian

Looks like this wasn't clear enough:
Quote:

Originally Posted by ItDontMeanAThing (Post 1367900)
Just to be clear, I'm interested in testing methods and interpreting results, not in different ways to hold a camera.


VTphotog Feb 11, 2014 12:06 AM

Looks like I wasn't clear enough. If you are comparing IS systems or cameras, you need a standardized, repeatable method. But that isn't what you are doing, so whatever method you develop is relevant only to yourself, under the conditions you set. There are too many variables for it to mean anything to anyone else.

brian

ItDontMeanAThing Feb 11, 2014 1:11 AM

Quote:

Originally Posted by VTphotog (Post 1367983)
Looks like I wasn't clear enough. If you are comparing IS systems or cameras, you need a standardized, repeatable method. But that isn't what you are doing, so whatever method you develop is relevant only to yourself, under the conditions you set. There are too many variables for it to mean anything to anyone else.

brian

You were perfectly clear. My response said your contribution had nothing to do with what I requested in post #1. Was I not clear?

TCav Feb 11, 2014 5:37 AM

Quote:

Originally Posted by ItDontMeanAThing (Post 1367954)
Hadn't thought about alternating testing between 'on' and 'off'. Thanks. Only reason I tested while 'off' was curiosity about IS effectiveness, so I won't be repeating that.

Both my education and career in data analysis say ignore the outliers and average the rest, assuming there is a central tendency. ...

I'm a big fan of the Standard Deviation. With only one test, the standard deviation is infinite. And with only three tests, chances are one of the tests will be outside one standard deviation from the mean. You can define an "Outlier" anyway you want, but my preference is that, for a very small sample size, an outlier is outside two standard deviations from the mean. If you're only going to test three times, it's unlikely that you'd get any outliers.

And since your objective is ...
Quote:

Originally Posted by ItDontMeanAThing (Post 1367900)
... to see how slow [you] could go without blur ...

... any results you get are significant because you got them, and shouldn't be discarded.


All times are GMT -5. The time now is 3:21 AM.

Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2018, vBulletin Solutions, Inc.
SEO by vBSEO 3.6.0 RC 2