Integer Maths vs floating point - page 3 - Script Writing - CHDK Forum

Integer Maths vs floating point

• 40 Replies
• 7750 Views

pigeonhill

• 1155
Re: Integer Maths vs floating point
« Reply #20 on: 27 / April / 2021, 01:59:36 »
@philmoz

I forgot to ask, when using imath there are limits one has to ensure numbers don’t break, eg sqrt

What are the limits in the fmath library, say, for sqrt?

Cheers

Garry

philmoz

• 3450
Re: Integer Maths vs floating point
« Reply #21 on: 27 / April / 2021, 02:01:46 »
@philmoz

I forgot to ask, when using imath there are limits one has to ensure numbers don’t break, eg sqrt

What are the limits in the fmath library, say, for sqrt?

Cheers

Garry

Everything in fmath uses double (64 bit) floating point operations so the only limits are what is inherent to the double data type.

Phil.

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

pigeonhill

• 1155
Re: Integer Maths vs floating point
« Reply #22 on: 27 / April / 2021, 02:22:39 »

pigeonhill

• 1155
Re: Integer Maths vs floating point
« Reply #23 on: 27 / April / 2021, 03:24:32 »
@philmoz

Phil

You have made me feel really good

Just did a quick test of a typical calculation and tested the result in Wolfram Alpha.

Here is the test script:

Code: [Select]
`--[[@fmath@chdk_version 1.5--]] u = 567 -- focus distance from the front principalf = 11000 -- simulating CHDK focal length, ie 11mm = 11000ff = fmath.new(f,1000) -- convert f to 11.000h = 1500 -- hyperfocal distancep = 3 -- pupil magndof = (ff*ff*p - ff*ff - ff*p*u + ff*u - h*p*u)/(p*(ff-h-u)) -- near DoF as measured from the front principalndof = ndof:tostr()print(ndof)sleep(5000)`
As I said before, this will transform my scripting of DoF stuff, ie not needing to use imath 'work arounds'.

Many thanks and I owe you a beer

pigeonhill

• 1155
Re: Integer Maths vs floating point
« Reply #24 on: 27 / April / 2021, 07:48:18 »
@philmoz

Just stumbled as I convert my imath based script to fmath.

I get the error "attempt to compare userdata with number", eg where ndof is a real

Code: [Select]
`if ndof < 45 then print(1) else print (2) end`
Does this mean ALL numbers in the script now need to be converted to real number format.

Or, could I convert reals back to 'just' normal integers? If so how should I best do this?

Cheers

Garry

philmoz

• 3450
Re: Integer Maths vs floating point
« Reply #25 on: 27 / April / 2021, 07:59:52 »
I get the error "attempt to compare userdata with number", eg where ndof is a real
Code: [Select]
`if ndof < 45 then print(1) else print (2) end`

@pigeonhill

It looks like you can't mix fmath and integer values in the comparison operators - this appears to be a Lua limitation.

You should be able to use:
if ndof:int() < 45 ...
or  if ndof < fmath.new(45) ...

Phil.
« Last Edit: 27 / April / 2021, 08:05:41 by philmoz »
CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

pigeonhill

• 1155
Re: Integer Maths vs floating point
« Reply #26 on: 27 / April / 2021, 08:04:51 »
Cheers

pigeonhill

• 1155
Re: Integer Maths vs floating point
« Reply #27 on: 27 / April / 2021, 11:55:52 »
@philmoz

Phil

Just to say that I've 'converted' my M3 focus bracketing script to use fmath and it, obviously, looks a bit of a mess ;-) :-)

When I get time I will look into refactoring/rewriting the code to look pleasing to the eye

Bottom line: as far as I'm concerned fmath is a great success.

Do you believe it is fit enough to put into the main trunk? Or will it be added to cameras as and when users request it?

Cheers

Garry

reyalp

• 14083
Re: Integer Maths vs floating point
« Reply #28 on: 27 / April / 2021, 13:49:57 »
Do you believe it is fit enough to put into the main trunk? Or will it be added to cameras as and when users request it?
It would not be camera specific. My initial impression is that it would be a good addition to the trunk. I was a bit concerned about how much size it might add to the Lua module, but it seems likes it's only a few KB, which IMO is a reasonable trade.

I would still like to add the ability for Lua to load C based modules for things like this and imath, but   it shouldn't be a show stopper for this feature.

It looks like you can't mix fmath and integer values in the comparison operators - this appears to be a Lua limitation.
Yep. Looks like they changed that in 5.2

Note that equality also compares type http://www.lua.org/manual/5.1/manual.html#2.8
Quote
A metamethod only is selected when both objects being compared have the same type and the same metamethod for the selected operation.

so fmath.new(1) == 1 will always be false. This could easily produce subtle bugs, because unlike the relational comparison, it will be false rather than raising an error.
« Last Edit: 27 / April / 2021, 14:03:22 by reyalp »
Don't forget what the H stands for.

pigeonhill

• 1155
Re: Integer Maths vs floating point
« Reply #29 on: 27 / April / 2021, 13:54:54 »
Quote
I would still like to add the ability for Lua to load C based modules for things like this and imath, but   it shouldn't be a show stopper for this feature.

That’s great to hear, although I appreciate time is tight and things like this are not necessarily a priority.

Anyway, I’m happy, with an M3 that has real maths