Integer Maths vs floating point - page 3 - Script Writing - CHDK Forum
« previous next »
Pages: 1 2 [3] 4 5   Go Down

Integer Maths vs floating point

  • 40 Replies
  • 9501 Views
*

Offline pigeonhill

  • *****
  • 1164
Re: Integer Maths vs floating point
« Reply #20 on: 27 / April / 2021, 01:59:36 »
Advertisements
@philmoz

I forgot to ask, when using imath there are limits one has to ensure numbers don’t break, eg sqrt

What are the limits in the fmath library, say, for sqrt?

Cheers

Garry
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Integer Maths vs floating point
« Reply #21 on: 27 / April / 2021, 02:01:46 »
Quote from: pigeonhill on 27 / April / 2021, 01:59:36
@philmoz

I forgot to ask, when using imath there are limits one has to ensure numbers don’t break, eg sqrt

What are the limits in the fmath library, say, for sqrt?

Cheers

Garry


Everything in fmath uses double (64 bit) floating point operations so the only limits are what is inherent to the double data type.


Phil.

Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline pigeonhill

  • *****
  • 1164
Re: Integer Maths vs floating point
« Reply #22 on: 27 / April / 2021, 02:22:39 »
 :)
Logged
*

Offline pigeonhill

  • *****
  • 1164
Re: Integer Maths vs floating point
« Reply #23 on: 27 / April / 2021, 03:24:32 »
@philmoz

Phil

You have made me feel really good  :D

Just did a quick test of a typical calculation and tested the result in Wolfram Alpha.

Here is the test script:

Code: [Select]
--[[
@fmath
@chdk_version 1.5
--]]
u = 567 -- focus distance from the front principal
f = 11000 -- simulating CHDK focal length, ie 11mm = 11000
ff = fmath.new(f,1000) -- convert f to 11.000
h = 1500 -- hyperfocal distance
p = 3 -- pupil mag

ndof = (ff*ff*p - ff*ff - ff*p*u + ff*u - h*p*u)/(p*(ff-h-u)) -- near DoF as measured from the front principal

ndof = ndof:tostr()

print(ndof)

sleep(5000)

As I said before, this will transform my scripting of DoF stuff, ie not needing to use imath 'work arounds'.

Many thanks and I owe you a beer  :)
Logged
*

Offline pigeonhill

  • *****
  • 1164
Re: Integer Maths vs floating point
« Reply #24 on: 27 / April / 2021, 07:48:18 »
@philmoz

Just stumbled as I convert my imath based script to fmath.

I get the error "attempt to compare userdata with number", eg where ndof is a real

Code: [Select]
if ndof < 45 then print(1) else print (2) end
Does this mean ALL numbers in the script now need to be converted to real number format.

Or, could I convert reals back to 'just' normal integers? If so how should I best do this?

Cheers

Garry
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Integer Maths vs floating point
« Reply #25 on: 27 / April / 2021, 07:59:52 »
Quote from: pigeonhill on 27 / April / 2021, 07:48:18
I get the error "attempt to compare userdata with number", eg where ndof is a real
Code: [Select]
if ndof < 45 then print(1) else print (2) end

@pigeonhill

It looks like you can't mix fmath and integer values in the comparison operators - this appears to be a Lua limitation.

You should be able to use:
      if ndof:int() < 45 ...
or  if ndof < fmath.new(45) ...


Phil.
« Last Edit: 27 / April / 2021, 08:05:41 by philmoz »
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline pigeonhill

  • *****
  • 1164
Re: Integer Maths vs floating point
« Reply #26 on: 27 / April / 2021, 08:04:51 »
Cheers
Logged
*

Offline pigeonhill

  • *****
  • 1164
Re: Integer Maths vs floating point
« Reply #27 on: 27 / April / 2021, 11:55:52 »
@philmoz

Phil

Just to say that I've 'converted' my M3 focus bracketing script to use fmath and it, obviously, looks a bit of a mess ;-) :-)

When I get time I will look into refactoring/rewriting the code to look pleasing to the eye  :D

Bottom line: as far as I'm concerned fmath is a great success.

Do you believe it is fit enough to put into the main trunk? Or will it be added to cameras as and when users request it?

Cheers

Garry
Logged
*

Offline reyalp

  • ******
  • 14125
Re: Integer Maths vs floating point
« Reply #28 on: 27 / April / 2021, 13:49:57 »
Quote from: pigeonhill on 27 / April / 2021, 11:55:52
Do you believe it is fit enough to put into the main trunk? Or will it be added to cameras as and when users request it?
It would not be camera specific. My initial impression is that it would be a good addition to the trunk. I was a bit concerned about how much size it might add to the Lua module, but it seems likes it's only a few KB, which IMO is a reasonable trade.

I would still like to add the ability for Lua to load C based modules for things like this and imath, but   it shouldn't be a show stopper for this feature.

Quote from: philmoz on 27 / April / 2021, 07:59:52
It looks like you can't mix fmath and integer values in the comparison operators - this appears to be a Lua limitation.
Yep. Looks like they changed that in 5.2

Note that equality also compares type http://www.lua.org/manual/5.1/manual.html#2.8
Quote
A metamethod only is selected when both objects being compared have the same type and the same metamethod for the selected operation.

so fmath.new(1) == 1 will always be false. This could easily produce subtle bugs, because unlike the relational comparison, it will be false rather than raising an error.
« Last Edit: 27 / April / 2021, 14:03:22 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline pigeonhill

  • *****
  • 1164
Re: Integer Maths vs floating point
« Reply #29 on: 27 / April / 2021, 13:54:54 »
Quote
I would still like to add the ability for Lua to load C based modules for things like this and imath, but   it shouldn't be a show stopper for this feature.

That’s great to hear, although I appreciate time is tight and things like this are not necessarily a priority.

Anyway, I’m happy, with an M3 that has real maths  :)
Logged
Pages: 1 2 [3] 4 5   Go Up
« previous next »
 

Related Topics


SimplePortal © 2008-2014, SimplePortal