Analyze picture in JPEG format

  • 12 Replies
  • 1387 Views
Analyze picture in JPEG format
« on: 26 / February / 2015, 21:49:29 »
Advertisements
Hello everyone:

I am currently doing a project related to CHDK. One of my target is to do some analysis to the picture that user have taken(shape detection, etc.). However, the picture is saved in JPEG mode, which cannot directly analyze. Can anyone offer me some idea of how to approach this target?

Thanks! :)

*

Offline reyalp

  • ******
  • 11514
Re: Analyze picture in JPEG format
« Reply #1 on: 26 / February / 2015, 22:37:05 »
I am currently doing a project related to CHDK. One of my target is to do some analysis to the picture that user have taken(shape detection, etc.). However, the picture is saved in JPEG mode, which cannot directly analyze. Can anyone offer me some idea of how to approach this target?
If you want to do it when the picture is captured, you can analyze the raw buffer.  Be warned that if you want to analyze the whole buffer it will be quite slow.

In the development branch, I have added the ability to read raw pixels from Lua script, discussed starting at  http://chdk.setepontos.com/index.php?topic=11081.msg119265#msg119265 but this is even slower than C code.

Otherwise, if you want to analyze jpegs saved on the card, you would either need to implement a jpeg decoder or figure out how to re-use the one in the canon firmware.

If you only need low resolution, you could look at the viewport buffer while the image is shown in review mode. This isn't available from script, but is already used for other purposes like PTP live view.

Using your own jpeg decode would likely be difficult because the amount of ram available to CHDK is likely to be less than a decoded image. Additionally, CHDK doesn't have standard runtime libraries available, so third party libraries will probably be difficult to port.

If analyzing the images off camera on a PC (or an SBC like raspberry pi or a smart phone) is an option, that would probably be much simpler, because you will have a lot more CPU and RAM, and will have a vast collection of imaging libraries available.
Don't forget what the H stands for.

Re: Analyze picture in JPEG format
« Reply #2 on: 26 / February / 2015, 23:18:40 »
I am currently doing a project related to CHDK. One of my target is to do some analysis to the picture that user have taken(shape detection, etc.). However, the picture is saved in JPEG mode, which cannot directly analyze. Can anyone offer me some idea of how to approach this target?
If you want to do it when the picture is captured, you can analyze the raw buffer.  Be warned that if you want to analyze the whole buffer it will be quite slow.

In the development branch, I have added the ability to read raw pixels from Lua script, discussed starting at  http://chdk.setepontos.com/index.php?topic=11081.msg119265#msg119265 but this is even slower than C code.

Otherwise, if you want to analyze jpegs saved on the card, you would either need to implement a jpeg decoder or figure out how to re-use the one in the canon firmware.

If you only need low resolution, you could look at the viewport buffer while the image is shown in review mode. This isn't available from script, but is already used for other purposes like PTP live view.

Using your own jpeg decode would likely be difficult because the amount of ram available to CHDK is likely to be less than a decoded image. Additionally, CHDK doesn't have standard runtime libraries available, so third party libraries will probably be difficult to port.

If analyzing the images off camera on a PC (or an SBC like raspberry pi or a smart phone) is an option, that would probably be much simpler, because you will have a lot more CPU and RAM, and will have a vast collection of imaging libraries available.

Thanks for your reply! :D I am quite OK with the delay since the analyze won't be too difficult. Do you have any idea on which function should I looked into if I want to do any "runtime" analysis on the raw picture? Also, what's the raw buffer's format, RGB or YUV?

*

Offline reyalp

  • ******
  • 11514
Re: Analyze picture in JPEG format
« Reply #3 on: 26 / February / 2015, 23:52:59 »
Thanks for your reply! :D I am quite OK with the delay since the analyze won't be too difficult.
Do you have any idea on which function should I looked into if I want to do any "runtime" analysis on the raw picture? Also, what's the raw buffer's format, RGB or YUV?
The raw buffer is bayer RGB. See http://chdk.wikia.com/wiki/Frame_buffers for more information.

In C code you can use get_raw_pixel from core/raw.c to read a single bayer element. To know which color a pixel is, you need to look at the camera_sensor.cfa_pattern and the coordinates.

In the trunk code, modules/rawhookops.c contains the Lua interface. Even if you don't want to use Lua, this has examples of getting RGB from the bayer pattern.

The raw buffer is only valid during the shooting process. To keep it from getting destroyed while you are working on it, the processing code should be in core/raw.c raw_process or use the raw hook from Lua.
Don't forget what the H stands for.


Re: Analyze picture in JPEG format
« Reply #4 on: 27 / February / 2015, 01:08:26 »
Thanks for your reply! :D I am quite OK with the delay since the analyze won't be too difficult.
Do you have any idea on which function should I looked into if I want to do any "runtime" analysis on the raw picture? Also, what's the raw buffer's format, RGB or YUV?
The raw buffer is bayer RGB. See http://chdk.wikia.com/wiki/Frame_buffers for more information.

In C code you can use get_raw_pixel from core/raw.c to read a single bayer element. To know which color a pixel is, you need to look at the camera_sensor.cfa_pattern and the coordinates.

In the trunk code, modules/rawhookops.c contains the Lua interface. Even if you don't want to use Lua, this has examples of getting RGB from the bayer pattern.

The raw buffer is only valid during the shooting process. To keep it from getting destroyed while you are working on it, the processing code should be in core/raw.c raw_process or use the raw hook from Lua.

I really appreciate your help!  :D One more question is that is it safe(valid) to directly change the content in buffer, like flip all the colors or change all red into blue? I saw the function of set_raw_pixel but it seems to be used only in the removement of bad pixels.

*

Offline reyalp

  • ******
  • 11514
Re: Analyze picture in JPEG format
« Reply #5 on: 27 / February / 2015, 01:49:51 »
I really appreciate your help!  :D One more question is that is it safe(valid) to directly change the content in buffer, like flip all the colors or change all red into blue?
Yes, I've done this extensively in my Lua raw hook scripts. The debug displays on the videos in http://chdk.setepontos.com/index.php?topic=11081.40 are all done that way.

If you call set_raw_pixel outside of the time the raw buffer is valid (in raw_process or the Lua raw hook) then the camera might be using the memory for something else. So don't do that ;)
Don't forget what the H stands for.

Re: Analyze picture in JPEG format
« Reply #6 on: 27 / February / 2015, 18:41:58 »
I really appreciate your help!  :D One more question is that is it safe(valid) to directly change the content in buffer, like flip all the colors or change all red into blue?
Yes, I've done this extensively in my Lua raw hook scripts. The debug displays on the videos in http://chdk.setepontos.com/index.php?topic=11081.40 are all done that way.

If you call set_raw_pixel outside of the time the raw buffer is valid (in raw_process or the Lua raw hook) then the camera might be using the memory for something else. So don't do that ;)

I've try it out and everything seems working well :xmas. But it seems that by modifying the raw_process it won't directly change the JPEG file it save. If I turned on the saving raw I can see the change I make in the raw file, but there's no effect on the JPEG file. Do you have any idea on how to change the JPEG at the same time?

Thanks!

*

Offline srsa_4c

  • ******
  • 3695
Re: Analyze picture in JPEG format
« Reply #7 on: 27 / February / 2015, 18:57:00 »
But it seems that by modifying the raw_process it won't directly change the JPEG file it save. If I turned on the saving raw I can see the change I make in the raw file, but there's no effect on the JPEG file. Do you have any idea on how to change the JPEG at the same time?
My guess: you're using black pixels.
The JPEG creation process may interpolate over pixels that are (near) black. Try using pixel values above the black level (search for CAM_BLACK_LEVEL in include/camera.h).


Re: Analyze picture in JPEG format
« Reply #8 on: 27 / February / 2015, 19:01:22 »
I really appreciate your help!  :D One more question is that is it safe(valid) to directly change the content in buffer, like flip all the colors or change all red into blue?
Yes, I've done this extensively in my Lua raw hook scripts. The debug displays on the videos in http://chdk.setepontos.com/index.php?topic=11081.40 are all done that way.

If you call set_raw_pixel outside of the time the raw buffer is valid (in raw_process or the Lua raw hook) then the camera might be using the memory for something else. So don't do that ;)

I've try it out and everything seems working well :xmas. But it seems that by modifying the raw_process it won't directly change the JPEG file it save. If I turned on the saving raw I can see the change I make in the raw file, but there's no effect on the JPEG file. Do you have any idea on how to change the JPEG at the same time?

Thanks!
Just ignore it... I figure out the problem. :P

Re: Analyze picture in JPEG format
« Reply #9 on: 01 / March / 2015, 01:30:10 »
Could anyone tell me more about the raw buffer's content? I am now trying to replace all the green or some other color that is near green to red. I am using the function of rawop_get_pixels_rgbg and rawop_set_pixels_rgbg(same content, just change the API to fit for C). I am using the filter of ((g1>800||g2>800)&&r<600&&b<600), but it seems it can only detect a really small amount of green object in the picture. Is there any fomular or some reference material to show the relation between actual color the rgbg code(for example, change rgbg into standard RGB)?

 

Related Topics