Live View format - RAW Shooting and Processing - CHDK Forum
supplierdeeply
« previous next »
Pages: [1]   Go Down

Live View format

  • 6 Replies
  • 8349 Views
*

Offline burglar

  • *
  • 23
Live View format
« on: 28 / March / 2018, 16:09:38 »
Advertisements
I'd like to do some analysis on the frames captured by live view but I am having hard time in decoding it so I wonder if there is some documentation about the raw format generated by the camera.

First of all, to get a frame I am using the Lua code:
Code: [Select]
con:live_dump_start(dumpfile)
con:live_get_frame(what)
con:live_dump_frame()
Already here I have some question. If I change the "what" I get different file size but from the lvdump function it seems that "what" should be 0=noframe; 1=vp; 5=vp+bm; 13=vp+bm+pal; 29=vp+bm+pal+bmo in all these cases the size should be width x height x bpp but the fact that I superimpose bm or not it shouldn't affect it.

Anyway I am reading the output file with this python:

Code: [Select]
frame_header = numpy.dtype([('magic','int32'),('header_size', 'int32'),('version_major', 'int32'),('version_minor','int32')])
frame_length = numpy.dtype([('length','int32')])
myFile = open('file.lvdump','r')
header=numpy.fromfile(myFile, dtype=frame_header, count=1)
length=numpy.fromfile(myFile, dtype=numpy.uint32, count=1)
frame=numpy.fromfile(myFile, dtype=numpy.ubyte, count=length)
myFile.close()
this works in the sense that I have in "frame" an array of the size written in the header, but I failed in converting it in RGB because I haven't understood the format. I thought it was in YUV8 but it doesn't seem so. So can someone point me to a simple function to decode it?
Many thanks!
« Last Edit: 28 / March / 2018, 16:14:48 by burglar »
Logged
*

Offline reyalp

  • ******
  • 14118
Re: Live View format
« Reply #1 on: 28 / March / 2018, 17:14:09 »
Quote from: burglar on 28 / March / 2018, 16:09:38
First of all, to get a frame I am using the Lua code:
Code: [Select]
con:live_dump_start(dumpfile)
con:live_get_frame(what)
con:live_dump_frame()
FWIW, you can use the lvdump -count=1 cli command to do the same thing.

You can also use lvdumpimg to dump live view and bitmap data to netpbm formats.

You can use cli:execute() to execute cli commands from lua code.

Quote
Already here I have some question. If I change the "what" I get different file size but from the lvdump function it seems that "what" should be 0=noframe; 1=vp; 5=vp+bm; 13=vp+bm+pal; 29=vp+bm+pal+bmo in all these cases the size should be width x height x bpp but the fact that I superimpose bm or not it shouldn't affect it.
lvdump saves a protocol dump, with a small header header for the file, and a length for each frame. Each protocol frame includes the live view protocol header, which describes the dimensions, position and format of the returned data, plus whichever frame buffers are selected. The live view, bitmap and (for digic 6) bitmap opacity are separate buffers, which is why the size changes based on 'what'. The protocol includes dimensions, position etc for each framebuffer in each frame, because some of them can change unpredictably at runtime. The protocol is described in:
https://app.assembla.com/spaces/chdk/subversion/source/HEAD/trunk/core/live_view.h

To summarize: Each frame includes a lv_data_header, which includes 2 or 3 lv_framebuffer_desc depending on protocol version. All the lv_framebuffer_desc are present regardless of which buffers are selected. To know the dimensions, offset and byte size of each framebuffer, you need to examine the corresponding lv_framebuffer_desc. In chdkptp, chdku.lua has code to bind these to lua friendly objects, see code starting around chdku.live_fields_v21

You can find the chdkptp code that converts these to RGB in https://app.assembla.com/spaces/chdkptp/subversion/source/HEAD/trunk/liveimg.c

If you want to extract the raw YUV or paletted data in Lua, code in gui_live.lua or chdku.lua may be a useful starting point.
Logged
Don't forget what the H stands for.
*

Offline burglar

  • *
  • 23
Re: Live View format
« Reply #2 on: 28 / March / 2018, 17:34:18 »
Thank you!

I know there are different methods, actually my first lua code was
Code: [Select]
local frame = con:get_live_data(nil, 1)
local pimg = liveimg.get_viewport_pimg(nil, frame, skip)
local lb = pimg:to_lbuf_packed_rgb(nil)but the liveimg.get_viewport_pimg returned invalid format for the frame, and this is why I started to wonder what I was doing wrong and why the frame was invalid for a function that is called apparently in the same way by lvdumpimg in the cli.lua.

Thank you for the explanation I read the gui_live.lua or chdku.lua and both uses the function I mentioned (actually they call the liveimg.c) so it didn't help because there is some piece that clearly I haven't saw.
I know that it works because when I run the lvdumping I obtain the netbpm file and if I run chdkptp with the gui I can start the live view, so chdk works, the camera works and the decoding seems to work. It seems that I am the only part of this problem not working correctly.
I will restart from the live_view.h file, it seems what I was looking for.
Thanks again.
Logged
*

Offline reyalp

  • ******
  • 14118
Re: Live View format
« Reply #3 on: 28 / March / 2018, 22:07:50 »
Quote from: burglar on 28 / March / 2018, 17:34:18
I know there are different methods, actually my first lua code was
Code: [Select]
local frame = con:get_live_data(nil, 1)
local pimg = liveimg.get_viewport_pimg(nil, frame, skip)
local lb = pimg:to_lbuf_packed_rgb(nil)
That seems like it should work. I did the following in chdkptp CLI and was able to load the resulting test.rgb in gimp as raw image data (720x480 rgb for this camera)

Code: [Select]
$ chdkptp -c -i
connected: Canon PowerShot ELPH 130 IS, max packet size 512
con> !frame = con:get_live_data(nil, 1)
con> !return frame
="userdata: 04CEFD30"
con> !pimg = liveimg.get_viewport_pimg(nil, frame, false)
con> !return pimg
="userdata: 04CB8958"
con> !lb = pimg:to_lbuf_packed_rgb(nil)
con> !fh=io.open('test.rgb','wb') lb:fwrite(fh) fh:close()

What model of camera are you using?

Are you using the official chdkptp distribution (if so, what version and platform?), or one of the python interfaces?

Quote
I know that it works because when I run the lvdumping I obtain the netbpm file and if I run chdkptp with the gui I can start the live view, so chdk works, the camera works and the decoding seems to work. It seems that I am the only part of this problem not working correctly.
If you are looking for RGB output to analyze, is there a reason not to just do
Code: [Select]
cli:execute('lvdumpimg -vp=file.ppm')
and load file.ppm in your python code?
Logged
Don't forget what the H stands for.
*

Offline burglar

  • *
  • 23
Re: Live View format
« Reply #4 on: 29 / March / 2018, 04:33:24 »
Quote from: reyalp on 28 / March / 2018, 22:07:50
That seems like it should work. I did the following in chdkptp CLI and was able to load the resulting test.rgb in gimp as raw image data (720x480 rgb for this camera)
Code: [Select]
$ chdkptp -c -i
connected: Canon PowerShot ELPH 130 IS, max packet size 512
con> !frame = con:get_live_data(nil, 1)
con> !return frame
="userdata: 04CEFD30"
con> !pimg = liveimg.get_viewport_pimg(nil, frame, false)
con> !return pimg
="userdata: 04CB8958"
con> !lb = pimg:to_lbuf_packed_rgb(nil)
con> !fh=io.open('test.rgb','wb') lb:fwrite(fh) fh:close()
Thanks, I will try this sequence.
Quote
What model of camera are you using?
I am using a CANON EOS M10, so the version of CHDK is not the official but a test version.
Quote
Are you using the official chdkptp distribution (if so, what version and platform?), or one of the python interfaces?
This problem is generated in the chdkptp.py, but in the installation process it downloads the trunk of chdkptp, so I think it should be the same. The only difference is that chdkptp.py applies some patch to the makefile in order to modify the target from the chdkptp executable to the library chdkptp.so but it should work in the same way.
Quote
If you are looking for RGB output to analyze, is there a reason not to just do
Code: [Select]
cli:execute('lvdumpimg -vp=file.ppm')
and load file.ppm in your python code?
Because my ultimate goal is not to pass by file but to dump the frame and have it as a numpy array to manipulate. I want to do some image detection to automatically align a telescope (so I will interface it with astrometry).
Thank you for your answers, I have now several things to try.
Logged
*

Offline burglar

  • *
  • 23
Re: Live View format
« Reply #5 on: 29 / March / 2018, 13:13:55 »
I was able to parse the data and reconstruct the YUV image, so I think it can be useful for other people to know how to do it in python.

This code is very raw, no control of the errors is done, the check about the presence of the buffers can be probably wrong etc. The only purpose is to explain with a proof-of-concept how the data are organized in the file. I have to thank @reyalp for the excellent answer that pointed me in the right direction to learn how to decode this file.

Code: [Select]
import matplotlib.pyplot as plt
import numpy
import cv2

fb_type = {0:12, # LV_FB_YUV8 8 bit per element UYVYYY, used for live view
           1:8, # LV_FB_PAL8 8 bit paletted, used for bitmap overlay. Note palette data and type sent separately
           2:16, # LV_FB_YUV8B 8 bit per element UYVY, used for live view and overlay on Digic 6
           3:16, # LV_FB_YUV8C 8 bit per element UYVY, used for alternate Digic 6 live view
           4:8 # LV_FB_OPACITY8 8 bit opacity / alpha buffer
          }
file_header_dtype = numpy.dtype([('magic','int32'),('header_size', 'int32'),('version_major', 'int32'),('version_minor','int32')])
frame_length_dtype = numpy.dtype([('length','int32')])
frame_header_dtype = numpy.dtype([('version_major','int32'),('version_minor', 'int32'),('lcd_aspect_ratio', 'int32'),
                            ('palette_type','int32'), ('palette_data_start','int32'), ('vp_desc_start','int32'),
                            ('bm_desc_start','int32'), ('bmo_desc_start','int32')])
block_description_dtype = numpy.dtype([('fb_type','int32'),('data_start','int32'),('buffer_width','int32'),
                                ('visible_width','int32'),('visible_height','int32'),('margin_left','int32'),
                                ('margin_top','int32'),('margin_right','int32'),('margin_bottom','int32')])

myFile = open('test.lvdump','r')
file_header=numpy.fromfile(myFile, dtype=file_header_dtype, count=1)
frame_length=numpy.fromfile(myFile, dtype=frame_length_dtype, count=1)
frame_header=numpy.fromfile(myFile, dtype=frame_header_dtype, count=1)
vp_description=numpy.fromfile(myFile, dtype=block_description_dtype, count=1)
vp_bpp = fb_type[int(vp_description['fb_type'])]
vp_frame_size=vp_description['buffer_width']*vp_description['visible_height']*vp_bpp/8 # in byte !

bm_description=numpy.fromfile(myFile, dtype=block_description_dtype, count=1)
bm_bpp = fb_type[int(bm_description['fb_type'])]
bm_frame_size=bm_description['buffer_width']*bm_description['visible_height']*bm_bpp/8

bmo_description=numpy.fromfile(myFile, dtype=block_description_dtype, count=1)
bmo_frame_size=bmo_description['buffer_width']*bmo_description['visible_height']*fb_type[int(bmo_description['fb_type'])]/8

if vp_description['data_start'] > 0:
    vp_raw_img=numpy.fromfile(myFile, dtype=numpy.uint8, count=vp_frame_size)
### WARNING: the extraction of the YUV format depends on the fb_type. Here I assume the type 2 that is UYVY but this must go in a proper function that takes into account the different cases.    y=vp_raw_img[1::2].reshape(int(vp_description['visible_height']),int(vp_description['buffer_width']))
    u=numpy.empty(vp_frame_size/2, dtype=numpy.uint8)
    u[0::2]=vp_raw_img[0::4]
    u[1::2]=vp_raw_img[0::4]
    u=u.reshape(int(vp_description['visible_height']),int(vp_description['buffer_width']))
    v=numpy.empty(vp_frame_size/2, dtype=numpy.uint8)
    v[0::2]=vp_raw_img[2::4]
    v[1::2]=vp_raw_img[2::4]
    v=v.reshape(int(vp_description['visible_height']),int(vp_description['buffer_width']))
    raw_yuv=np.dstack((y,u,v))[:,0:int(vp_description['visible_width']),:]
    vp_rgb=cv2.cvtColor(raw_yuv, cv2.COLOR_YUV2BGR)
if bm_description['data_start'] > 0:
    bm_raw_img=numpy.fromfile(myFile, dtype=numpy.uint8, count=bm_frame_size)
    y=bm_raw_img[1::2].reshape(int(bm_description['visible_height']),int(bm_description['buffer_width']))
    u=numpy.empty(bm_frame_size/2, dtype=numpy.uint8)
    u[0::2]=bm_raw_img[0::4]
    u[1::2]=bm_raw_img[0::4]
    u=u.reshape(int(bm_description['visible_height']),int(bm_description['buffer_width']))
    v=numpy.empty(bm_frame_size/2, dtype=numpy.uint8)
    v[0::2]=bm_raw_img[2::4]
    v[1::2]=bm_raw_img[2::4]
    v=v.reshape(int(bm_description['visible_height']),int(bm_description['buffer_width']))
    raw_yuv=np.dstack((y,u,v))[:,0:int(bm_description['visible_width']),:]
    bm_rgb=cv2.cvtColor(raw_yuv, cv2.COLOR_YUV2BGR)
if bmo_description['data_start'] >0: # Not used at the moment
    bmo_raw_img=numpy.fromfile(myFile, dtype=numpy.int32, count=bmo_frame_size)
myFile.close()

plt.figure(figsize=(20,10))
plt.imshow(numpy.clip(vp_rgb.astype(numpy.uint16)+bm_rgb.astype(numpy.uint16),0,255).astype(numpy.uint8))
Logged
*

Offline reyalp

  • ******
  • 14118
Re: Live View format
« Reply #6 on: 29 / March / 2018, 13:15:39 »
Quote from: burglar on 29 / March / 2018, 04:33:24
I am using a CANON EOS M10, so the version of CHDK is not the official but a test version.
This is a digic 6 camera, so the format is different from earlier cams. This should be supported as long a you are using recent versions of CHDK and chdkptp, but your mention of "invalid format" suggests something might not be current.

Quote
This problem is generated in the chdkptp.py, but in the installation process it downloads the trunk of chdkptp, so I think it should be the same. The only difference is that chdkptp.py applies some patch to the makefile in order to modify the target from the chdkptp executable to the library chdkptp.so but it should work in the same way.
I'd suggest making double checking that it's actually current. The official chdkptp source is at https://app.assembla.com/spaces/chdkptp/subversion/source/HEAD/trunk - there are some other forks on github but I don't know anything about how current or not they are.

Another possibility would be bugs in the M10 port, but if the standalone chdkptp CLI lvdumpimg works, that suggests the port is OK.

Quote
Because my ultimate goal is not to pass by file but to dump the frame and have it as a numpy array to manipulate. I want to do some image detection to automatically align a telescope (so I will interface it with astrometry).
While slightly inefficient, it seems like going through a temporary file should work. However, you might be better off working directly with Y values rather than converting to RGB.

This sounds like an interesting project, any details you care to share on the forum would be welcome.
Logged
Don't forget what the H stands for.
Pages: [1]   Go Up
« previous next »
 

Related Topics
SimplePortal 2.3.6 © 2008-2014, SimplePortal