Size-optimized badpixel format - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: [1] 2   Go Down

Size-optimized badpixel format

  • 10 Replies
  • 4850 Views
*

Offline srsa_4c

  • ******
  • 4451
Size-optimized badpixel format
« on: 14 / November / 2013, 16:37:46 »
Advertisements
I'm working on more efficient badpixel formats. The results will surely be limited by my less than optimal math skills, but at least I now have something that appears to work. I got ideas from previous threads in this forum, thanks for those who mentioned them.

Because my limited skills, I'm only using halfword units, not a bitstream.
The common part of the schemes is that one badpixel coordinate (x) is relative, not absolute. I'm also assuming that the y coordinate fits in 12bits (AFAIK it's true for all current Canon compacts).

Description of the v4 method (v4 is what I call it):
Badpixel descriptors are stored in 16bit halfwords.

bit11...bit0: [y dimension, except when 0x0fff] 0xfff means dummy badpixel, only needed when x distance from previous badpixel exceeds 14.
bit15..bit12: IF not dummy:
             0...14: x distance from previous badpixel (or dummy or the "0" coordinate of the x axis)
                 15: last badpixel in a group of badpixels

This scheme can "compress" like this:
- RLE compressed badpixel groups: 4 bytes before, 4 bytes after, however group length is not limited by 8
- single badpixels: 4 bytes before, 2 bytes after
- overhead: long x "gaps" without badpixels will result in added dummy badpixel descriptors

Example CMOS case (Ixus310): 153180 bytes -> 103400 bytes
Example CCD case (Ixus870): 80128 bytes -> 40270 bytes

I have some ideas that can improve the compression of badpixel groups, it's in progress.

Attached is a patch for trunk (for testing only, replaces existing routines, NOT compatible with stock badpixel.bins), and a command line utility for transforming between the current and "v4" format. Both can contain errors, and the method is fixed to the above mentioned 12+4 bit split.

If something like this ever gets in, I would change the badpixel.bin file name AND use a proper file header - current method needs CAM_ACTIVE_AREA_X1 to work correctly.

Critique, suggestions, etc. are welcome. Sorry about the messy coding style.
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Size-optimized badpixel format
« Reply #1 on: 14 / November / 2013, 16:45:44 »
Another option that occurred to me when I was looking at the RLE bug last night was that the RLE concept could be extended to the X co-ordinate the same way it is applied to Y.

The top three bits of the X co-ordinate would hold the number of Y values following -1 - so 0 means 1 Y value, etc.

This would be backwards compatible with existing badpixel.bin files and fairly easy to implement.

It may not save as much space as your method though.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline srsa_4c

  • ******
  • 4451
Re: Size-optimized badpixel format
« Reply #2 on: 14 / November / 2013, 17:02:57 »
Quote from: philmoz on 14 / November / 2013, 16:45:44
Another option that occurred to me when I was looking at the RLE bug last night was that the RLE concept could be extended to the X co-ordinate the same way it is applied to Y.

The top three bits of the X co-ordinate would hold the number of Y values following -1 - so 0 means 1 Y value, etc.

This would be backwards compatible with existing badpixel.bin files and fairly easy to implement.
I guess that would allow for RLE in one direction only (that's not much limitation though).
Quote
It may not save as much space as your method though.
I'm not against competing compression methods :) We could have more of them, either user-selectable or fixed for a camera model.
Perhaps a badpixel.bin collection could aid testing.
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Size-optimized badpixel format
« Reply #3 on: 15 / November / 2013, 03:39:54 »
This patch adds additional RLE compression on the X co-ordinate - up to 8 entries using the same co-ordinate can be compressed, with the length in the top 3 bits.

On my G12, with 2,486 bad pixels, it reduces the badpixel.bin file size from 9,868 to 8,500 bytes.

On my IXUS310 (68,498 bad pixels), the file size drops from 153,180 to 89,686 bytes.

The code should be compatible with existing badpixel.bin files for patching bad pixels during image capture.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline srsa_4c

  • ******
  • 4451
Re: Size-optimized badpixel format
« Reply #4 on: 15 / November / 2013, 11:23:17 »
Quote from: philmoz on 15 / November / 2013, 03:39:54
This patch adds additional RLE compression on the X co-ordinate - up to 8 entries using the same co-ordinate can be compressed, with the length in the top 3 bits.
OK, I misunderstood your intention. I thought you wanted to collect badpixels along the other axis.

Quote
On my G12, with 2,486 bad pixels, it reduces the badpixel.bin file size from 9,868 to 8,500 bytes.

On my IXUS310 (68,498 bad pixels), the file size drops from 153,180 to 89,686 bytes.
That's impressive.

Quote
The code should be compatible with existing badpixel.bin files for patching bad pixels during image capture.
If testing proves that, I see no reason not to add this to trunk. edit: Actually, I do see one. A CHDK downgrade to 1.2 on the card would made the badpixel file fail to work - unless the changes get backported.

v5 (only command line utility ATM):
Example CMOS case (Ixus310): 153180 bytes -> 103400 (v4) -> 79702 (v5)
Another CMOS (sx230): 66224 bytes -> 40238 (v4) -> 34862 (v5)
Example CCD case (Ixus870): 80128 bytes -> 40270 (v4) -> 40100 (v5)
« Last Edit: 16 / November / 2013, 10:17:57 by srsa_4c »
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Size-optimized badpixel format
« Reply #5 on: 16 / November / 2013, 17:41:15 »
Quote from: srsa_4c on 15 / November / 2013, 11:23:17
Quote
The code should be compatible with existing badpixel.bin files for patching bad pixels during image capture.
If testing proves that, I see no reason not to add this to trunk. edit: Actually, I do see one. A CHDK downgrade to 1.2 on the card would made the badpixel file fail to work - unless the changes get backported.

Yes, the change needs to go into both trunk and 1.2 versions.

I've tested this on my IXUS310 which has over 64,000 bad pixels and it reliably fixes bad pixels using both the old and new badpixel.bin files.

I'll add this to SVN for both 1.2 & 1.3 soon.

You're v5 results are looking very good - nicely done.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline srsa_4c

  • ******
  • 4451
Re: Size-optimized badpixel format
« Reply #6 on: 17 / November / 2013, 16:08:37 »
Quote from: philmoz on 16 / November / 2013, 17:41:15
I'll add this to SVN for both 1.2 & 1.3 soon.

You're v5 results are looking very good - nicely done.
It's still vapourware though. Your results with the double RLE are pretty close with much less code complexity...
Additional concern is the size of DNG module - I'm now thinking about putting only the interpreter(s) into it, and create a separate module that can create those formats - from a regular badpixel.bin and maybe from the text based badpixel file.

v5 description - units are halfwords:

bit11...bit0: [y dimension when < 0xfff, 0xfff = dummy badpixel descriptor]

normal mode:
bit14..bit12: IF bit15=0: x distance (0..7) from the previous badpixel
              IF bit15=1: same column, nr of all badpixels in the current group (1..8)
bit15:        as above

0xfff mode:
bit15..bit12: x distance + 8


v5 is better on the typical CMOS data, v4 is better on CCD data with only a few hundreds or thousands of badpixels.
Logged
*

Offline srsa_4c

  • ******
  • 4451
Re: Size-optimized badpixel format
« Reply #7 on: 18 / November / 2013, 17:50:20 »
v6

                      uncomp    RLE1   RLE2      v4      v5      v6
ixus310 sample (CMOS):273992  153180  89686  103400   79702   76592
sx230 sample (CMOS)  : 97496   66224          40238   34862   33322
ixus870 sample (CCD) : 80540   80128          40270   40100   40102
a3200 iso1600 (CCD)  : 13712                   6866    6878    8934

description, units are halfwords

bit11...bit0: [y dimension when < 0xfe0, 0xfe0..0xfff = x jump]

normal mode:
bit14..bit12: nr of all badpixels in the current group, including the current bp (1..8)
bit15:        0: badpixel is in the same column
              1: badpixel is in the next column (bigger x steps need x jump)

x jump mode:
bit15...bit0: 0xffe0 = jump 2, 0xffe1 = jump 3, ... 0xffff = jump 33
Logged
*

Offline srsa_4c

  • ******
  • 4451
Re: Size-optimized badpixel format
« Reply #8 on: 25 / November / 2013, 17:42:17 »
v7, variable unit length (probably the last method I ever come up with)
One drawback: bigger code size, may be too slow for the camera (never tested)
                      uncomp    RLE1   RLE2      v4      v5      v6      v7
ixus310 sample (CMOS):273992  153180  89686  103400   79702   76592   65750
sx230 sample (CMOS)  : 97496   66224          40238   34862   33322   28970
ixus870 sample (CCD) : 80540   80128          40270   40100   40102   32522
a3200 iso1600 (CCD)  : 13712                   6866    6878    8934    6684

description, units are either 16 or 12 bit long

bit11..bit8:  y dimension high order nibble (in normal mode), skipped in 12bit spec mode
bit7...bit0:  y dimension low order byte (normal mode), y jump low order byte (spec mode)
             
bit15:        0: spec (special) mode
              1: normal mode (unit is 16bit)

bit14..bit12 (norm): nr of badpixels ahead (1..8)

bit14 (spec): 0: same column, most significant 2 bits of y increment (1..1024) in bit13..bit12, unit is 12bit
              1: x distance +1 in bit13..bit12, unit is 16bit

bit13..bit12 (spec): when bit14=0: bit5..bit4 of the y increment value
                     when bit14=1: x distance +1

exception1: 16bit units between 0xffe0...0xfffe: x increase by 5..35

exception2: 16bit unit with 0xffff as value is ignored

exception3: if a normal mode unit shows less or same y dimension than the previous, it's in the next column (x+1)

determination of 12bit spec mode: bit15=0 and bit14=0

Command line utility attached, see disclaimers in the first post.
« Last Edit: 25 / November / 2013, 18:04:21 by srsa_4c »
Logged
*

Offline nocaC_230

  • *
  • 27
Re: Size-optimized badpixel format
« Reply #9 on: 26 / November / 2013, 08:46:53 »
@srsa_4c,

I have "badpixcodec.c" compiled with Cygwin and everything works perfectly.
I have created a new "badpixel.bin" with half the size of the previous version.

But what do i have to do now to use the new v7 badpixel.bin (With the new "badpixel.bin" the camera {SX230HS 1.00c CHDK trunk3230} crash) ?

Ciao
« Last Edit: 26 / November / 2013, 09:40:05 by nocaC_230 »
Logged
Pages: [1] 2   Go Up
« previous next »
 

Related Topics
SimplePortal 2.3.6 © 2008-2014, SimplePortal