Understanding Flash filesystems and storage

SRobertJames

2

« on: 09 / March / 2010, 09:41:52 »

Advertisements

I'm trying to understand some details of flash storage. My understanding was that flash requires an erase-before-write, which is done by writing 1111s to the data, and only that can be written on. But, as an experiment, I deleted some pictures from a flash SD card, loaded the card's image up in a hex editor, and, voila! the original data bits were still there - just as if they were on a magnetic device.

When does deleting pictures do a "erase" (overwrite with 1111s) and when does it just mark them as deleted (like magnetic media)?
If they are just marked as deleted, how are they ever overwritten? Does the camera just erase them on the fly?
Most importantly: Where in the firmware is this all handled?

Logged

peabody

167

Re: Understanding Flash filesystems and storage

« Reply #1 on: 09 / March / 2010, 12:33:01 »

I've been looking at this issue too. My understanding is the same as yours - a write operation can be performed only on a block that has been fully erased beforehand. Every SD card has a micro-controller built in. This controller takes care of all the low-level operations of the card, including wear leveling, erasing, reading and writing. So this means that the camera, or card reader, or whatever the card is plugged into, can communicate with the card on a higher level with standardized commands, and not have to deal with all the minutia.

There are high-level commands which instruct the controller to erase a block, or range of blocks, of memory, and in theory doing that would speed up subsequent writes to those blocks. That's because if you just try to write to a block and it isn't erased, then the controller has to erase it on-the-fly before performing the write, and you have to wait while it does that. (The card's controller always responds to commands with completion codes, so you would just be waiting longer for that code to come back.)

However, it's not clear just how the controller knows whether a block is erased or not. Of course it could keep track in its ram of what it has erased during the current session, but what you really need is some flag for each block that survives a power cycle and can be quickly read by the controller. You have to believe there is such a mechanism, but I can't find any confirmation of that. This is a key point, because if the controller has no quick way to determine if a block is erased, then it would have to automatically erase everying on-the-fly before writing, and erasing a card in advance would yield no benefit.

There is an SD card formatting program at sdcard.org (under "Consumers") which claims to erase a card as part of the formatting process. But it tells me my "interface device" doesn't support that operation. Also, the program has been known at times to mess up cards and leave hundreds of MB unaccessible.

On the other hand, my Canon A590IS has an option to "low-level" format a card. It's possible that's an erase operation. I just don't know.

I found the Wikipedia entries for "secure digital" and "flash memory" to be quite good.

Let me know if you find answers to this. I'd really like to know if erasing a card ahead of time might speed up the write process, and if it does, how I can do such an erase.

Logged

reyalp

14118

Re: Understanding Flash filesystems and storage

« Reply #2 on: 10 / March / 2010, 01:46:43 »

Quote from: SRobertJames on 09 / March / 2010, 09:41:52

But, as an experiment, I deleted some pictures from a flash SD card, loaded the card's image up in a hex editor, and, voila! the original data bits were still there - just as if they were on a magnetic device.

This is a filesystem level thing. Deleting in FAT doesn't erase the data, it just marks the clusters as free. This is a higher level thing that doesn't know or care about the underlying technology.

If an erase+write sequence is needed, it would be done when those sectors were used for a new file (whether by the SD controller, the card itself or the firmware I don't know.)

Slightly OT, but the firmware does have code for erasing vs. writing on board flash (where the firmware and other permanent settings are stored). Unused space in this flash is indeed initialized to all 1s

Logged

Don't forget what the H stands for.

SRobertJames

2

Re: Understanding Flash filesystems and storage

« Reply #3 on: 11 / March / 2010, 13:53:23 »

Perhaps this is what's going on. At one level, the flash *memory* is an array of bytes. You can't write to a byte which hasn't been erased yet. You also need to worry about wear leveling.

However, the flash *controller* hides this. It wants to present the appearance of a conventional disk, where you can write as much as you want. So, it presents a different array, which is writeable, where you don't need to worry about wear leveling. It keeps some metadata in the *memory* array, which tells it where to map the *virtual array*. Imagine a hidden mini filesystem. Or virtual memory.

To give an example, let's say we have a brand new 1GB Flash Card. We write 0xAABBCCDD to address 0. That means address 0 of the virtual array. The controller might map this to address 1000 of the physical memory, so it writes 0xAABBCCDD to value 1000 of the memory. It also writes to its allocation table, which it stores at address 0 of the memory, that virtualaddrss 0 corresponds to physicaladdress 1000.

When we tell it to read address 0, it looks in the table (which it stores at physicaladdress 0) that virtualaddress0 maps to physicaladdress1000

Now, later on, we write 0x11223344 to address 0, the controller sees in its table that address 0 is already written to, not erased. So, it instead appends the table (not overwrites - it appends - the table can be changed through append operations only) that virtualaddress0 now maps to physicaladdress2000. It then writes 0x11223344 to physicaladdress2000.

The end results are that the host app can use the Flash SD as if it was magnetic media, and the controller handles all of the flash peculiarites.

This also has great impact for data recovery and forensics. If we can bypass the virtualization layer, and get directly at the physical memory, we can see a whole of new things!

So, the question is, is there a way (without opening up the card and extracting the memory), to access - at least at the read level - the physical memory itself.

Logged