It's not the physical level, it's the filesystem level which gets fragmented. Erasing existing files doesn't make their data go away, deleted directory entries remain, FAT chains are only marked as deleted, fragments of new files will be scattered through the whole area as fragmentation grows. The card doesn't know about the filesystem.