One can write more blocks per pass in order to reduce
the total number of passes.
Trading resets for writes is effective when writing blocks is
cheaper than reseting the device being probed.
This patch dynamically balances the number of writes and
resets while probing.
The effectiveness of this balance is shown below:
A good 256MB drive produced the following measurements:
Probe time: 2.89 seconds
Probe read op: count=64, total time=0.13s, avg op time=2.06ms
Probe write op: count=48, total time=1.41s, avg op time=29.47ms
Probe reset op: count=8, total time=1.35s, avg op time=168.48ms
The results from previous commit (see git log):
Probe time: 47.57 seconds
Probe read op: count=2014, total time=1.72s, avg op time=0.85ms
Probe write op: count=2003, total time=45.32s, avg op time=22.62ms
Probe reset op: count=3, total time=0.53s, avg op time=175.66ms
Moreover, this patch spaces more uniformly
the blocks write_test_blocks() writes to improve
the effectiveness of each pass.
In order to reduce probing time, one needs to time
reads, writes, and resets to evaluate how to change
the probing algorithm.
This patch adds these measurements.
A good 256MB drive produced the following measurements:
Probe time: 47.57 seconds
Probe read op: count=2014, total time=1.72s, avg op time=0.85ms
Probe write op: count=2003, total time=45.32s, avg op time=22.62ms
Probe reset op: count=3, total time=0.53s, avg op time=175.66ms
Of all three operations called on the flash drive being probed
(i.e. reading blocks, writing blocks, and resetting the drive),
the most time-consuming operation is reliably resetting the drive.
This patch reduces the number of resets writing and reading
more blocks.
This option instructs f3probe to trade speed for less memory usage,
that is, f3probe minimizes use of memory.
Currently, this option only drops the use of the bitmap of
the safe device.
Nevertheless, this is not negligible memory.
For example, a 1TB drive whose block size is 512 Bytes requires
256MB of RAM for this bitmap:
1TB / 512Byte/Block = 2^31Block
2^31Block / 1Block/bit = 2^31bit
2^31bit / 8bit/Byte = 2^28Byte = 256MB
To put it in context, 256MB of RAM was all that
Raspberry Pi Model A had.
The current reset method isn't supported for all USB drives,
what leads to wrong conclusions about some fake drives.
The option --manual-reset allows users to unplug and plug back
the USB drive being tested to manually reset the drive.
This patch makes f3probe read and save in memory
the content of all written blocks before writting them
during the probe, and restore the original content of those blocks
after probing the device.
In order words, f3probe behaves likes the probed device were
read only.
All the necessary memory to protect the probed device is
preallocated, so an on-going probe does not fail due to
lack of memory.
Although this feature adds an important protection layer,
users should be conscious that things can go wrong, and
the data will be lost.
This patch also adds option --debug-keep-file to help debugging
the new code this patch adds, and option --destructive to disable
the protection.
search_wrap() was considering @high_bit to be in blocks,
but it was in bytes.
This patch also improves the unit test to catch the fixed bug, and
to use real geometries.
This patch makes file devices use the same geometry model that
probe_device() probes for.
This change helps one to test whatever probe_device() finds in
the field.
The code is based on udev_example.c available on the following page:
libudev and Sysfs Tutorial
http://www.signal11.us/oss/udev/
f3probe nows requires library libudev.
This patch also updates the following files to reflect
the dependency of f3probe on libudev, and
the experimental status of f3probe: Makefile and README.
The internal cache of the USB drive was keeping all recent writes,
so subsequents reads were correct even when the underlying
storage was faulty.
The code is based on usbreset.c available on the following page:
How do you reset a USB device from the command line?
http://askubuntu.com/questions/645/how-do-you-reset-a-usb-device-from-the-command-line
- Addopt -ggdb to add debugging information to binary.
- Add cscope target to help navigating the code.
- Add cscope.out to .gitignore.
- Add *.swp to .gitignore.
- Change compiling flag -Wpedantic to -pedantic.
Flag -Wpedantic is not available in older versions of gcc,
what makes the life of some users unnecessarily harder.
Not to mention that both flags work fine for F3.
- Allow make variable CC to be set externally.
Some platforms favor other C compilers.
For example, FreeBSD defaults to clang.
- Allow make variable CFLAGS to be extended externally.
This helps packaging.
- Target clean to also remove *.d files.
The three first improvements were suggested by
Thomas Fischer and Uffe Jakobsen.
The discussion is available here:
https://github.com/AltraMayor/f3/issues/4
This patch improves the progress percentage and estimate of
time to finish when option --end-at requires f3write to write
less than the amount of free space available.
The previous version of f3write.h2w didn't recognize
parameters --start-at= and --end-at=.
The new code of f3write.h2w also avoids overwritting
the environment variable PATH.
This parameter allows users to execute F3 concurrently.
Moreover, users Mark and Wolfi have asked for similar parameters to
help analyzing very large flash drives.
This rewrite of the Makefile takes advantage of the fact that
the Makefile has been unified for all supported operating systems
to properly express source-file dependencies.
The source now identifies the operating system with the help of
macros defined by GCC.
The macros are documented here:
http://sourceforge.net/p/predef/wiki/OperatingSystems/
This patch was inspired by a suggestion from Max Justicz.
Now F3 and h2testw share the same file format.
Guenter Knauf has suggested this feature and
got the random-number generator used in h2testw from
Harald Bogeholz, the author of h2testw.
This change also addresses the issue that
the original random-number generator only fills
the first 32 bits of the 64 bits of the random numbers on
64-bit machines.
Besides the lower entropy on 64-bit machines, this issue was
making files generated on 32-bit and 64-bit machines incompatible.
Anselm Distelrath was the first to report this issue.
Two things could go wrong when .fff files are not fully read:
1. The constraint "assert(*read_all || errno == EIO);"
could fail because, according to the manual,
feof(3) does not set errno(3), but gettimeofday(2),
which is called after fread(3), does.
2. The tail message "NOT fully read" was never issued because
the test was "read_all", and variable read_all was a pointer.
Thus, the test was always true.
The not-fully-read error message now adds the output of
strerror(3) to guide the diagnose of the failure.
These bugs must be rare because they have been lurking around for
more than two years according to git's log.
They were only found due to a bug report by John Lussmyer on
September 21st, 2013.
This parameter became important because F3 now can be used on
very large storages and one may want to resume the test process
from a failure or interruption.
The filenames were from 0001.fff to 9999.fff.
This limit hasn't been a problem, but Anselm Distelrath reported
on February 8th, 2013 that isn't enough to test his system:
a RAID 6 with 18 hard drives of 3TB each.
Now filenames are from 1.fff to <INT_MAX>.fff.
