Block Storage Benchmark

This benchmark suite uses fio, scripts for automation and graph/PDF 
generators for reporting to replicate the SNIA Solid State Storage 
(SSS) Performance Test Specification (PTS) Enterprise v1.1. This specification
includes 8 test each measuring different block storage performance 
characteristics. The specification is available here:

http://snia.org/sites/default/files/SSS_PTS_Enterprise_v1.1.pdf

This benchmark suite uses gnuplot and wkhtmltopdf to generate test reports 
based on the SNIA specification. An example report is available here:

https://github.com/cloudharmony/block-storage/blob/master/reports/sample-report.pdf?raw=true

A few notes on cloud/virtualization limitations as they pertain to SNIA test 
specifications:

1. Due to virtualization, direct access to hardware commands like ATA secure 
   erase are generally unavailable in cloud environments. Thus, the recommended  
   Purge methods prescribed by the test specification are generally not 
   feasible in a cloud environment. As a workaround, where supported, full 
   device TRIM is applied prior to preconditioning and testing cycles (using 
   `blkdiscard` for devices and `fstrim` for mounted volumes). If ATA secure 
   erase and TRIM are not supported, devices are zero filled as a last resort. 
   The purge method used is documented in test output. If test targets are 
   rotational devices, zero fill formatting is used instead. Thus, the purge 
   order of precedence is: 1) ATA secure erase; 2) TRIM (SSD only) 3) zero 
   fill. These can be overridden using the nosecureerase, notrim and nozerofill
   test parameters.
2. The test specification requirement for write caches to be disabled cannot 
   be guaranteed in a cloud environment.
3. Often virtual machine block storage devices are partial logical volumes
   on a physical drive that may contain additional logical volumes in use by 
   other users. Because of this, testing may not be across the full LBA of  
   the physical drive (i.e. ActiveRange < 100%). Additionally, the possible  
   shared state of the physical drive may contribute to higher variability
   and difficulty or inability to achieve the prescribed +/-10% condition for 
   Steady State verification. This may also restrict the ability to 
   consistently reproduce results across multiple tests


TESTING PARAMETERS
The SNIA test specification allow for some user configurable parameters. These 
are described below. To start testing, use `run.php`. For runtime options, use 
`run.php --help`. Each of the parameters below may be specified using CLI 
arguments (e.g. `run.php --test=iops`) or `bm_param_` prefixed environment 
variables (e.g. `export bm_param_test=iops`)

* active_range              LBA range to use for testing represented as a 
                            percentage. Default is 100, or 100% per the SNIA
                            test specification. To test across a smaller 
                            range, this parameter may be reduced. If the
                            test targets are not devices, test size is total 
                            free space - 100 MB. WARNING: if multiple volume
                            type targets are specified, active range will be 
                            limited to the least amount of free space on all 
                            targets
                            
* collectd_rrd              If set, collectd rrd stats will be captured from 
                            --collectd_rrd_dir. To do so, when testing starts,
                            existing directories in --collectd_rrd_dir will 
                            be renamed to .bak, and upon test completion 
                            any directories not ending in .bak will be zipped
                            and saved along with other test artifacts (as 
                            collectd-rrd.zip). User MUST have sudo privileges
                            to use this option
                            
* collectd_rrd_dir          Location where collectd rrd files are stored - 
                            default is /var/lib/collectd/rrd
                            
* fio                       Optional explicit path to the fio command - 
                            otherwise fio in PATH will be used
                            
* fio_*                     Optional fio runtime parameters. By default, fio 
                            parameters are generated by the test script per 
                            the SNIA test specification. Use this parameter to 
                            override default fio settings 
                            (e.g. fio_ioengine=sync)
                            
* font_size                 The base font size pt to use in reports and graphs. 
                            All text will be relative to this size (i.e. 
                            smaller, larger). Default is 9. Graphs use this 
                            value + 4 (i.e. default 13). Open Sans is included
                            with this software. To change this, simply replace
                            the reports/font.ttf file with your desired font
                            
* highcharts_js_url         URL to highcharts.js. Highcharts is used to render
                            3D charts in reports. Use 'no3dcharts' to disable 
                            3D charts. Default for this parameter is
                            http://code.highcharts.com/highcharts.js
                            
* highcharts3d_js_url       URL to highcharts-3d.js. Highcharts is used to 
                            render 3D charts in reports. Use 'no3dcharts' to 
                            disable 3D charts. Default for this parameter is
                            http://code.highcharts.com/highcharts-3d.js
                            
* jquery_url                URL for jquery. jquery is used by Highcharts.
                            Highcharts is used to render 3D charts in reports. 
                            Use 'no3dcharts' to disable 3D charts. Default for 
                            this parameter is
                            http://code.jquery.com/jquery-2.1.0.min.js
                            
* meta_compute_service      The name of the compute service this test pertains
                            to. Used for report headers. May also be specified 
                            using the environment variable bm_compute_service
                            
* meta_compute_service_id   The id of the compute service this test pertains
                            to. Added to saved results. May also be specified 
                            using the environment variable 
                            bm_compute_service_id

* meta_cpu                  CPU descriptor to use for report headers. If not 
                            specified, it will be set using the 'model name' 
                            attribute in /proc/cpuinfo
                            
* meta_drive_interface      Optional drive interface descriptor to use for 
                            report headers (e.g. SATA 6Gb/s)

* meta_drive_model          Optional drive model descriptor to use for report
                            headers (e.g. Intel DC S3700)

* meta_drive_type           Optional drive type descriptor to use for report 
                            headers (e.g. SATA, SSD, PCIe)
                            
* meta_instance_id          The compute service instance type this test pertains 
                            to (e.g. c3.xlarge). Used for report headers
                            
* meta_memory               Memory descriptor to use for report headers. If not 
                            specified, the amount of memory will be used (as 
                            reported by 'free')
                            
* meta_notes_storage        Optional notes to display in the Storage Platform 
                            header column
                            
* meta_notes_test           Optional notes to display in the Test Platform 
                            header column
                            
* meta_os                   Operating system descriptor to use for report 
                            headers. If not specified, this meta data will be 
                            derived from the first line of /etc/issue
                            
* meta_provider             The name of the cloud provider this test pertains
                            to. Used for report headers. May also be specified 
                            using the environment variable bm_provider
                            
* meta_provider_id          The id of the cloud provider this test pertains
                            to. Added to saved results. May also be specified 
                            using the environment variable 
                            bm_provider_id

* meta_region               The compute service region this test pertains to. 
                            Used for report headers. May also be specified 
                            using the environment variable bm_region
                            
* meta_resource_id          An optional benchmark resource identifiers. Added 
                            to saved results. May also be specified using the 
                            environment variable bm_resource_id
                            
* meta_run_id               An optional benchmark run identifiers. Added to 
                            saved results. May also be specified using the 
                            environment variable bm_run_id
                            
* meta_storage_config       Storage configuration descriptor to use for report 
                            headers. If not specified, 'Unknown' will be 
                            displayed in this column
                            
* meta_storage_vol_info     If testing is against a volume, this optioanl 
                            parameter may be used to designate setup of that 
                            volume (e.g. Raid, File System, etc.). Only 
                            displayed when targets are volumes. If this
                            parameter is not specified the file system type 
                            for each target volume will be included in this 
                            column
                            
* meta_test_id              Identifier for the test. Used for report headers. 
                            If not specified, this header column will be blank
                            
* meta_test_sw              Name/version of the test software. Used for report 
                            headers. If not specified, this header column will 
                            be blank
                            
* no3dcharts                Don't generate 3D charts. Unlike 2D charts rendered 
                            with the free chart utility gnuplot, 3D charts are 
                            rendered using highcharts - a commercial javascript
                            charting tool. highcharts is available for free for 
                            non-commercial and development use, and for a 
                            nominal fee otherwise. See http://www.highcharts.com
                            for additional licensing information
                            
* nojson                    Don't generate JSON result or fio output files
                            
* nopdfreport               Don't generate PDF version of test report - 
                            report.pdf. (wkhtmltopdf dependency removed if 
                            specified)
                            
* noprecondition            Don't perform the default 2X 128K sequential 
                            workload independent preconditioning (per the SNIA 
                            test specification). This step precedes workload 
                            dependent preconditioning
                            
* nopurge                   Don't require a purge for testing. If this 
                            parameter is not set, and at least 1 target could 
                            not be purged, testing will abort. This parameter
                            is implicit if --nosecureerase, --notrim and 
                            --nozerofill are all specified
                            
* norandom                  Don't test using random (less compressible) data.
                            Use of random data for IO is a requirement of the 
                            SNIA test specification
                            
* randommap                 Random maps are allocated at init time to track 
                            written to blocks and duplicate block writes. When 
                            used, random maps must be allocated in memory at 
                            init time. The memory allocation for these can be 
                            problematic for large test volume (e.g. 16TB volume
                             = 4.2GB memory). If this option is not set, random
                            fio tests will be executed using the --norandommap
                            and --randrepeat=0 fio options
                            
* noreport                  Don't generate html or PDF test reports - 
                            report.zip and report.pdf (gnuplot, wkhtmltopdf and
                            zip dependencies removed if specified)
                            
* nosecureerase             Don't attempt to secure erase device targets prior 
                            to test cycles (this is the first choice - hdparm 
                            dependency removed if specified). This parameter is
                            implicit if --secureerase_pswd is not provided
                            
* notrim                    Don't attempt to TRIM devices/volumes prior to 
                            testing cycles (util-linux dependency removed if 
                            specified)
                            
* nozerofill                Don't zero fill rotational devices (or SSD devices 
                            when TRIM is not supported) prior to testing 
                            cycles. Zero fill applies only to device targets
                            

* oio_per_thread            The outstanding IO per thread (a.k.a. queue depth). 
                            This translates to the fio `iodepth` parameter. 
                            Total outstanding IO for a given test is 
                            `threads` * `threads_oio`. Per the SNIA test 
                            specification, this is a user definable parameter. 
                            For latency tests, this parameter is a fixed value
                            of 1. Default value for this parameter is 64
                            
* output                    The output directory to use for writing test artifacts 
                            (JSON and reports). If not specified, the current 
                            working directory will be used
                            
* precondition_passes       Number of passes for workload independent 
                            preconditioning. Per the SNIA test specification 
                            the default is 2X. Use this or the --noprecondition
                            argument to change this default behavior

* savefio                   Include results from every fio test job in save 
                            output
                            
* secureerase_pswd          In order for ATA secure erase to be attempted for 
                            device purge (prior to test invocation), you must
                            first set a security password using the command:
                            sudo hdparm --user-master u --security-set-pass [pswd] /dev/[device]
                            The password used should be supplied using this 
                            test parameter. If it is not supplied, ATA secure
                            erase will not be attempted. If this parameter is 
                            not specified, the hdparm dependent will be removed

* sequential_only           If set, all random tests will be executed using 
                            sequential IO instead
                            
* skip_blocksize            block sizes to skip during testing. This argument 
                            may be repeated. Valid options are:
                            1m, 128k, 64k, 32k, 16k, 8k, 512b
                            
* skip_workload             workloads to skip during testing. This argument may
                            be repeated. Valid options are:
                            100/0, 95/5, 65/35, 50/50, 35/65, 5/95
                            
* ss_max_rounds             The maximum number of test cycle iterations to 
                            allow for steady state verification. Default is 
                            x=25 (per the SNIA test specification). If steady 
                            state cannot be reached within this number of test 
                            cycles (per the ss_verification ratio), testing 
                            will terminate, and results will be designated as 
                            having not achieved steady state. This parameter 
                            may be used to increase (or decrease) the number of 
                            test cycles. A minimum value of 5 is permitted 
                            (the minimum number of cycles for the measurement 
                            window)
                            
* ss_verification           Ratio to utilize for steady state verification. The 
                            default is 10 or 10% per the SNIA test 
                            specification. In order to achieve steady state 
                            verification, the variance between the current test 
                            cycle loop and the 4 that precede it cannot exceed 
                            this value. In cloud environments with high IO 
                            variability, it may be difficult to achieve the 
                            default ratio and thus this value may be increased 
                            using this parameter
                            
* target                    REQUIRED: The target device or volume to use for 
                            testing. This parameter may reference either the 
                            physical device (e.g. /dev/sdc) or a mounted volume
                            (e.g. /ssd). TRIM will be attempted using 
                            `blkdiscard` for a device and `fstrim` for a volume
                            if the targets are non-rotational. For rotational 
                            devices, a zero fill will be used (i.e. 
                            `dd if=/dev/zero`). Multiple targets may be 
                            specified each separated by a comma. When multiple 
                            targets are specified, the `threads` parameter 
                            represents the number of threads per target (i.e. 
                            total threads = # of targets * `threads`). Multiple 
                            target tests provide aggregate metrics. With the 
                            exception of latency tests, if multiple targets are 
                            specified, they will be tested concurrently. 
                            Sufficient permissions for the device/volume must 
                            exist for the user that initiates testing. 
                            WARNING: If a device is specified (e.g. /dev/sdc),
                            all data on that device will be erased during the 
                            course of testing. All targets must be of the same 
                            type (device or volume). WARNING: if multiple 
                            volume type targets are specified, active range 
                            will be limited to the least amount of free space 
                            on all targets
                            
* target_skip_not_present   If set, targets specified that do not exist will be
                            ignored (so long as at least 1 target exists)
                            
* test                      The SNIA SSS PTS tests to perform. One or more of 
                            the following:
                              iops:       IOPS Test - measures IOPS at a range 
                                          of random block sizes and read/write 
                                          mixes
                              throughput: Throughput Test - measures 128k and 
                                          1m sequential read and write 
                                          throughput (MB/s) in steady state
                              latency:    Latency Test - measures IO response 
                                          times for 3 block sizes (0.5k, 4k and 
                                          8k), and 3 read/write mixes (100/0, 
                                          65/35 and 0/100). If multiple 
                                          `target` devices or volumes are 
                                          specified, latency tests are 
                                          performed sequentially
                              wsat:       Write Saturation Test - measures how 
                                          drives respond to continuous 4k 
                                          random writes over time and total GB 
                                          written (TGBW). NOTE: this 
                                          implementation uses the alternate 
                                          steady state test method (1 minute SS
                                          checks interspersed by 30 minute WSAT
                                          test intervals)
                              hir:        Host Idle Recovery - observes whether 
                                          the devices utilizes background 
                                          garbage collection wherein 
                                          performance increases with the 
                                          introduction of host idle times 
                                          between periods of 4k random writes
                              xsr:        Cross Stimulus Recovery - tests how 
                                          the device handles transitions from 
                                          large block sequential writes to 
                                          small block random writes and back
                                          NOT YET IMPLEMENTED
                              ecw:        Enterprise Composite Workload - 
                                          measures performance in a mixed IO 
                                          environment
                                          NOT YET IMPLEMENTED
                              dirth:      Demand Intensity / Response Time 
                                          Histogram - measures performance 
                                          degradation when a device is subject 
                                          to a super saturating IO load
                                          NOT YET IMPLEMENTED
                            Multiple tests may be specified each separated by 
                            a comma. Default for this parameter is iops. Some 
                            tests like 'hir' are specific to SSD type devices
                            
* threads                   The number of threads to use for the test cycle. 
                            Per the SNIA test specification, this is a user 
                            definable parameter. The default value for this 
                            parameter is the number of CPU cores. This 
                            parameter may contain the token {cpus} which will 
                            be replaced with the number of CPU cores present. 
                            It may also contain a mathematical expression in 
                            conjunction with {cpus} - e.g. {cpus}/2. If 
                            `target` references multiple devices or volumes, 
                            this parameter signifies the number of threads per 
                            device. Thus total threads is # of targets * 
                            `threads`. Latency tests are fixed at 1 thread. 
                            This parameter is used to define the fio --numjobs
                            argument
                            
* threads_per_core_max      Max number of threads per CPU core - default is 2.
                            If this parameter causes the number of threads per
                            target to be less than 1, threads per target will 
                            be increased to 1
                            
* threads_per_target_max    Max number of threads per target - default is 8
                            
* timeout                   Max time to permit for testing in seconds. Default 
                            is 24 hours (86400 seconds)
                            
* trim_offset_end           When invoking a blkdiscard TRIM, offset the lenth
                            by this number of bytes
                            
* verbose                   Show verbose output - warning: this may produce a 
                            lot of output
                            
* wd_test_duration          The test duration for workload dependent test 
                            iterations in seconds. Default is 60 per the SNIA 
                            test specification
                            
* wd_sleep_between          Optional sleep duration (seconds) to apply between 
                            each workload dependent test. This may be a formula
                            containing the following dynamic values:
                              {duration} => duration of the last test (secs)
                              {size}     => size of targets (GB)
                              {volumes}  => number of target volumes
                            Ternary operations are also supported. This 
                            parameter may be useful for credit based storage 
                            platforms like Amazon EC2 EBS General Purpose 
                            storage 
                            (see: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)
                            For example, per the Amazon EC2 EBS documentation
                            to ensure testing occurred within EBS burst 
                            capacity, the following formula could be set for
                            this parameter: 
                            --wd_sleep_between "{duration}*({size} >= 1000 ? 0 : ({size} >= 750 ? 0.33 : ({size} >= 500 ? 1 : ({size} >= 250 ? 3 : ({size} >= 100 ? 9 : 100)))))"
                            A shortcut to this formula is --wd_sleep_between ebs
                            
DEPENDENCIES
This benchmark suite uses the following packages:

  fio         Performs actual testing - version 2.1+ required
  gnuplot     Generates graphs per the SNIA test specification. These graphs
              are used in the PDF report
  hdparm      Used for ATA secure erase (when supported)
  php-cli     Test automation scripts (/usr/bin/php)
  util-linux  For TRIM operations using `blkdiscard` and `fstrim` (when 
              supported). Not required if test targets are rotational
  wkhtmltopdf Generates PDF version of report - download from 
              http://wkhtmltopdf.org
  zip         Archives HTML test report into a single zip file


TEST ARTIFACTS
Upon successful completion of testing, the following artifacts will be 
produced in the working directory ([test] replaced with one of the test 
identifiers - e.g. report-iops.json):

  [test].json        JSON formatted job results for [test]. Each test provides 
                     different result metrics. This file contains a single 
                     level hash of key/value pairs representing these metrics
  fio-[test].json    JSON formatted fio job results for [test]. Jobs are 
                     in run order each with a unique job name. For workload 
                     independent preconditioning, the job name uses the 
                     format 'wipc-N', where N is the preconditioning pass
                     (i.e. N=1 is first pass, N=2 second). Unless 
                     --precondition_passes is otherwise specified, only 2 wipc 
                     JSON files should be present (each representing one of the 
                     2X preconditioning tests). For workload dependent 
                     preconditioning and other testing, the job name is set by
                     the test. For example, for the IOPS test, the job name 
                     format is 'xN-[rw]-[bs]-rand', where N is iteration 
                     number (1-25+), [rw] is the read/write ratio (separated by 
                     underscore) and [bs] is block size. Jobs that fall within
                     the steady state measurement window will have the suffix 
                     '-ssmw' (e.g. x5-0_100-4k-rand-ssmw). there may be up to 
                     10 fio-[test].json corresponding with each of the tests 
                     and 2 files for throughput: test-throughput-128k.json and
                     test-throughput-1024k.json
  report.zip         HTML test report (open index.html). The report design
                     and layout is based on the SNIA test specification. In 
                     addition, this archive contains source gnuplot scripts and 
                     data files for report graphs. Graphs are rendered in svg 
                     format
  report.pdf         PDF version of the test report (wkhtmltopdf used to 
                     generate this version of the report)
                     
  collectd-rrd.zip   collectd RRD files (see --collectd_rrd)
                     
                     
SAVE SCHEMA
The following columns are included in CSV files/tables generated by save.sh. 
Indexed MySQL/PostgreSQL columns are identified by *. Columns without 
descriptions are documented as runtime parameters above. Data types and 
indexing used is documented in save/schema/*.json. Columns can be removed using 
the save.sh --remove parameter

COMMON COLUMNS (included in all CSV/tables):
active_range
benchmark_version: [benchmark version]
collectd_rrd: [URL to zip file containing collectd rrd files]
fio_version: [fio version]
iteration: [iteration number (used with incremental result directories)]
meta_compute_service
meta_compute_service_id*
meta_cpu: [CPU model info]
meta_cpu_cache: [CPU cache]
meta_cpu_cores: [# of CPU cores]
meta_cpu_speed: [CPU clock speed (MHz)]
meta_drive_interface
meta_drive_model
meta_drive_type
meta_instance_id*
meta_hostname: [system under test (SUT) hostname]
meta_memory
meta_memory_gb: [memory in gigabytes]
meta_memory_mb: [memory in megabyets]
meta_os_info: [operating system name and version]
meta_provider
meta_provider_id*
meta_region*
meta_resource_id
meta_run_id
meta_storage_config*
meta_storage_vol_info
meta_test_id*
meta_test_sw
noprecondition
nopurge
norandom
nosecureerase
notrim
nozerofill
oio_per_thread: [queue depth per thread]
precondition_passes
purge_methods: [purge methods used for each device]
results_pdf: [URL to results PDF file (if --store option used)]
results_zip: [URL to results ZIP file (if --store option used)]
skip_blocksize
skip_workload
# Note: ss_* columns are not part of fio or wsat tables
ss_average: [mean value of y axis in steady state window]
ss_largest_data_excursion: [largest data excusion in steady state window]
ss_largest_slope_excursion: [largest slope excusion in steady state window]
ss_linear_fit_formula: [steady state linear fit formula]
ss_max_data_excursion: [max allowed data excusion in steady state window]
ss_max_rounds
ss_max_slope_excursion: [max allowed slope excusion in steady state window]
ss_rounds: [steady state window rounds (e.g. 1 to 5)]
ss_slope: [slope in steady state window]
ss_start: [steady state window start round]
ss_stop: [steady state window end round]
ss_verification
ss_y_intercept: [y intercept in steady state window]
target
target_count: [number of targets]
target_size_gb: [average target size in gigabytes]
target_sizes: [sizes of each target]
test*: [test identifier (e.g. iops)]
test_started*: [when the test started]
test_stopped: [when the test ended]
threads: [number of threads per target]
threads_total: [total number of threads]
timeout
wd_sleep_between
wd_test_duration

FIO CSV/TABLES
fio_command: [complete fio command]
iodepth_level_1: [distribution of IO with IO depth <= 1]
iodepth_level_2: [distribution of IO with IO depth 2]
iodepth_level_4: [distribution of IO with IO depth 4]
iodepth_level_8: [distribution of IO with IO depth 8]
iodepth_level_16: [distribution of IO with IO depth 16]
iodepth_level_32: [distribution of IO with IO depth 32]
iodepth_level_gte64: [distribution of IO with IO depth >= 64]
jobname: [fio job name]
latency_us_2: [distribution of IO with latency <= 2 microseconds]
latency_us_4: [distribution of IO with latency >2 and <=4 microseconds]
latency_us_10: [distribution of IO with latency >4 and <=10 microseconds]
latency_us_20: [distribution of IO with latency >10 and <=20 microseconds]
latency_us_50: [distribution of IO with latency >20 and <=50 microseconds]
latency_us_100: [distribution of IO with latency >50 and <=100 microseconds]
latency_us_250: [distribution of IO with latency >100 and <=250 microseconds]
latency_us_500: [distribution of IO with latency >250 and <=500 microseconds]
latency_us_750: [distribution of IO with latency >500 and <=750 microseconds]
latency_us_1000: [distribution of IO with latency >750 and <=1000 microseconds]
latency_ms_2: [distribution of IO with latency >1000 microseconds and <=2 milliseconds]
latency_ms_4: [distribution of IO with latency >2 and <=4 milliseconds]
latency_ms_10: [distribution of IO with latency >4 and <=10 milliseconds]
latency_ms_20: [distribution of IO with latency >10 and <=20 milliseconds]
latency_ms_50: [distribution of IO with latency >20 and <=50 milliseconds]
latency_ms_100: [distribution of IO with latency >50 and <=100 milliseconds]
latency_ms_250: [distribution of IO with latency >100 and <=250 milliseconds]
latency_ms_500: [distribution of IO with latency >250 and <=500 milliseconds]
latency_ms_750: [distribution of IO with latency >500 and <=750 milliseconds]
latency_ms_1000: [distribution of IO with latency >750 and <=1000 milliseconds]
latency_ms_2000: [distribution of IO with latency >1000 and <2000 milliseconds]
latency_ms_gte2000: [distribution of IO with latency >=2000 milliseconds]
majf: [major page faults]
minf: [minor page faults]
read_io_bytes: [read IO - KB]
read_bw: [read bandwidth - KB/s]
read_iops: [read IOPS]
read_runtime: [read runtime - ms]
read_slat_min: [min read submission latency - μs]
read_slat_max: [max read submission latency - μs]
read_slat_mean: [mean read submission latency - μs]
read_slat_stddev: [read submission latency standard deviation - μs]
read_clat_min: [min read completion latency - μs]
read_clat_max: [max read completion latency - μs]
read_clat_mean: [mean read completion latency - μs]
read_clat_stddev: [read completion standard deviation - μs]
read_clat_percentile_1: [1st percentile read completion latency - μs]
read_clat_percentile_5: [5th percentile read completion latency - μs]
read_clat_percentile_10: [10th percentile read completion latency - μs]
read_clat_percentile_20: [20th percentile read completion latency - μs]
read_clat_percentile_30: [30th percentile read completion latency - μs]
read_clat_percentile_40: [40th percentile read completion latency - μs]
read_clat_percentile_50: [50th percentile read completion latency - μs]
read_clat_percentile_60: [60th percentile read completion latency - μs]
read_clat_percentile_70: [70th percentile read completion latency - μs]
read_clat_percentile_80: [80th percentile read completion latency - μs]
read_clat_percentile_90: [90th percentile read completion latency - μs]
read_clat_percentile_95: [95th percentile read completion latency - μs]
read_clat_percentile_99: [99th percentile read completion latency - μs]
read_clat_percentile_99_5: [99.5th percentile read completion latency - μs]
read_clat_percentile_99_9: [99.9th percentile read completion latency - μs]
read_clat_percentile_99_95: [99.95th percentile read completion latency - μs]
read_clat_percentile_99_99: [99.99th percentile read completion latency - μs]
read_lat_min: [min total read latency - μs]
read_lat_max: [max total read latency - μs]
read_lat_mean: [mean total read latency - μs]
read_lat_stddev: [total read latency standard deviation - μs]
read_bw_min: [min read bandwidth - KB/s]
read_bw_max: [max read bandwidth - KB/s]
read_bw_agg: [aggregate read bandwidth - KB/s]
read_bw_mean: [mean read bandwidth - KB/s]
read_bw_dev: [read bandwidth standard deviation - KB/s]
started: [when the fio job started]
stopped: [when the fio job ended]
usr_cpu: [user CPU usage]
sys_cpu: [system CPU usage]
write_io_bytes: [write IO - KB]
write_bw: [write bandwidth - KB/s]
write_iops: [write IOPS]
write_runtime: [write runtime - ms]
write_slat_min: [min write submission latency - μs]
write_slat_max: [max write submission latency - μs]
write_slat_mean: [mean write submission latency - μs]
write_slat_stddev: [write submission latency standard deviation - μs]
write_clat_min: [min write completion latency - μs]
write_clat_max: [max write completion latency - μs]
write_clat_mean: [mean write completion latency - μs]
write_clat_stddev: [write completion standard deviation - μs]
write_clat_percentile_1: [1st percentile write completion latency - μs]
write_clat_percentile_5: [5th percentile write completion latency - μs]
write_clat_percentile_10: [10th percentile write completion latency - μs]
write_clat_percentile_20: [20th percentile write completion latency - μs]
write_clat_percentile_30: [30th percentile write completion latency - μs]
write_clat_percentile_40: [40th percentile write completion latency - μs]
write_clat_percentile_50: [50th percentile write completion latency - μs]
write_clat_percentile_60: [60th percentile write completion latency - μs]
write_clat_percentile_70: [70th percentile write completion latency - μs]
write_clat_percentile_80: [80th percentile write completion latency - μs]
write_clat_percentile_90: [90th percentile write completion latency - μs]
write_clat_percentile_95: [95th percentile write completion latency - μs]
write_clat_percentile_99: [99th percentile write completion latency - μs]
write_clat_percentile_99_5: [99.5th percentile write completion latency - μs]
write_clat_percentile_99_9: [99.9th percentile write completion latency - μs]
write_clat_percentile_99_95: [99.95th percentile write completion latency - μs]
write_clat_percentile_99_99: [99.99th percentile write completion latency - μs]
write_lat_min: [min total write latency - μs]
write_lat_max: [max total write latency - μs]
write_lat_mean: [mean total write latency - μs]
write_lat_stddev: [total write latency standard deviation - μs]
write_bw_min: [min write bandwidth - KB/s]
write_bw_max: [max write bandwidth - KB/s]
write_bw_agg: [aggregate write bandwidth - KB/s]
write_bw_mean: [mean write bandwidth - KB/s]
write_bw_dev: [write bandwidth standard deviation - KB/s]

HIR CSV/TABLES:
hir_steady_state_iops: [mean 4k rand write IOPS during steady state phase of preconditioning]
hir_wait_5s_iops: [mean 4k rand write IOPS during testing with 5 second wait intervals]
hir_wait_10s_iops: [mean 4k rand write IOPS during testing with 10 second wait intervals]
hir_wait_15s_iops: [mean 4k rand write IOPS during testing with 15 second wait intervals]
hir_wait_25s_iops: [mean 4k rand write IOPS during testing with 25 second wait intervals]
hir_wait_50s_iops: [mean 4k rand write IOPS during testing with 50 second wait intervals]

IOPS CSV/TABLES:
iops_1m_100_0: [mean 1m read IOPS in steady state]
iops_128k_100_0: [mean 128k read IOPS in steady state]
iops_64k_100_0: [mean 64k read IOPS in steady state]
iops_32k_100_0: [mean 32k read IOPS in steady state]
iops_16k_100_0: [mean 16k read IOPS in steady state]
iops_8k_100_0: [mean 8k read IOPS in steady state]
iops_4k_100_0: [mean 4k read IOPS in steady state]
iops_512b_100_0: [mean 512b read IOPS in steady state]
iops_1m_95_5: [mean 1m 95/5 rw IOPS in steady state]
iops_128k_95_5: [mean 128k 95/5 rw IOPS in steady state]
iops_64k_95_5: [mean 64k 95/5 rw IOPS in steady state]
iops_32k_95_5: [mean 32k 95/5 rw IOPS in steady state]
iops_16k_95_5: [mean 16k 95/5 rw IOPS in steady state]
iops_8k_95_5: [mean 8k 95/5 rw IOPS in steady state]
iops_4k_95_5: [mean 4k 95/5 rw IOPS in steady state]
iops_512b_95_5: [mean 512b 95/5 rw IOPS in steady state]
iops_1m_65_35: [mean 1m 65/35 rw IOPS in steady state]
iops_128k_65_35: [mean 128k 65/35 rw IOPS in steady state]
iops_64k_65_35: [mean 64k 65/35 rw IOPS in steady state]
iops_32k_65_35: [mean 32k 65/35 rw IOPS in steady state]
iops_16k_65_35: [mean 16k 65/35 rw IOPS in steady state]
iops_8k_65_35: [mean 8k 65/35 rw IOPS in steady state]
iops_4k_65_35: [mean 4k 65/35 rw IOPS in steady state]
iops_512b_65_35: [mean 512b 65/35 rw IOPS in steady state]
iops_1m_50_50: [mean 1m 50/50 rw IOPS in steady state]
iops_128k_50_50: [mean 128k 50/50 rw IOPS in steady state]
iops_64k_50_50: [mean 64k 50/50 rw IOPS in steady state]
iops_32k_50_50: [mean 32k 50/50 rw IOPS in steady state]
iops_16k_50_50: [mean 16k 50/50 rw IOPS in steady state]
iops_8k_50_50: [mean 8k 50/50 rw IOPS in steady state]
iops_4k_50_50: [mean 4k 50/50 rw IOPS in steady state]
iops_512b_50_50: [mean 512b 50/50 rw IOPS in steady state]
iops_1m_35_65: [mean 1m 35/65 rw IOPS in steady state]
iops_128k_35_65: [mean 128k 35/65 rw IOPS in steady state]
iops_64k_35_65: [mean 64k 35/65 rw IOPS in steady state]
iops_32k_35_65: [mean 32k 35/65 rw IOPS in steady state]
iops_16k_35_65: [mean 16k 35/65 rw IOPS in steady state]
iops_8k_35_65: [mean 8k 35/65 rw IOPS in steady state]
iops_4k_35_65: [mean 4k 35/65 rw IOPS in steady state]
iops_512b_35_65: [mean 512b 35/65 rw IOPS in steady state]
iops_1m_5_95: [mean 1m 5/95 rw IOPS in steady state]
iops_128k_5_95: [mean 128k 5/95 rw IOPS in steady state]
iops_64k_5_95: [mean 64k 5/95 rw IOPS in steady state]
iops_32k_5_95: [mean 32k 5/95 rw IOPS in steady state]
iops_16k_5_95: [mean 16k 5/95 rw IOPS in steady state]
iops_8k_5_95: [mean 8k 5/95 rw IOPS in steady state]
iops_4k_5_95: [mean 4k 5/95 rw IOPS in steady state]
iops_512b_5_95: [mean 512b 5/95 rw IOPS in steady state]
iops_1m_0_100: [mean 1m write IOPS in steady state]
iops_128k_0_100: [mean 128k write IOPS in steady state]
iops_64k_0_100: [mean 64k write IOPS in steady state]
iops_32k_0_100: [mean 32k write IOPS in steady state]
iops_16k_0_100: [mean 16k write IOPS in steady state]
iops_8k_0_100: [mean 8k write IOPS in steady state]
iops_4k_0_100: [mean 4k write IOPS in steady state]
iops_512b_0_100: [mean 512b write IOPS in steady state]

LATENCY CSV/TABLES:
latency_8k_100_0_mean: [mean 8k read latency]
latency_8k_100_0_max: [max 8k read latency]
latency_4k_100_0_mean: [mean 4k read latency]
latency_4k_100_0_max: [max 8k read latency]
latency_512b_100_0_mean: [mean 512b read latency]
latency_512b_100_0_max: [max 512b read latency]
latency_8k_65_35_mean: [mean 8k 65/35 rw latency]
latency_8k_65_35_max: [max 8k 65/35 rw latency]
latency_4k_65_35_mean: [mean 4k 65/35 rw latency]
latency_4k_65_35_max: [max 4k 65/35 rw latency]
latency_512b_65_35_mean: [mean 512b 65/35 rw latency]
latency_512b_65_35_max: [max 512b 65/35 rw latency]
latency_8k_0_100_mean: [mean 8k write latency]
latency_8k_0_100_max: [max 8k write latency]
latency_4k_0_100_mean: [mean 4k write latency]
latency_4k_0_100_max: [max 4k write latency]
latency_512b_0_100_mean: [mean 512b write latency]
latency_512b_0_100_max: [max 512b write latency]

THROUGHPUT CSV/TABLES:
throughput_1024k_100_0: [mean 1024k read throughput - MB/s]
throughput_1024k_0_100: [mean 1024k write throughput - MB/s]
throughput_128k_100_0: [mean 128k write throughput - MB/s]
throughput_128k_0_100: [mean 128k write throughput - MB/s]

WSAT CSV/TABLES
wsat_iops: [mean 4k write IOPS in steady state]


USAGE
# perform IOPS test against device /dev/sdc
./run.sh --target=/dev/sdc --test=iops

# perform IOPS, Latency and Throughput tests against /dev/sdc and /dev/sdd 
# concurrently using a maximum of [num CPU cores]*2 and 32 OIO per thread
./run.sh --target=/dev/sdc --target=/dev/sdd --test=iops --test=latency --test=throughput --threads="{cpu}*2" --oio_per_thread=32

# perform IOPS test against device /dev/sdc but skip the purge step
./run.sh --target=/dev/sdc --test=iops --nopurge

# perform IOPS test against device /dev/sdc but skip the purge and workload 
# independent preconditioning
./run.sh --target=/dev/sdc --test=iops --nopurge --noprecondition

# perform 5 iterations of the same IOPS test above
for i in {1..5}; do mkdir -p ~/block-storage-testing/$i; ./run.sh --target=/dev/sdc --test=iops --nopurge --noprecondition --output ~/block-storage-testing/$i; done


# save.sh saves results to CSV, MySQL, PostgreSQL, BigQuery or via HTTP 
# callback. It can also save artifacts (PDF and ZIP reports) to S3, Azure
# Blob Storage or Google Cloud Storage

# save results to CSV files
./save.sh

# save results from 5 iterations text example above
./save.sh ~/block-storage-testing

# save results to a PostgreSQL database
./save --db postgresql --db_user dbuser --db_pswd dbpass --db_host db.mydomain.com --db_name benchmarks

# save results to BigQuery and artifacts (PDF and ZIP reports) to S3
./save --db bigquery --db_name benchmark_dataset --store s3 --store_key THISIH5TPISAEZIJFAKE --store_secret thisNoat1VCITCGggisOaJl3pxKmGu2HMKxxfake --store_container benchmarks1234