NVMe/Health
Checking drive health
SATA-based
Use the smartctl command to get a wealth of information about the device’s performance and health.
# smartctl -A /dev/sda smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-62-generic] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0 5 Reallocated_Sector_Ct 0x0032 100 100 010 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 41 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2 171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 173 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 1 174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 194 Temperature_Celsius 0x0022 065 059 000 Old_age Always - 35 (Min/Max 21/41) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0 202 Unknown_SSD_Attribute 0x0030 100 100 001 Old_age Offline - 0 206 Unknown_SSD_Attribute 0x000e 100 100 000 Old_age Always - 0 246 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 145599393 247 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 4550280 248 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 582524 180 Unused_Rsvd_Blk_Cnt_Tot 0x0033 000 000 000 Pre-fail Always - 1260 210 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
While smartctl might not know all vendor-specific smart values, typically you can Google the drive model along with “smart attributes” and find documents like this to get more details.
NVMe
- If you move to newer generation NVMe-based flash storage, smartctl won’t work anymore – at least it doesn’t work for the packages available for Ubuntu 16.04 (what I’m running).
- It looks like support for NVMe in Smartmontools is coming, and it would be great to get a single tool that supports both SATA and NVMe flash storage.
- In the meantime, you can use the nvme tool available from the nvme-cli package.
- It provides some basic information for NVMe devices.
Installed devices
# nvme list Node SN Model Namespace Usage Format FW Rev --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 S4J4NG0M900897N Samsung SSD 970 EVO Plus 2TB 1 940,20 GB / 2,00 TB 512 B + 0 B 2B2QEXM7 /dev/nvme1n1 S4J4NG0M802465V Samsung SSD 970 EVO Plus 2TB 1 928,61 GB / 2,00 TB 512 B + 0 B 2B2QEXM7
SMART information
# nvme smart-log /dev/nvme0 Smart Log for NVME device:nvme0 namespace-id:ffffffff critical_warning : 0 temperature : 34 C available_spare : 100% available_spare_threshold : 10% percentage_used : 0% data_units_read : 3,465,389 data_units_written : 9,014,689 host_read_commands : 89,719,366 host_write_commands : 134,671,295 controller_busy_time : 310 power_cycles : 11 power_on_hours : 21 unsafe_shutdowns : 8 media_errors : 0 num_err_log_entries : 1 Warning Temperature Time : 0 Critical Composite Temperature Time : 0 Temperature Sensor 1 : 34 C Temperature Sensor 2 : 47 C Temperature Sensor 3 : 0 C Temperature Sensor 4 : 0 C Temperature Sensor 5 : 0 C Temperature Sensor 6 : 0 C
Additional SMART Information
(not all devices support it)
# nvme smart-log-add /dev/nvme0 Additional Smart Log for NVME device:nvme0 namespace-id:ffffffff key normalized raw program_fail_count : 100% 0 erase_fail_count : 100% 0 wear_leveling : 62% min: 1114, max: 1161, avg: 1134 end_to_end_error_detection_count: 100% 0 crc_error_count : 100% 0 timed_workload_media_wear : 100% 37.941% timed_workload_host_reads : 100% 51% timed_workload_timer : 100% 446008 min thermal_throttle_status : 100% 0%, cnt: 0 retry_buffer_overflow_count : 100% 0 pll_lock_loss_count : 100% 0 nand_bytes_written : 100% sectors: 16185227 host_bytes_written : 100% sectors: 6405605
TMP
Some of this information is self-explanatory, and some of it isn’t. After looking at the NVMe specification document, here is my read on some of the data:
- Available Spare. Contains a normalized percentage (0 to 100%) of the remaining spare capacity that is available.
- Available Spare Threshold. When the Available Spare capacity falls below the threshold indicated in this field, an asynchronous event completion can occur. The value is indicated as a normalized percentage (0 to 100%).
- (Note: I’m not quite sure what the practical meaning of “asynchronous event completion” is, but it looks like something to avoid!)
- Percentage Used. Contains a vendor specific estimate of the percentage of the NVM subsystem life used, based on actual usage and the manufacturer’s prediction of NVM life.
- (Note: the number can be more than 100% if you’re using storage for longer than its planned life.)
- Data Units Read/Data Units Written. This is the number of 512-byte data units that are read/written, but it is measured in an unusual way. The first value corresponds to 1000 of the 512-byte units. So you can multiply this value by 512000 to get value in bytes. It does not include meta-data accesses.
- Host Read/Write Commands. The number of commands of the appropriate type issued. Using this value, as well as one below, you can compute the average IO size for “physical” reads and writes.
- Controller Busy Time. Time in minutes that the controller was busy servicing commands. This can be used to gauge long-term storage load trends.
- Unsafe Shutdowns. The number of times a power loss happened without a shutdown notification being sent. Depending on the NVMe device you’re using, an unsafe shutdown might corrupt user data.
- Warning Temperature Time/Critical Temperature Time. The time in minutes a device operated above a warning or critical temperature. It should be zeroes.
- Wear_Leveling. This shows how much of the rated cell life was used, as well as the min/max/avg write count for different cells. In this case, it looks like the cells are rated for 1800 writes and about 1100 on average were used
- Timed Workload Media Wear. The media wear by the current “workload.” This device allows you to measure some statistics from the time you reset them (called the “workload”) in addition to showing the device lifetime values.
- Timed Workload Host Reads. The percentage of IO operations that were reads (since the workload timer was reset).
- Thermal Throttle Status. This shows if the device is throttled due to overheating, and when there were throttling events in the past.
- Nand Bytes Written. The bytes written to NAND cells. For this device, the measured unit seems to be in 32MB values. It might be different for other devices.
- Host Bytes Written. The bytes written to the NVMe storage from the system. This unit also is in 32MB values. The scale of these values is not very important, as they are the most helpful for finding the write amplification of your workload. This ratio is measured in writes to NAND and writes to HOST. For this example, the Write Amplification Factor (WAF) is 16185227 / 6405605 = 2.53
As you can see, the NVMe command line tools provide a lot of good information for understanding the health and performance of NVMe devices. You don’t need to use vendor-specific tools (like isdct).
https://www.percona.com/blog/2017/02/09/using-nvme-command-line-tools-to-check-nvme-flash-health/