diff options
authorKeith Busch <>2020-05-08 13:04:06 -0700
committerChristoph Hellwig <>2020-05-12 18:02:24 +0200
commitb69e2ef24b7b4867f80f47e2781e95d0bacd15cb (patch)
parent59c7c3caaaf8750df4ec3255082f15eb4e371514 (diff)
nvme-pci: dma read memory barrier for completions
Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads. Commit 324b494c2862 inlined verifying a new completion with its handling. At least one architecture was observed to access the contents out of order, resulting in the driver using stale data for the completion. Add a dma read barrier before reading the completion queue entry and after the condition its contents depend on to ensure the read order is determinsitic. Reported-by: John Garry <> Suggested-by: Will Deacon <> Signed-off-by: Keith Busch <> Tested-by: John Garry <> Acked-by: Will Deacon <> Reviewed-by: Sagi Grimberg <> Signed-off-by: Christoph Hellwig <>
1 files changed, 5 insertions, 0 deletions
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index e13c370de830..3726dc780d15 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq)
while (nvme_cqe_pending(nvmeq)) {
+ /*
+ * load-load control dependency between phase and the rest of
+ * the cqe requires a full read memory barrier
+ */
+ dma_rmb();
nvme_handle_cqe(nvmeq, nvmeq->cq_head);