Hi all,
I am writing RDMA_WRITE latency(Round Trip Latency) application using OFED user verbs and kernel verbs.
Since, in RDMA_WRITE, no work completion is generated on res-ponder side.
I am using technique like polling on the last byte of rdma target buffer on the responder side,
to find RDMA_WRITE completion.
Steps followed in user space rdma_write
1. allocated memory using malloc,
2. registered memory using ibv_reg_mr, REMOTE access is enabled,
3. requested rdma_write from initiator, completion entry is success,
4. responder is polling(spinning) on the last byte of receive buffer to be set, responder is able to see last byte set.
Steps followed in kernel space rdma_write
1. allocated memory using kmalloc,
2. registered memory using ib_reg_phys_mr, REMOTE access is enabled,
3. requested rdma_write from initiator, completion entry is success,
4. responder is polling(spinning) on the last byte of receive buffer to be set. Responder is not able to find buffer updatation.
If i use ib_dma_sync_single_for_cpu() before accessing the buffer on Responder side, Responder is able to see updated data(last byte set).
Please help me to understand following,
Is this buffer updation problem is because of CPU cache, or some other.
If it is CPU cache problem, then how user space is able to get updated data without any explicit dma synchronization.
Test setup is
arch: x86_64
Linux kernel :2.6.18-194.el5 (RHEL 5.5)
OFED : 1.5.4
Thanks in advance