qemu/qemu.git, branch stable-4.1

Update version for 4.1.1 release

2019-11-14T18:04:03+00:00

Signed-off-by: Michael Roth

mirror: Keep mirror_top_bs drained after dropping permissions

2019-11-12T22:31:49+00:00

mirror_top_bs is currently implicitly drained through its connection to the source or the target node. However, the drain section for target_bs ends early after moving mirror_top_bs from src to target_bs, so that requests can already be restarted while mirror_top_bs is still present in the chain, but has dropped all permissions and therefore runs into an assertion failure like this: qemu-system-x86_64: block/io.c:1634: bdrv_co_write_req_prepare: Assertion `child->perm & BLK_PERM_WRITE' failed. Keep mirror_top_bs drained until all graph changes have completed. Cc: qemu-stable@nongnu.org Signed-off-by: Kevin Wolf Reviewed-by: Max Reitz (cherry picked from commit d2da5e288a2e71e82866c8fdefd41b5727300124) Signed-off-by: Michael Roth

block/create: Do not abort if a block driver is not available

2019-11-12T22:31:38+00:00

The 'blockdev-create' QMP command was introduced as experimental feature in commit b0292b851b8, using the assert() debug call. It got promoted to 'stable' command in 3fb588a0f2c, but the assert call was not removed. Some block drivers are optional, and bdrv_find_format() might return a NULL value, triggering the assertion. Stable code is not expected to abort, so return an error instead. This is easily reproducible when libnfs is not installed: ./configure [...] module support no Block whitelist (rw) Block whitelist (ro) libiscsi support yes libnfs support no [...] Start QEMU: $ qemu-system-x86_64 -S -qmp unix:/tmp/qemu.qmp,server,nowait Send the 'blockdev-create' with the 'nfs' driver: $ ( cat << 'EOF' {'execute': 'qmp_capabilities'} {'execute': 'blockdev-create', 'arguments': {'job-id': 'x', 'options': {'size': 0, 'driver': 'nfs', 'location': {'path': '/', 'server': {'host': '::1', 'type': 'inet'}}}}, 'id': 'x'} EOF ) | socat STDIO UNIX:/tmp/qemu.qmp {"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 4}, "package": "v4.1.0-733-g89ea03a7dc"}, "capabilities": ["oob"]}} {"return": {}} QEMU crashes: $ gdb qemu-system-x86_64 core Program received signal SIGSEGV, Segmentation fault. (gdb) bt #0 0x00007ffff510957f in raise () at /lib64/libc.so.6 #1 0x00007ffff50f3895 in abort () at /lib64/libc.so.6 #2 0x00007ffff50f3769 in _nl_load_domain.cold.0 () at /lib64/libc.so.6 #3 0x00007ffff5101a26 in .annobin_assert.c_end () at /lib64/libc.so.6 #4 0x0000555555d7e1f1 in qmp_blockdev_create (job_id=0x555556baee40 "x", options=0x555557666610, errp=0x7fffffffc770) at block/create.c:69 #5 0x0000555555c96b52 in qmp_marshal_blockdev_create (args=0x7fffdc003830, ret=0x7fffffffc7f8, errp=0x7fffffffc7f0) at qapi/qapi-commands-block-core.c:1314 #6 0x0000555555deb0a0 in do_qmp_dispatch (cmds=0x55555645de70 , request=0x7fffdc005c70, allow_oob=false, errp=0x7fffffffc898) at qapi/qmp-dispatch.c:131 #7 0x0000555555deb2a1 in qmp_dispatch (cmds=0x55555645de70 , request=0x7fffdc005c70, allow_oob=false) at qapi/qmp-dispatch.c:174 With this patch applied, QEMU returns a QMP error: {'execute': 'blockdev-create', 'arguments': {'job-id': 'x', 'options': {'size': 0, 'driver': 'nfs', 'location': {'path': '/', 'server': {'host': '::1', 'type': 'inet'}}}}, 'id': 'x'} {"id": "x", "error": {"class": "GenericError", "desc": "Block driver 'nfs' not found or not supported"}} Cc: qemu-stable@nongnu.org Reported-by: Xu Tian Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Eric Blake Reviewed-by: John Snow Signed-off-by: Kevin Wolf (cherry picked from commit d90d5cae2b10efc0e8d0b3cc91ff16201853d3ba) Signed-off-by: Michael Roth

vhost: Fix memory region section comparison

2019-11-12T22:23:54+00:00

Using memcmp to compare structures wasn't safe, as I found out on ARM when I was getting falce miscompares. Use the helper function for comparing the MRSs. Fixes: ade6d081fc33948e56e6 ("vhost: Regenerate region list from changed sections list") Cc: qemu-stable@nongnu.org Signed-off-by: Dr. David Alan Gilbert Message-Id: <20190814175535.2023-4-dgilbert@redhat.com> Reviewed-by: Michael S. Tsirkin Signed-off-by: Michael S. Tsirkin (cherry picked from commit 3fc4a64cbaed2ddee4c60ddc06740b320e18ab82) Signed-off-by: Michael Roth

memory: Provide an equality function for MemoryRegionSections

2019-11-12T22:23:54+00:00

Provide a comparison function that checks all the fields are the same. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Philippe Mathieu-Daudé Message-Id: <20190814175535.2023-3-dgilbert@redhat.com> Reviewed-by: Michael S. Tsirkin Signed-off-by: Michael S. Tsirkin (cherry picked from commit 9366cf02e4e31c2a8128904d4d8290a0fad5f888) Signed-off-by: Michael Roth

memory: Align MemoryRegionSections fields

2019-11-12T22:23:49+00:00

MemoryRegionSection includes an Int128 'size' field; on some platforms the compiler causes an alignment of this to a 128bit boundary, leaving 8 bytes of dead space. This deadspace can be filled with junk. Move the size field to the top avoiding unnecessary alignment. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Philippe Mathieu-Daudé Message-Id: <20190814175535.2023-2-dgilbert@redhat.com> Reviewed-by: Michael S. Tsirkin Signed-off-by: Michael S. Tsirkin (cherry picked from commit 44f85d3276397cfa2cfa379c61430405dad4e644) Signed-off-by: Michael Roth

tests: make filemonitor test more robust to event ordering

2019-11-12T17:59:58+00:00

The ordering of events that are emitted during the rmdir test have changed with kernel >= 5.3. Semantically both new & old orderings are correct, so we must be able to cope with either. To cope with this, when we see an unexpected event, we push it back onto the queue and look and the subsequent event to see if that matches instead. Tested-by: Peter Xu Tested-by: Wei Yang Tested-by: Cornelia Huck Reviewed-by: Thomas Huth Signed-off-by: Daniel P. Berrangé (cherry picked from commit bf9e0313c27d8e6ecd7f7de3d63e1cb25d8f6311) Signed-off-by: Michael Roth

block: posix: Always allocate the first block

2019-11-12T17:59:58+00:00

When creating an image with preallocation "off" or "falloc", the first block of the image is typically not allocated. When using Gluster storage backed by XFS filesystem, reading this block using direct I/O succeeds regardless of request length, fooling alignment detection. In this case we fallback to a safe value (4096) instead of the optimal value (512), which may lead to unneeded data copying when aligning requests. Allocating the first block avoids the fallback. Since we allocate the first block even with preallocation=off, we no longer create images with zero disk size: $ ./qemu-img create -f raw test.raw 1g Formatting 'test.raw', fmt=raw size=1073741824 $ ls -lhs test.raw 4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw And converting the image requires additional cluster: $ ./qemu-img measure -f raw -O qcow2 test.raw required size: 458752 fully allocated size: 1074135040 When using format like vmdk with multiple files per image, we allocate one block per file: $ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat $ ls -lhs test*.vmdk 4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk 4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk 4.0K -rw-r--r--. 1 nsoffer nsoffer 353 Aug 27 03:23 test.vmdk I did quick performance test for copying disks with qemu-img convert to new raw target image to Gluster storage with sector size of 512 bytes: for i in $(seq 10); do rm -f dst.raw sleep 10 time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw done Here is a table comparing the total time spent: Type Before(s) After(s) Diff(%) --------------------------------------- real 530.028 469.123 -11.4 user 17.204 10.768 -37.4 sys 17.881 7.011 -60.7 We can see very clear improvement in CPU usage. Signed-off-by: Nir Soffer Message-id: 20190827010528.8818-2-nsoffer@redhat.com Reviewed-by: Max Reitz Signed-off-by: Max Reitz (cherry picked from commit 3a20013fbb26d2a1bd11ef148eefdb1508783787) Signed-off-by: Michael Roth

file-posix: Handle undetectable alignment

2019-11-12T17:59:58+00:00

In some cases buf_align or request_alignment cannot be detected: 1. With Gluster, buf_align cannot be detected since the actual I/O is done on Gluster server, and qemu buffer alignment does not matter. Since we don't have alignment requirement, buf_align=1 is the best value. 2. With local XFS filesystem, buf_align cannot be detected if reading from unallocated area. In this we must align the buffer, but we don't know what is the correct size. Using the wrong alignment results in I/O error. 3. With Gluster backed by XFS, request_alignment cannot be detected if reading from unallocated area. In this case we need to use the correct alignment, and failing to do so results in I/O errors. 4. With NFS, the server does not use direct I/O, so both buf_align cannot be detected. In this case we don't need any alignment so we can use buf_align=1 and request_alignment=1. These cases seems to work when storage sector size is 512 bytes, because the current code starts checking align=512. If the check succeeds because alignment cannot be detected we use 512. But this does not work for storage with 4k sector size. To determine if we can detect the alignment, we probe first with align=1. If probing succeeds, maybe there are no alignment requirement (cases 1, 4) or we are probing unallocated area (cases 2, 3). Since we don't have any way to tell, we treat this as undetectable alignment. If probing with align=1 fails with EINVAL, but probing with one of the expected alignments succeeds, we know that we found a working alignment. Practically the alignment requirements are the same for buffer alignment, buffer length, and offset in file. So in case we cannot detect buf_align, we can use request alignment. If we cannot detect request alignment, we can fallback to a safe value. To use this logic, we probe first request alignment instead of buf_align. Here is a table showing the behaviour with current code (the value in parenthesis is the optimal value). Case Sector buf_align (opt) request_alignment (opt) result ====================================================================== 1 512 512 (1) 512 (512) OK 1 4096 512 (1) 4096 (4096) FAIL ---------------------------------------------------------------------- 2 512 512 (512) 512 (512) OK 2 4096 512 (4096) 4096 (4096) FAIL ---------------------------------------------------------------------- 3 512 512 (1) 512 (512) OK 3 4096 512 (1) 512 (4096) FAIL ---------------------------------------------------------------------- 4 512 512 (1) 512 (1) OK 4 4096 512 (1) 512 (1) OK Same cases with this change: Case Sector buf_align (opt) request_alignment (opt) result ====================================================================== 1 512 512 (1) 512 (512) OK 1 4096 4096 (1) 4096 (4096) OK ---------------------------------------------------------------------- 2 512 512 (512) 512 (512) OK 2 4096 4096 (4096) 4096 (4096) OK ---------------------------------------------------------------------- 3 512 4096 (1) 4096 (512) OK 3 4096 4096 (1) 4096 (4096) OK ---------------------------------------------------------------------- 4 512 4096 (1) 4096 (1) OK 4 4096 4096 (1) 4096 (1) OK I tested that provisioning VMs and copying disks on local XFS and Gluster with 4k bytes sector size work now, resolving bugs [1],[2]. I tested also on XFS, NFS, Gluster with 512 bytes sector size. [1] https://bugzilla.redhat.com/1737256 [2] https://bugzilla.redhat.com/1738657 Signed-off-by: Nir Soffer Signed-off-by: Kevin Wolf (cherry picked from commit a6b257a08e3d72219f03e461a52152672fec0612) Signed-off-by: Michael Roth

block/file-posix: Let post-EOF fallocate serialize

2019-11-12T17:59:58+00:00

The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit c8bb23cbdbe32f. We can work around it by treating post-EOF fallocates as serializing up until infinity (INT64_MAX in practice). Cc: qemu-stable@nongnu.org Signed-off-by: Max Reitz Message-id: 20191101152510.11719-4-mreitz@redhat.com Signed-off-by: Max Reitz (cherry picked from commit 292d06b925b2787ee6f2430996b95651cae42fce) Signed-off-by: Michael Roth