[libvirt-users] Create qcow2 v3 volumes via libvirt

[Gionatan Danti]

Hi all,
on a fully patched CentOS 7.4 x86-64, I see the following behavior:

- when creating a new volumes using vol-create-as, the resulting file is
a qcow2 version 2 (compat=0.10) file. Example:

[***@gdanti-lenovo vmimages]# virsh vol-create-as default zzz.qcow2
8589934592 --format=qcow2 --backing-vol /mnt/vmimages/centos6.img
Vol zzz.qcow2 created
[***@gdanti-lenovo vmimages]# file zzz.qcow2
zzz.qcow2: QEMU QCOW Image (v2), has backing file (path
/mnt/vmimages/centos6.img), 8589934592 bytes
[***@gdanti-lenovo vmimages]# qemu-img info zzz.qcow2
image: zzz.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 196K
cluster_size: 65536
backing file: /mnt/vmimages/centos6.img
backing file format: raw
Format specific information:
compat: 0.10
refcount bits: 16

- when creating a snapshot, the resulting file is a qcow2 version 3
(comapt=1.1) file. Example:

[***@gdanti-lenovo vmimages]# virsh snapshot-create-as centos6left
--disk-only --no-metadata snap.qcow2
Domain snapshot snap.qcow2 created
[***@gdanti-lenovo vmimages]# file centos6left.snap.qcow2
centos6left.snap.qcow2: QEMU QCOW Image (v3), has backing file (path
/mnt/vmimages/centos6left.qcow2), 8589934592 bytes
[***@gdanti-lenovo vmimages]# qemu-img info centos6left.snap.qcow2
image: centos6left.snap.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 196K
cluster_size: 65536
backing file: /mnt/vmimages/centos6left.qcow2
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false

From what I know, this is a deliberate decision: compat=1.1 requires
relatively recent qemu version, and creating a new volume play on the
"safe side" of compatibility.

It is possible to create a new volume using qcow2 version 3 (compat=1.1)
format *using libvirt/virsh* (I know I can do that via qemu-img)? Any
drawback on using version 3 format?

Thanks.

[Gionatan Danti]

Post by Gionatan Danti
Hi all,
- when creating a new volumes using vol-create-as, the resulting file
8589934592 --format=qcow2 --backing-vol /mnt/vmimages/centos6.img
Vol zzz.qcow2 created
zzz.qcow2: QEMU QCOW Image (v2), has backing file (path
/mnt/vmimages/centos6.img), 8589934592 bytes
image: zzz.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 196K
cluster_size: 65536
backing file: /mnt/vmimages/centos6.img
backing file format: raw
compat: 0.10
refcount bits: 16
- when creating a snapshot, the resulting file is a qcow2 version 3
--disk-only --no-metadata snap.qcow2
Domain snapshot snap.qcow2 created
centos6left.snap.qcow2: QEMU QCOW Image (v3), has backing file (path
/mnt/vmimages/centos6left.qcow2), 8589934592 bytes
image: centos6left.snap.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 196K
cluster_size: 65536
backing file: /mnt/vmimages/centos6left.qcow2
backing file format: qcow2
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
From what I know, this is a deliberate decision: compat=1.1 requires
relatively recent qemu version, and creating a new volume play on the
"safe side" of compatibility.
It is possible to create a new volume using qcow2 version 3
(compat=1.1) format *using libvirt/virsh* (I know I can do that via
qemu-img)? Any drawback on using version 3 format?
Thanks.

Hi all,
anyone with some thoughts on the matter?

Another question: how reliable are qcow2 ver2/3 files nowadays? Are you
using them in production environments?
At the moment, I am using RAW files and filesystem-level snapshot to
manage versioning; however, as virt-manager has direct support for
managing qcow2 internal snapshots, it would be easier to deploy qcow2
disks.

What strikes me is that, if thing have not changed, Red Hat support
policy was to *not* support internal snapshots. So, are they reliable
enough for production VMs?

Thanks.

[Daniel P. Berrangé]

Post by Gionatan Danti
Another question: how reliable are qcow2 ver2/3 files nowadays? Are you
using them in production environments?

qcow2 is widely used in production at large scale in general. Just not
with internal snapshots - almost everything uses external snapshots,
aka backing file chains.

Post by Gionatan Danti
At the moment, I am using RAW files and filesystem-level snapshot to manage
versioning; however, as virt-manager has direct support for managing qcow2
internal snapshots, it would be easier to deploy qcow2 disks.
What strikes me is that, if thing have not changed, Red Hat support policy
was to *not* support internal snapshots. So, are they reliable enough for
production VMs?

The QEMU community still tends to discourage use of internal snapshots.
There are not even any QMP monitor commands to use them - you are forced
to use the legacy HMP interface to QEMU for mgmt. All of the workaround
providing interesting block storage mgmt is focused on external snapshots
(aka the backing_file option). There are some technical downsides to
internal snapshots IIUC, such as inability to free the space used by the
internal snapshot when it is deleted, loading/saving snapshots blocks
execution of the guest OS, and probably more I've forgotten about.

The only nice thing about internal snapshots is simplicity of mgmt, and
that is a very nice thing indeed, which is why virt-manager has code
to support that - it was much easier to add that code for external
snapshots. Just a shame about all the downsides :-(

Regards,
Daniel

[Gionatan Danti]

Post by Daniel P. Berrangé
qcow2 is widely used in production at large scale in general. Just not
with internal snapshots - almost everything uses external snapshots,
aka backing file chains.
The QEMU community still tends to discourage use of internal snapshots.
There are not even any QMP monitor commands to use them - you are forced
to use the legacy HMP interface to QEMU for mgmt. All of the workaround
providing interesting block storage mgmt is focused on external snapshots
(aka the backing_file option). There are some technical downsides to
internal snapshots IIUC, such as inability to free the space used by the
internal snapshot when it is deleted, loading/saving snapshots blocks
execution of the guest OS, and probably more I've forgotten about.
The only nice thing about internal snapshots is simplicity of mgmt, and
that is a very nice thing indeed, which is why virt-manager has code
to support that - it was much easier to add that code for external
snapshots. Just a shame about all the downsides :-(

So internal snapshots remain something very useful for lab/tests, but
are not recommended for regular use in production environment, right?

Thanks.

[Eric Blake]

Post by Gionatan Danti

Post by Daniel P. Berrangé
qcow2 is widely used in production at large scale in general. Just not
with internal snapshots - almost everything uses external snapshots,
aka backing file chains.
The QEMU community still tends to discourage use of internal snapshots.
There are not even any QMP monitor commands to use them - you are forced
to use the legacy HMP interface to QEMU for mgmt. All of the workaround
providing interesting block storage mgmt is focused on external snapshots
(aka the backing_file option). There are some technical downsides to
internal snapshots IIUC, such as inability to free the space used by the
internal snapshot when it is deleted, loading/saving snapshots blocks
execution of the guest OS, and probably more I've forgotten about.
The only nice thing about internal snapshots is simplicity of mgmt, and
that is a very nice thing indeed, which is why virt-manager has code
to support that - it was much easier to add that code for external
snapshots. Just a shame about all the downsides :-(

So internal snapshots remain something very useful for lab/tests, but
are not recommended for regular use in production environment, right?

That's fairly accurate. Also, mixing internal and external snapshots at
the same time is likely to trigger some known data-loss problems, if you
are not extremely careful, so for now, just pick one or the other and
stick to it (I hope to someday enhance qemu to refuse operations that
would risk data loss, or to perform a slower version of the operation
with the same result instead of its current fast but wrong
implementations, when dealing with mixed-internal/external backing chains).

https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg00865.html

[Gionatan Danti]

Post by Eric Blake
That's fairly accurate. Also, mixing internal and external snapshots
at the same time is likely to trigger some known data-loss problems,
if you are not extremely careful, so for now, just pick one or the
other and stick to it (I hope to someday enhance qemu to refuse
operations that would risk data loss, or to perform a slower version
of the operation with the same result instead of its current fast but
wrong implementations, when dealing with mixed-internal/external
backing chains).
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg00865.html

Ok, I think I'll continue to use RAW images + filesystem snapshots +
external snapshot when required.
It is unfortunate that we have no GUI to manage external snapshots. I
even remember that external snapshot remove was not supported - I had to
use the --metadata flag and a plain "rm" to remove them (or using
--no-metadata in the first place)[1]. Does it remain the current
behavior?

Thanks.

[1]
https://www.redhat.com/archives/libvirt-users/2015-September/msg00037.html

[Gionatan Danti]

Post by Gionatan Danti
Ok, I think I'll continue to use RAW images + filesystem snapshots +
external snapshot when required.
It is unfortunate that we have no GUI to manage external snapshots. I
even remember that external snapshot remove was not supported - I had to
use the --metadata flag and a plain "rm" to remove them (or using
--no-metadata in the first place)[1]. Does it remain the current behavior?
Thanks.
[1]
https://www.redhat.com/archives/libvirt-users/2015-September/msg00037.html

Any hope to get virsh external snapshot creation/deletion fixed?
Or, even better, to have virt-manager support for external snapshots?
Thanks.

[Kashyap Chamarthy]

Post by Gionatan Danti

Post by Gionatan Danti
Ok, I think I'll continue to use RAW images + filesystem snapshots +
external snapshot when required.
It is unfortunate that we have no GUI to manage external snapshots. I
even remember that external snapshot remove was not supported - I had to
use the --metadata flag and a plain "rm" to remove them (or using
--no-metadata in the first place)[1]. Does it remain the current behavior?

Yes, you still have to clean up external snapshot metadata manually, as
the link below indicates. Deleting external snapshots is a non-trivial
problem. For example, if you have a disk image chain like:

[A] <-- [B] <-- [C] <-- [D]

Now if you no longer need image [B], what is the most efficient way to
get rid of it? Three possibilities:

- Merge [B] into [A], and rewrite the backing chain of [C]
to point to [A]. Now can you discard image [B].
- Merge [A] into [B], call the new image [AB]. Now you can
discard [A].
- Merge [B] into [C], and rewrite the backing chain of [C] to point to
[A]. Now you can discard image [B].

It depends on how large each of the disk images are, and based on the
size, merge / coalesce them accordingly, to arrive at the quickest
solution.

Note that existing commands (`virsh blockpull` and `virsh blockcommit`,
which is relatively faster) will allow you handle it manually or script
it -- which many people already do it.

A couple of references (that are also easily scriptable):

http://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-including-current-active-disk
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

Post by Gionatan Danti

Post by Gionatan Danti
[1]
https://www.redhat.com/archives/libvirt-users/2015-September/msg00037.html

Any hope to get virsh external snapshot creation/deletion fixed?

I don't see any problem creation external snapshots (or more correctly:
"overlays"). Deletion of external snapshots involves a bit more work,
but it is scriptable as well.


[...]

[Gionatan Danti]

Post by Kashyap Chamarthy
Yes, you still have to clean up external snapshot metadata manually, as
the link below indicates. Deleting external snapshots is a non-trivial
[A] <-- [B] <-- [C] <-- [D]
Now if you no longer need image [B], what is the most efficient way to
- Merge [B] into [A], and rewrite the backing chain of [C]
to point to [A]. Now can you discard image [B].
- Merge [A] into [B], call the new image [AB]. Now you can
discard [A].
- Merge [B] into [C], and rewrite the backing chain of [C] to point to
[A]. Now you can discard image [B].
It depends on how large each of the disk images are, and based on the
size, merge / coalesce them accordingly, to arrive at the quickest
solution.

True.

Post by Kashyap Chamarthy
Note that existing commands (`virsh blockpull` and `virsh blockcommit`,
which is relatively faster) will allow you handle it manually or script
it -- which many people already do it. >
http://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-including-current-active-disk
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

Post by Gionatan Danti

Post by Gionatan Danti
[1]
https://www.redhat.com/archives/libvirt-users/2015-September/msg00037.html

Any hope to get virsh external snapshot creation/deletion fixed?

"overlays"). Deletion of external snapshots involves a bit more work,
but it is scriptable as well.

Reading old mailing list posts, I was under impression that some work
was put to automatically and trasparently issue the best
blockpull/blockcommit command. If not - fair enough ;)

Thanks.

[Paul O'Rorke]

I have been using internal snapshots on production qcow2 images for a
couple of years, admittedly as infrequently as possible with one
exception and that exception has had multiple snapshots taken and
removed using virt-manager's GUI.

Post by Daniel P. Berrangé
There are some technical downsides to
internal snapshots IIUC, such as inability to free the space used by the
internal snapshot when it is deleted,

This might explain why this VM recently kept going into a paused state
and I had to extend the volume to get it to stay up.  This VM is used
for testing our software in SharePoint and we make heavy use of
snapshots.  Is there nothing I can to do recover that space?

What would be the best practice then for a VM that needs to be able to
create and remove snapshots on a regular basis?

*Paul O'Rorke*
*Tracker Software Products (Canada) Limited *
www.tracker-software.com <http://www.tracker-software.com/>
Tel: [phone]
Fax: [phone]

<http://www.tracker-software.com/>

Support:
http://www.tracker-software.com/support
Download latest Releases
http://www.tracker-software.com/downloads/

Post by Daniel P. Berrangé

Post by Daniel P. Berrangé
qcow2 is widely used in production at large scale in general. Just not
with internal snapshots - almost everything uses external snapshots,
aka backing file chains.
The QEMU community still tends to discourage use of internal snapshots.
There are not even any QMP monitor commands to use them - you are forced
to use the legacy HMP interface to QEMU for mgmt. All of the workaround
providing interesting block storage mgmt is focused on external snapshots
(aka the backing_file option). There are some technical downsides to
internal snapshots IIUC, such as inability to free the space used by the
internal snapshot when it is deleted, loading/saving snapshots blocks
execution of the guest OS, and probably more I've forgotten about.
The only nice thing about internal snapshots is simplicity of mgmt, and
that is a very nice thing indeed, which is why virt-manager has code
to support that - it was much easier to add that code for external
snapshots. Just a shame about all the downsides :-(

So internal snapshots remain something very useful for lab/tests, but
are not recommended for regular use in production environment, right?
Thanks.

[Eric Blake]

Post by Paul O'Rorke
I have been using internal snapshots on production qcow2 images for a
couple of years, admittedly as infrequently as possible with one
exception and that exception has had multiple snapshots taken and
removed using virt-manager's GUI.

Post by Daniel P. Berrangé
There are some technical downsides to
internal snapshots IIUC, such as inability to free the space used by the
internal snapshot when it is deleted,

This is not an insurmountable difficulty, just one that no one has spent
time coding up.

Post by Paul O'Rorke
This might explain why this VM recently kept going into a paused state
and I had to extend the volume to get it to stay up.  This VM is used
for testing our software in SharePoint and we make heavy use of
snapshots.  Is there nothing I can to do recover that space?

If you have no internal snapshots, you can do a 'qemu-img convert' to
copy just the portion of the image that is actively in use; the copy
will use less disk space than the original because it got rid of the
now-unused space. 'virt-sparsify' from libguestfs takes this one step
further, by also removing unused space within the guest filesystem itself.

In fact, even if you do have internal snapshots, there is probably a
sequence of 'qemu-img convert' invocations that can ultimately convert
all of your internal snapshots into an external chain of snapshots; but
I don't have a ready formula off-hand to point to (experiment on an
image you don't care about, before doing it on your production image).

Post by Paul O'Rorke
What would be the best practice then for a VM that needs to be able to
create and remove snapshots on a regular basis?

In a managed environment, external snapshots probably have the most
support for creating and later merging portions of a chain of snapshots,
although we could still improve libvirt to make this feel like more of a
first class citizen.

[Paul O'Rorke]

Thanks for the suggestions Eric,

great stuff and most appreciated.

I must play in my sandbox...

*Paul O'Rorke*
*Tracker Software Products (Canada) Limited *
www.tracker-software.com <http://www.tracker-software.com/>
Tel: [phone]
Fax: [phone]

<http://www.tracker-software.com/>

Support:
http://www.tracker-software.com/support
Download latest Releases
http://www.tracker-software.com/downloads/

Post by Eric Blake

Post by Paul O'Rorke
I have been using internal snapshots on production qcow2 images for a
couple of years, admittedly as infrequently as possible with one
exception and that exception has had multiple snapshots taken and
removed using virt-manager's GUI.

Post by Daniel P. Berrangé
There are some technical downsides to
internal snapshots IIUC, such as inability to free the space used by the
internal snapshot when it is deleted,

This is not an insurmountable difficulty, just one that no one has
spent time coding up.

Post by Paul O'Rorke
This might explain why this VM recently kept going into a paused
state and I had to extend the volume to get it to stay up.  This VM
is used for testing our software in SharePoint and we make heavy use
of snapshots.  Is there nothing I can to do recover that space?

If you have no internal snapshots, you can do a 'qemu-img convert' to
copy just the portion of the image that is actively in use; the copy
will use less disk space than the original because it got rid of the
now-unused space.  'virt-sparsify' from libguestfs takes this one step
further, by also removing unused space within the guest filesystem itself.
In fact, even if you do have internal snapshots, there is probably a
sequence of 'qemu-img convert' invocations that can ultimately convert
all of your internal snapshots into an external chain of snapshots;
but I don't have a ready formula off-hand to point to (experiment on
an image you don't care about, before doing it on your production image).

Post by Paul O'Rorke
What would be the best practice then for a VM that needs to be able
to create and remove snapshots on a regular basis?

In a managed environment, external snapshots probably have the most
support for creating and later merging portions of a chain of
snapshots, although we could still improve libvirt to make this feel
like more of a first class citizen.

[Paul O'Rorke]

Thanks for the suggestions Eric, great stuff and most appreciated.

I will have a play in my sandbox...

*Paul O'Rorke*
*Tracker Software Products (Canada) Limited *
www.tracker-software.com <http://www.tracker-software.com/>
Tel: [phone]
Fax: [phone]

<http://www.tracker-software.com/>

Support:
http://www.tracker-software.com/support
Download latest Releases
http://www.tracker-software.com/downloads/

Post by Eric Blake

Post by Paul O'Rorke
I have been using internal snapshots on production qcow2 images for a
couple of years, admittedly as infrequently as possible with one
exception and that exception has had multiple snapshots taken and
removed using virt-manager's GUI.

Post by Daniel P. Berrangé
There are some technical downsides to
internal snapshots IIUC, such as inability to free the space used by the
internal snapshot when it is deleted,

This is not an insurmountable difficulty, just one that no one has
spent time coding up.

Post by Paul O'Rorke
This might explain why this VM recently kept going into a paused
state and I had to extend the volume to get it to stay up.  This VM
is used for testing our software in SharePoint and we make heavy use
of snapshots.  Is there nothing I can to do recover that space?

If you have no internal snapshots, you can do a 'qemu-img convert' to
copy just the portion of the image that is actively in use; the copy
will use less disk space than the original because it got rid of the
now-unused space.  'virt-sparsify' from libguestfs takes this one step
further, by also removing unused space within the guest filesystem itself.
In fact, even if you do have internal snapshots, there is probably a
sequence of 'qemu-img convert' invocations that can ultimately convert
all of your internal snapshots into an external chain of snapshots;
but I don't have a ready formula off-hand to point to (experiment on
an image you don't care about, before doing it on your production image).

Post by Paul O'Rorke
What would be the best practice then for a VM that needs to be able
to create and remove snapshots on a regular basis?

In a managed environment, external snapshots probably have the most
support for creating and later merging portions of a chain of
snapshots, although we could still improve libvirt to make this feel
like more of a first class citizen.

[Paul O'Rorke]

Hi Eric and list,

I had another production VM start pausing itself.  This one had been
running for more than 4 years on a 60G LVM volume.  It has had the
occasional snapshot during that time though all have been "removed"
using the virt-manager gui so I used qemu-img as you suggested.

# qemu-img convert /dev/trk-kvm-02-vg/rt44 -O qcow2 /mnt/scratch/rt44.qcow2

dd'd the qcow2 image back on to the LV after testing it boots OK
directly from the image and it is in production again.

The VM itself reports ample space available:

$ df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                             3.9G     0  3.9G   0% /dev
tmpfs                            789M  8.8M  780M   2% /run
/dev/mapper/RT--vg-root           51G   21G   28G  42% /
tmpfs                            3.9G     0  3.9G   0% /dev/shm
tmpfs                            5.0M     0  5.0M   0% /run/lock
tmpfs                            3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1                        472M  155M  293M  35% /boot
192.168.0.16:/volume1/fileLevel  8.1T  2.5T  5.6T  31% /mnt/nfs/fileLevel
tmpfs                            789M     0  789M   0% /run/user/1000

I would prefer to not get caught out again with this machine pausing,
how can I determine how much space is being used up by 'deleted'
internal snapshots?  Do you have any suggested reading on this?

If I extend the LVM volume but not the guest file system will snapshots
be "at the end" of the LV and "outside" the guest file system?

If I were to expand the guest's ext4 file system I would want to do it
unmounted and from a live CD but I'm having a heck of a time getting my
live distro to use the virtio disk drivers.  Any advice there?

sincerely

*Paul O'Rorke*
*Tracker Software Products (Canada) Limited *
www.tracker-software.com <http://www.tracker-software.com/>
Tel: [phone]
Fax: [phone]

<http://www.tracker-software.com/>

Support:
http://www.tracker-software.com/support
Download latest Releases
http://www.tracker-software.com/downloads/

Post by Eric Blake

Post by Paul O'Rorke
I have been using internal snapshots on production qcow2 images for a
couple of years, admittedly as infrequently as possible with one
exception and that exception has had multiple snapshots taken and
removed using virt-manager's GUI.

Post by Daniel P. Berrangé
There are some technical downsides to
internal snapshots IIUC, such as inability to free the space used by the
internal snapshot when it is deleted,

This is not an insurmountable difficulty, just one that no one has
spent time coding up.

Post by Paul O'Rorke
This might explain why this VM recently kept going into a paused
state and I had to extend the volume to get it to stay up.  This VM
is used for testing our software in SharePoint and we make heavy use
of snapshots.  Is there nothing I can to do recover that space?

If you have no internal snapshots, you can do a 'qemu-img convert' to
copy just the portion of the image that is actively in use; the copy
will use less disk space than the original because it got rid of the
now-unused space.  'virt-sparsify' from libguestfs takes this one step
further, by also removing unused space within the guest filesystem itself.
In fact, even if you do have internal snapshots, there is probably a
sequence of 'qemu-img convert' invocations that can ultimately convert
all of your internal snapshots into an external chain of snapshots;
but I don't have a ready formula off-hand to point to (experiment on
an image you don't care about, before doing it on your production image).

Post by Paul O'Rorke
What would be the best practice then for a VM that needs to be able
to create and remove snapshots on a regular basis?

In a managed environment, external snapshots probably have the most
support for creating and later merging portions of a chain of
snapshots, although we could still improve libvirt to make this feel
like more of a first class citizen.

[Eric Blake]

Post by Paul O'Rorke
Hi Eric and list,
I had another production VM start pausing itself.  This one had been
running for more than 4 years on a 60G LVM volume.  It has had the
occasional snapshot during that time though all have been "removed"
using the virt-manager gui so I used qemu-img as you suggested.
# qemu-img convert /dev/trk-kvm-02-vg/rt44 -O qcow2 /mnt/scratch/rt44.qcow2

To make sure I understand, /dev/trk-kvm-02-vg/rt44 was using qcow2 on
top of a block device, both before and after you used qemu-img convert
to compact out the wasted space? Also, do you know if you were using
qcow2v2 or v3 prior to running out of space? qcow2v2 was the default in
CentOS 6 for historical reasons, but qemu doesn't support efficient
space reclamation on those older images the way it does on qcow2v3.

Did you save the full image anywhere, so we can do additional postmortem
analysis on it? Not that I'm asking to see the image itself, in case
that would give away private information; but even something like
running 'qcheck' from https://github.com/jnsnow/qcheck.git might shed
some insight into the storage being used by the image that reached capacity.

Post by Paul O'Rorke
dd'd the qcow2 image back on to the LV after testing it boots OK
directly from the image and it is in production again.
$ df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                             3.9G     0  3.9G   0% /dev
tmpfs                            789M  8.8M  780M   2% /run
/dev/mapper/RT--vg-root           51G   21G   28G  42% /
tmpfs                            3.9G     0  3.9G   0% /dev/shm
tmpfs                            5.0M     0  5.0M   0% /run/lock
tmpfs                            3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1                        472M  155M  293M  35% /boot
192.168.0.16:/volume1/fileLevel  8.1T  2.5T  5.6T  31% /mnt/nfs/fileLevel
tmpfs                            789M     0  789M   0% /run/user/1000
I would prefer to not get caught out again with this machine pausing,
how can I determine how much space is being used up by 'deleted'
internal snapshots?  Do you have any suggested reading on this?

The highest write offset (block.<num>.allocation in
virConnectGetAllDomainStats()) should tell you how much of the
underlying block device is "in use", that is, the highest offset that is
being written to. It doesn't count for holes earlier in the image,
especially where space could be reclaimed by reusing those holes, but
does give a good indication when you might need to resize a storage
volume to account for growing amounts of metadata that aren't being cleared.

And it may be that qemu still needs some patches to more fully trim and
reuse the space previously occupied by a deleted internal snapshot.

Post by Paul O'Rorke
If I extend the LVM volume but not the guest file system will snapshots
be "at the end" of the LV and "outside" the guest file system?

qcow2 is a mapped file format. Resizing the LVM volume does NOT change
the amount of disk space seen by the guest. Clusters may appear in a
different order in the host's raw storage than the order in which they
are visited in the guest ('qemu-img map' can show you the mapping), but
the guest does not care because it is always presented a logical view of
a linear array of storage, regardless of how clusters are fragmented
across the host device. Right now, qcow2 does not have any way to
constrain the clusters used by guest data to appear at any particular
offset (although there has been talk of adding a new mode to qemu to
operate on a fully-preallocated image where all guest clusters except
maybe the first occur in linear order at the beginning of the file, and
all qcow2 metadata except for the leading header be placed at the end of
the file at offsets higher than the guest clusters - in such a layout,
as long as you don't use internal snapshots, then all further qcow2
metadata writes would be beyond the linear region reserved for
guest-visible clusters).

Post by Paul O'Rorke
If I were to expand the guest's ext4 file system I would want to do it
unmounted and from a live CD but I'm having a heck of a time getting my
live distro to use the virtio disk drivers.  Any advice there?

virt-resize from libguestfs-tools is your friend!

[Daniel P. Berrangé]

Post by Gionatan Danti
Hi all,
- when creating a new volumes using vol-create-as, the resulting file is a
8589934592 --format=qcow2 --backing-vol /mnt/vmimages/centos6.img
Vol zzz.qcow2 created

Yes, for sake of backcompat we default to v2 unless something
requires v3. You can't use vol-create-as to create a v3 image,
you need to use the XML input.

Either force the use of v3:

<volume>
<name>demo.qcow2</name>
<capacity>1073741824</capacity>
<target>
<format type='qcow2'/>
<compat>1.1</compat>
</target>
</volume>

Or request some feature that implies v3:


<volume>
<name>demo.qcow2</name>
<capacity>1073741824</capacity>
<target>
<format type='qcow2'/>
<features>
<lazy_refcounts/>
</features>
</target>
</volume>

and pass this XML to virsh vol-create

Post by Gionatan Danti
It is possible to create a new volume using qcow2 version 3 (compat=1.1)
format *using libvirt/virsh* (I know I can do that via qemu-img)? Any
drawback on using version 3 format?

The only drawback is OS compatibility.

Regards,
Daniel

[Gianluca Cecchi]

Post by Daniel P. Berrangé

Post by Gionatan Danti
Hi all,
- when creating a new volumes using vol-create-as, the resulting file is

a

Post by Gionatan Danti
8589934592 --format=qcow2 --backing-vol /mnt/vmimages/centos6.img
Vol zzz.qcow2 created

Yes, for sake of backcompat we default to v2 unless something
requires v3. You can't use vol-create-as to create a v3 image,
you need to use the XML input.

BTW: in virt-manager of CentOS 7.4 by default qcow2 disks (see below for
zzz3.qcow2) are created with compatibility 1.1 (probably indirectly because
of its lazy refcounts feature enabled, as Daniel wrote):

[***@c7client images]# qemu-img info zzz3.qcow2
image: zzz3.qcow2
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 516K
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true
[***@c7client images]#

The same if I create a volume with a backing file from virt-manager (not
important if this is 0.10 or 1.1):

[***@c7client images]# qemu-img info zzz4.qcow2
image: zzz4.qcow2
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 196K
cluster_size: 65536
backing file: /var/lib/libvirt/images/zzz2.qcow2
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: true
[***@c7client images]#


But I don't know if internally virt-manager uses virsh or qemu-img
actually....

My tests with:
libvirt-client-3.2.0-14.el7_4.9.x86_64
virt-manager-1.4.1-7.el7.noarch

The same default to 0.10 format seems true with virsh from Fedora 27 and
libvirt-client-3.7.0-4.fc27.x86_64

Gianluca

[Daniel P. Berrangé]

Post by Gianluca Cecchi

Post by Daniel P. Berrangé

Post by Gionatan Danti
Hi all,
- when creating a new volumes using vol-create-as, the resulting file is

a

Post by Gionatan Danti
8589934592 --format=qcow2 --backing-vol /mnt/vmimages/centos6.img
Vol zzz.qcow2 created

Yes, for sake of backcompat we default to v2 unless something
requires v3. You can't use vol-create-as to create a v3 image,
you need to use the XML input.

BTW: in virt-manager of CentOS 7.4 by default qcow2 disks (see below for
zzz3.qcow2) are created with compatibility 1.1 (probably indirectly because

Yes, virt-manager explicitly turns on lazy refcounts if it sees a new
enough QEMU, so will get v3


Regards,
Daniel