[libvirt-users] how "safe" is blockcommit ?

[Lentes, Bernd]

Hi,

currently i'm following https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit. I 'm playing around with it and it seems to be quite nice.
What i want is a daily consistent backup of my image file of the guest.
I have the idea of the following procedure:

- Shutdown the guest (i can live with a downtime of a few minutes, it will happen in the night).
And i think it's the only way to have a real clean snapshot
- create a snapshot with snapshot-create-as: snapshot-create-as guest testsn --disk-only
- start the guest again. Changes will now go into the overlay, as e.g. inserts in a database
- rsync the base file to a cifs server. With rsync not the complete, likely big file is transferred but just the delta
- blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait --verbose --pivot
- delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata
- remove the overlay

Is that ok ? How "safe" is blockcommit on a running guest ? It's possible that during the rsync, when the guest is running, some inserts are done in a database.
Is it safe to copy the new sectors (i assume that's what blockcommit does) under a running database ?
Or is it only safe doing blockcommit on a stopped guest ?

Thanks for any answer.

Bernd

P.S. Is the same procedure possible when the guest disk(s) reside directly in a plain logical volume, without a file system in-between ?
--
Bernd Lentes
Systemadministration
Institut für Entwicklungsgenetik
Gebäude 35.34 - Raum 208
HelmholtzZentrum münchen
[ mailto:***@helmholtz-muenchen.de | ***@helmholtz-muenchen.de ]
phone: [phone]
fax: [phone]
[ http://www.helmholtz-muenchen.de/idg | http://www.helmholtz-muenchen.de/idg ]

wer Fehler macht kann etwas lernen
wer nichts macht kann auch nichts lernen


Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias H. Tschoep, Heinrich Bassler, Dr. rer. nat. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671

[Eric Blake]

Post by Lentes, Bernd
Hi,
currently i'm following https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit. I 'm playing around with it and it seems to be quite nice.
What i want is a daily consistent backup of my image file of the guest.
- Shutdown the guest (i can live with a downtime of a few minutes, it will happen in the night).
And i think it's the only way to have a real clean snapshot

Not the only way - you can also use qemu-guest-agent with a trusted
guest to quiesce all filesystem I/O after freezing database operations
at a consistent point, for a clean snapshot of a live guest. But
shutting down is indeed safe, and easier to reason about than worrying
whether your qga interaction is properly hooked into all necessary
places for a live quiesce.

Post by Lentes, Bernd
- create a snapshot with snapshot-create-as: snapshot-create-as guest testsn --disk-only
- start the guest again. Changes will now go into the overlay, as e.g. inserts in a database
- rsync the base file to a cifs server. With rsync not the complete, likely big file is transferred but just the delta

We're also trying to add support for incremental backups into a future
version of libvirt on top of the qemu 3.0 feature of persistent bitmaps
in qcow2 images, which could indeed guarantee that you transfer only the
portions of the guest disk that were touched since the last backup. But
as that's still something I'm trying to code up, your solution of using
rsync to pick out the deltas is as good as anything you can get right now.

Post by Lentes, Bernd
- blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait --verbose --pivot
- delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata
- remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?

Yep, that's the right way to do it! It's perfectly safe - the guest
doesn't care whether it is reading/writing from the backing file or the
overlay, and even if the blockcommit action is aborted, you can restart
it for the same effects.

(Okay, if you want to get technical, you need to know that committing
from 'Base <- mid <- top' down to 'Base' leaves 'mid' in an inconsistent
state - but that's not something the guest can see through 'top'; and
your specific case is just committing to the immediate backing layer
rather than skipping layers like that).

Post by Lentes, Bernd
It's possible that during the rsync, when the guest is running, some inserts are done in a database.

As long as the backing file is read-only during the rsync (which it is,
since all your guest writes are going to the overlay), then nothing the
guest can do will interfere with what rsync can see. Just be sure that
you don't start the blockcommit until after rsync is done.

Post by Lentes, Bernd
Is it safe to copy the new sectors (i assume that's what blockcommit does) under a running database ?
Or is it only safe doing blockcommit on a stopped guest ?

Live blockcommit is safe, and exists so you don't have to stop the guest.

For a bit more insight into what is going on under the hood, the slides
from my KVM Forum talk from a few years back may give some nice insights:
http://events17.linuxfoundation.org/sites/events/files/slides/2015-qcow2-expanded.pdf

Post by Lentes, Bernd
Thanks for any answer.
Bernd
P.S. Is the same procedure possible when the guest disk(s) reside directly in a plain logical volume, without a file system in-between ?

Live blockcommit works onto any host storage protocol (whether
filesystem, block device via LVM, or even remote access such as NBD or
ceph). The key is that your overlay is a qcow2 file that is tracking
the deltas during the time in which you are capturing your backup of the
backing file, and then blockcommit safely writes those deltas back into
the backing file prior to no longer needing the overlay.

[Gionatan Danti]

Post by Eric Blake
We're also trying to add support for incremental backups into a future
version of libvirt on top of the qemu 3.0 feature of persistent
bitmaps in qcow2 images, which could indeed guarantee that you
transfer only the portions of the guest disk that were touched since
the last backup. But as that's still something I'm trying to code up,
your solution of using rsync to pick out the deltas is as good as
anything you can get right now.

Plain rsync is going to be very slow for large file transfer. I strongly
suggest to experiment with rsync "--inplace no-whole-file" options to
really transfer/write changed bits only. If that does not cut in, you
can try with bdsync[1] or blocksync[2]

[1] https://github.com/TargetHolding/bdsync
[2] https://github.com/shodanshok/blocksync

Post by Eric Blake

Post by Lentes, Bernd
- blockcommit the overlay: blockcommit guest /path/to/testsn --active
--wait --verbose --pivot
- delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata
- remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?

Yep, that's the right way to do it! It's perfectly safe - the guest
doesn't care whether it is reading/writing from the backing file or
the overlay, and even if the blockcommit action is aborted, you can
restart it for the same effects.

I found the "manual overlay rm" a bit of a pain. What is the reason why
a successfull blockcommit does not delete the just-inactivate overlay
file?

Thanks.

[Peter Krempa]

[...]

Post by Eric Blake

Post by Lentes, Bernd
- blockcommit the overlay: blockcommit guest /path/to/testsn
--active --wait --verbose --pivot
- delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata
- remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?

Yep, that's the right way to do it! It's perfectly safe - the guest
doesn't care whether it is reading/writing from the backing file or
the overlay, and even if the blockcommit action is aborted, you can
restart it for the same effects.

I found the "manual overlay rm" a bit of a pain. What is the reason why a
successfull blockcommit does not delete the just-inactivate overlay file?

It was not implemented because currently we don't track what happens to
the backing chain after a successful commit, but rather we redetect it
afterwards.

The commit operation can remove more than one image from the backing
chain and thus tracking what happened is very important if you want to
remove the files afterwards.

I'm currently working on supporting -blockdev which includes full
backing chain tracking so the implementation of deletion of the images
obsoleted with block commit will become much simpler [*].

Peter

[*] I'm not promising to implement that feature soon though.

[Lentes, Bernd]

Post by Eric Blake

Post by Lentes, Bernd
Hi,
currently i'm following
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit. I 'm
playing around with it and it seems to be quite nice.
What i want is a daily consistent backup of my image file of the guest.
- Shutdown the guest (i can live with a downtime of a few minutes, it will
happen in the night).
And i think it's the only way to have a real clean snapshot

Not the only way - you can also use qemu-guest-agent with a trusted
guest to quiesce all filesystem I/O after freezing database operations
at a consistent point, for a clean snapshot of a live guest. But
shutting down is indeed safe, and easier to reason about than worrying
whether your qga interaction is properly hooked into all necessary
places for a live quiesce.

Post by Lentes, Bernd
- create a snapshot with snapshot-create-as: snapshot-create-as guest testsn --disk-only
- start the guest again. Changes will now go into the overlay, as e.g. inserts
in a database
- rsync the base file to a cifs server. With rsync not the complete, likely big
file is transferred but just the delta

We're also trying to add support for incremental backups into a future
version of libvirt on top of the qemu 3.0 feature of persistent bitmaps
in qcow2 images, which could indeed guarantee that you transfer only the
portions of the guest disk that were touched since the last backup. But
as that's still something I'm trying to code up, your solution of using
rsync to pick out the deltas is as good as anything you can get right now.

Post by Lentes, Bernd
- blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait
--verbose --pivot
- delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata
- remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?

Yep, that's the right way to do it! It's perfectly safe - the guest
doesn't care whether it is reading/writing from the backing file or the
overlay, and even if the blockcommit action is aborted, you can restart
it for the same effects.
(Okay, if you want to get technical, you need to know that committing
from 'Base <- mid <- top' down to 'Base' leaves 'mid' in an inconsistent
state - but that's not something the guest can see through 'top'; and
your specific case is just committing to the immediate backing layer
rather than skipping layers like that).

Post by Lentes, Bernd
It's possible that during the rsync, when the guest is running, some inserts are
done in a database.

As long as the backing file is read-only during the rsync (which it is,
since all your guest writes are going to the overlay), then nothing the
guest can do will interfere with what rsync can see. Just be sure that
you don't start the blockcommit until after rsync is done.

Post by Lentes, Bernd
Is it safe to copy the new sectors (i assume that's what blockcommit does) under
a running database ?
Or is it only safe doing blockcommit on a stopped guest ?

Live blockcommit is safe, and exists so you don't have to stop the guest.
For a bit more insight into what is going on under the hood, the slides
http://events17.linuxfoundation.org/sites/events/files/slides/2015-qcow2-expanded.pdf

Post by Lentes, Bernd
Thanks for any answer.
Bernd
P.S. Is the same procedure possible when the guest disk(s) reside directly in a
plain logical volume, without a file system in-between ?

Live blockcommit works onto any host storage protocol (whether
filesystem, block device via LVM, or even remote access such as NBD or
ceph). The key is that your overlay is a qcow2 file that is tracking
the deltas during the time in which you are capturing your backup of the
backing file, and then blockcommit safely writes those deltas back into
the backing file prior to no longer needing the overlay.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. [phone]
Virtualization: qemu.org | libvirt.org

Hi Eric,

a big thanks to this outstanding clear and thorough answer.
It's a pleasure to get such information from the developers themselves, and so quick !

Bernd


Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias H. Tschoep, Heinrich Bassler, Dr. rer. nat. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671

[Lentes, Bernd]

Post by Eric Blake

Post by Lentes, Bernd
Hi,
currently i'm following
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit. I 'm
playing around with it and it seems to be quite nice.
What i want is a daily consistent backup of my image file of the guest.
- Shutdown the guest (i can live with a downtime of a few minutes, it will
happen in the night).
And i think it's the only way to have a real clean snapshot

Not the only way - you can also use qemu-guest-agent with a trusted
guest to quiesce all filesystem I/O after freezing database operations
at a consistent point, for a clean snapshot of a live guest. But
shutting down is indeed safe, and easier to reason about than worrying
whether your qga interaction is properly hooked into all necessary
places for a live quiesce.

Post by Lentes, Bernd
- create a snapshot with snapshot-create-as: snapshot-create-as guest testsn
--disk-only
- start the guest again. Changes will now go into the overlay, as e.g. inserts
in a database
- rsync the base file to a cifs server. With rsync not the complete, likely big
file is transferred but just the delta

We're also trying to add support for incremental backups into a future
version of libvirt on top of the qemu 3.0 feature of persistent bitmaps
in qcow2 images, which could indeed guarantee that you transfer only the
portions of the guest disk that were touched since the last backup. But
as that's still something I'm trying to code up, your solution of using
rsync to pick out the deltas is as good as anything you can get right now.

Post by Lentes, Bernd
- blockcommit the overlay: blockcommit guest /path/to/testsn --active --wait
--verbose --pivot
- delete the snapshot: snapshot-delete guest --snapshotname testsn --metadata
- remove the overlay
Is that ok ? How "safe" is blockcommit on a running guest ?

Yep, that's the right way to do it! It's perfectly safe - the guest
doesn't care whether it is reading/writing from the backing file or the
overlay, and even if the blockcommit action is aborted, you can restart
it for the same effects.
(Okay, if you want to get technical, you need to know that committing
from 'Base <- mid <- top' down to 'Base' leaves 'mid' in an inconsistent
state - but that's not something the guest can see through 'top'; and
your specific case is just committing to the immediate backing layer
rather than skipping layers like that).

Post by Lentes, Bernd
It's possible that during the rsync, when the guest is running, some inserts are
done in a database.

As long as the backing file is read-only during the rsync (which it is,
since all your guest writes are going to the overlay), then nothing the
guest can do will interfere with what rsync can see. Just be sure that
you don't start the blockcommit until after rsync is done.

Post by Lentes, Bernd
Is it safe to copy the new sectors (i assume that's what blockcommit does) under
a running database ?
Or is it only safe doing blockcommit on a stopped guest ?

Live blockcommit is safe, and exists so you don't have to stop the guest.

Hi,

i thought i got the procedure i needed, but a problem arise.
My guests are resources in a pacemaker cluster. I start/stop the guests through the cluster.
But when i stop the guest via pacemaker, libvirt doesn't know the guest any longer.
After a successfull stop "virsh list --all" doesn't show any guest. So i can't take a sanpshot with libvirt.
What is about qemu-img ? Could i still use my procedure ?
- stop the guest via cluster
- snapshot with qemu-img
- start the guest via cluster
- rsync the snapshot to a file server
- commit the snapshot with qemu-img to a running guest (i think this will be the problem)

Thanks for any thoughts.

Bernd


Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: NN
Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter
Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Dr. rer. nat. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671