[libvirt-users] live migration and config

Post by Dmitry Melekhov
After some mistakes yesterday we ( me and my colleague ) think that it
will be wise for libvirt to check config file existence on remote side

Which config file?

Post by Dmitry Melekhov
and through error if not,
before migrating, otherwise migration will fail and VM fs can be
damaged, because it is sort of remove of power plug...
We missed twice yesterday :-(
Could you tell me is there already such option or any plans to
implement this?

You didn't really say anything about what you did and what happened.
Thus it's impossible to help you.

Jirka

Dmitry Melekhov

2018-09-13 14:38:57 UTC

Post by Dmitry Melekhov
After some mistakes yesterday we ( me and my colleague ) think that it
will be wise for libvirt to check config file existence on remote side

Which config file?

VM config file, namely qemu.
We forgot to mount shared storage (namely gluster volume), on which we
have config files, twice...
And tried to live migrate VM to another host, where there is no config
file, so VM can't be started,
so it died. Twice :-(

You didn't really say anything about what you did and what happened.
Thus it's impossible to help you.

I hope I explained.

Thank you!

Post by Jiri Denemark
Jirka

Jiri Denemark

2018-09-13 14:57:16 UTC

Post by Dmitry Melekhov
After some mistakes yesterday we ( me and my colleague ) think that it
will be wise for libvirt to check config file existence on remote side

Which config file?

VM config file, namely qemu.

Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ?

Post by Dmitry Melekhov
We forgot to mount shared storage (namely gluster volume), on which we
have config files, twice...

If you're talking about the domain XML file, then you should never store
them on a shared storage. All XML files in /etc/libvirt are supposed to
be local to each host. Sharing them may cause changes done on one host
will not be seen by libvirtd on the other host and they can even get
lost when libvirtd on the other host decides to write its version of the
domain definition to the file.

And migration is designed so that the domain definition does not need to
be present on the destination host. In fact, even if it's present,
libvirtd will ignore it until you stop the migrated domain and start it
again.

Post by Dmitry Melekhov
And tried to live migrate VM to another host, where there is no config
file, so VM can't be started, so it died. Twice :-(

Error messages or logs would help. Otherwise we have no clue what failed
and why.

Jirka

Dmitry Melekhov

2018-09-13 15:37:00 UTC

Post by Dmitry Melekhov
After some mistakes yesterday we ( me and my colleague ) think that it
will be wise for libvirt to check config file existence on remote side

Which config file?

VM config file, namely qemu.

Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ?

Yes and no, xml, but at another location.

Post by Dmitry Melekhov
We forgot to mount shared storage (namely gluster volume), on which we
have config files, twice...

If you're talking about the domain XML file, then you should never store
them on a shared storage. All XML files in /etc/libvirt are supposed to
be local to each host.

They are not stored in /etc/libvirt , but in other location, which is
suitable for sharing.

Post by Jiri Denemark
Sharing them may cause changes done on one host
will not be seen by libvirtd on the other host and they can even get
lost when libvirtd on the other host decides to write its version of the
domain definition to the file.
And migration is designed so that the domain definition does not need to
be present on the destination host. In fact, even if it's present,
libvirtd will ignore it until you stop the migrated domain and start it
again.

Sounds strange, because if there is no config file on target host
migration fails, always.

Post by Dmitry Melekhov
And tried to live migrate VM to another host, where there is no config
file, so VM can't be started, so it died. Twice :-(

Error messages or logs would help. Otherwise we have no clue what failed
and why.

Unfortunately, my colleague left today for 2 weeks vacations and I have
no access to these hosts,
so I will be able to look in logs or do some tests only in October.
So, I'll write there again after 2 weeks or so.

Thank you!

Jiri Denemark

2018-09-14 11:43:27 UTC

Post by Dmitry Melekhov
After some mistakes yesterday we ( me and my colleague ) think that it
will be wise for libvirt to check config file existence on remote side

Which config file?

VM config file, namely qemu.

Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ?

Yes and no, xml, but at another location.

Post by Dmitry Melekhov
We forgot to mount shared storage (namely gluster volume), on which we
have config files, twice...

If you're talking about the domain XML file, then you should never store
them on a shared storage. All XML files in /etc/libvirt are supposed to
be local to each host.

They are not stored in /etc/libvirt , but in other location, which is
suitable for sharing.

I'm even more confused now. How could libvirt even know about such
files? Unless you use transient domains, i.e., point libvirt to the
right XML when starting the domain.

Post by Dmitry Melekhov
Unfortunately, my colleague left today for 2 weeks vacations and I have
no access to these hosts,
so I will be able to look in logs or do some tests only in October.
So, I'll write there again after 2 weeks or so.

Yeah, it would be nice if you could share the actual domain XML and the
command you used for migrating it to the other host, too.

Jirka

Dmitry Melekhov

2018-09-14 12:00:43 UTC

Post by Dmitry Melekhov
After some mistakes yesterday we ( me and my colleague ) think that it
will be wise for libvirt to check config file existence on remote side

Which config file?

VM config file, namely qemu.

Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ?

Yes and no, xml, but at another location.

Post by Dmitry Melekhov
We forgot to mount shared storage (namely gluster volume), on which we
have config files, twice...

If you're talking about the domain XML file, then you should never store
them on a shared storage. All XML files in /etc/libvirt are supposed to
be local to each host.

They are not stored in /etc/libvirt , but in other location, which is
suitable for sharing.

I'm even more confused now. How could libvirt even know about such
files? Unless you use transient domains, i.e., point libvirt to the
right XML when starting the domain.

Yes, it transient, really, we use pacemaker's VirtualDomain
http://www.linux-ha.org/wiki/VirtualDomain_(resource_agent)

pacemaker creates domain and destroys it.

As I can see from sources migration looks like:
virsh ${VIRSH_OPTIONS} migrate --live $migrate_opts $DOMAIN_NAME
$remoteuri $migrateuri &

Don't know which exact options it uses though.

Do we need xml on another host if domain is transient for live migration?

Yeah, it would be nice if you could share the actual domain XML and the
command you used for migrating it to the other host, too.

Don't sure I can do this, I know English a little bit ( my colleague
can't write ;-) ), but this is his cluster, not my, so I prefer to wait
2 weeks,
not try to request access....

Thank you!

Jiri Denemark

2018-09-14 12:15:02 UTC

Post by Dmitry Melekhov
After some mistakes yesterday we ( me and my colleague ) think that it
will be wise for libvirt to check config file existence on remote side

Which config file?

VM config file, namely qemu.

Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ?

Yes and no, xml, but at another location.

Post by Dmitry Melekhov
We forgot to mount shared storage (namely gluster volume), on which we
have config files, twice...

If you're talking about the domain XML file, then you should never store
them on a shared storage. All XML files in /etc/libvirt are supposed to
be local to each host.

They are not stored in /etc/libvirt , but in other location, which is
suitable for sharing.

I'm even more confused now. How could libvirt even know about such
files? Unless you use transient domains, i.e., point libvirt to the
right XML when starting the domain.

Yes, it transient, really, we use pacemaker's VirtualDomain
http://www.linux-ha.org/wiki/VirtualDomain_(resource_agent)
pacemaker creates domain and destroys it.
virsh ${VIRSH_OPTIONS} migrate --live $migrate_opts $DOMAIN_NAME
$remoteuri $migrateuri &
Don't know which exact options it uses though.
Do we need xml on another host if domain is transient for live migration?

No, the XML is only needed when starting the domain. At the beginning of
migration libvirt transfers the XML to libvirtd on the other host so
that QEMU with the right parameters can be started there.

Just to make sure, the disk images used by the domain are stored on a
shared storage accessible from both hosts, right?

Jirka

Dmitry Melekhov

2018-09-14 12:24:27 UTC

Post by Dmitry Melekhov
After some mistakes yesterday we ( me and my colleague ) think that it
will be wise for libvirt to check config file existence on remote side

Which config file?

VM config file, namely qemu.

Do you mean the domain XML definition found in /etc/libvirt/qemu/*.xml ?

Yes and no, xml, but at another location.

Post by Dmitry Melekhov
We forgot to mount shared storage (namely gluster volume), on which we
have config files, twice...

If you're talking about the domain XML file, then you should never store
them on a shared storage. All XML files in /etc/libvirt are supposed to
be local to each host.

They are not stored in /etc/libvirt , but in other location, which is
suitable for sharing.

I'm even more confused now. How could libvirt even know about such
files? Unless you use transient domains, i.e., point libvirt to the
right XML when starting the domain.

No, the XML is only needed when starting the domain. At the beginning of
migration libvirt transfers the XML to libvirtd on the other host so
that QEMU with the right parameters can be started there.

Well, then it is really strange...

Post by Jiri Denemark
Just to make sure, the disk images used by the domain are stored on a
shared storage accessible from both hosts, right?

Yes, we use gluster and connection over gfapi, so VM storage is
accessible from all hosts.
Usually, when xml are available, migration works without problems.

Just thought - it is possible that problem is in pacemaker's
VirtualDomain and it needs xml on all hosts,
there is such comment :

# During a probe, it is permissible for the config file to not be
# readable (it might be on shared storage not available during the
# probe). In that case, we're
# unable to get the domain name. Thus, we also can't check whether the
# domain is running. The only thing we can do here is to assume that
# it is not running.

We have to check, but looks like we have to ask in pacemaker mail list,
it has to check if xml is available.

Thank you!

Dmitry Melekhov

2018-10-04 05:22:12 UTC