[libvirt-users] Permission problem with /dev/net/tun

[Thomas Karcher]

Hi lxc folks,

the symptom my libvirt LXC container suffers from is:

***@depot:/dev/net# ls -la
total 0
drwxr-xr-x 2 root root 40 Jun 29 16:26 .
drwxr-xr-x 5 root root 480 Jun 29 16:26 ..

***@depot:/dev/net# mknod tun c 10 200
mknod: `tun': Operation not permitted

The host is an up-to-date AMD64 Ubuntu raring on 3.8.0-25-generic that
was formerly installed from precise and then upgraded. The guest is
Ubuntu precise; however, I see the same symptom in another raring
container on the same host.

What I tried to resolve this:

1) On the host, I echoed various stuff to the cgroup device files:

cd /sys/fs/cgroup/devices/libvirt/lxc
echo "c 10:200 rwm" > devices.allow
echo "c 10:200 rwm" > depot/devices.allow
echo a > depot/devices.allow

... and I see the successful results in depot/devices.list, but no
success.

2) I inserted a line "/dev/net/tun rwk," into
/etc/apparmor.d/abstractions/lxc/container-base - no change. (I know,
it seems kind of pointless - because it's about permissions to a
device, not a path. You may deduce my desperation from this ...

SELinux is not active. Mounts on /dev look normal to me:

devfs on /dev type tmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
devpts on /dev/ptmx type devpts
(rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)

Now comes the weird part: Once I umount /dev inside the container, the
"hidden" /dev appears which contains a usable /dev/net/tun. So the
mknod problem is probably due to the dropped capabilities - but
why/how mounts the container a more restricted /dev on top of the
prepared one ...?


Thanks & regards
Thomas

[Daniel P. Berrange]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi lxc folks,
total 0
drwxr-xr-x 2 root root 40 Jun 29 16:26 .
drwxr-xr-x 5 root root 480 Jun 29 16:26 ..
mknod: `tun': Operation not permitted
The host is an up-to-date AMD64 Ubuntu raring on 3.8.0-25-generic that
was formerly installed from precise and then upgraded. The guest is
Ubuntu precise; however, I see the same symptom in another raring
container on the same host.
cd /sys/fs/cgroup/devices/libvirt/lxc
echo "c 10:200 rwm" > devices.allow
echo "c 10:200 rwm" > depot/devices.allow
echo a > depot/devices.allow
... and I see the successful results in depot/devices.list, but no
success.
2) I inserted a line "/dev/net/tun rwk," into
/etc/apparmor.d/abstractions/lxc/container-base - no change. (I know,
it seems kind of pointless - because it's about permissions to a
device, not a path. You may deduce my desperation from this ...
devfs on /dev type tmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
devpts on /dev/ptmx type devpts
(rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
Now comes the weird part: Once I umount /dev inside the container, the
"hidden" /dev appears which contains a usable /dev/net/tun. So the
mknod problem is probably due to the dropped capabilities - but
why/how mounts the container a more restricted /dev on top of the
prepared one ...?

Allowing the container direct access to the hosts' /dev would be
a security flaw, so libvirt sets up a private /dev for the container.
Allowing the container to use mknod would also be insecure, so we
blocking mknod using both cgroups device ACL, and also droping the
CAP_MKNOD capability.

http://libvirt.org/drvlxc.html#devnodes

Any device that the container is authorized to access per the XML
configuration, will be pre-created in the container's /dev. To
explicitly allow /dev/net/tun you need to tell libvirt about it.

http://libvirt.org/formatdomain.html#elementsHostDevCaps


Daniel

[Thomas Karcher]

Hi Daniel,

Jun 29 16:26 . drwxr-xr-x 5 root root 480 Jun 29 16:26 ..
not permitted

Allowing the container direct access to the hosts' /dev would be a
security flaw, so libvirt sets up a private /dev for the
container. Allowing the container to use mknod would also be
insecure, so we blocking mknod using both cgroups device ACL, and
also droping the CAP_MKNOD capability.
http://libvirt.org/drvlxc.html#devnodes

Good to know.

Any device that the container is authorized to access per the XML
configuration, will be pre-created in the container's /dev. To
explicitly allow /dev/net/tun you need to tell libvirt about it.
http://libvirt.org/formatdomain.html#elementsHostDevCaps

Thanks!

I extended the 'devices' section as follows:

<hostdev mode='capabilities' type='misc'>
<source>
<char>/dev/net/tun</char>
</source>
</hostdev>

... because even though /dev/net/tun is used for networking, it
appears as a character device. (Btw: The documentation says in the
hostdev section: ''For block/character device passthrough mode is
always "capabilities" and type is "block" for a block device, "char"
for a character device and "net" for a host network interface.'' When
I specify type='char', I get an error from virsh.)

With this XML, I can define the container. But upon start, I get the
following error message:

Fehler: internal error guest failed to start: PATH=/bin:/sbin
TERM=linux container=lxc-libvirt
container_uuid=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_UUID=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_NAME=depot /sbin/init
2013-07-08 21:36:50.735+0000: 1: info : libvirt version: 1.0.2
2013-07-08 21:36:50.735+0000: 1: error :
lxcContainerSetupHostdevCapsMisc:1490 : Unable to create device
/dev/net/tun: No such file or directory
2013-07-08 21:36:50.744+0000: 19537: info : libvirt version: 1.0.2
2013-07-08 21:36:50.744+0000: 19537: error : virCommandWait:2287 :
internal error Child process (ip link set veth6 netns 19538)
unexpected exit status 2: RTNETLINK answers: No such process

2013-07-08 21:36:50.786+0000: 19537: error : virCommandWait:2287 :
internal error Child process (ip link del veth4) unexpected exit
status 1: Cannot find device "veth4"

On the host, /dev/net/tun exists as character device:

***@main:~# ls -la /dev/net/tun
crw-rw-rwT 1 root root 10, 200 Jul 8 23:45 /dev/net/tun

What am I doing wrong ...?


Thanks
Thomas

[Gao feng]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Daniel,

Jun 29 16:26 . drwxr-xr-x 5 root root 480 Jun 29 16:26 ..
not permitted

Allowing the container direct access to the hosts' /dev would be a
security flaw, so libvirt sets up a private /dev for the
container. Allowing the container to use mknod would also be
insecure, so we blocking mknod using both cgroups device ACL, and
also droping the CAP_MKNOD capability.
http://libvirt.org/drvlxc.html#devnodes

Good to know.

Any device that the container is authorized to access per the XML
configuration, will be pre-created in the container's /dev. To
explicitly allow /dev/net/tun you need to tell libvirt about it.
http://libvirt.org/formatdomain.html#elementsHostDevCaps

Thanks!
<hostdev mode='capabilities' type='misc'>
<source>
<char>/dev/net/tun</char>
</source>
</hostdev>
... because even though /dev/net/tun is used for networking, it
appears as a character device. (Btw: The documentation says in the
hostdev section: ''For block/character device passthrough mode is
always "capabilities" and type is "block" for a block device, "char"
for a character device and "net" for a host network interface.'' When
I specify type='char', I get an error from virsh.)
With this XML, I can define the container. But upon start, I get the
Fehler: internal error guest failed to start: PATH=/bin:/sbin
TERM=linux container=lxc-libvirt
container_uuid=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_UUID=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_NAME=depot /sbin/init
2013-07-08 21:36:50.735+0000: 1: info : libvirt version: 1.0.2
lxcContainerSetupHostdevCapsMisc:1490 : Unable to create device
/dev/net/tun: No such file or directory
2013-07-08 21:36:50.744+0000: 19537: info : libvirt version: 1.0.2
internal error Child process (ip link set veth6 netns 19538)
unexpected exit status 2: RTNETLINK answers: No such process
internal error Child process (ip link del veth4) unexpected exit
status 1: Cannot find device "veth4"
crw-rw-rwT 1 root root 10, 200 Jul 8 23:45 /dev/net/tun
What am I doing wrong ...?

You are right, it should be char device.

libvirt lxc should create "net" directory atomically for the tun device.

[Gao feng]

This helper function is used to create parent directroy for
the hostdev which will be added to the container. if the
parent directory of this hostdev doesn't exist, the mknod of
the hostdev will fail.

Signed-off-by: Gao feng <***@cn.fujitsu.com>
---
src/lxc/lxc_container.c | 17 +++++++++++++++++
src/lxc/lxc_container.h | 2 ++
2 files changed, 19 insertions(+)

diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c
index c8420db..b954107 100644
--- a/src/lxc/lxc_container.c
+++ b/src/lxc/lxc_container.c
@@ -1544,6 +1544,23 @@ cleanup:
}


+int lxcContainerSetupHostdevCapsMakePath(char *dev)
+{
+ int ret = 0;
+ char *dir = NULL;
+
+ if ((dir = strrchr(dev, '/'))) {
+ *dir = '\0';
+ if ((virFileMakePath(dev) < 0) && (errno != EEXIST))
+ ret = -1;
+
+ *dir = '/';
+ }
+
+ return ret;
+}
+
+
static int lxcContainerSetupHostdevCapsStorage(virDomainDefPtr vmDef ATTRIBUTE_UNUSED,
virDomainHostdevDefPtr def ATTRIBUTE_UNUSED,
virSecurityManagerPtr securityDriver ATTRIBUTE_UNUSED)
diff --git a/src/lxc/lxc_container.h b/src/lxc/lxc_container.h
index ada72f7..f168703 100644
--- a/src/lxc/lxc_container.h
+++ b/src/lxc/lxc_container.h
@@ -63,6 +63,8 @@ int lxcContainerStart(virDomainDefPtr def,

int lxcContainerAvailable(int features);

+int lxcContainerSetupHostdevCapsMakePath(char *dev);
+
virArch lxcContainerGetAlt32bitArch(virArch arch);

#endif /* LXC_CONTAINER_H */

[Gao feng]

Create parent directroy for hostdev atomically when we
start a lxc domain or attach a hostdev to a lxc domain.

Signed-off-by: Gao feng <***@cn.fujitsu.com>
---
src/lxc/lxc_container.c | 42 ++++++++++++++++++++++++++++--------------
src/lxc/lxc_driver.c | 14 ++++++++++++++
2 files changed, 42 insertions(+), 14 deletions(-)

diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c
index b954107..a204789 100644
--- a/src/lxc/lxc_container.c
+++ b/src/lxc/lxc_container.c
@@ -1569,14 +1569,15 @@ static int lxcContainerSetupHostdevCapsStorage(virDomainDefPtr vmDef ATTRIBUTE_U
int ret = -1;
struct stat sb;
mode_t mode;
+ char *dev = def->source.caps.u.storage.block;

- if (def->source.caps.u.storage.block == NULL) {
+ if (dev == NULL) {
virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s",
_("Missing storage host block path"));
goto cleanup;
}

- if (virAsprintf(&src, "/.oldroot/%s", def->source.caps.u.storage.block) < 0) {
+ if (virAsprintf(&src, "/.oldroot/%s", dev) < 0) {
virReportOOMError();
goto cleanup;
}
@@ -1591,19 +1592,25 @@ static int lxcContainerSetupHostdevCapsStorage(virDomainDefPtr vmDef ATTRIBUTE_U
if (!S_ISBLK(sb.st_mode)) {
virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
_("Storage source %s must be a block device"),
- def->source.caps.u.storage.block);
+ dev);
+ goto cleanup;
+ }
+
+ if (lxcContainerSetupHostdevCapsMakePath(dev) < 0) {
+ virReportError(errno,
+ _("Failed to create directory for device %s"),
+ dev);
goto cleanup;
}

mode = 0700 | S_IFBLK;

- VIR_DEBUG("Creating dev %s (%d,%d)",
- def->source.caps.u.storage.block,
+ VIR_DEBUG("Creating dev %s (%d,%d)", dev,
major(sb.st_rdev), minor(sb.st_rdev));
- if (mknod(def->source.caps.u.storage.block, mode, sb.st_rdev) < 0) {
+ if (mknod(dev, mode, sb.st_rdev) < 0) {
virReportSystemError(errno,
_("Unable to create device %s"),
- def->source.caps.u.storage.block);
+ dev);
goto cleanup;
}

@@ -1626,14 +1633,15 @@ static int lxcContainerSetupHostdevCapsMisc(virDomainDefPtr vmDef ATTRIBUTE_UNUS
int ret = -1;
struct stat sb;
mode_t mode;
+ char *dev = def->source.caps.u.misc.chardev;

- if (def->source.caps.u.misc.chardev == NULL) {
+ if (dev == NULL) {
virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s",
_("Missing storage host block path"));
goto cleanup;
}

- if (virAsprintf(&src, "/.oldroot/%s", def->source.caps.u.misc.chardev) < 0) {
+ if (virAsprintf(&src, "/.oldroot/%s", dev) < 0) {
virReportOOMError();
goto cleanup;
}
@@ -1648,19 +1656,25 @@ static int lxcContainerSetupHostdevCapsMisc(virDomainDefPtr vmDef ATTRIBUTE_UNUS
if (!S_ISCHR(sb.st_mode)) {
virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
_("Storage source %s must be a character device"),
- def->source.caps.u.misc.chardev);
+ dev);
+ goto cleanup;
+ }
+
+ if (lxcContainerSetupHostdevCapsMakePath(dev) < 0) {
+ virReportError(errno,
+ _("Failed to create directory for device %s"),
+ dev);
goto cleanup;
}

mode = 0700 | S_IFCHR;

- VIR_DEBUG("Creating dev %s (%d,%d)",
- def->source.caps.u.misc.chardev,
+ VIR_DEBUG("Creating dev %s (%d,%d)", dev,
major(sb.st_rdev), minor(sb.st_rdev));
- if (mknod(def->source.caps.u.misc.chardev, mode, sb.st_rdev) < 0) {
+ if (mknod(dev, mode, sb.st_rdev) < 0) {
virReportSystemError(errno,
_("Unable to create device %s"),
- def->source.caps.u.misc.chardev);
+ dev);
goto cleanup;
}

diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c
index 1a6d086..2661c1b 100644
--- a/src/lxc/lxc_driver.c
+++ b/src/lxc/lxc_driver.c
@@ -3616,6 +3616,13 @@ lxcDomainAttachDeviceHostdevStorageLive(virLXCDriverPtr driver,
goto cleanup;
}

+ if (lxcContainerSetupHostdevCapsMakePath(dst) < 0) {
+ virReportSystemError(errno,
+ _("Unable to create directroy for device %s"),
+ dst);
+ goto cleanup;
+ }
+
mode = 0700 | S_IFBLK;

VIR_DEBUG("Creating dev %s (%d,%d)",
@@ -3720,6 +3727,13 @@ lxcDomainAttachDeviceHostdevMiscLive(virLXCDriverPtr driver,
goto cleanup;
}

+ if (lxcContainerSetupHostdevCapsMakePath(dst) < 0) {
+ virReportSystemError(errno,
+ _("Unable to create directroy for device %s"),
+ dst);
+ goto cleanup;
+ }
+
mode = 0700 | S_IFCHR;

VIR_DEBUG("Creating dev %s (%d,%d)",

[Daniel P. Berrange]

Post by Gao feng
Create parent directroy for hostdev atomically when we
start a lxc domain or attach a hostdev to a lxc domain.
---
src/lxc/lxc_container.c | 42 ++++++++++++++++++++++++++++--------------
src/lxc/lxc_driver.c | 14 ++++++++++++++
2 files changed, 42 insertions(+), 14 deletions(-)

ACK


Daniel

[Thomas Karcher]

Guys,

thank you for your help and your effort! (Didn't try the solution yet,
but I will.)


Thomas

[Daniel P. Berrange]

Post by Gao feng
This helper function is used to create parent directroy for
the hostdev which will be added to the container. if the
parent directory of this hostdev doesn't exist, the mknod of
the hostdev will fail.
---
src/lxc/lxc_container.c | 17 +++++++++++++++++
src/lxc/lxc_container.h | 2 ++
2 files changed, 19 insertions(+)
diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c
index c8420db..b954107 100644
--- a/src/lxc/lxc_container.c
+++ b/src/lxc/lxc_container.c
}
+int lxcContainerSetupHostdevCapsMakePath(char *dev)
+{
+ int ret = 0;
+ char *dir = NULL;
+
+ if ((dir = strrchr(dev, '/'))) {
+ *dir = '\0';

Modifying arguments that are passed into a function is bad
practice, so I'm changing this to strdup the arg thus:


+int lxcContainerSetupHostdevCapsMakePath(const char *dev)
+{
+ int ret = -1;
+ char *dir, *tmp;
+
+ if (VIR_STRDUP(dir, dev) < 0)
+ return -1;
+
+ if ((tmp = strrchr(dir, '/'))) {
+ *tmp = '\0';
+ if (virFileMakePath(dir) < 0) {
+ virReportSystemError(errno,
+ _("Failed to create directory for '%s' dev '%s'"),
+ dir, dev);
+ goto cleanup;
+ }
+ }
+
+ ret = 0;
+
+cleanup:
+ VIR_FREE(dir);
+ return ret;
+}


Daniel