Integrate Capabilities in Open-Lambda #301

m0mosenpai · 2025-05-15T08:46:52Z

Upon looking at the code (and the workshop slides), I found 3 possible places where we might drop privileges:

Inside freshProc() [src/worker/sandbox/sock.go] right before exec-ing the server.py.
Inside fork_server() [min-image/runtimes/server.py] right before the while loop that forks a new Zygote
Inside start_container() [ min-image/runtimes/server.py] right before forking a Leaf to run the user code.

Some general questions I have:

When and on what basis is either of web_server.py or fork_server.py invoked? (EDIT: I meant web_server() and fork_server() functions in server.py my bad)
I also noticed that the base directory for OL gets initialized with 700 permissions and since we run everything as root, it makes it so that I have to su before I can navigate to the directory and create a lambda function with my code. Is it possible that none of this is run as root right from the beginning itself so this issue doesn't occur either? Or maybe the base directory can be initialized with more relaxed permissions?
Which capabilities do we really need at each step? One plan for finding that out was that maybe we can drop all privileges at each step and strace the system calls (or if we know ones that are needed already) to see which capabilities they need and then iteratively add and test them to see if all things work as expected. Which also brings me to another question
Is there a good way I can test all of my changes? Test cases or things I should look out for?

min-image/runtimes/python/ol.c

tylerharter · 2025-06-08T19:22:23Z

Great progress! Let's do one case at a time, each in different PRs:

How about this first? "right before forking a Leaf to run the user code". But is it before, or after forking?

For you four questions:

I'm a little confused, I don't think we have files named web_server.py or fork_server.py
OL needs root sometimes to manipulate cgroups and namespaces. Maybe there's something to improve here, but probably as a separate effort (after capability dropping)
Unfortunately, creating namespaces and cgroups requires CAP_SYS_ADMIN (https://man.archlinux.org/man/clone3.2.en). CAP_SYS_ADMIN is very broad and is sometimes called the "new root", so we cannot precisely drop to the exact privileges we need. But something is better than nothing, and this is a great step if CAP_SYS_ADMIN gets broken into finer-grained permissions in the future.
You can see some of the CI tests we run (https://github.com/open-lambda/open-lambda/blob/main/.github/workflows/ci.yml#L64) and do those locally. You should eventually add some tests that try to use capabilities that are not allowed (we should see the invocation fail).

tylerharter

Great start!

tylerharter · 2025-06-08T19:23:28Z

min-image/runtimes/python/ol.c

+
+  capList[0] = cap;
+  if (cap_set_flag(caps, CAP_EFFECTIVE, 1, capList, setting) == -1) {
+    cap_free(caps);


One of the nice things about a goto is it gives you a single place to clean things up, like this:

if (caps != NULL)
cap_free(caps);

@tylerharter I actually didn't get what you mean here. I tried to use the goto statement already. Is there an even better way to do it?

You have cap_free in multiples places. What I mean is that cap_free can be after "out:". You'll need to check whether it is NULL before freeing there, of course.

tylerharter · 2025-06-08T19:25:04Z

min-image/runtimes/python/ol.c

@@ -8,6 +9,71 @@
 #include <sys/types.h>
 #include <sys/wait.h>
 #include <seccomp.h>
+#include <sys/capability.h>
+
+static int modify_cap_impl(int cap, int setting) {


To what extent can we just create Python wrappers around the cap API (can_get_proc, cap_set_flag, etc), and then have the logic using it in Python code only?

@tylerharter I couldn't actually find a native way to do this in Python (atleast at a granular level) and then came across the seccomp code in this file. It feels like we should be able to do a lot with the wrappers - like we have good amount of control over how we want to expose these functions to Python. Is there a concern around doing it in this way?

We may have to do it in a different way (or use an external module?) for dropping privileges in sock.go since these would only work in Python.

We only plan to support Python in the forseeable future, and the more code we have in Python (rather than C), the more people will be able to understand and contribute.

@tylerharter I see. Based on my research, I couldn't find any official Python libraries that would give us control over capabilities. There are some community wrappers available but I'm assuming they use similar C code underneath. I'm also concerned that if we rely on them, we could open a channel for untrustworthy libraries/ packages that aren't well maintained or supported. What are your thoughts on that?

tylerharter · 2025-06-08T19:26:42Z

min-image/runtimes/python/server.py

@@ -208,6 +208,7 @@ def main():
            file.write(pid)
            print(f'server.py: joined cgroup, close FD {fd_id}')

+    # drop privileges here


Can we make capability dropping optional, via config, like it is for seccomp? https://github.com/open-lambda/open-lambda/blob/main/min-image/runtimes/python/server.py#L189

@tylerharter I added the command line argument for enabling capabilities as well. However, we'll need to pass the flag on to everywhere we drop capabilities - to check it and drop only if it's true. Wondering what would be the best way to do that.

m0mosenpai · 2025-06-10T16:35:43Z

@tylerharter My bad, I meant to say web_server and fork_server functions in server.py. I edited the original comment. I couldn't understand which one was being invoked when.

Okay sure, that makes sense. I'll start with CAP_SYS_ADMIN then. Thanks for pointing me to the tests.

m0mosenpai · 2025-07-14T21:59:45Z

@tylerharter "How about this first? "right before forking a Leaf to run the user code". But is it before, or after forking?"

Taking this scenario first, I'm not sure exactly which one would be better or what the difference in the impact would be - dropping capabilities before vs after?

tylerharter

Looking good! Looking forward to seeing the next iteration.

tylerharter · 2025-08-01T17:57:08Z

min-image/Dockerfile

@@ -1,9 +1,10 @@
 FROM ubuntu:22.04

+RUN apt-get autoremove


Let's leave this out of this PR. Maybe you can drop me a note about why you want this?

@tylerharter upon setting up OpenLambda for the first time, the DockerFile errored out due to some dependency or cleanup issue and suggested to run this to remove any conflicting stale package in the instance. I don't remember the exact error now but I adding this should make sure none of that happens

tylerharter · 2025-08-01T17:57:31Z

min-image/runtimes/python/ol.c

@@ -1,4 +1,5 @@
-#include <Python.h>
+#include <python3.13/Python.h>
+// #include <Python.h>


Better to delete old code than comment out.

@tylerharter This is only here for testing reasons. I think due to how Python is set up on my system, I have to make that particular change (mentioning the exact version in the include statement) or my C linter complains. I'll remove this in the final iteration

tylerharter · 2025-08-01T17:58:51Z

min-image/runtimes/python/ol.c

@@ -8,6 +9,71 @@
 #include <sys/types.h>
 #include <sys/wait.h>
 #include <seccomp.h>
+#include <sys/capability.h>
+
+static int modify_cap_impl(int cap, int setting) {


We only plan to support Python in the forseeable future, and the more code we have in Python (rather than C), the more people will be able to understand and contribute.

tylerharter · 2025-08-01T18:00:17Z

min-image/runtimes/python/ol.c

+
+  capList[0] = cap;
+  if (cap_set_flag(caps, CAP_EFFECTIVE, 1, capList, setting) == -1) {
+    cap_free(caps);


You have cap_free in multiples places. What I mean is that cap_free can be after "out:". You'll need to check whether it is NULL before freeing there, of course.

tylerharter · 2025-08-01T18:01:44Z

min-image/runtimes/python/server.py

@@ -177,15 +185,16 @@ def main():

    global bootstrap_path

-    if len(sys.argv) < 2:
-        print("Expected execution: chroot <path_to_root_fs> python3 server.py <path_to_bootstrap.py> [cgroup-count] [enable-seccomp]")
+    if len(sys.argv) < 3:


Let's add a TODO to move to named args (in a future PR) now that we're getting more than a couple.

tylerharter · 2025-08-01T18:03:53Z

min-image/runtimes/python/server.py

+    assert return_val == 0
+
+    # set CAP_SYS_ADMIN
+    return_val = ol.modify_caps(21, 1);


Can we avoid hardcoding this? Anyway we can expose some consts (like CAP_SYS_ADMIN) from our ol module to Python?

Wait, aren't we always doing this, regardless of cmd line args?

@tylerharter Right, for that, I was concerned that I would have to pass that boolean flag down the function calls. enable-seccomp bool is only needed in the beginning where it is "enabled" and then that's it. But since we would have to drop privileges at certain points in the code, we would need the drop-capabilities bool value to propagate down.

I left it as it is for now for testing while I thought of a good solution for it or if you have any suggestions.

tylerharter · 2025-08-01T18:08:38Z

min-image/runtimes/python/server.py

        print("    cgroup-count: number of FDs (starting at 3) that refer to /sys/fs/cgroup/..../cgroup.procs files")
-        print("    enable-seccomp: true/false to enable or disables seccomp filtering")
+        print("    enable-seccomp: true/false to enable or disable seccomp filtering")
+        print("    enable-capabilities: true/false to enable or disable capability dropping")


I wonder if we should call it drop-capabilities or something like that? I think it is intuitive that "enable-seccomp" is going to restrict you. But it's less obvious whether "enable-capabilities" is going to give you more capabilities, or turn on logic that limits your capabilities.

That makes sense, I'll change the name

basic code to modify/drop privileges

5d88f1a

m0mosenpai commented May 15, 2025

View reviewed changes

min-image/runtimes/python/ol.c Show resolved Hide resolved

tylerharter requested changes Jun 8, 2025

View reviewed changes

m0mosenpai closed this Jun 10, 2025

m0mosenpai reopened this Jun 10, 2025

add argument to make capabilities optional

e90441f

set CAP_SYS_ADMIN after forking a leaf

a3987d6

tylerharter requested changes Aug 1, 2025

View reviewed changes

tylerharter reviewed Aug 1, 2025

View reviewed changes

Integrate Capabilities in Open-Lambda #301

Are you sure you want to change the base?

Integrate Capabilities in Open-Lambda #301

Conversation

m0mosenpai commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

tylerharter commented Jun 8, 2025

Uh oh!

tylerharter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

m0mosenpai Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

m0mosenpai commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

m0mosenpai commented Jul 14, 2025

Uh oh!

tylerharter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

m0mosenpai commented May 15, 2025 •

edited

Loading

m0mosenpai Jul 14, 2025 •

edited

Loading

m0mosenpai commented Jun 10, 2025 •

edited

Loading