What happens when setting the O_CREAT bit but only providing two parameters for open()?

Question

Here, I was asking about an assignment of implementing the sys_open syscall for OS161 which will be called by the open function.

The definition of the open function is as follows:

int open(const char *filename, int flags, ...);

and the man page looks like this:

Synopsis

#include <unistd.h>
#include <fcntl.h>

int
open(const char *filename, int flags);
int
open(const char *filename, int flags, mode_t mode);

Since it's a variadic function, it can take as many arguments as passed, but only the cases with 2 or 3 arguments are defined.

My question was how to determine the number of arguments passed to open, is it 2 or 3. The answer way that if the O_CREAT bit is set in the flags, then a mode is provided.

But I've seen some code within the OS that looks like this:

fd = open(filename, O_WRONLY|O_CREAT|O_TRUNC);

In this case, the value of the mode would be whatever was in the a2 register and not a valid value.

1 - What happens in this case and what should I do in such case? How the arguments are passed? Does it pass a zero in the a2 register (not likely to happen)? Or does it just leave whatever value was in the register? At any case, how can I determine whether the last argument is valid or not?

I used to let the sys_open function receive 3 arguments and only use the last one (mode) if the O_CREAT bit is set. But now since the function can be called with the bit set and with no third argument, I can still use the value of mode even if it's invalid.

2 - What should the logic be for handling the arguments?

The way the syscall is issued is detailed in the question mentioned above.

Answer 1

The POSIX documentation simply states that the:

mode shall be set to the value of the argument following the oflag argument taken as type mode_t modified as follows: ...

It does not specify what happens when you omit that parameter so you would be well advised to ensure it's there (on the client side). The most likely case will be that it will use what is where the parameter would be (whether in register or memory, depending on calling convention), and the chances of this being the correct value would be slim.

---

The Linux documentation has this to say on the matter (my emphasis):

The mode argument must be supplied if O_CREAT or O_TMPFILE is specified in flags; if it is not supplied, some arbitrary bytes from the stack will be applied as the file mode.

Now, granted, that's not a standards document where the term "must" is a solid requirement that you do it (or else all bets are off as to the effect), but you should probably read it in that way. Hence, again, you should make sure you supply it.

So I would say that the way you handle it is as follows:

• If on the client/caller side, make sure you supply the mode. Anything else is breaking the contract.
• On the server/callee side, assume that the caller has supplied it since, if they haven't, they're breaking the contract. There appears to be no suitable error return in POSIX (in either the "shall fail" or "may fail" section) that would be suitable for indicating this issue, even if you could detect it.