Since C++17 the order of evaluation is specified such that the operands of = are evaluated right-to-left and those of << are evaluated left-to-right, matching the associativity of these operators. (But this doesn't apply to all operators, e.g. + and other arithmetic operators.)
So in
addOne(x) = x;
first the value of the right-hand side is evaluated, yielding 5. Then the function addOne is called and it doesn't matter what it does with x since it returns a reference to it, to which the right-hand value 5 is assigned.
Formally, evaluating the right-hand side first means that we replace the lvalue x by the (pr)value it holds (lvalue-to-rvalue conversion). Then we call addOne(x) to modify the object that the lvalue x refers to.
So, imagining temporary variables to hold the results of the individual evaluations, the line is equivalent to (except for extra copies introduced by the new variables, which don't matter in the case of int):
int t = x;
int& y = addOne(x);
y = t; // same as x = t, because y will refer to x
Then in the line
std::cout << x << ' ' << addOne(x);
we first evaluate and output x, resulting in 5, and then call addOne, resulting in 6.
So the line is equivalent to (simplified, knowing that operator<< will return std::cout again):
int t1 = x;
std::cout << t1 << ' ';
int t2 = addOne(x);
std::cout << t2;
The output 5 6 is the only correct one since C++17.
Before C++17, the evaluation order of the two sides of the assignment operator was unsequenced.
Having a scalar modification unsequenced with a value computation on the same scalar (on the right-hand side of your assignment) causes undefined behavior normally.
But since you put the increment of x into a function, an additional rule saying that the execution of a function body is merely indeterminately sequenced with other evaluations in the calling context saves this. It means that the line wont have undefined behavior anymore, but the order in which the evaluations of the two sides of the assignment happen could be either left-first or right-first.
This means we won't know whether x is evaluated first and then addOne(x) or the other way around.
Therefore after the line, x may be 5 or 6.
6 would be obtained if the evaluation happened equivalently to
int& y = addOne(x);
int t = x;
y = t;
Then in the line
std::cout << x << ' ' << addOne(x);
pre-C++17 the same issue applied. The evaluations of the arguments to << were indeterminately sequenced, rather than left-to-right and so addOne(x) could be evaluated before the left-hand x, i.e. in addition to the previous order, the evaluation could also be equivalent to
int t2 = addOne(x);
int t1 = x;
std::cout << t1 << ' ' << t2;
In this case x is first incremented and then its new value is printed twice.
Therefore possible program output could be either of the following:
5 6
6 6
6 7
7 7
(Technically the int t2 = addOne(x) are two evaluations: One call to addOne returning a reference and then the lvalue-to-rvalue conversion. These could happen interleaved with the other evaluations, but this doesn't give any new program outputs.)
You can specify to use C++17 (or newer versions like C++20) with the -std=c++17 flag to the compiler if you are using GCC or Clang and /std:c++17 if you are using MSVC. Which standard version is chosen by-default depends on the compiler and compiler version.
