This might be one of those problems that looks really simple on the surface but it's surprisingly complicated to prove. There might be a simpler proof, but I couldn't come up with one.
Let's stick to the "ideally rigid" scenario. At an instinctive level, I think everybody will agree that the friction "ought to be zero". It took me several rounds to find a semi-formal way to prove it (I invite anybody to improve this response with a more formal proof).
I'm going to sketch a proof relying on the principle of Virtual Work. For a system to be in equilibrium, the virtual work from all the forces (due to any infinitesimal motion) must be exactly zero.
The cylinder is unconstrained to rotate in place. The normal forces don't produce any virtual work because they are normal to the virtual movement. The weight doesn't produce virtual work because the center of mass doesn't move under a rotation. The only virtual work comes from the friction forces. However, that means that the virtual work from the friction on the left wall has to exactly cancel the virtual work form the friction from the right side, and that can't happen because both of them act in the opposite direction of the virtual rotation (both virtual works have to be negative). The only explanation is that that the friction forces must be exactly zero.
Now, let's assume for the sake of argument that you know for a fact there is a frictional force. Can't that happen? Yes, but only if the apparatus is no longer ideally rigid. It can happen if the cylinder is not just resting on top of the wedge, but it's actually... wedged in.
Wedging requires that the wedge deforms to make space for the cylinder (and/or the cylinder has to squeeze, which is a harder problem to work with).
In a basic view of the problem, the wedge acts like a loaded spring. The load on the spring (wedge) will be due to the weight of the cylinder. But if we push the cylinder down to wedge it in, then there would be an additional load due to excess potential energy transferred to the wedge.
In that scenario, there will be friction, and the friction is countering the additional restorative force from the wedge trying to return the additional potential energy.
A much simpler version of the same problem is a box on a horizontal surface with friction, pushed against a horizontal spring. The friction between the box and the surface can be calculated as $F=k\times x$ where $x$ is the distance the box moved while being pushed against the spring and $k$ is the stiffness of the spring.
Bonus content:
Hey, look, that's an underdeterminated (unidimensional) static problem but we still managed to solve it!
Being statically underdeterminated doesn't mean that you "cannot know the answer", I mean, the Universe obviously knows the answer somehow... It just means that the laws of static are not enough to give the answer to you. You need more sophisticated tools, and more importantly - you need more information about the system. In the simplified example, the additional information is: "one of the components is an ideal elastic object of stiffness $k$ and the "more sophisticated tool" is "Hooke's law for springs".
How about the original problem?
That's harder, because the wedge is not necessarily a simple spring. How it reacts to the wedging depends on a lot of details about the cylinder, the wedge and the environment. In the real world - even assuming a rigid cylinder - the answer might depend on such things as the exact material used for the wedge, its construction, temperature gradients present at the time, the scale of the forces involved and thickness of the walls of the wedge.
A typical (academic perhaps) version of the problem might be:
The wedge is made of a perfectly isotropic elastic material [the material is expected to follow a generalized version of Hooke's law within the expected range of forces it will be subjected to. Also, it deforms the same way in all directions] of elastic modulus $E$ [a generalized version of $k$ for springs]. The cylinder has a weight of $F$; you may consider the cylinder rigid (because it's rigid enough that its own deformations won't contribute significantly to the value of the force). The cylinder was placed on top of the wedge and then pushed down an additional vertical distance $n$. You also need more details on the geometry of the problem. For example: The walls of the wedge are $h$ thick and the cylinder is $d$ in diameter.
Now we're closer to something manageable.
I'll throw in one more simplification for now: "the walls of the wedge are relatively thin compared to their height" (if the problem is numerical, the measurements bear that). This is commonly called "slenderness" and it's an important simplification that allows us to use comparatively simpler tools to analyze the problem. We also want the wedging distance to be relatively small compared to the dimensions of the apparatus. In these conditions, the two sides of the wedge can be analyzed exactly like cantilever beams.
With that information, an undergrad engineering student can determine the force responding to the deformation of the wedge walls due to the wedging ($F = \epsilon\times 3EI/L^3$. $\epsilon$ is the deformation normal to the wall due to the wedging; $L$ and $I$ can be calculated from the geometry of the problem). After that, if you want, you can then determine the frictional force required by using conventional static methods.
If you can't make the slenderness simplification, you will have to resort to other methods. I would have to look it up, but there should be some approximate formulas available in the literature. Or you might have to use numerical simulations (finite element analysis) to determine the deformation suffered by the wedge. If the wedge is not elastic (or the forces exceed the elastic limit for the material), that throws more complications into the problem. You might get to the point where you have to "build it and check it out in a lab".
I liked your thought processes on these parts of your question; let me try to answer them:
What does this mean physically? If I set this up and measured F and N, would I really get nonreproducible results?
Assuming the wedged-in scenario: maybe. As mentioned, the answer depends a lot on tiny details of the apparatus. If you use the same apparatus every time and you wedge it in the same distance, under the same conditions, and you don't exceed the elastic limit, you should get the same results. As an opposite example, if the cylinder is wedged in and you use untreated wood parts, even if everything else stays the same the results might change by the hour depending on the temperature and humidity fluctuations throughout the day.
Would the results depend on how hard I jammed the cylinder into the corner?
Yes.
The system would then have a memory of how it was prepared, which seems surprising to me, since this information would have to be stored somehow in its physical state, and that would presumably be because there was some microscopic displacement of one surface relative to the other.
The information is stored mostly on the deformation of the wedge walls due to the total force applied. It will be small, but it probably won't be microscopical.
But in theory I can make the surfaces arbitrarily smooth and hard, while still maintaining a nonzero coefficient of static friction. After all, we know that static friction can exist, and even be quite strong, between surfaces such as optically flat glass plates.
The harder you make the materials, the less deformation it will take to store the additional energy. But the amount of friction in our wedged-in example depends only indirectly on that and it's affected by other factors. However, if all the materials are perfectly (ideally) rigid, then jamming is not possible and there won't be friction.
Or does the system actually not have any memory, meaning that if I try to jam the cylinder down harder, it will just spring back into a standard state when I let go?
That's exactly what would happen with an elastic wedge and a null or negligible friction coefficient.
