The problem is happening because you are using algebraic notation in RPN mode. Never do that. In RPN, function arguments are always taken from the stack, and are never put in parentheses after the function (because THAT is algebraic notation).
However, sometimes you have to tell Prime how many arguments to take from the stack. This is done by typing the desired COUNT of arguments (not the arguments themselves) after the function in parentheses. For example, if you want the F1 function to take ONE number off the stack as its argument, then type F1(1). This LOOKS like algebraic notation, but it's not; the "(1)" here doesn't mean "perform F1 on an argument of 1" but rather it means "take 1 number off the stack and perform F1 on it."
In RPN mode, FOO(3) doesn't mean "perform the FOO function on an argument of 3" but rather "perform the FOO function on the bottom 3 levels of the stack." And those 3 levels get "consumed" by the function and removed from the stack, of course.
Functions which always have the same number of arguments don't need to be told how many arguments to take off the stack, e.g. SIN works without needing to type SIN(1). But user-defined functions are unpredictable, so Prime needs you to type F1(1) if F1 has one argument.
Hope that made sense!
Suggestion: If you want to create user-defined functions for use in RPN mode, you can avoid the need to type the number of arguments by using the Define key (Shift x-t-theta-n) instead of the Function app. For example, if you Define 'foo' to be 'X^2+1' then in RPN you can just type foo and it'll take its argument from the stack without you needing to type foo(1).
Disclaimer: I don't work for HP. I'm just another happy HP calculator user.
Thanks a lot for your quick and good answer. I have understood it. I also tried another way which also works: I can enter 'F1(2)' - ENTER and then EVAL. But you are right, I should be very cautious when using RPN after having entered an algebraic equation.