The risk-reward relation depends on what you are changing. In the most cases people ask about, it is not linear but I will give examples of both.
Nonlinear case 1: As you diversify your portfolio, the firm-specific risks of various stocks cancel each other out without necessarily affecting the expected return of the portfolio. Reduction in risk without any loss in returns--very nonlinear.
Nonlinear case 2: If you are changing the weights in your portfolio to move along the efficient frontier, then you the risk-reward relation is a hyperbola, which is nonlinear.
Nonlinear case 3: If you are changing the weights in your portfolio to move away from the efficient frontier, then you increase risk without adding a fully compensatory amount of return. There could be many paths along the risk-reward plane, but generally it will not be linear in the sense that it will not be on the same line as your initial, efficient, portfolio and your savings account.
Linear case 1: The most common sense in which we think of the risk-reward relation being linear is when the thing you are changing is the size of your investment. If you take money out of savings to put in your fully diversified portfolio without changing the relative weights, your expected returns will increase linearly.
Linear case 2: If you believe the CAPM, then the expected return of an asset stock is linearly proportional to the market risk of the firm. If you could change the market risk of a single asset without changing anything else, then you would linearly change its expected return.
The general rule about the risk/reward relation is this: If you are changing the size of your investment, the relation is linear. If you are changing its composition, the relation is nonlinear