If I understand your situation correctly, you are using the scheduler to produce template transactions in the register, then manually updating the amounts to reflect the actual specifics (tips and so on) before recording the updated transaction to the register. The problem is how to efficiently calculate the updated amounts before saving the transaction.
One of the GnuCash Tips of the Day says:
As you enter amounts in the register, you can use the GnuCash
calculator to add, subtract, multiply and divide. Simply type the
first value, then select '+', '-','*', or '/'. Type the second value
and press Enter to record the calculated amount.
As you have discovered, you can group operands together using brackets as well as using the 4 basic arithmetic operators. However, I don't think that GnuCash allows for higher mathematics in the register. I think that GnuCash is intended as a coordinated system to store transactions created with input by the user or transactions imported using the Import wizards. With some exceptions (eg. the Loan Repayment Calculator), GnuCash isn't intended to perform complex calculations, it's meant to store your numbers.
Given the limited functionality available in the data entry fields of the account register, I can't offer you an elegant solution where GnuCash does all of the work. I can, however, suggest 3 workarounds:
You can eyeball the result of the calculation that Gnucash performs based on the limited formula that you can enter into the amount fields of the register. If the calculated result is 10.63, but you know that the taxation authority requires the amount rounded to a multiple of 5 cents, you can just manually overtype the calculated result with your own figure (10.65). You will probably need to correct an imbalance split as well. (You are probably already doing this.)
You could use an external spreadsheet program to perform all of the rounding and other mathematical functions required, to the point where the spreadsheet finalises the numbers in the correct format. You then either (a) manually transcribe those spreadsheet-calculated figures into your template transaction, or (b) save the worksheet as CSV data and import it into GnuCash as a complete transaction.
If you really, really want GnuCash to do all of the calculations, you can effectively force it to produce a number rounded to the nearest 5 cents by
a. enter your formula into the register AND DIVIDE BY FIVE; eg (1234+tip)*0.011/5
b. press ENTER or TAB to force GnuCash to calculate a RESULT rounded to the nearest cent
c. go back to the previous field and without over-writing multiply RESULT by 5
d. press ENTER or TAB to force GnuCash to calculate the answer to your original calculation, rounded to the nearest 5 cents.
Remember, I warned you up front that this was not going to be an elegant solution!