After few days of troubling results and re-designing the whole RF board I finally have some good news. A completely changed tx mixer, outputs a signal that more or less represent a sine wave. The screen is from actual tx mixer output, before any filtering or smoothing:
This is 3.5 Mhz + tuning offset pure carrier, no filters on the scope, also using lower frequency as the bandwidth of my scope is quite limited (15 years old machine).
Original schematics used 74LVC4066 in balanced mode, with 4 phase quadrature clock, but I was not able to tune it. I had to go for the most commonly used design for such mixer:
But it is now clear why this design is the most popular – fewer parts and stable operation in 1-30Mhz range. It is a winner!