Thanks alot for the responses guys.
My stick is old and isnt the most precise anymore. so I dailed in some deadzones and it seems better,
I will try with the rudder trim, I was trying to trim my ailerons all the time.
glad to hear its not my hardware malfunctioning but that it is suppose to happen
once again, thanks everyone!