Is it rude to refuse tea?

Offering it means you're being received as a guest. Accepting means you recognise the relationship. It creates a shared pause — business may follow, or it may not.

After the first glass, you can leave without offence. Before the first glass is finished — that's rude. Drinking all three glasses and buying nothing is completely normal.