For example in a tree, the water is lifted from the high concentration in the soil to the low concentration higher up in the tree. But at the end of that process the water has been elevated, which should take energy (=mgh), but it seems like it kind of gets lifted for free without spending any energy?
Similarly, dipping a paper towel into a bowl of water, the water "climbs" the towel (by capillary action?) and absorbs upwards, meaning the water was lifted upwards (so gained potential energy) seemingly for free?
What you are describing is not osmosis, it is capillary action. Capillary action is caused by the forces between the water molecules and the molecules of the tube overcoming the force of gravity. You can read more here: https://chem.libretexts.org/Bookshelves/Physical_and_Theoretical_Chemistry_Textbook_Maps/Supplemental_Modules_(Physical_and_Theoretical_Chemistry)/Physical_Properties_of_Matter/States_of_Matter/Properties_of_Liquids/Capillary_Action
Briefly, the water molecules are attracted to the molecules of the tube by adhesive force. The liquid molecules are also attracted to each other by cohesive force. The interplay of these forces causes capillary action.
However, it seems that tree sap moves by more than just capillary action. If you scroll down part way in this book they talk about it a bit: https://pressbooks.online.ucf.edu/phy2053bc/chapter/cohesion-and-adhesion-in-liquids-surface-tension-and-capillary-action/
So is it ultimately down to electromagnetic attraction on the microscopic scale?
Generally, yes, but in the case of trees there is also negative pressure (vacuum) exerted from transpiration when water leaves the top of the tree and "pulls" other water up behind it.