The amount of consumed memory is dictated by the remoting technology, and also for RDP, by the Operating System of the remote host. Windows 8 / 2012 sessions do take upwards of 150Mb, whereas the previous versions would consume around half of that.
This is something that we have no control over. You can look at Large Memory Aware Application or use the 64bit edition of Remote Desktop Manager.
This situation is something that we also experience under those conditions (our testing environment is virtualized and has minimal memory for each guests).
In the Experience tab of your RDP session, you need to disable everything that is not used. A good method is to select the Modem connection speed, this will disable all settings that are related to appearance. You can then turn on a single setting to see if it affects the performance negatively. Repeat until you find the combination that works best for you.
1.Disable your Anti-virus and test the connection to see of this make a difference. If you use Trend Micro Business Security, add RemoteDesktopManager.exe, RemoteDesktopManager64.exe, and Embedded32.exe to the Process Exception List under Preferences - Global Settings and updated the Trend client. If you use Symantec Norton Internet Security, ensure you allow Remote Desktop Manager to access the internet in the Program Rules.
2.Uncheck all settings in the Local Resources tab.
There are known security updates that have broken connectivity to a host, but they have been fixed in ulterior patches. If you keep the hosts always up to date then it should be something else.
Sometimes the mapping of local devices will cause issues. Uncheck all settings in the Local Resources tab.