You can profile the filer's memory consumption using massif (part of valgrind). Run it like this:
$ rox --massif -n
Use it a bit to get the memory consumption up and then quit. You should get a .ps and a .txt file showing memory usage. That should tell you where the memory is being allocated.
It's also possible to get plain valgrind to report memory still in use at exit, but the output can be confusing because there are several caches (in GTK and ROX-Filer) which are intentionally not freed.
If you are having trouble understanding the results, put the txt file up on the web somewhere and post a link to the rox-devel mailing list.