First, I would try something like adding this directive, and testing a 404 to see if it works:
ErrorDocument 404 "Sorry, no file here"
If you don't get that error, then in all likelihood your .htaccess file isn't being processed. You need to make sure that it's enabled on your server, etc. It is also possible that your ErrorDocument directive is specified somewhere that it isn't being read for the context where you are testing.
If you do get that error, the next thing I would do is make sure that your error documents are where you think they are. You mention that you uploaded them to the root directory of your project, so I would try browsing to them and see what happens.