Databases ultimately store data on disk. Database performance is not really affected by disk space and disk usage (except in certain extreme cases, such as running out of disk space or severely fragmenting the disk space, neither of which is an issue for your question).
Database performance is driven by what happens in memory, particularly by the time taken to load data from disk into memory. There are various data caches in memory, such as the page cache. When a page is not available, then the engine has to fetch it from disk -- and that takes time.
A table that just sits around on disk never being used should not use any space in memory. That means that other tables used by other queries can fill up memory, with no problem. Of course, when that table is modified, then one or more pages will be loaded into memory. From what you describe, this would not use up much of the in-memory cache.
The basic answer to your question is that a large table hanging around on disk will not affect performance. Modifications to the table would, of course, affect performance because that work would be done by processors and memory that could be used for other queries.
