If you are talking about the agent.exe then this could be working as designed depending on the components you have selected to be installed (like EPS and AV).
However:
One thing to check on the core would be the \AgentBehaviors folder on the core for a LDtmp folder with a lot of tmp files. You can delete this tmp files and then rebuild the agent, that might also help with size.