A container should have the minimum amount of code / libraries as possible to fulfil it's single purpose.
Reasons for a smaller image:
- Faster in the network (deploy, reschedule, update)
- Increased I/O performance
- Smaller attack surface. Easier to audit.
Many containers are based of debian/buildessentials which is often unnecessary for runtime. Use different containers for building and running.
You don't have to use Dockerfile. Maybe creating a tar with something like buildroot and importing it via docker import. See the talk from Redbeard: Best Practices For Containerized Environments.
Also have a look at Alpine. A minimalist Linux distribution based on busybox, musl-libc, a new package manager called apk (not the Android one) and OpenRC as init system. Some Thoughts on the Use of Alpine Linux in Docker Images.