Use a light-weight base image, most popular is alpine based images.
Specify work directory in Dockerfile and add the application files to that work directory.
Break down copying application resources from host to container to optimize usage of build cache and avoid rebuilds of image layers. This will minimize cache busting.