As technology companies race to release their next features, any delay in productivity can be extremely detrimental, making an efficient development build process essential.
Companies that use Kubernetes and Docker in production environments most likely use Docker for local development. Docker-compose, a tool for defining and running multi-container Docker applications, ensures consistent local development processes, which makes handling application dependencies easier. Docker-compose lets engineers choose the specific versions of dependencies to be run alongside the application container—as opposed to having each engineer install dependencies manually on their machine. Having consistent versions of dependencies is crucial for a fast development cycle as there is little chance of having compatibility issues due to incorrect versions of the dependencies.
DoorDash faced a similar problem when our team had been dealing with a slow build time in our local environment, which had slowed the development process significantly. Whenever any engineer added new code to the codebase, the build integration process would take four to five minutes to finish. It didn’t matter if we were changing 20 files or just adding one line of code to create a quick log, it would still take roughly five minutes to complete the build. Every engineer knows that it's hard to remain focused and productive if any part of the development cycle pauses their thought process.
The problem with our build process
We use Gradle to compile our application then use Docker to run it and its dependencies. However, due to the lack of a uniform build process for local development, we were unable to achieve efficient development.
One problem was that the Dockerfile our team was using was the exact replica of the production Dockerfile. Our production Dockerfile is meant to compile the application from scratch as we're not using the Gradle build cache in production. By using the production Dockerfile locally, we were not taking advantage of the Gradle cache and we were also downloading dependencies that are not needed in our local environment. Figure 1, below, is a representation of the Dockerfile that was causing delays:
Another issue was that the team was using the following Makefile target to run our build process, in which we actually compile our application using a local Gradle instance before we execute the Docker build:
.PHONY: build
build:
git submodule update --init --recursive --remote
./gradlew clean installDist
docker-compose buildThe idea was to use the Gradle cache before we build the entire image for faster build time. However, this part of the process became a waste of effort as we were executing the same command inside the Dockerfile, as shown in Figure 1, above. Basically, our Dockerfile used to have this command:
RUN ./gradlew clean installDist test ktlintCheck --no-daemon && \
mv build/install/* /home/ && \
rm -r buildNot only was this command similar to what we just executed outside the Dockerfile, but it also included a test and format check along with the build. The worst part was that it didn't use the cache from the local machine so it had to re-download all the dependencies of our application.
Solution: Update our build scripts and Dockerfile
We resolved the build time issue by examining our build script and Dockerfile and reorganizing the way we built our application and its dependencies.
How we updated our Dockerfile for local development
- We removed all unnecessary third party applications that we don't use locally. For example, we removed the New Relic agent that we definitely need in production but is not needed in a local environment.
- We removed the Gradle installation. To use Gradle cache properly, we used the Gradle installation in our local machine and built the application outside our docker build.
- Since we're not compiling the application inside the Dockerfile anymore, we had to copy the files from the build context to the appropriate directory inside our image.
Separating build and unit test executions
In our Makefile, we separated our build and test executions to give us more flexibility. The team constantly writes and runs unit tests via IDE during development so there's no need to rerun them for every local build. Also, running every unit test for each local build is not practical, especially if the change in the code is minimal. Of course, every member of the team still runs the complete build—which runs all the tests, checks the format, and builds before committing the changes to our repository.
Made sure that we're using the Gradle cache properly
We enabled the Gradle cache by putting org.gradle.caching=true in our gradle.properties. Since we don't have complex tasks in our build.gradle, this was all we needed to do to efficiently use the Gradle cache. For complex tasks in build.gradle, such as constantly copying or moving files around, optimizing those tasks can help gain the greatest benefit from the Gradle cache.
Results
As shown in Figure 2, below, our Dockerfile is leaner than the previous version (Figure 1). By making these changes, we cut our build time from roughly five minutes to an average of 45 seconds, roughly an 85% decrease.
Considering we execute build commands a couple of times a day, this improvement is a huge win for the team, since it saves us a lot of time and keeps everyone focused on the tasks at hand.
This kind of issue can happen to any team that's new to containerization or to microservices. If managing a Dockerfile, it is always a good idea to revisit it every now and then to look for potential improvements. Dockerfile best practices and Gradle build cache are excellent documentations to gain a better understanding of specific build issues.
