Gradle taking time up front seemingly doing nothing

Gradle seem to be really slow for me and I have no idea why. Whenever I run a build it takes like 30 seconds for me, according to the output 25 seconds of those it's doing nothing.

Building like this:

gradlew build --parallel --offline

This is happening to me now with Kotlin, it was the same when I was using Java only, incremental builds don't seem to do anything either

EDIT: I have the gradle daemon enabled

Every subsequent build right at the start takes as long or longer than starting the daemon apparently doing nothing

This is my Project: https://github.com/forsakenharmony/GameProt