Year Calculation Returns Incorrect Result from Date Subtraction

For any Java dates d1 and d2 my requirement is to get the Ceiling of Years in between, without accounting for time (day only).

Examples:

May-01-2017 -> Dec-30-2021:  YR = 5 (4 + remainder)
May-01-2017 -> Apr-30-2021:  YR = 4
May-01-2017 -> May-01-2021:  YR = 4 (exact)

For some reason, for the 3rd case above, 5/1/17 -> 5/1/21, I get 5 years instead of 4. The reason is, the # of days is 1461 instead of 1460.

SimpleDateFormat sdf = new SimpleDateFormat("MM/dd/yyyy");
Date d1 = sdf.parse("05/01/2017");
Date d2 = sdf.parse("05/01/2021");

int daysDifference = getDaysBetweenWithoutTime(d1, d2); 
int yearCount =  (int)Math.ceil(((float)daysDifference)/365);   
// This returns YearCount = 5


// Get Days Between without Time
    private int getDaysBetweenWithoutTime(Date d1, Date d2)
    {
        SimpleDateFormat dateFormat = new SimpleDateFormat("MM/dd/yyyy");
        int diff = -1;

        try
        {
            Date dateStartWithoutTime = dateFormat.parse(dateFormat.format(d1));
            Date dateEndWithoutTime = dateFormat.parse(dateFormat.format(d2));

            diff = (int) ((dateEndWithoutTime.getTime() - dateStartWithoutTime.getTime()) / 
                     (1000 * 60 * 60 * 24));
        }
        catch (ParseException pe)
        {
            // Empty
        }

        return diff;
    }