Yes, the difference between start and finish is what is important. That's why I show the start-finish difference in my chart using the two different sources.
I never found the elevations given by my GPS unit very reliable, either in absolute elevation or in elevation difference between two points.