I think I first heard the concerto (at least, I first really listened to it) when I was home from college and heard the Cleveland Philharmonic, a semi-professional orchestra in which my uncle Frank Griesinger played clarinet, with a distinguished Cleveland surgeon, Dr. Jerome Gross, playing the solo.
Gross was a great amateur, an amateur who could really play. He owned a Stradivarius (which he kindly let me play once), and played chamber music with George Szell. As I remember it, it was a pretty good rendition, probably as good as in Brahms's time, though we've raised the bar since then.
Then, and now, the part that got me was right after the cadenza in the first movement. A cadenza in the late Romantic, of course, has to be agitated, virtuosic, dazzling. And it is in Brahms, whichever version of the cadenza anyone plays (there are many versions). Certainly it was, when Julia Fischer played it. Fast notes in triple and duple and quadruple and sextuples; the whole range of the fiddle; double stops and even little fugues; all from one little instrument. How could it all come from those four strings, I think? Amazing: playing on the edge of technique the whole way through.
And then the orchestra comes back in, and Brahms does the most amazing thing. Bravado yields to utter calm. The orchestra, barely a presence, supports the violinist as she plays the most sublime, high, quiet passage. It's just the very simple theme that starts the movement, as simple as a melody can get, just notes from the D major triad. D, then up to the F sharp, back to the D, past a little off-note, the B, down the triad to A and F sharp, exactly what you'd write if you didn't really feel confident modulating out of the home key.
Then come those very high, simple, D major triad notes of the theme in the solo violin. No tricks at all. The orchestra (the string parts are below the solo violin in this example) are to play pianissimo, super-quiet. Yes, they do some interesting things (those sharps indicate that they're not really sticking to the D major triad), but mostly we listen to the soloist's simplicity.
Quiet, nothing new, nothing flashy (though, come to think of it, way up there on the fingerboard and not so easy to keep in tune -- and woe betide you if you let it get out of tune, for everyone would notice!).
As T. S. Eliot writes in a mystical vein, it might be the axis of eternity, "the still point of the turning world." Or as Wordsworth writes, also feeling the mystery of quietude, it's a point at which "we see into the life of things." A moment of great change, but change without the fanfare that sometimes we insist upon.