Browser caching is great, but enough users will reach your website with an empty cache that you must have acceptable performance (whatever that means to you...) for an empty cache. The links that follow give reasons for the problem and quantitative data.
The original 2007 study by the YUI performance team showing that only 80% of requests have cached resources, and 40-60% of unique users experience a page request with a cold cache in a given day.
Facebook reran the YUI study in 2015. It appears that things have gotten marginally better since 2007, but over 40% of unique users are getting at least one cold hit, and 15% of all requests have an cache. Predictably, mobile caches have lower hit rates.
Wide ranging discussion by Steve Souders in 2012 of why caching is not as effective as it could be. Includes that statistic that for users whose caches are full, the typical time to fill is 4 hours of active browsing, or 20 hours of clock time.
Steve Souders asks why 40-60% of unique users have an empty cache at least once per day. The culprit seems to be cache eviction: browsers caches are typically too small, given the space available on desktop machines, and arguably cache eviction policies are no longer appropriate (scripts should be more aggresively cached than images, for instance).
Steve Souders tests the percentage of sites from the HTTP Archive that use any given version of JQuery from the Google Hosted Libraries CDN. There is substantial version fragmentation and no version is referenced by more than 2% of sites. Without consolidation, it seems unlikely that including JQuery through an CDN will create a substantial caching effect.
Souders' 2013 is from 2013 and replicates a 2011 investigation by Steve Webster. It's debatable whether the situation got worse or better in the meantime. On the plus side: more websites loaded some version of JQuery from the Google Hosted Libraries in 2013, and more websites used one of the top 10 most common versions of JQuery. However, the most used version of JQuery (1.4.2 in both years) was less popular in 2013.
This version is quite old: one consequence of relying on the crowd for caching is that you only get the benefits if you use the same version that most people are using, potentially leaving you stuck with an outdated version.