I was hired to be FanGraphs’ Lead Prospect Analyst just after the 2016 Draft and took my first run at evaluating the entire minor leagues on my own the following winter. Enough time has now passed that many of the players from that era of prospecting have had big league careers transpire (or not). Hindsight allows me to have a pretty definitive idea of whether my call on a player was right or wrong in a binary sense, and gauge the gap between my evaluation and what the player ultimately became. Looking back allows me to assess my approach to grading and ranking players so that I might begin to establish some baselines of self-assessment and see how I perform compared to my peers at other publications. I spent time this offseason compiling the various Top 100 prospect rankings from seven years ago for the purposes of such a self-assessment. Below are the results of that exercise and my thoughts on them.

There are absolutely deeper avenues of retrospective analysis that can be done with prospect lists than what I have attempted below, approaches that could educate us about prospects themselves, and probably also about prospect writers. Before we get to a couple of big, fun tables and my notes, I want to quickly go over why I took the approach I did here, discuss its flaws, and posit other potential methods (while also including some thoughts about their limitations).

Below, I have compiled the 2017 Top 100 prospect rankings from the publications where it was basically at least one person’s full-time job to do this: FanGraphs, Baseball Prospectus, ESPN, MLB Pipeline, and Baseball America. Of course, more players from the 2017 rookie class than were ranked on Top 100 lists have turned into good players, and so I’ve also included players who might be thought of as collective whiffs, guys who produced enough that they clearly should have been on Top 100 lists by my measure of production (i.e., they performed like a 50-grade player or better). I’ve also included anyone who I graded as a 50, but who was ranked just outside the arbitrary 100-player cutoff.

My goal here was to reflect on my work, not to create metrics that measure these publications’ performance against one another, and so I haven’t done that; the Top 100 rankings are the only comparative aspect of this piece, which you’ll find next to one another in the table below. Everyone has their fair share of hits and misses; readers can get a sense of those on a case-by-case basis as they play with the table.

Eventually, someone could pool the lists and look at how the rankings correlate with WAR (either career WAR or WAR during a specific window of time, which is how I would do it) to compare everyone’s work, but I don’t believe quantitative analysis of this kind can be done on all the players who appeared on the 2017 rankings just yet. As you’ll see in the table below, a lot of the younger players from this prospect class are still in the early stages of their major league careers. As you’ll also see in the table, many of the guys who have had six- or seven-year big league tenures have had wild peaks and valleys in performance during that span. Folks inclined to do comparative analysis of this sort will want to wait and see how things pan out for guys like Ke’Bryan Hayes, Leody Taveras, and others like them before pursuing something like that. You can also really only do comparative analysis using rankings rather than grades, because while each publication tends to use the 20-to-80 scouting scale in some way, the approaches different people and publications take and the way the scale is applied varies pretty significantly from shop to shop (which I think is cool and good, by the way).

There’s nothing stopping me from looking at my own grades, though. So in that vein, career-to-date WAR for players who graduated from rookie status in 2017 (and only for those players) is included in the first table. In the second table further down the page, I go into deeper, more meaningful analysis of players from 2017’s prospect lists whose careers are now sufficiently long to do so, whose pre-free agency service time is complete. Part of my approach to forecasting players has been to focus on the window of service time prior to a player’s free agency, and now members of the rookie class of 2017 like Cody Bellinger, Matt Chapman, Jordan Montgomery, and others have played for long enough to be subject to this kind of analysis.

Why do I have this six-to-seven year cutoff? For one, I think it’s a fool’s errand to try to decide which of these guys is going to have a 15-year career (or more) when so few of them do. Teams also aren’t digging back into a free agent target’s scouting reports from seven years ago when he was at Double-A and using that information to decide whether or not to sign him, so why should a scouting report from when the player was at Double-A attempt to predict what kind of player he’ll be after he hits free agency? We should care about it as baseball fans, of course, but as a prospect analyst I believe I should not.

Here are our collective Top 100 lists:

Some thoughts on this table before we get to the next one: Sort the table by career WAR and, boy oh boy, this feels bad. As a group, we missed entirely on half of the top six players from this class. My 55 FV tier ran all the way back to 72nd overall, which is far too many guys graded that highly. There aren’t 70 guys in the minors at any given time who are going to be top 10 players at their position across my multi-year window, and that is essentially what I was saying with a 55 grade. I know there are readers who think I’m too pragmatic or even miserly with my projections, but at this stage I was handing out big grades to too many players. Now my 55 FV tier tends to run 20-30 players deep.

I was also not properly assessing the importance of health, proximity, and stability as it pertains to pitching prospects. There are far too many young, lower-level pitching prospects absolutely stuffed on my list. Contemporary approaches to pitcher deployment have teams spreading innings across more guys, which flattens the distribution of production across the league’s arms. Teams need a lot of pitchers to get through a big league season, but it’s rare for them to lean on any one guy enough to make him as valuable as the top hitters in the league. As such, I should have more pitchers than hitters on my team-by-team prospect lists, but the Top 100 and the peak of each team list should mostly be occupied by hitters. Modern, scientific approaches to developing pitchers also began to take hold around this time and pitchers’ collective ability to get better quickly exploded. In a way, this made guys who throw 95 and have a good slider surprisingly fungible. At this time, I was also nowhere near understanding the impact of fastball shape and angle, and it shows on this list. I also had not yet developed the list-making processes and data infrastructure that I now have in place in the background while I’m working on all of this stuff.

Let’s dive deeper into the rookie class from this season. Below are the players who exhausted their rookie eligibility in 2017 who were also on a Top 100 list (not just mine) that year with their career WAR, WAR rank among all qualified position players or pitchers since 2016 (to capture cups of coffee from the prior year), my projected FV grade from the time, and how I believe the player actually performed on the FV scale during his “window of evaluation.” Obviously, click through to the player pages for more stats, nerd. I’ve also included some players who haven’t graduated, but whose careers have stalled for one reason or another. In cases where a player quickly fell out of baseball entirely, they were given a “bust” designation; others who have hung around the upper levels of the minors have been given an “org” designation. I’ve also included thoughts on a couple of the notable collective whiffs:

Source