Vetting the Indians’ Bullpen

It seems to be fairly accepted wisdom that the Indians were not good last season.  There’s an advanced sabermetric statistic that does a nice job of conveying this.

It’s called “losses”, and the Indians had 93 of them.

But, for whatever reason, the Indians actually had some pretty good results out of their bullpen in 2010.  In an otherwise pathetic campaign, we were treated to a breakout performance from Chris Perez (2-2, 1.71 ERA), respectable bouncebacks from Raffy Perez (6-1, 3.25 ERA) and Jensen Lewis (4-2, 2.97 ERA), and reasonably effective showings from Joe Smith, Vinnie Pestano, Frank Hermann and Justin Germano.  And, in case you forgot, Andy Marte flat-out dominated.

Overall, the Indians ended the 2010 season with a 3.83 ERA out of the bullpen, good for sixth in the American League.  That figure put us ahead of the pricey relief corps of Boston, Detroit and Los Angeles.  The only team with a better bullpen ERA than the Indians who didn’t make the playoffs was the White Sox, who boasted Chris Sale, Matt Thornton and Bobby Jenks.  Further, considering that most of our 2010 bullpen will be back this season—we lost Kerry Wood, whom we’re not likely to miss—it’s not entirely silly to think that our bullpen could be a strength for us in 2011.

This is, of course, where I rain on the collective parade.  While it’s true that our bullpen had some pretty good results last year, it’s also true that the underlying performances were not nearly as stellar.  Depending on how you look at it, they might have been flat-out bad.

Let’s look at a table with each player’s ERA and xFIP—an ERA estimator that tells you what a pitcher’s ERA should be based on peripheral statistics like walks and strikeouts. (I wrote an intro to some of the statistics used in this piece here.)

Chris Perez1.714.302.59
Rafael Perez3.254.291.04
Jensen Lewis2.974.701.73
Vinnie Pestano3.604.240.64
Frank Herrmann4.034.820.79
Justin Germano2.164.192.03
Joe Smith3.834.410.58
Hector Ambriz5.594.64-0.95
Tony Sipp4.144.660.52


So read the table like this: “Chris Perez allowed 1.71 earned runs per nine innings pitched last year, but he was lucky, because the way he pitched probably merited a 4.30 ERA.”  I’ve subtracted each player’s ERA from xFIP to give you a feel for who was lucky and who wasn’t—a positive value is lucky, a negative is unlucky.

At first glance, this table will be jarring.  For one thing, nearly every member of the bullpen looks to have outperformed his peripheral statistics.  The only pitcher who didn’t was Hector Ambriz, and if we’re counting on a big contribution from him this season, we’re in more trouble than I thought considering his reconstructed elbow and all-around badness.

Second though, is that when we look at these players based on xFIP, they all look really similar.  The best xFIP performance was Pestano at 4.24 and the worst was Hermann at 4.82.  That’s not a drastic spread; they’re all “clumped together”.

And the reason for that “data-clump” is that xFIP assumes that pitchers don’t really control how many HRs they let up—that it’s going to happen to all pitchers at a league-average rate.  I’m not sure I agree with that assumption, so instead let’s look at the group by FIP, which does include HR-rate as an input and see how the group looks from this angle:

Chris Perez1.713.541.83
Rafael Perez3.253.770.52
Jensen Lewis2.973.410.44
Vinnie Pestano3.602.88-0.72
Frank Herrmann4.034.490.46
Justin Germano2.164.672.51
Joe Smith3.834.650.82
Hector Ambriz5.595.35-0.24
Tony Sipp4.145.321.18


Certainly they look better here, though still pretty lucky: most of the guys still had an ERA below their FIP.  Still, this tells me that our bullpen was better than average at preventing home runs last season.  If you believe limiting HR is a “skill”—that is, that it can be repeated—then we should see the guys outperform xFIP again this season.  If you think that the low HR-rates were more a product of luck and small sample size, then you’re looking at the potential for a pretty large regression in 2011.

But really, home runs aren’t the driving force behind most ERA estimators; they’re fairly rare events compared to the nuts and bolts of pitching a baseball game. What really affects these sorts of statistics are two basic inputs: walks and strikeouts.

Let’s see how the Indians’ bullpen did last year by these metrics.  First, let’s look at how many walks our bullpen gave up per nine innings pitched compared with the rest of the AL:

Blue Jays3.62
White Sox3.79
Red Sox3.98


Not so good for the Tribe, sitting with the third worst walk-rate in the AL last season.  The biggest offenders were Joe Smith, Jensen Lewis and Tony Sipp—all guys we are probably going to be counting on this season.

Now here are the strikeout rates for the bullpens, based again on how many strikeouts each recorded per nine innings pitched:

White Sox9.05
Red Sox7.56
Blue Jays7.49


Fifth worst.  So you combine those two rates, and look at the ratio of strikeouts-to-walks to get the whole picture:

White Sox2.39
Blue Jays2.07
Red Sox1.90


Fourth worst in the league, and well below average.  This could become a problem going forward.  It’s hard to sustain any sort of success when you strikeout fewer than two batters for each batter you walk.

Don’t believe me?  Fine, I’ll give you more tables.

Here are the AL bullpens ranked by their “ability” to strand runners on base (that’s a good thing, from a pitching perspective, though whether it’s an “ability” or just luck is an open debate):

Red Sox74.2%
White Sox73.9%
Blue Jays73.0%


Hmm. Above average.  That’s odd, considering that they were below average at the things that would end innings quickly—striking batters out and not walking them.  This is sounding more and more like a fluke to me.

One more table.  This one presents the BABiP for opposing batters—that’s batting average on balls in play.  Pitchers don’t have a lot of control over this stat, and usually things normalize at around .290 for relief pitchers.  If it’s much lower than that, you’re getting lucky; if it’s much higher, you’re letting up a bunch of “groundballs with eyes.”

Red Sox0.280
Blue Jays0.291
White Sox0.314


Yep, it looks like we were above average in the “luck stats” and below average in the “skill stats”.  Does this mean we’re destined to have a crummy bullpen this year?  Not necessarily.  Our guys might really break the mold—we might have pitchers who can consistently limit opposing BABiP, who can strand base runners, and who can suppress HR/FB rate better than anyone else.

But when I hear fans talking about the bullpen as the strength of the team, as if we’re destined to do as well in 2011 as we did in 2010, well, I think they might be setting themselves up for some disappointment.  The underlying numbers just don’t suggest that our bullpen is remotely good.  According to WAR—a limited statistic in this regard to be sure—we had the second worst bullpen in the AL last season.  It just doesn’t inspire a lot of confidence when you look past the luck and try to project how the group might perform this season.

On the other hand, I’ve been the one who’s been telling you all off-season to believe in Matt LaPorta and Michael Brantley, so I might not have the bona fides required to tell others to temper their enthusiasm in regard to lofty expectations.

  • Lyon

    I never expect anything from the bullpen, it can be so hit or miss. It’s better to expect nothing and be surprised than to rely on it and be very disappointed.

  • BAJ22

    “There’s an advanced sabermetric statistic that does a nice job of conveying this.

    It’s called “losses”, and the Indians had 93 of them.”

    Jon, coming from you that really struck my funny bone!!
    And you didn’t disappoint with the advanced sabermetric stats later in the article!

    Great stuff!!

  • NJ

    Based on the stats, Perez had a ridiculously lucky ’10. His BABIP was something like .220 and his HR/FB was around 5%, both of which are extremely low.

    Regression towards the mean will be the order of the day.

  • JM

    I’m just hoping for them to be an average team. I can’t take another bad season.

  • Karsten

    #3 Nope. He is Pure Rage. One of the intangible stats that your silly little Sabermetric charts cannot account for is the fear that he strikes into the hearts of the opposition. And luck is actually an ingredient in grape gatorade. He shares with the whole ‘pen on occasion, contrary to popular belief.

  • Charles

    How much of the Indians’ pitchers outperforming their FIP and xFIP is due to playing in what was probably the best pitcher’s park in the AL last year?

  • Jon

    @ Charles,

    Where are you getting that it was the best pitcher’s park in the AL last season? In 2009, Aaron Gleeman (whom I respect) had them at a 1.03 (slightly hitter friendly), and in 2010 ESPN (whose method I don’t know) had Progressive ranked 21st out of the 30 stadiums at 0.95 (slightly pitcher friendly).

    But I’m not sure that park factors would affect ERA anymore than FIP or xFIP. Why would they? I’ll look into it.

  • Charles

    At a mark of 93 for both, the Indians had the lowest pitching park factor, and tied with Seattle for lowest batting park factor. And it would affect ERA more than FIP because park factors have very little effect in K’s and BB’s. And it’s going to cause almost everyone’s xFIP to rise, as the park will suppress HR-rate.

    So after looking up some numbers, there is a slight correlation – .36 between park factor and outperforming FIP in 2010 for AL teams. Not much but there’s something there.