Or is it? Commenter Bret thinks they might have a case, but I'd contend that not only are the not the worst pen ever, but they're not even the worst pen in the league this very year! The stats and the team's rankings in the majors, and in the AL:
They're walking too many batters and are giving up a lot of home runs (largely due to a relative lack of groundballs), but missing bats is a good sign. The relievers have allowed 43% of inherited runners to score (that includes runners inherited from previous relievers as well as starters), which is higher than the league average of 34% but not the highest in baseball (the Nationals, at an astounding 62%), the AL (the Angels, at 56%), or even the AL East (the Jays, at 47%). The bullpen just plain isn't pitching that poorly.
It does seem like they're pitching really poorly though, because of the way that they've blown games. Their -1.95 Win Probability Added^ is 3rd worst in the majors, and their -1.39 clutch^ score is dead last. Part of that - a big part - is Mike Gonzalez's terrible start to the season, where he blew two saves and racked up -1.18 WPA all by himself before going on the DL.
Another part of it is that the team has played in so many close games, so relievers are always coming in in higher leverage situations - the highest in the majors, in fact. O's relievers as a whole are entering games with a leverage index of 1.60, which is akin to what closers tend to face over a full season (the average leverage index on entering a game for pitchers with at least 20 saves last year was 1.62). That means that when they mess up - and all pitchers mess up, including average-ish at best ones like the Birds' current crew - it hurts.
There's not a lot of quality depth out there right now, with Albers (5.17 xFIP), Mickolio (5.91), Ohman (6.66), and Meredith (5.08) having stunk up the joint, but all of those guys are better than those numbers. Until they turn it around, it might not hurt to go more to Berken (my boy with the 3.79 xFIP), Hendrickson (2.81), Castillo (1.07), and JJ* to slam the door (2.97).
Reliever performance is volatile though, so 5 more innings could flip things all around. For now, feel free to freak out when they turn a 3-2 lead into a 4-3 deficit, but keep in mind that if the offense had managed 5 runs the team would still be ahead.
* Jim Johnson is a good relief pitcher. Full stop.
[Still in the digression, but non italicized for easier reading.]
So far this year, for the second year in a row, he's upped his strike-outs rate quite a bit while walking fewer batters. The absurd .447 BABIP^ won't continue. There is no real reason to think that he can't come in in the ninth inning with the team up by one to three runs.
JJ in save situations, career: 3.55 FIP
In non-save situations: 3.91 FIP
In low leverage situations, career: 4.28 xFIP
In medium leverage situations: 4.77 xFIP
In high leverage situations: 4.02 xFIP
Please stop overreacting to five games at the end of 2009. It's irrational and stupid, and if the Orioles signed Gonzalez for $12 M because they didn't think JJ "had what it takes to be a Closer" based on the end of last season (not accounting for the benefit of some sort of inside info.), then they also acted stupidly.