Homeowners, and at times the government, have long complained that banks and other companies that service mortgages aren’t good at their job of collecting monthly payments, modifying loans and processing foreclosures. Now, a new cast of characters are piling on the criticism: the servicer’s own clients, the investors that actually own the mortgages.

Servicers handle the day-to-day of working with homeowners on behalf of the investors, who bought bundled mortgages from Wall Street. But investors are now threatening servicers with legal action. Just like homeowners, investors are frustrated by the poor job in modifying loans that servicers have been doing. They also say servicers are looking out for themselves, not investors’ interests as their contracts typically require.

Investor Bill Frey, who runs the securities firm Greenwich Financial Services, says servicers view investors as “a Thanksgiving turkey to be carved up and shared amongst themselves.”

Investors can range from foreign governments and hedge funds to college endowments and pension funds. During the housing bubble, they gobbled up AAA-rated bonds created by pools of mortgages. Now that defaults and foreclosure are mounting, investors argue that flaws in how loans are serviced are costing them billions of dollars.

They say servicers have often dragged out foreclosures to rack up fees and refused to reduce second mortgages to make modifications sustainable. Investors often prefer modifications to foreclosures. But for modifications that won’t ultimately prevent a homeowner from defaulting, investors still prefer quick foreclosures so they can recoup their money and move on.

“Terminal indecision is not good,” says Frey. “If it can be fixed, fix it. If it can’t, nix it.”

Servicers have been slow to modify mortgages—something we’ve writtenabout many times — and when they do modify loans, homeowners are still saddled with other debt from second mortgages and home equity lines. Even after modifications under the government’s program, homeowners typically still must spend almost two-thirds of their income to pay off their mortgage and other loans, like credit cards or second mortgages.

The current mortgage paperwork scandal adds more fuel to the fire as major servicers have halted foreclosures because of potential paperwork irregularities around the country. Concerns are also growing that banks may not have properly transferred loans into the mortgage pools in the first place. “This deficient approach undermines the integrity and the operational framework of the housing finance and mortgage system as it exists today,” the Association of Mortgage Investors wrote in a press release.

(For more on the growing scandal, check out our recent explanation of the main players involved.)

The Mortgage Bankers Association, which represents most major servicers, did not respond to ProPublica’s request for comment.

Investors from across the country have been coordinating legal strategies for over a year ago, with the effort ramping up in early spring, according to Frey. Since then, more and more investors have formed a loose consortium, gaining momentum “like a snowball going downhill,” he says. In the last month alone, the group added other investors that own an additional $100 billion in mortgage bonds.

They have not filed any suits yet, Frey says, because the group is first trying to grow even more. Also, since each investor group has different, nonmortgage business with the banks, some investors have conflicting interests in how to proceed, he says. The consortium now represents investors that own more than $600 billion in mortgage securities, which is around a third of the entire mortgage securitization market. The group includes 65 major mortgage investors; Bloomberg reported that large investment companies including Black Rock, PIMCO and Fortress are part of the effort, as are the quasi-governmental Fannie Mae and the Federal Home Loan Banks, which both own private securitized loans.

Coordinating investors is no easy task, since the mortgage bonds were sliced and diced to be sold off to investors around the world. To assert legal rights, investors must coordinate to prove that they collectively represent a certain percentage of each mortgage pool, or in some cases, a certain percentage of each slice of each mortgage pool. (The Wall Street Journal and Bloomberg both describe how Texas-based attorney Talcott Franklin is coordinating a clearinghouse to keep track of the various investments.)

Once investors have standing in each pool, they have the legal right to pressure servicers and trustees to improve or face litigation. The group says they have the legal authority to act in over 2,300 deals.

Investors say servicers must reduce or cancel second mortgages entirely before adjusting the primary loan, since that follows the legal pecking order of how loans should be paid off. But investors say servicers have are dragging their feet in reducing second mortgages to protect their own books, since the largest servicers — Bank of America, Citigroup, JPMorgan Chase and Wells Fargo — also own almost 60 percent of the $1 trillion second lien market.

A Congressional Oversight Panel concluded in April that there is “tension” between Treasury’s goal of supporting reductions to second mortgages and Treasury’s interest in ensuring that writing down second liens doesn’t severely weaken banks’ balance sheets. The panel wrote than when a servicer owns the second lien, the “inexorable conflict of interest” will more likely lead to modifications on the first loan, “as it benefits the bank at the expense of the mortgage-backed security investors.”

We’ve previously reported that mortgages servicers frequently tell homeowners that investors are the roadblock to loan modifications, even though few mortgage deals actually restrict modifications.

Servicers are also supposed to act like watchdogs and report back to investors when they identify loans they suspect didn’t meet the lending standards promised when the bonds were initially sold to investors. If the banks did misrepresent the quality of the loans initially, the banks would have to buy back the invalid mortgages from the investors. But in many cases, the servicers are subsidiaries of the banks that sold the bonds, which investors say helps explain why servicers have been dragging their feet. Bloomberg noted an analyst’s report that said mortgage repurchases could total over $179 billion.

According to an investor letter cited in the Wall Street Journal, in some mortgage pools that have high default rates, the banks have not repurchased any loans when the servicers are subsidiaries of the banks that sold the bonds.

Investors say this is all no small matter. Since the country’s mortgage market is heavily dependent on government support right now, they insist servicers make good on their contracts before start buying loans and supporting the mortgage market again.