Mortgage Bankers are lenders that are large enough to originate loans and create pools of loans, which are then sold directly to Fannie Mae, Freddie Mac, Ginnie Mae, jumbo loan investors, and others. Any company that does this is considered to be a mortgage banker.
Some companies don't sell directly to those major investors, but sell their loans to the mortgage bankers. They often refer to themselves as mortgage bankers as well. Since they are actually engaging in the selling of loans, there is some justification for using this label. The point is that you cannot reliably determine the size or strength of a particular lender based on whether or not they identify themselves as a mortgage banker.
An institution that lends their own money and originates loans for itself is called a portfolio lender. This is because they are lending for their own portfolio of loans and not worried about being able to immediately sell them on the secondary market. Because of this, they don't have to obey Fannie/Freddie guidelines and can create their own rules for determining credit worthiness. Usually these institutions are larger banks and savings & loans.
Quite often only a portion of their loan programs are a portfolio product. If they are offering fixed rate loans or government loans, they are certainly engaging in mortgage banking as well as portfolio lending.
Once a borrower has made the payments on a portfolio loan for over a year without any late payments, the loan is considered seasoned. Once a loan has a track history of timely payments it becomes marketable, even if it does not meet Freddie/Fannie guidelines.
Selling these seasoned loans frees up more money for the portfolio lender to make additional loans. If they are sold, they are packaged into pools and sold on the secondary market. You will probably not even realize your loan is sold because, quite likely, you will still make your loan payments to the same lender, which has now become your servicer.
Lenders are considered to be direct lenders if they fund their own loans. A direct lender can range anywhere from the biggest lender to a very tiny one. Banks and savings & loans obviously have deposits with which they can fund loans, but they usually use warehouse lines of credit for drawing the money to fund the loans. Smaller institutions also have warehouse lines of credit from which they draw money to fund loans.
Direct lenders usually fit into the category of mortgage bankers or portfolio lenders, but not always.
Correspondent is usually a term that refers to a company that originates and closes home loans in their own name, then sells them individually to a larger lender, called a sponsor. The sponsor acts as the mortgage banker, re-selling the loan to Ginnie Mae, Fannie Mae, or Freddie Mac as part of a pool. The correspondent may fund the loans themselves or funding may take place from the larger company. Either way, the sponsor usually underwrites the loan.
It is almost like being a mortgage broker, except that there is usually a very strong relationship between the correspondent and their sponsor.
Mortgage Brokers are companies that originate loans with the intention of brokering them to lending institutions. A broker has established relationships with these companies. Underwriting and funding takes place at the larger institutions. Many mortgage brokers are also correspondents.
Mortgage brokers deal with lending institutions that have a wholesale loan department.
Most mortgage bankers and portfolio lenders also act as wholesale lenders, catering to mortgage brokers for loan origination. Some wholesale lenders do not even have their own retail branches, relying solely on mortgage brokers for their loans. These wholesale divisions offer loans to mortgage brokers at a lower cost than their retail branches offer them to the general public. The mortgage broker then adds on his fee. The result for the borrower is that the loan costs about the same as if he obtained a loan directly from a retail branch of the wholesale lender.
Banks and savings & loans usually operate as portfolio lenders, mortgage bankers, or some combination of both.
Credit Unions usually seem to operate as correspondents, although a large one could act as a portfolio lender or a mortgage banker.