It is often said that immigrants “do the jobs that Americans won’t do.” While there are no major immigrant-dominated jobs in the U.S.—even about half of drywall installers are native-born—the claim does contain a kernel of truth. For over 50 years, a growing percentage of native-born American men have dropped out of the labor force altogether. For these men, every available job is a job they won’t do. Rather than focus on reversing the trend of idleness among native men, American politicians and business leaders have bandaged the problem with immigrant labor. A steady supply of new immigrants means less need for low-skill native workers, and the idleness problem is left to fester.

According to the Census Bureau’s Current Population Survey (CPS), just 5 percent of 25- to 64-year-old men in 1962 were “out of the labor force” at the time of the survey, meaning they were neither working nor looking for work. The number rose slowly but steadily to 16 percent by 2015.

So who are these working-age men who are not even looking for work? They are primarily lower-skill natives. In 1994, the first year that the CPS recorded immigration status, 33 percent of native high-school-dropout men were not in the labor force. By 2015, that number was up to 40 percent. For black American dropouts in particular, labor force absence is now 53 percent—meaning fewer than half are even looking for a job. By contrast, the percentage of low-skill immigrant men who are out of the labor force has consistently fallen in the 10 to 15 percent range since 1994.

These are not new observations. Scholars from William Julius Wilson on the left to Charles Murray on the right have warned about the decline of work among low-skill natives. “Many of today’s problems in the inner-city ghetto neighborhoods—crime, family dissolution, welfare, low levels of social organization, and so on—are fundamentally a consequence of the disappearance of work,” according to Wilson. And it’s not just the inner city that suffers. In any community where the social and economic capital fostered by work weakens, dysfunction creeps in.

Unfortunately, the popular media are reluctant to cover the drip-drip-drip of a problem that slowly worsens over many years. They focus instead on the ups and downs of the official unemployment rate, which includes only people who are actively looking for work. With the unemployment rate currently down to 5 percent, many Americans seem unaware that low-skill native men have been working less than ever in the past few years.

The steady decline of work has no simple explanation. Progressive analysts tend to blame the loss of good-paying union jobs, while conservatives focus more on the expansion of the welfare state and weakened social sanctions against idleness. And immigration itself may have exacerbated the problem by pushing down the available wages for native men who are “on the bubble” between work and non-work.

The goal here is not to resolve the debate over causes, but rather to point out how immigration functions as a means to ignore the problem. Instead of searching for ways to get native men back to work—whether through higher wages, less access to welfare, social pressure, or some other means—politicians and businessmen have brought in immigrants to do the work instead. Imagine how their focus would change if there were no supply of new immigrants to harvest their vegetables, weed their gardens, or hang their drywall. They would likely take a much greater interest in getting idle American men back to work.

Even limited immigration restrictions seem to spur employer outreach to natives. When many of its workers were deported after a high-profile raid in 2006, a Georgia poultry processor advertised not just higher wages, but free transportation and lodging for workers, according to the Wall Street Journal. Similarly, after stepped-up enforcement helped reduce Arizona’s illegal immigrant population, the Journal reported that wages went up 10 percent and 15 percent for the state’s construction and farm workers, respectively. The extent to which immigration has depressed wages nationally is a matter of scholarly debate, but wage hikes were only part of a concerted effort by Arizona employers to convince people to work for them. “Now you have to put out feelers, buy ads, go on Craigslist, tap job agencies just to get a few men,” the paper quoted one business owner saying.

Of course, immigration restriction alone will not solve the problems endemic to the American underclass, but it restores the incentive to help. For those who are serious about getting low-skill natives back to work and into the mainstream of American society, advocating more low-skill immigration seems difficult to justify.

One tempting justification is that low-skill natives might respond to immigrant competition by pursuing more education and graduating to skilled work. But while more schooling could certainly benefit some people, it’s naïve to think it can lift all boats. A non-trivial portion of native men will always lack the academic skills that are rewarded in our knowledge economy, and the challenge is to ensure they still have a valued place in society.

It may also be tempting to view immigrants as redeemers of the American underclass, coming to share a stronger worth ethic that will inspire low-skill natives. The problem is that hard-working immigrants tend to have not-so-hard-working children and grandchildren. For example, just 8 percent of working-age male immigrants from Latin America are out of the labor force, but the number among U.S.-born Hispanics is 17 percent, which is higher than the national average. It seems that immigrant families eventually assimilate to the native work ethic, growing the underclass and fueling calls for even more immigration.

It’s time to break the cycle.

Jason Richwine is a public-policy analyst in Washington, D.C.