When the cosmetics heir Leonard A. Lauder gave $131 million to the Whitney Museum of American Art two years ago, it was the biggest donation in the institution’s history, and it came with one important stipulation: The Whitney could not sell its popular but cramped home on Madison Avenue in the foreseeable future.

Now an expansion plan that involves building a second museum downtown has opened a rift on the museum’s 45-member board...