On the evening of July 25, 1964, Pop artist Andy Warhol and five associates armed with a single camera and a bag of 33-minute film cartridges entered an office on the 41st floor of the Time-Life Building in Manhattan. For the next six hours, Warhol and his crew pointed their camera at the top third of the Empire State Building, documenting it first in decaying daylight, then awash in its electric glow, and finally swallowed by the darkness of twilight. The camera never moves, and neither does the building.

Andy Warhol, Empire, 1964, © The Andy Warhol Museum

The resulting eight-hour Empire (Warhol slowed the footage down from 24 frames per second to 16) is boring, beguiling, and divisive. But 50 years on there’s an air of inevitability about Warhol’s cinematic portrait of the skyscraper. After all, how could an artist so obsessed with cultural icons—Campbell Soup cans, Brillo boxes, Marilyn Monroe—resist the one towering over Manhattan?

“The culprit is the Empire State Building itself,” says Jonas Mekas, the writer, filmmaker, and patron saint of the underground cinema, about the making of the film. Empire began with Mekas and a young acolyte, John Palmer, running an errand. Sometime in mid-July 1964, they walked out of Mekas’s building at 414 Park Avenue South to mail copies of Film Culture magazine from the Empire State Building post office. (Mekas edited the publication and ran the Film-Makers’ Cooperative, which was Warhol’s film school, from his loft.) They headed north up Park, turned west, and wham! “There it was, the Empire State Building, in its glory, just in front of us as we were walking,” the 91-year-old Mekas remembers. “So we stopped and admired it and I think I said, ‘This is really a subject for Warhol.’ And John agreed.”

Mekas encouraged Palmer to share the idea with Warhol (Palmer also hung out at the Factory), and later that evening he did. The artist immediately went for it—so much so that they were filming only days later. “That’s really his kind of image,” Mekas adds.

Mekas and Palmer were part of Warhol’s Empire crew. (The others were poet Gerard Melanga, Mekas’s girlfriend Marie Desert, and Henry Romney, the head of Information Services for Rockefeller Foundation, which occupied the Time-Life space.) Palmer was on set as the writer (though the script eventually got scrapped), while Mekas’s role was that of a technician. “I set up the camera and framed the shot more or less the way I thought Andy would like,” he says. “So I called Andy to take a look. And there was no change. It’s OK, go ahead, that’s it.” He also purchased the film and was responsible for changing the magazines every 33 minutes.

But Mekas was there as a journalist, too, gathering material for his Movie Journal column, which was carried in the Village Voice. His entry, published June 30, 1964, not only takes us inside the shoot, it’s the source of one Warhol’s most indelible exclamations: “The Empire State Building is a star!” Indeed, Empire is a moving-image version of his silkscreens: A static image mechanically reproduced to such a dizzying extent that it simultaneously reinforces its subject’s icon status while blanching it of all meaning.

“I have seen the film at least three times, and it really gets a little bit boring at the beginning,” Mekas says. “But after a point, you give up and you begin to sort of relax and it becomes a little hypnotic, all those dots, the changing of the lights. And then, after two hours, the lights go on! And then everybody applauds because this is a huge event!” The trick is staying around for those first two hours, which not everyone did when Empire premiered in March 1965. Some 200 people turned out for the screening, and about half made it through to the end. A few dozen moviegoers stormed out and demanded a refund.

But as the years go by, Empire exists as something more than just art: It’s as a window on a bygone New York. That view from the 41st floor at the Time-Life Building? Gone, thanks to the 600-feet-tall 1166 Avenue of the Americas built in 1974. Mekas’ Park Avenue loft? Destroyed to make way for the Gansevoort Park Avenue NYC. The Empire State Building post office? Closed.

It’s unlikely Warhol thought he was capturing a vision of his city as evanescent as his silver balloons. Yet that has become as much a part of Empire’s legacy as it’s length and audacity. “It’s like nothing else,” Mekas says. “It’s a monumental, unique document and film.”