Smith shaped Port with iconic restaurant, hotel

It’s fair to say that Smith Bros. Restaurant at the corner of Grand Avenue and Franklin Street in downtown Port Washington was a landmark.

Crowds of people would gather in the bar waiting for a table on a Friday night while others stood in line outside the restaurant waiting their turn.

Buses of tourists would clog the intersection, dropping off visitors hungry for the restaurant’s signature planked whitefish and lemon meringue pie.

And in the midst of it all would be Lloyd Smith, whose family built the restaurant from the ground up and operated a successful commercial fishing operation as well.

Mr. Smith — who died Friday, Sept. 11, at age 88 — would often pop out from his office, visit with his wife Toni, the restaurant’s hostess, and mingle with diners.

He was the face of the popular eatery for decades, but his impact on the city didn’t stop there. Recognizing the need for a hotel downtown, Mr. Smith persuaded his family to build the Harborside Motor Inn — today the Harborview — in 1973 and operated it until his retirement in 1998.

In the process, Mr. Smith helped Port Washington transition from a manufacturing community to one fueled by tourism.

“He was integral in shaping downtown,” longtime downtown businessman Merton Lueptow said. “He had two things that drew people to Port Washington — the best known restaurant in all the U.S. and the hotel — and he was very, very active in the downtown to make sure we had things for (visitors) to do.”

NewPort Shores restaurant owner John Weinrich, who began his career in the kitchen at Smith Bros., put it simply.

“Lloyd was a legend,” he said. “Smith Bros. was the place to go, and he was the face of it.”

It was Mr. Smith’s abiding love for Port Washington that shaped him, longtime friend Rick Smith said.

“He was Port Washington born and bred,” Rick Smith said. “He took the family legacy and carried it into the future.”

That family legacy ran deep. Mr. Smith was born into the Smith Bros. commercial fishing family, started by William Smith and his son Gilbert near Cedar Grove in 1848.

  The family moved the fishing operation to Port Washington in 1899. The business expanded into canning of domestic caviar, and in 1924, when a storm washed the fish shanties along Sauk Creek into the harbor, Mr. Smith’s aunt Evelyn Smith set up a fryer in an old harness shop in the heart of downtown and began frying the fish the family caught. 

That was the start of Smith Bros. Restaurant, which grew through the years, eventually expanding into California.

But it was in Port Washington that Mr. Smith made his mark.

Ozaukee Press publisher Bill Schanen III, who was editor of the newspaper when Smith Bros. Fish Shanty restaurant was in its heyday, credits Smith with being largely responsible for keeping Port Washington’s downtown vital with “his steady management and bold investment in the iconic restaurant that attracted a continuous stream of visitors to the city.”

He added, “The huge restaurant, that could seat scores of diners on two floors and frequently had lines of people waiting for tables to enjoy the reliably wonderful fish dinners, gave the city its identity. Mention Port Washington and people would immediately associate it with Smith Bros. The Fish Shanty logo with its fisherman carrying a giant sturgeon was the city’s semi-official image.”

Weinrich agreed, saying no matter where you traveled in the country, you could always find someone who knew Smith Bros. Restaurant.

“That’s why people came to Port Washington,” he said. “It wasn’t such a big deal then to walk on the beach — they came to Smith Bros. to eat. In the course of a day, we would have thousands of customers.”

Mr. Smith began his career at the restaurant sweeping floors when he was 13, but he wasn’t sure the family business was his calling. He got a degree in government at Beloit College in 1953, but after he crossed a professor whose signature he needed to get into graduate school, he returned to the family business.

One notable night — Nov. 16, 1953 — Mr. Smith closed the restaurant and headed home only to get a call hours later that the restaurant was on fire.

“I joined the family and watched it burn,” Mr. Smith wrote in a memoir.

The next day, the family agreed to rebuild the restaurant. During construction, Mr. Smith earned a bachelor’s degree in hotel and restaurant management from Michigan State University. And when he returned, he became assistant manager and later manager of the restaurant.

Mr. Smith later recognized the need for a hotel in downtown, and he sold the family on the concept. When the Harborside opened in 1973, Mr. Smith became its general manager.

“He told me he built the hotel not because the family needed it but because Port Washington needed it,” Rick Smith said. “He was a big promoter of Port, and he knew this would be a big thing for the city.”

Mr. Smith was a dedicated man who worked tirelessly, his daughter Margaret Smith said.

“I’ve never seen anyone work as hard as my dad —seven days a week, five nights a week, he was at the restaurant,” she said. “I don’t know where he got the energy. He was literally there all the time.”

There was only one day in all those years that Mr. Smith took a day off for relaxation, she said. He took his family to brunch, then stopped at a friends.

“(Her brother) David, mom and I just looked at each other and wondered, when is Dad going back to work?” Ms. Smith said. “One day off —it was such a treat.”

Even when he retired from the restaurant in 1988, Mr. Smith couldn’t stay away, she said. Less than six months later, he and his wife bought the hotel from the other family members. In 1993, encouraged by the growth of tourism in the city, they expanded the 60-room hotel , adding36 rooms with panoramic views of the city’s waterfront. 

Mr. Smith ran the hotel until 1998, when he and his wife sold it. 

It wasn’t just in his business that Mr. Smith had an impact on the community. He was a founding member of the Ozaukee and Port Washington Tourism Councils, and was active in the Port Chamber of Commerce and on the Fish Day Committee.

“When bus groups would come to town, he’d hop on board and give them a tour around town,” his daughter said. “He was a real salesman for Port Washington.

“Port Washington was in his blood.”

Mr. Smith would often attend Common Council and Plan Commission meetings, offering his advice to officials and encouraging them to retain the city’s small-town charm, and he was instrumental in getting the city to pass a room tax to help fund tourism efforts, Rick Smith said.

And his legacy continues in other ways, big and small. Mr. Smith was a mentor to many, Weinrich said, adding he gave him the iconic Smith Bros. recipes.

“He was very generous to everyone,” he said. 

Today, the Smith Bros. name lives on in the coffeehouse that now stands where the Smith Bros. market once operated, and the iconic neon Smith Bros. sign still stands atop the building.