Colby's is housed within the historic Odd Fellows Building, constructed in 1895. Over the years, the building has served a variety of purposes, including as a professional space, clubhouse, and specialty retail location. Designed in the Renaissance Revival and Italian Revival styles, it is listed on the National Register of Historic Places and recognized as one of the most architecturally intricate buildings in downtown Owensboro. Its construction aligned with the city's rapid growth in the 1890s, when Owensboro became a vital center for manufacturing, trade, and transportation along the Ohio River.