The park is located along the Stanislaus River near the town of Ripon, California. The park's 258 acres protect a fine example of the threatened and still declining riparian oak woodland, which once flourished throughout California's Central Valley. Caswell is home to several endangered animal species, including the riparian brush rabbit which is not known to occur anywhere else.

The Native Americans who lived along this river and collected acorns among these ancient groves were Yokuts. In the early 1800s, Spanish explorers traversed this area, and fur trappers found the river bountiful.

Thomas Caswell, landowner, enjoyed this wonderful forest and felt it should be preserved. In 1950 the children and grand children donated 134 acres to the people of California. Additional donations and state purchases brought Caswell to its current size of 258 acres. Caswell Memorial State Park was open to the public in 1958.