The name 'Baexem' is first mentioned in a charter from June 1244. Recently there was an extraordinary discovery nearby the castle: a signet ring from the year 1293 from the knight Lenvalus from Baexem.
From excavations revealed that earlier the castle consisted of a large dungeon. The foundations thereof are below the eastern tip of the present building. In 1987 the current owners Jan and Theo Peters bought half of the castle that was uninhabitable. In cooperation with the Municipality and the State the restoration has started in 1992. Over time, the castle was 13 times rebuilt and in the 17th century it received its present form. The moat and gatehouse have disappeared in the course of the 19th century.