River deltas commonly have a heterogeneous substratum of alternating peat, clay and sand deposits. This has important consequences for the river bed development and in particular scour hole formation. When the substratum consists of a poorly erodible top layer, erosion is retarded. Upon breaking through a resistant top layer and reaching an underlying layer with higher erodibilty, deep scour holes may form within a short amount of time. The unpredictability and fast development make these scour holes difficult to manage while stability of dikes and infrastructure may be at stake. In this paper we determine how subsurface lithology controls the bed elevation in net incising river branches, particularly focusing on scour hole initiation, growth rate and direction. For this, the Rhine-Meuse Estuary forms an ideal study site, as over 100 scour holes have been identified in this area and over 40 years of bed level data and thousands of core description are available. It is shown that the subsurface lithology plays a crucial role in the emergence of scour holes, their shape and evolution. Although most scour holes follow the characteristic exponential development of fast initial growth and slower final growth, temporally strong variations are observed, with sudden growth rates of several meters per year in depth and tens of meters in extent. In addition, we could relate the characteristic build-up of the subsurface lithology to typical scour hole development like large elongated expanding scour holes or confined scour holes with steep slopes. As river deltas commonly have a heterogeneous substratum and often face channel bed erosion, the observations likely apply to many delta rivers. These findings call for good knowledge of the subsurface lithology as without, scour hole development is hard to predict and can lead to sudden failures of nearby infrastructure and flood defence works.