The marine pelagic brown claystone was widely recovered in the deep South China Sea (SCS) by International Ocean Discovery Program (IODP) Expedition 349, 367, 368 and 368X. The continuous deposition of brown claystone in Miocene in the SCS may represent a special sedimentary environment. IODP Site U1503A provides the ideal brown and green claystone of the early-mid Miocene to study the origin of the red coloring for sediments and their implications to the paleoclimate and paleoceanography. In this paper, the reflectance a*, grain size, clay minerals, major and trace elements, and Sr-Nd isotope analyses in brown and green claystone from U1503A have been conducted to understand the sedimentary condition and controlling factors of SCS marine brown claystone. The results show that the sediments color transition was caused by the variations of Fe (III) content. The provenance was relatively stable and sediment was mainly supplied from South China and Luzon. The source areas underwent strong chemical and physical weathering since 17 Ma, which is related to Miocene Climatic Optimum events. The data of redox proxies suggest that both brown and green claystone were deposited in the oxic condition, while the sedimentary environment of brown claystone was more oxidizing. We suggest this difference in the redox condition was mainly controlled by the variation of the oxygen-rich bottom current in the northern SCS. We attribute this variation to the larger-scale deep-water circulation change in the Antarctic and western Pacific due to glacial-deglacial cycle.