Vast quantities of marine debris have beached at remote islands in the western Indian Ocean such as Seychelles, but little is known about where this debris comes from. To identify these sources and temporal patterns in accumulation rate, we carried out global Lagrangian particle tracking experiments incorporating surface currents, waves, and variable windage, beaching, and sinking rates, taking into account both terrestrial (coastal populations and rivers) and marine (fisheries and shipping) sources of debris. Our results show that, whilst low-buoyancy terrestrial debris may originate from the western Indian Ocean (principally Tanzania, Comoros, and Seychelles), most terrestrial debris beaching at remote western Indian Ocean islands drifts from the eastern and northern Indian Ocean, primarily Indonesia and, to a lesser extent, India and Sri Lanka. Purse-seine fragments beaching at Seychelles are likely associated with fishing activity in the western Indian Ocean, but longline fragments may also be swept from the southeastern Indian Ocean. The entire of Seychelles is at very high risk from waste discarded from shipping routes transiting the Indian Ocean, and comparison with observations suggests that many bottles washing up on beaches may indeed originate from these routes. Our analyses indicate that marine debris accumulation at Seychelles (and the Outer Islands in particular) is likely to be strongly seasonal, peaking during February-April, and this pattern is driven by local monsoonal winds. This seasonal cycle may be amplified during positive Indian Ocean Dipole phases and El-Ni\~{n}o events. These results underline the vulnerability of small island developing states to marine plastic pollution, and are a crucial first step towards improved management of the issue. The Lagrangian trajectories used in this study are available for download, and our analyses can be rerun under different parameters using the associated scripts.