Upper-Extremity motor impairment affects millions of Americans due to cerebrovascular incidents, spinal cord injuries, or brain trauma. Current therapy practices used to assist these individuals in regaining motor functionality often require extensive time at rehabilitation facilities with potentially prohibitive travel or financial costs. This work presents a mobile low-cost field programmable gate array (FPGA)-smart rehabilitation system that can be used in home environments. The prototype is a rehabilitation table instrumented with a capacitive sensor array (CSA) to track upper-extremity motions of the user through proximity or touch. In addition, inertial measurement units (IMUs) are placed on the affected upper limb and combined with the CSA data with our sensor fusion signal processing architecture. Motions are classified and evaluated using multi-task convolutional recurrent neural networks with three additional motion quality output classes to personalize recognition based on the particular motor skills of each patient. The prototype achieves above 99% accuracy with 32-bit fixed-point format implementation for recognizing dynamic motions and identifying unnatural characteristics (i.e., tremor or limited flexion and extension) in upper limb motions based on sensor values. The convolutional recurrent neural network (C-RNN) fusion classification network is implemented on a 200 MHz Zynq ZCU104 FPGA using an HLS-based design optimized with pipelining and parallelism techniques and achieves 5.4x speedup compared to ARM® Cortex-A53 implementation running at an operating frequency of 1.3 GHz. The prototype is also demonstrated to perform the machine learning classification in real-time.