Video S1. Magnetic actuation of the metamaterial on a flat surface. 

A prominent problem in computer vision is occlusion, which occurs when an object’s key features temporarily disappear behind another crossing body, causing the computer to struggle with image detection. While the human brain is capable of compensating for the invisible parts of the blocked object, computers lack such scene interpretation skills. Cloud computing using convolutional neural networks is typically the method of choice for handling such a scenario. However, for mobile applications where energy consumption and computational costs are critical, cloud computing should be minimized. In this regard, we propose a computer vision sensor capable of efficiently detecting and tracking covered objects without heavy reliance on occlusion handling software. Our edge-computing sensor accomplishes this task by self-learning the object prior to the moment of occlusion and uses this information to “reconstruct” the blocked invisible features. Furthermore, the sensor is capable of tracking a moving object by predicting the path it will most likely take while travelling out of sight behind an obstructing body. Finally, sensor operation is demonstrated by exposing the device to various simulated occlusion events. Keywords:  Computer vision, occlusion handling, edge computing, object tracking, dye sensitized solar cell. Corresponding author Email: [email protected]  

Human receives and transmits various information from the outside world through different sensory systems. The sensory neurons integrate various sensory inputs into a synthetical perception to monitor complex environments, and this fundamentally determines the way how we perceive the world. Developing multifunctional artificial sensory elements that can integrate multisensory perception plays a vital role in future intelligent perception systems, whereas prior spiking neurons reported can only handle single-mode physical signals. Here, we present a bio-inspired haptic-temperature fusion spiking neuron based upon a serial connection of piezoresistive sensor and VO2 volatile memristor. The artificial sensory neuron is capable of detecting and encoding pressure and temperature inputs based on the voltage dividing effect and the intrinsic thermal sensitivity of metal-insulator transition in VO2. Recognition of Braille characters is achieved through multiple piezoresistive sensors, taking advantage of the spatial integration capabilities of such spiking neurons. Notably, the traditionally separate haptic and temperature signals can be fused physically in the sensory neuron when synchronizing the two sensory cues, which is able to recognize multimodal haptic/temperature patterns. The artificial multisensory neuron thus provides a promising approach towards e-skin, neuro-robotics and human-machine interaction technologies.

Authors   Xulei Wu1,2, Bingjie Dang2, Hong Wang5, Xiulong Wu1,*, and Yuchao Yang2,3,4,* 1School of Electronics and Information Engineering, Anhui University, Hefei 230601, China. 2Key Laboratory of Microelectronic Devices and Circuits (MOE), School of Integrated Circuits, Peking University, Beijing 100871, China.

Magnetic miniature robots (MMRs) are mobile actuators that can exploit their size to non-invasively access highly confined, enclosed spaces. By leveraging on such unique abilities, MMRs have great prospects to transform robotics, biomedicine and materials science. As having high dexterity is critical for MMRs to enable their targeted applications, existing MMRs have developed numerous soft-bodied gaits to locomote in various environments. However, there exist two critical limitations that have severely restricted their dexterity: (i) MMRs capable of multimodal soft-bodied locomotion have only demonstrated five-degrees-of-freedom (five-DOF) motions because the sixth-DOF rotation about their net magnetic moment axis is uncontrollable; (ii) six-DOF MMRs have only realized one mode of soft-bodied, swimming locomotion. Here we propose a six-DOF MMR that can execute seven modes of soft-bodied locomotion and perform 3-dimensional pick-and-place operations. By optimizing its harmonic magnetization profile, our MMR can produce 1.41-63.9 folds larger sixth-DOF torque than existing MMRs with similar profiles, without compromising their traditional five-DOF actuation capabilities. The proposed MMR demonstrated unprecedented dexterity; it could jump through narrow slots to reach higher grounds; use precise orientation control to roll, two-anchor crawl and swim across tight openings with strict shape constraints; perform undulating crawling across three different planes in convoluted channels. Keywords: Magnetic materials; soft actuators; miniature robots; locomotion. Corresponding author(s) Email:   [email protected]  

Abstract

"AI & Drug Discovery" mode has significantly promoted drug development and achieved excellent performance, especially with the rapid development of deep learning, making remarkable contributions to protecting human physiological health. However, due to the "black-box" characteristic of the deep learning model, the decision route and predicted results in different research stages assisted by deep models are usually unexplainable, limiting their application in practice and more in-depth research of drug discovery. Focusing on the drug molecules, we propose an explainable fragment-based molecular property attribution technique for analyzing the influence of particular molecule fragments on properties and the relationship between the molecular properties in this paper. Quantitative experiments on 42 benchmark property tasks demonstrate that 325 attribution fragments, which account for 90% of the overall attribution results obtained by the proposed method, have positive relevance to the corresponding property tasks. More impressively, most of the attribution results randomly selected are consistent with the existing mechanism explanations. The discovery mentioned above provides a reference standard for assisting researchers in developing more specific and practical drug molecule studies, such as synthesizing molecular with the targeted property using a fragment obtained from the attribution method.Corresponding author(s) Email:   [email protected]  

AbstractIn recent years, there has been a growing interest in the development of universal soft grippers that can handle objects of varying form factors (including flat objects), surface condition (including moistened or oily objects), and mechanical properties (deformable and fragile). Yet, there is no single gripper that can gently grip objects with such a wide range of properties. In this paper, we present a soft gripper that combines granular jamming (GJ) and electroadhesion (EA) to gently grasp and release a large set of diverse objects. The gripper can operate in GJ mode only, in EA mode only, or in a combination mode that simultaneously activates GJ and EA. In GJ mode, the gripper can grasp objects with different surface properties, lift objects 38 times its own weight using negative pressure, and release objects by applying positive pressure, but has difficulty in handling flat and fragile objects. In EA mode, the gripper can manipulate flat and fragile objects but encounters difficulties with different surface properties such as oily or moistened. In the combination mode, the gripper can generates grasping forces up to 35% higher than in the GJ mode for all object sizes and certain shapes such as a cylinder.IntroductionThe softness of the human hand is a critical factor that allows us to hold, lift, and manipulate a variety of objects and has inspired roboticists to incorporate softness in gripper design and materials. The compliance of soft materials enables passive adaptation of the gripper during grasping operations allowing manipulation of a wide range of objects without bringing additional control complexities.[1,2] In recent years, there has been a growing interest in the development of universal soft grippers that can work with objects of different form factors, rigidity, surface properties, and level of fragility.[3–6] A possible approach to create such highly versatile grippers is to combine different gripping technologies that complement their individual limitations.[1,3,7–10] Yet, it is still challenging to develop a single gripper that can grasp and release objects of different form factor including flat objects, surface conditions (wet, porous, oily, and powdered), and mechanical properties (fragile and deformable).In this paper, we present a soft gripper capable of manipulating different objects with varying physical properties, such as shape, surface conditions, and rigidity. The proposed gripper combines two different technologies: granular jamming to control stiffness and electroadhesion to control adhesion. Here we show that not only does this combination mutually compensate for the limitations of each individual technology, but it also makes the gripper capable of performing multi-stage grasping tasks that consist of diverse grasping and releasing operations on objects made of different material, surface, and shape. The manipulation of a book is an example of multi-stage operation that requires grasping and turning a rigid cover and flipping through single pages.Granular jamming (GJ) enables reversible stiffness change between soft and rigid configurations by means of negative pressure[5,11,12]. High compliance in the soft state allows a GJ gripper to envelope the manipulated object by pressing on it. When negative pressure is applied, the gripper becomes stiff and holds the encaged object.[2] Variable stiffness can also be achieved by integrating phase-change materials that vary mechanical properties under thermal stimulation.[7,13] However, granular jamming offers comparatively faster response time (~100ms), independence from environmental temperature, higher lifting force, easier fabrication, higher robustness, and lower cost.[2,5,14,15] The grasping force produced by granular jamming is sufficient to grasp objects of different morphologies, almost independently of the surface conditions of the object.[5,11,16] The grasping force of GJ grippers can vary from 0.09 to 1.2 kN.[12] GJ has been combined with soft pneumatic actuators to provide more dexterous grasp and lift heavier objects because of the enhanced holding forces.[10,17] However, GJ grippers cannot lift flat objects, such as a sheet of paper. Also, the grasping performance of delicate, fragile and easily deformable objects such as a thin layer of cloth, an egg, or water balloons, as well as larger objects than the active area of the granular bag can be challenging and have not been demonstrated so far.Electroadhesion (EA) instead is an adhesive technology that leverages the shear force generated by electrostatic forces.[18] Electroadhesive pads have been combined with different actuation technologies, such as dielectric elastomer actuation,[1] soft pneumatic actuation,[19] layer jamming,[20] and Fin-Ray structured actuation[21]. The enhanced shear force makes EA-based grippers capable of delicately grasping both flat and fragile objects without squeezing or breaking them.[1,22–26] While the adhesive force of EA pads can be tuned by regulating electrical input, EA effectiveness is highly dependent on the environmental and surface conditions of the object being grasped.[18] In particular electroadhesion is less effective for objects that are greasy, rough, or wet.[2] An additional challenge of soft grippers that rely on electroadhesion is the residual electrostatic charge that remains for a few seconds after removing the voltage and can result in difficult release of light objects.[27]

IntroductionReal world Time on Treatment (rwToT), also known as real world time to treatment discontinuation (rwTTD), is defined as the length of time observed in real world data (as distinct from controlled clinical trials) from initiation of a medication to discontinuation of that medication1,2. The ending of the treatment can be caused by adverse events, deaths, switches of treatment and loss of follow up. Because time to treatment discontinuation can be readily obtained from electronic medical records, this effectiveness endpoint is convenient to evaluate the efficacy of a drug that is already approved for public use3. It is often used as a surrogate effectiveness endpoint, showing high correlation to progression-free survival and moderate-to-high correlation to overall survival4,5. As rwTTD is an important metric for drug effectiveness, it is routinely reported during the post-clinical trial phase2,4,6–9.Calculation of rwTTD in patient population is often equivalent to constructing a (Kaplan-Meier) KM curve, with each point representing the proportion of patients that are still on treatment at a specific time point 1. Either the entire curve, or mean rwTTD, restricted mean10, or the time point at which a specific portion of the patients (e.g. , 50%) dropping treatment is of interest. Currently, there is no existing machine learning scheme established to predict such a curve, or the midpoint, as the vast majority of the machine learning models have been focused on predicting individuals’ behavior rather than population-level behavior. Such a machine learning scheme, if established, has many meaningful clinical applications. For instance, given observed clinical parameters and outcomes in clinical trials, how do we derive expected time-to-treatment in the real-world? Given the rwTTD for a drug on one patient population, how can we predict the rwTTD when applying this drug to another population (e.g. , for a different disease)?This study establishes a machine learning framework to infer population-wise rwTTD. We showed that population-wise curve prediction differs substantially from aggregating all individuals’ results. Our framework models the population-wise curve and is generic to diverse base-learners for predicting rwTTD. We demonstrated the effectiveness of this framework based on both simulated data and real world Electronic medical records (EMR) data for pembrolizumab-treated cancer populations7,11,12. The study opens a new direction of modeling population-level rwTTD, which has great values for directing post-clinical stage drug administrations. This machine learning scheme will also have meaningful implications to population-based predictions for other problems, as machine learning algorithms have so far been focused on predictions for individual samples.

This Supporting Information includes: a comparison of the REAL (Robot Ear Accomplished by Laser) with a typical vibration measuring system (Laser Doppler Vibrometers, LDV), frequency response of various materials on REAL and real-time analysis of REAL audio neural network model. Xiaoping Hong  Email:    [email protected]

AbstractMotile cilia move in an asymmetric pattern and implement a metachronal wave (MCW) to facilitate fluid movement in a viscous environment. Studies have been conducted to mimic MCW movement of motile cilia, but the fabrication process was too complicating or there were difficulties in accurately mimicking the shape of the cilia. To overcome these limitations, we introduce a self-assembly method to fabricate a reprogrammable magnetically actuated self-assembled (RMS) cilia array that can be reprogrammed by changing the magnetization direction through additional magnetization. Using the RMS cilia array, a unilateral cilia array (UCA) channel and a bilateral cilia array (BCA) channel were constructed, and the motion and fluid flow of the RMS cilia array were analyzed by applying different magnetic fields (strike magnetic field and rotating magnetic field). When a rotating magnetic field was applied to the UCA channel, a distinct MCW appeared. In the BCA channel test, fluid pumping was observed when a strike magnetic field, whereas fluid mixing was observed when a rotating magnetic field was applied. Based on these results, it is expected that the proposed RMS cilia array and magnetic field actuation method can be applied to lab-on-a-chip or microfluidic channels for fluid mixing and pumping.1. IntroductionCilia are hair-like, microtubule-based structures that have various distributions with a length of approximately 3–200 µm and an aspect ratio ranging from 10 to 100, depending on the location where they are found, and are divided into primary cilia and motile cilia.[1–5] Motile cilia can move objects or mix fluids by moving mucus or body fluids in the human body.[6,7] Among these motile cilia, the cilia that are found in the fallopian tube of the female reproductive system help the movement of the ovary, and the cilia existing in the lungs mix settled dust and bacteria through mucociliary clearance and move them out of the body.[8–10] The environment in which motile cilia move is normally filled with fluid with a low Reynolds number. In such an environment, the viscous force is generally more dominant than the inertial force, and has a significant influence on the fluid flow.[11–13] To be helpful in this environment, the cilium moves in an asymmetrical pattern comprising an effective stroke and a recovery stroke, creating a net fluid flow. In an effective stroke, the cilium moves in an arc that is fully stretched, while in the recovery stroke, the cilium returns to the starting point in a bent state as if swinging, which increases the moving area of the cilium.[14–16] In addition, when several cilia gather to form a cilia array, they move in a sequential pattern that forms a wave called the metachronal wave (MCW), which helps move the fluid faster and more efficiently because of their asymmetrical motion.[17–20]Several studies have reported mimicking the asymmetric motion of cilia and the MCW motion to efficiently pump or mix fluids in microfluidic devices with low Reynolds numbers.[21–23] To mimic cilia motion, many actuation methods have been used; actuation via a magnetic field is the most used method among them.[24–27] In addition, diverse manufacturing methods exist for magnetically actuated artificial cilia, and, the fabrication method using self-assembly has the advantages of simplicity and capability to mimic the appearance of natural cilia. However, it is difficult to program the magnetization direction, thus limiting the implementation of the MCW of the cilia.[28–31]Nevertheless, studies have been conducted to imitate the metachronal wave of cilia by fabricating artificial cilia using the molding method, which facilitates reprogramming.[32,33] Nelson formed a cilia array using a molding method and then reprogrammed the cilia array, which moved the cilia array to form an MCW.[34] Sitti fabricated micro-cilia using a mold, magnetized each cilium independently, and attached them to form a cilia array with the desired arrangement that implements the MCW.[35] However, these studies actuated the cilia array using only a rotating external magnetic field, and because the cilia array was fabricated using the molding method, several complex steps were required for making the cilia array.

Manipulation strategies based on the passive dynamics of soft-bodied interactions provide robust performances with limited sensory information. They utilise the kinematic structure and passive dynamics of the body to adapt to objects of varying shapes and properties. However, these soft passive interactions make the state of the robotic device influenced by the environment, making control generation and state estimation difficult. This work presents a closed-loop framework for dynamic interaction-based grasping that relies on two novelties: (i) a wrist-driven passive soft anthropomorphic hand that can generate robust grasp strategies using one-step kinaesthetic teaching and (ii) a learning-based perception system that uses temporal data from sparse tactile sensors to predict and adapt to failures before it happens. With the anthropomorphic soft design and wrist-driven control, we show that controllers can be generated robust to novel objects and location uncertainty. With the learning-based high-level perception system and 32 sensing receptors, we show that failures can be predicted in advance, further improving the robustness of the entire system by more than doubling the grasping success rate. From over 1000 real-world grasping trials, both the control and perception framework are also seen to be transferable to novel objects and conditions. Corresponding author(s) Email:  _ [email protected]  _

In existing surgery process, surgeons need to manually adjust the laparoscopes to provide a better field of view during operation, which may distract surgeons and slow down the surgery process. This paper presents a data-driven control method that uses a continuum laparoscope to adjust the field of view by tracking the surgical instruments. A Koopman-based system identification method is firstly applied to linearize the nonlinear system. Shifted Chebyshev polynomials are used to construct observation functions that transfer low-dimension observations to high-dimension ones. The Koopman operator is approximated using a finite-dimensional estimation method. An optimal controller is further developed according to the trained linear model. Furthermore, a learning-based pose estimation framework is designed to detect keypoints on surgical instruments and provide visual feedback for adjusting the laparoscope. Compared with other detection methods, the proposed scheme achieves a higher detection precision and provides more optional keypoints for tracking. Simulation and experiments validate the feasibility of the proposed control method. Experiment results show that the proposed method can automatically adjust the field of continuum laparoscope through tracking surgical instruments in a timely manner and the number of surgical tools is not limited.

While most deep learning approaches are developed for single images, in real world applications, images are often obtained as a series to inform decision making. Due to hardware (memory) and software (algorithm) limitations, few methods have been developed to integrate multiple images so far. In this study, we present an approach that seamlessly integrates deep learning and traditional machine learning models, to study multiple images and score joint damages in rheumatoid arthritis. This method allows the quantification of joining space narrowing to approach the clinical upper limit. Beyond predictive performance, we integrate the multilevel interconnections across joints and damage types into the machine learning model and reveal the cross-regulation map of joint damages in rheumatoid arthritis. Corresponding author(s) Email:   [email protected] or [email protected]

The metaverse, where the virtual and real world are fused, is currently under rapid development. Immersive and vivid experience in the metaverse requires human-machine interaction devices that, unlike those currently available, are simultaneously imperceptible, convenient to use, inexpensive, and safe. In this study, we propose and realize an optical-nanofiber-based gesture-recognition wristband that can accurately recognize gestures and use them to interact with a robotic hand. Requiring only three optical-nanofiber-based pressure sensors, the wristband is simple in structure, convenient to use, and remarkably imperceptible to the user. With the assistance of a machine-learning algorithm, a maximum recognition accuracy of 94% is achieved for testers with different physiques. A robotic hand can be remotely controlled by the wristband through gestures. The wristband has broad application prospects and is a promising solution for advanced human-machine-interaction devices.

This Supporting Information includes the extended description of the superposition state of the asymmetric double-well system in vacuum system and in solution, truth tables for the residue pairs and their corresponding quantum logic gates, and figures for the double well potential energy surfaces and transmission spectra of the residue pairs.Corresponding Authors Email: [email protected] and [email protected]   

Abstract:There is a lack of reliable prognostic biomarkers for hypoxic-ischemic (HI) brain injury in preterm infants. Herein, spectrally detailed wavelet scalograms (WSs), derived from the 1024 Hz sampled electroencephalograms (EEG) of preterm fetal sheep after HI (n ¼ 7), are infused into a high-performance deep convolutional neural network (CNN) pattern classifier to identify high-frequency spike transient biomarkers. The deep WS-CNN pattern classifier identifies EEG spikes with remarkable accuracy of 99.81 ¼ 0.15% (area under curve, AUC ¼ 1.000), cross-validated across 5010 EEG waveforms, during the first 6 h post-HI (42 h total), an important clinical period for diagnosis of HI brain injury. Further, a feature-fusion strategy is introduced to extract the spectrally dominant features of the raw EEG epochs to form robust 3D input matrix sets to be infused into the deep 2D-CNNs for pattern classification. The results show that the proposed WS-CNN approach is less sensitive to the potential morphological variations of spikes across all subjects compared to other deep CNNs and spectral-fuzzy classifiers, allowing the user to flexibly choose an approach depending on their computational requirements. Collectively, the data provide a reliable framework that could help support well-timed diagnosis of at-risk neonates in clinical practice.

In this study, we utilize simple light-emitting diodes (LEDs) and photodetectors (PDs) combined with an intelligent shape decoding framework to enable 3D shape sensing of a self-contained flexible substrate. Finite element analysis (FEA) is leveraged to optimize the LED-PD layout and enrich ground-truth data from sparse to dense points for model training. The mapping from light intensities to overall sensor shape was achieved with an autoregression-based model that considers temporal continuity and spatial locality. The sensing framework was evaluated on an A5-sized flexible sensor prototype and a fish-shaped prototype, where sensing accuracy (RMSE = 0.27 mm) and repeatability (Δ light intensity < 0.31% over 1000 cycles) were tested underwater. We validate an affordable alternative to FBG sensors with high-order sensing outputs, where demonstrations are supplemented in the below videos.