Results
Our analysis of simulated datasets showed the rate of true positives (probability of a core taxon assigned as such or signal) is close to one in many cases and appears to provide support for the ability of those methods to correctly assign core taxa (Figure 1a, denoted in blue). Furthermore, the rate of false positives (probability of a non-core taxon assigned as core member or noise) is close to zero in many cases seemingly providing additional support for core assignment methods (Figure 2b, denoted in blue). However, when examined individually these two metrics only tell half the story, as we are concerned with the ability of a given method to accurately identify the core taxa (i.e. true positives), while not over inflating membership through inclusion of non-core taxa (i.e. false positives); thus, being able to discern signal from noise.
The net assignment scores for simulations revealed the inability of the methods to accurately assign core membership (Figure 1c). The net assignment value quantifies the absolute difference in true positives (signal) and false positives (noise), with a net assignment value of 25 meaning the method assigned all of the correct taxa to the core with no erroneous assignments and smaller values indicating poorer performance in accurate core assignment. Our results show that rarely did the methods accurately recover the correct number and identity of core taxa (those simulated to be included in the core). In general, a large difference in the abundance of core and non-core taxa (πcorenon-core, with varying degrees of precision), led to the greatest success in accurate identification of the correct 25 core taxa (Figure 1c, right side x-axis, success denoted by dark blue squares, white and red indicate poor performance). When comparing results of the four core assignment criteria, the proportion of sequence replicates and proportion of sequence reads and replicates methods most often accurately assigned the 25 core taxa, with multiple instances of a net assignment value >24 (Figure 1c). The two methods tha0t utilized the proportion of replicates produced similar results in our simulations. They were followed by the hard cutoff method and then the cumulative proportion of sequence reads method (Figure 1c). All methods, with the exception of the cumulative proportion of sequence reads, were able to accurately recover the known core in some circumstances (net assignment value >24). However, they did so for different ranges of parameter combinations, suggesting each method may better suited to different taxon distributions.
Even though core methods accurately assigned core membership in some circumstances, the same methods produced negative net assignment values in other situations, consistent with overestimation of core membership. Core inclusion was most severely overestimated in the cumulative proportion of sequence reads and hard cutoffs methods in simulations with low πcore to πnon-core ratio and high precision (parameterized by θ). This overestimation manifested as a high false positive rate (noise) in certain simulated communities. In general, the methods based on proportionality tended to assign the smallest set of core taxa and possessed the best net assignment value (i.e. correct assignment of known core taxa and limited erroneous assignment of non-core taxa to the core) and as such could be considered the most conservative.
For the two published datasets, the four core methods led to different conclusions, with the inferred core corresponding to 1.21%-15.74% of total taxa (Table 2). All methods assigned taxa with high abundance to the core, though methods differed in their assignments with respect to CV among replicates (Figure 2). More specifically, the cumulative proportion of sequence reads method and the proportion of sequence replicates method included highly abundant taxa regardless CV in both datasets. The method based on proportionality of replicates and sequence reads selected only abundant taxa with a relatively low CV in the human microbiome dataset (Figure 2a) and selected abundant taxa regardless of CV in the Arabidopsis dataset (Figure 2b). Finally, the core method that uses both the proportion of reads and replicates appear to arbitrarily exclude taxa with relatively high mean abundance and low CV, taxa that fit multiple criteria for core membership. This is especially evident in the Human Microbiome Project dataset. These exclusions highlight problems associated with assigning continuously distributed count data into core and non-core groups.
Examination of core assignments in the published datasets showed that co-assignment (i.e. common core assignment by multiple methods) varied depending on the dataset (Figure 3). The Human Microbiome Project dataset yielded 176 core assignments that were assigned by all four methods (9.5% of total unique core assignments). The Arabidopsisdataset produced 165 core assignments that were shared among all four methods (8.1% of total unique core assignments). These common core assignments equivalate to 1.49% and 1.1% of the total number of taxa in each taxon table, respectively. For the Arabidopsis dataset, 758 taxa (37.2% of total unique core assignments) were assigned to the core by two methods and 322 taxa (15.8% of total unique core assignments) by three methods. As for the Human Microbiome Project dataset, 404 taxa (21.8% of total unique core assignments) were assigned to the core by three methods, and 530 taxa ( 28.6% of total unique core assignments) were assigned by two methods.
Comparisons of differences in beta-diversity between assigned cores and the full datasets, showed that in some cases the core datasets matched the entire dataset, but this was not always true. The entireArabidopsis dataset showed both developmental stage and genotype to be significant in structuring the community (p=0.001); this was true for both the Bray-Curtis and Jaccard dissimilarity indices. The taxon table including only taxa assigned by all four methods matched these results (p=0.001) when using Bray-Curtis dissimilarity, but the Jaccard index only resulted in a significant effect of developmental stage (p=0.001) with genotype not significant predictor (p=0.132). Beta-diversity analysis of the core communities based on each of the four core-assignment methods separately mostly produced the same effects on beta-diversity as observed for the entire dataset, except the hard cutoff method. However, this method had comparable results to the taxon table created from taxa co-assigned by all four methods, with developmental stage being significant for both the Bray-Curtis and Jaccard dissimilarity indices (p=0.001), and genotype being significant for the Bray-Curtis index (p=0.001) but not Jaccard (p=0.148).
As for the Human Microbiome Project dataset, estimates of beta-diversity were affected by the use of core taxa or all taxa, raising concern for interpretation and the validity of core assignments. The full Human Microbiome Project stool dataset showed both sex and sequencing center to be significant (p<0.01), while visit number was shown to be statistically insignificant (p>0.05). These results were true for both Bray-Curtis and Jaccard dissimilarity indices. When examining only taxa assigned by all four core assignment methods, visit number, sex, and sequencing center were all significant (p<0.05) with Bray-Curtis dissimilarity, but only sequencing center was significant (p<0.001) with the Jaccard index. Results of beta-diversity analysis based on the core communities determined by each of the four core-assignment methods were similar to the results from the full dataset for both dissimilarity indices, except for the proportion of replicates reps and reads method. While the proportion of replicates and reads agreed with the others on the significance of sequencing center, this core assignment method showed visit number to be significant (p<0.05) for Bray-Curtis dissimilarity and insignificant for Jaccard dissimilarity (p>0.05). As for sex, Jaccard dissimilarity was insignificant (p>0.05) and Bray-Curtis dissimilarity was significant (p< 0.01).
Comparison of core assignments to taxa deemed important by their degree centrality revealed further disagreement. The Arabidopsis dataset produced 2,258 taxa that were deemed important, either by any of the four core assignment methods or by the cooccurrence network (Figure 4a,c). Of these 2,258 taxa, 1655 (73.3%) were uniquely assigned by the core methodologies, while 222 (9.8%) were assigned by the network alone. A small number of taxa, 381 (16.9%), was identified by both core assignment methods and the network analysis. The average degree centrality of taxa assigned as core by any method was 9.4, while the average degree centrality of non-core taxa was 0.25. This dataset produced large number of taxa deemed important by solely core assignment, with 1655 core taxa possessing zero significant edges in the network. The top 62 taxa, determined by degree centrality, were identified by the core assignment methods as well. The taxa with the highest degree centrality not picked up by core methods had a degree centrality of 118. On the other hand, the human microbiome project dataset produced very different results, with 3,181 taxa being identified as important by either the any of the core assignment methods or the network (Figure 4b,d), and almost half, 1586 (49.86%), identified by both core assignment methods and the network analysis. Of these 3,181 taxa, only a small portion of 264 (8.3%) were uniquely assigned by the core methodologies, while 1331 (41.84%) were assigned by the network alone. The average degree centrality of taxa assigned as core by any method was 22.9, while the average degree centrality of non-core taxa was 2.2. In the human microbiome network, the top 61 taxa, in terms of degree centrality, were identified by both the cooccurrence network and core assignment methods. The taxa with the highest degree centrality not picked up by core methods had a degree centrality of 98.