Due to their pivotal roles in tumor progression and therapy resistance, cancer-associated fibroblasts (CAF) are considered key therapeutic targets with loss of stromal androgen receptor (AR) a poorly understood hallmark of aggressive prostate cancer (PCa). A paucity of pre-clinical models however has hampered functional studies of CAF heterogeneity. We demonstrate that our newly generated CAF biobank contains three FAP⁺-fibroblast subtypes, each with unique molecular and functional traits. Cultures with an early-activated phenotype expressed the highest levels of AR and exhibited AR-dependent growth, whereby AR inhibition suppressed their migration. Consistently, stromal cells expressing early-activation markers co-expressed nuclear AR in clinical specimens and were enriched in pre-neoplastic lesions/low-grade PCa. Conversely, myofibroblastic CAF (myCAF) expressed low AR levels in vitro and in vivo, were insensitive at the proliferative and migratory levels to AR signaling modulation and significantly promoted PCa cell invasion in 3D composite collagen networks. Accordingly, myCAF constituted the predominant CAF subpopulation in stromogenic high-grade PCa and were enriched in aggressive disease states in PCa single cell atlases and castration-resistant LACP9 patient-derived xenografts. Exacerbation of the myCAF state upon castration of LAPC9-bearing hosts underscored these findings. Mechanistically, AR loss in myCAF was driven by an NFκB-TGFβ1-YAP1 axis, whose combined pharmacological or genetic targeting synergistically repressed myofibroblastic hallmarks and impaired autophagic flux, effects that were potentiated by enzalutamide resulting in enhanced myCAF cell death. Collectively, data herein provide a mechanistic rationale for stromal AR loss in aggressive PCa and suggest that adjuvant targeting of the YAP1-TGFβ signaling axis may improve patient outcome.