APOBEC enzymes, including APOBEC3B (A3B), have been linked to recurrent mutational patterns in breast cancers (BC). Currently, there is considerable level of evidence that A3B mRNA expression is associated with poor prognosis in ER-positive BC. Since the clinical relevance of A3B at the protein level has not been well studied, we evaluated A3B protein expression by immunohistochemistry on tumor tissue microarrays. Samples from 646 invasive, non-metastatic primary ER-positive BC patients who did not receive any systemic endocrine therapy as standard of care treatment, allowed us to study the pure prognostic impact of A3B protein expression on disease-free survival (DFS), metastatic disease-free survival (MFS), breast cancer specific-survival (BCSS), and overall survival (OS). Furthermore, 220 tumor specimens from BC patients who relapsed and did receive tamoxifen as first-line palliative treatment were included to study the association of A3B protein expression and progression free survival (PFS). Cox regression models were used to determine the impact of A3B protein expression on prognosis and PFS. We found that A3B expression associates with larger and higher grade tumors. Furthermore, univariable analyses showed that high A3B expression was associates with a shorter DFS (HR = 1.68; 95%CI = 1.31–2.14), MFS (HR = 1.93; 95%CI = 1.45–2.56), BCSS (HR = 2.72; 95%CI = 1.89–3.90), and OS (HR = 1.89; 95%CI = 1.40–2.56). Also, a shorter PFS (HR = 1.41; 95%CI = 1.03–1.92) was found. After adjustment for confounders, the observed associations remained significant. To conclude, in ER-positive BC, A3B protein expression is a marker for poor disease outcome and for rapid progression during endocrine treatment.