Inclusion Criteria:

• Parts 1, 2, and 5: Participants with histologically or cytologically confirmed metastatic castration-resistant prostate cancer (mCRPC) who are refractory to a novel antiandrogen therapy (abiraterone acetate and/or enzalutamide, apalutamide, or darolutamide) and have failed at least 1 (but not more than 2) taxane regimens including for metastatic hormone-sensitive prostate cancer (mHSPC) (or who are deemed medically unsuitable to be treated with a taxane regimen or have actively refused treatment with a taxane regimen). Note: A taxane regimen is defined as a minimum exposure of 2 cycles of a taxane. Any NHT that has been administered and has been stopped for reasons other than progression will not be counted as an additional line of treatment.
  1. Dose exploration phase: Novel antiandrogen therapy must have been given for treatment of metastatic disease.
  2. Dose expansion phase: participants must not have had more than 2 NHTs and 2 taxane regimens in any setting, and an additional up to 2 other systemic anti-cancer treatments are allowed (eg, anti-PD1, PARP inhibitors, radioligand therapies, sipuleucel-T, experimental agents) Note: Combinations are considered one systemic anti-cancer treatment.
• Part 3: Participants with histologically or cytologically confirmed mCRPC who are refractory to a NHT (abiraterone acetate, enzalutamide, apalutamide, or darolutamide) given in any disease setting and who are deemed medically unsuitable to be treated with a taxane regimen or have actively refused treatment with a taxane regimen.
• Parts 4A and 4B:
  1. Participants with histologically or cytologically confirmed mCRPC who have received no or 1/2 prior NHT (given in any disease setting depending on the part), and no or 1 taxane (given for hormone sensitive prostate cancer).
  2. Dose-expansion phase: at least 1 prior NHT must have been given; 0-1 prior PARP inhibitors are acceptable.
  3. 4A: Participants planning to receive abiraterone acetate for the first time (participants who received prior abiraterone acetate are not eligible). Participants may have had exposure to up to 2 NHTs with a similar mechanism of action (apalutamide, enzalutamide or darolutamide) in the non-mCRPC and mCRPC setting.
  4. 4B: Participants planning to receive enzalutamide for the first time (participants who received prior enzalutamide/apalutamide or daralutamide are not eligible).
• All parts:
• Participants must have undergone bilateral orchiectomy or be on continuous androgen-deprivation therapy with a gonadotropin releasing hormone (GnRH) agonist or antagonist.
• Total serum testosterone <= 50 ng/dL or 1.7 nmol/L.
• Evidence of progressive disease, defined as 1 or more Prostate Cancer Working Group 3 (PCWG3) criteria:
  1. PSA level >= 1 ng/mL that has increased on at least 2 successive occasions at least 1 week apart.
  2. nodal or visceral progression as defined by Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 with PCGW3 modifications.
  3. appearance of 2 or more new lesions in bone scan.
• Eastern Cooperative Oncology Group performance status of 0-1.
• Adequate organ function, defined as follows:
  1. Hematological function:
     1. absolute neutrophil count >= 1 x 10^9/L (without growth factor support within 7 days from screening assessment).
     2. platelet count >= 75 x 10^9/L (without platelet transfusion within 7 days from screening assessment).
     3. hemoglobin >= 9 g/dL (90 g/L) (without blood transfusion within 7 days from screening assessment).
  2. Renal function:

     1. estimated glomerular filtration rate based on Modification of Diet in Renal Disease calculation >= 30 ml/min/1.73 m^2.

  3. Hepatic function:
     1. aspartate aminotransferase (AST) and alanine aminotransferase (ALT) < 3 x upper limit of normal (ULN) (or < 5 x ULN for participants with liver involvement).
     2. total bilirubin (TBL) < 1.5 x ULN (or < 2 x ULN for participants with liver metastases).
  4. Cardiac function:
     1. left ventricular ejection fraction > 50% (2-D transthoracic echocardiogram [ECHO] is the preferred method of evaluation; multi-gated acquisition scan is acceptable if ECHO is not available).
     2. Baseline electrocardiogram (ECG) QTcF <= 470 msec (average of triplicate values).

Exclusion Criteria:

• Pathological finding consistent with pure small cell, neuroendocrine carcinoma of the prostate or any other histology different from adenocarcinoma.
• Radiation therapy within 4 weeks of first dose (or local or focal radiotherapy within 2 weeks of first dose).
• Untreated central nervous system (CNS) metastases or leptomeningeal disease. Participants with a history of treated CNS metastases are eligible if there is radiographic evidence of improvement upon the completion of CNS-directed therapy and no evidence of interim progression between the completion of CNS-directed therapy and the screening radiographic study.
• Participants with symptoms and/or clinical signs and/or radiographic signs that indicate an acute and/or uncontrolled active systemic infection within 7 days prior to the first dose of investigational product administration.
• Confirmed history or current autoimmune disease or other diseases resulting in permanent immunosuppression or requiring permanent immunosuppressive therapy.
• History of arterial or venous thrombosis (eg, stroke, transient ischemic attack, pulmonary embolism, or deep vein thrombosis); for arterial thrombosis within 12 months of AMG 509 initiation; for venous thrombosis, 6 months and stable on anti-coagulation.
• Myocardial infarction and/or symptomatic congestive heart failure (New York Heart Association > class II) within 12 months of first dose of AMG 509 with the exception of ischemia or non-ST segment elevation myocardial infarction controlled with stent placement and confirmed by a cardiologist more than 6 months prior to first dose of AMG 509.
• Any anti-cancer therapy or immunotherapy within 4 weeks of start of first dose, not including luteinizing hormone-releasing hormone (LHRH)/GnRH analogue (agonist/antagonist). Participants on a stable bisphosphonate or denosumab regimen for >= 30 days prior to enrollment are eligible.
• Prior PSMA radionuclide therapy within 2 months prior to AMG 509 unless participant received < 2 cycles (Note: a participant cannot have received PSMA radionuclide therapy < 35 days prior to enrollment if 1 cycle was given). Parts 3 and 4: prior PSMA radionuclide therapy is prohibited.