Trouble modifying CSV with switch statement ?

I have a simple csv-file like this:

Company;Department
aaa;abc
bbb;xyz

With this code

$b=import-csv test.csv -Delimiter ';' 
$b | %{
    switch ($_.department){
        'abc' {$_.company="companyX"}
        default {$_.company="companyY"}
    }
}

I get this error:

Property 'company' cannot be found on this object; make sure it exists and is settable.
At line:3 char:16
+         'abc' {$_.company="companyX"}
+                ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : PropertyAssignmentException

Changing the switch to an If like this:

$b | %{
    if($_.DEPARTMENT -match 'a*'){$_.COMPANY="companyX"}
    if($_.DEPARTMENT -match 'x*'){$_.COMPANY="companyY"}
}

This works but all rows are changed

Company         Department                                                                           
-------         ----------                                                                           
companyY        abc                                                                                  
companyY        xyz                                                                                  

What am I missing here?

I would like the switch approach to work and also that only the "current" row is changed.