A primary key constraint that is referenced by a foreign key cannot be removed until the foreign key constraint has first been deleted.
If a column was first defined as not null, and then as a primary key, the not null constraint on the column cannot be removed until the primary key constraint has been removed. Removing a primary key constraint does not remove the NOT NULL part of the constraint. That must be removed separately.