Instrument transformer is used to measure the signal parameters (voltage and current) to indicate in the meters (voltage and current) where as protection transformer measuring signal is used for relay operation for protection of transformer.
Instrument transformer can be operated at near core saturation limit as the meter indicates up to rated values but the protection transformer should be available for 50 % higher than rated current so these should be operated below the core saturation.
The accuracy should be high for Instrument transformers where as in protection it can be somewhat less.