One of the partners we had integrated with wanted us to send the extended characters in the output file.

Those characters would look something like this  »    «

It was a easy change for us in code, so we made, deployed to production as well.. Later we realized that a different character shows along with the ones intended to show.

The file had an output as   «   instead of just «.

To add more to the perplexity, This character «  would not show up when the file is opened in text editors like notepad or notepad++, but shows up in other editors like “7-Edit”. So at first we ignored considering that problem is with the editor.

Later we heard back from the client with the same issue and they wanted us to fix it.

On deep diving, we realized that, in-fact it was an issue in one of our pipeline component, where we converted the output message to UTF8 encoding using the code below

byteArray = Encoding.UTF8.GetBytes(strMessage);

pInMsg.BodyPart.Data = new MemoryStream(byteArray);

But instead it should be left to default encoding.

byteArray = Encoding.GetEncoding(0).GetBytes(strMessage);

pInMsg.BodyPart.Data = new MemoryStream(byteArray);

So once the code was changed, the issue got fixed and made our client happySmile

Articles referred :-