[Mono-devel-list] Crypto inconsistency between Mono and MS.NET

[Jon Larimer]

Hello,

I'm working on getting some code running on the Microsoft.NET platform
to talk to it's Linux counterpart running on Mono. We were using
Rijndael in CBC mode to encrypt packets before transmitting data over
the network. Mono was decrypting every packet after the first one
received "wrong", it produce gibberish, so I dug in to try to figure
out the problem.

I found out that there's some inconsistency between Mono and
Microsoft.NET's TransformFinalBlock() method. I believe that
Microsoft's implementation is the broken one. It seems like after
TransformFinalBlock() is called on an ICryptoTransform object, the
temporary buffer used for CBC transforms get overwritten with the
original IV in Microsoft's implementation. The result of this is that
you use TransformFinalBlock() to encrypt a block of data twice, the
resulting crypto text will be exactly the same, defeating the purpose
of using CBC in the first place. In the Mono implementation, the CBC
temporary buffer is not modified after a call to TransformFinalBlock.

Anyways, if you run the attached code in the Microsoft.NET and Mono
platforms, you can see the difference:


Microsoft.NET:
--------------
TransformBlock test:
  First round of CBC:
  6F 5A A1 28 67 88 40 B9 5F 8B 0E 61 37 57 52 A1

  Second round of CBC:
  41 24 AA 82 5F 92 1E FF D9 DA F4 F0 F3 42 84 92


TransformFinalBlock test:
  First round of CBC:
  6F 5A A1 28 67 88 40 B9 5F 8B 0E 61 37 57 52 A1

  Second round of CBC:
  6F 5A A1 28 67 88 40 B9 5F 8B 0E 61 37 57 52 A1


Mono:
-----
TransformBlock test:
  First round of CBC:
  6F 5A A1 28 67 88 40 B9 5F 8B 0E 61 37 57 52 A1

  Second round of CBC:
  41 24 AA 82 5F 92 1E FF D9 DA F4 F0 F3 42 84 92


TransformFinalBlock test:
  First round of CBC:
  6F 5A A1 28 67 88 40 B9 5F 8B 0E 61 37 57 52 A1

  Second round of CBC:
  41 24 AA 82 5F 92 1E FF D9 DA F4 F0 F3 42 84 92


In the copy compiled with Visual Studio and ran on Microsoft.NET, the
results of the CBC operation is the same twice in a row when using
TransformFinalBlock.

The code I was working on used TransformBlock() to encrypt individual
blocks and then used TransformFinalBlock() on the last one. The
problem manifested itself on Mono, when the application received and
decrypted a packet, then failed to decrypt the next packet received --
because the .NET framework reset the CBC temporary buffer with the IV,
and the Mono version had not.

I eventually fixed my problem by adding the padding myself and not
calling TransformFinalBlock().

Now, my question is, should Mono adopt the (IMHO) broken Microsoft
behavior in an effort to be compatible, or stay (IMHO) correct? I can
submit a patch to mimic the Microsoft behavior if necessary.

Can anyone forward this to someone responsible for
System.Security.Cryptography at Microsoft to see what they think? It's
a security issue if using TransformFinalBlock() on an ICryptoTransform
object in CBC mode behaves as if it's EBC, unless of course it's
supposed to and it's just not documented. :-)

Thanks,
-jon

Jon Larimer
jlarimer at gmail.com
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: TransformTest.cs
Url: http://lists.ximian.com/pipermail/mono-devel-list/attachments/20041020/237d5d7d/attachment.pl