Wednesday, May 11, 2011

Still Can't Figure out ZLib and Ruby

‹prev | My Chain | next›

I am about ready to give up on Ruby and Zlib decompression in SPDY. But one more try.

For the past few nights, I have been trying to figure out how to decompress the second of two packets sent to my SPDY backend. What this boils down to is:
$: << 'lib' << '../lib'                                                                                                                    
require 'spdy'
d1 = "8\xEA\xDF\xA2Q\xB2b\xE0a`\x83\xA4\x17\x06{\xB8\vu0,\xD6\xAE@\x17\xCD\xCD\xB1.\xB45\xD0\xB3\xD4\xD1\xD2\xD7\x02\xB3,\x18\xF8Ps,\x83\x9Cg\xB0?\xD4"
d2 = "b\xE0\x82\xC7\x1630\xDC\a&\xC4\x89\nX,\xC1\xC8\xA3\x9F\x96X\x96\t\x8C("
SPDY::Zlib.inflate d1
=> "\x00\f\x00\x06accept\x00?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\x00\x0Eaccept-charset\x00\x1EISOon/xhtm"
SPDY::Zlib.inflate d2
RuntimeError: invalid stream
from /home/cstrom/repos/spdy/lib/spdy/compressor.rb:35:in `inflate'
from (irb):6
from /home/cstrom/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in `<main>'
Per a comment to last night's post, it seems that I might somehow need to remember the "dictionary" being used with the compressed data between the two packets. SPDY::Zlib.inflate is trying to recreate the whole setup each time through and it is clearly not working.

So I give this a shot:
DICT = \
"optionsgetheadpostputdeletetraceacceptaccept-charsetaccept-encodingaccept-" \
"languageauthorizationexpectfromhostif-modified-sinceif-matchif-none-matchi" \
"f-rangeif-unmodifiedsincemax-forwardsproxy-authorizationrangerefererteuser" \
"-agent10010120020120220320420520630030130230330430530630740040140240340440" \
"5406407408409410411412413414415416417500501502503504505accept-rangesageeta" \
"glocationproxy-authenticatepublicretry-afterservervarywarningwww-authentic" \
"ateallowcontent-basecontent-encodingcache-controlconnectiondatetrailertran" \
"sfer-encodingupgradeviawarningcontent-languagecontent-lengthcontent-locati" \
"oncontent-md5content-rangecontent-typeetagexpireslast-modifiedset-cookieMo" \
"ndayTuesdayWednesdayThursdayFridaySaturdaySundayJanFebMarAprMayJunJulAugSe" \
"pOctNovDecchunkedtext/htmlimage/pngimage/jpgimage/gifapplication/xmlapplic" \
"ation/xhtmltext/plainpublicmax-agecharset=iso-8859-1utf-8gzipdeflateHTTP/1" \
".1statusversionurl\0"


CHUNK = 10*1024
in_buf = FFI::MemoryPointer.from_string(d1)
out_buf = FFI::MemoryPointer.new(CHUNK)
zstream = FFI::Zlib::Z_stream.new
zstream[:next_in] = in_buf
zstream[:next_out] = out_buf
result = FFI::Zlib.inflateInit(zstream)

result = FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)

result = FFI::Zlib.inflateSetDictionary(zstream, DICT, DICT.size)
result = FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)
out_buf.get_bytes(0, zstream[:total_out])
=> "\x00\f\x00\x06accept\x00?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\x00\x0Eaccept-charset\x00\x1EISOon/xhtm"
So far, so good. That is nearly an exact duplication of how SPDY::Zlib.inflate does its thing. But how to re-use the stream with the dictionary set?

I fiddle around a bit, but nothing seems to work. This is my best guess:
# Try to re-use zstream to retain dictionary / context

zstream[:next_in] = in_buf
zstream[:next_out] = out_buf
result = FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)
=> -5

result = FFI::Zlib.inflateInit(zstream)
result = FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)
=> -5
Bah!

More than a little disillusioned, I give the non-FFI Zlib a try, but with similar luck:
require 'zlib'
zstream = Zlib::Inflate.new
buf = zstream.inflate(d1)
Zlib::NeedDict: need dictionary
from (irb):48:in `inflate'
from (irb):48
from /home/cstrom/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in `
'
buf = zstream.inflate(d1)
Zlib::DataError: invalid distance too far back
from (irb):51:in `inflate'
from (irb):51
from /home/cstrom/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in `
'

# Crazy, I have to send in an empty string?!

ruby-1.9.2-p0 > buf = zstream.inflate("")
=> "\x00\f\x00\x06accept\x00?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\x00\x0Eaccept-charset\x00\x1EISOhtml,ap"


# But it does not work on subsequent data
ruby-1.9.2-p0 > buf = zstream.inflate(d2)
Zlib::DataError: invalid distance too far back
from (irb):59:in `inflate'
from (irb):59
from /home/cstrom/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in `
'
ruby-1.9.2-p0 > buf = zstream.inflate("")
=> ""
Grr.... Once again, I am thwarted and have to call it a night without good resolution. I think tomorrow I will try this in a different language.


Day #16

2 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete