The buffers positions will be advanced to reflect the bytes read and the characters written, but their marks and limits will not be modified. </

so if we have [A0][A1][B0], invoking decode method will decode first char successfully but it will fail on the second char with CR_MALFORMED result, my point here is the buffer will still have 1 byte remaining so we can resume decoding again when more bytes available. I have setup a test unit to simulate the network packets.


//Snip of NioSaxSource
private CharsetDecoder decoder;
private ByteBuffer buf;
private final CharBuffer tmp = CharBuffer.allocate(1);
private int remaning;

public final boolean hasCharacter() {
return buf != null && remaning != buf.remaining() && buf.
hasRemaining();
}

public final char decode() {
if (hasCharacter()) {
tmp.rewind();
CoderResult result = decoder.decode(buf, tmp, true);
if (result.isError()) {
remaning = buf.remaining();
} else {
remaning = 0;
return tmp.get(0);
}
}
return NOT_ENOUGH_DATA;
}


And the test unit

@Test
public void testRandomSequence() throws Exception {
byte[] b = createXMLUTF8ByteArray();
Random rand = new Random();
List parts = new ArrayList();
int min = 0, max = 0;
while (max != b.length) {
max = rand.nextInt(b.length - min) + min;
if (!parts.contains(max)) {
if (max == b.length - 1) {
max += 1;
}
parts.add(max);
}
min = max;
}
System.out.println(parts + " " + b.length);
start();
min = 0;
ByteBuffer f = ByteBuffer.allocate(b.length);
source.setByteBuffer(f);
for (Integer i : parts) {
f.mark();
f.limit(i);
f.position(min);
f.put(b, min, i - min);
f.reset();
min = i;
parser.parse(source);
}
stop();
}


Heres the problem, say you receive from the network a couple of UTF-16 A & B characters. This would be 4 bytes in total:

[A0][A1][B0][B1]

Due to the network fragmenting the packet the second one is only partially received (the second byte is still in transit). In this case our ByteBuffer contains:

[A0][A1][B0]

Then the nio API would fail as the second character is incomplete. What I do is to decode up the the beginning of the partial but leave it in the ByteBuffer – hence the NOT_ENOUGH_DATA state. Then when you return that buffer to NIO, it then appends the next block from the network which happens to have [B1], the char is then complete and it can be decoded.


// Snip of NioSaxSource
public final boolean isValid(final char c) {
return c != NOT_ENOUGH_DATA && c != INVALID_CHAR;
}

public final char decode() {
if (!hasCharacter()) {
return NOT_ENOUGH_DATA;
}
b.rewind();
//Where decoder = charset.newDecoder() and b = CharBuffer.allocate(1)
CoderResult result = decoder.decode(buffer, b, true);
if (result.isError()) {
return INVALID_CHAR;
}
return b.get(0);
}


Thanks again for sharing

Not right
Tatoo is a general parser generator that is able to produce NIO based push parsers.
see http://portal.acm.org/citation.cfm?id=1529707

Rémi

As for licence, it’s BSD.

I will look into more details of into your project.
BTW, I also have a project which contains some Core libraries. If you are interested you can have a look at
http://code.google.com/p/jlibs/

Also, the output had to be DOM as that is then passed on to other frameworks (specifically JAXB in my case) so a non-standard framework would be out.