Java Scanner ignores first token if it is empty

Question

I'm trying to read an InputStream of String tokens with a Scanner. Every token ends with a comma ,. An empty string "" is also a valid token. In that case the whole token is just the comma that ends it.

The InputStream is slowly read from another process, and any tokens should be handled as soon as they have been fully read. Therefore reading the whole InputStream to a String is out of the question.

An example input could look like this:

ab,,cde,fg,

If I set the delimiter of the Scanner to a comma, it seems to handle the job just fine.

InputStream input = slowlyArrivingStreamWithValues("ab,,cde,fg,");

Scanner scan = new Scanner(input);
scan.useDelimiter(Pattern.quote(","));
while (scan.hasNext()) {
    System.out.println(scan.next());
}

output:

ab

cde
fg

However the problems appear when the stream begins with an empty token. For some reason Scanner just ignores the first token if it is empty.

/* begins with empty token */
InputStream input = slowlyArrivingStreamWithValues(",ab,,cde,fg,");
...

output:

ab

cde
fg

Why does Scanner ignore the first token? How can I include it?

As a side note, if you're trying to implement a CSV parser, STOP. There are endless parsers out there ([apache-csv](https://commons.apache.org/proper/commons-csv/), to name one). It's going to get very very complicated on some edge cases. — Mordechai, Jan 02 '18 at 04:43
Possible duplicate of [Tokenising a String containing empty tokens](https://stackoverflow.com/questions/12395862/tokenising-a-string-containing-empty-tokens) — NineBerry, Jan 02 '18 at 04:44

Tim Biegeleisen · Accepted Answer · 2018-01-02T04:59:33.903

1

Try using a lookbehind as the pattern:

(?<=,)

and then replace comma with empty string with each token that you match. Consider the following code:

String input = ",ab,,cde,fg,";
Scanner scan = new Scanner(input);
scan.useDelimiter("(?<=,)");
while (scan.hasNext()) {
    System.out.println(scan.next().replaceAll(",", ""));
}

This outputs the following:

(empty line)
ab

cde
fg

Demo

edited Jan 02 '18 at 04:59

answered Jan 02 '18 at 04:49

Tim Biegeleisen

502,043
27
286
360

In this case the input is an InputStream, so string manipulation is out of the question. – Tuupertunut Jan 02 '18 at 04:51
I might leave this answer up in the event that you can read into memory. – Tim Biegeleisen Jan 02 '18 at 04:54
That is also out of the question as the tokens are slowly coming from another process and should be handled as soon as they come. Maybe I should edit the question. – Tuupertunut Jan 02 '18 at 04:56
@Tuupertunut Tokenizing on a positive lookbehind seems promising, q.v. my updated answer. – Tim Biegeleisen Jan 02 '18 at 05:01
@Tuupertunut I may have introduced another problem, by the way. Now, a trailing (i.e. final) comma may not be captured. But perhaps you don't care about that. – Tim Biegeleisen Jan 02 '18 at 05:07

score 0 · Answer 2 · answered Jan 02 '18 at 05:16

0

It's easier if you write it yourself, without using Scanner:

static List<String> getValues(String source){
    List<String> list = new ArrayList<String>();
    for(int i = 0; i < source.length();i++){
        String s = "";
        while(source.charAt(i) != ','){
            s+=source.charAt(i++);
            if(i >= source.length()) break;
        }
        list.add(s);
    }
    return list;
}

For example, if source = ",a,,b,,c,d,e", the output will be "", "a", "", "c", "d", "e".

answered Jan 02 '18 at 05:16

Lorenzo Notaro

89
8

The OP is using a stream, not a fixed string, q.v. the comments under my answer. – Tim Biegeleisen Jan 02 '18 at 05:17
I saw the comments right after I posted my answer. Anyway, string manipulation could be used once there is a string buffer, so I was suggesting to make one first and work on that. – Lorenzo Notaro Jan 02 '18 at 05:23
Just doing `String#split()` on comma would already give the OP its desired behavior. – Tim Biegeleisen Jan 02 '18 at 05:27
This approach could possibly be used if string iteration was replaced with inputstream iteration. – Tuupertunut Jan 02 '18 at 05:31

Java Scanner ignores first token if it is empty

2 Answers2

Demo